Logging

Logs are one of the four pillars of observability. Instana supports log consumption from various sources by using tracers, container sensors, and OpenTelemetry. Instana focuses in collecting logs to support the observability and address related use cases effectively.

Instana automatically gathers application and service logs, correlates them with metrics and traces, and enhances traces or calls with extra log messages to provide deeper insights.

Collect OpenTelemetry logs.
Collect application and service logs automatically. For more information, see Log messages.
Correlate logs with metrics and traces by using the logs within the context of Kubernetes and containers. For more information, see Analyze Kubernetes logs.
Enrich traces or calls with extra messages. For more information, see Instana tracing SDK.
Integrate with a dedicated logging product. For more information, see Logging integrations.

To use Logs with Instana, you have the following options:

Managing and viewing logs

From the navigation menu in the Instana UI, click Logs.

To analyze logs in detail when troubleshooting, click Analyze Logs and carry over the selected timeframe to your further investigation.

Summary

The Summary tab offers you a quick overview of the Current retention Period, Current month log volume, and Log count distribution for the selected timeframe. Click on the pencil icon of the Current retention period for configuring extended log retention. For more information, see Configuring extended log retention. Click on the magnifier glass icon of the Current month log volume for viewing log volume report. For more information, see Viewing log volume report.

Only users with Log retention period and Access log volume report permissions can access the respective functions by using the button icons on the related cards. These permissions are available only after the logging add-on is activated for the tenant unit.

Inspect the Log count distribution for the selected time frame by interacting with the graph:

Click a legend item to view detailed information about a specific combination of log levels or isolate a single log level for a more detailed graph.
Hover over the graph to view a detailed analysis list for a specific time point.

Click a time point on the graph to access the following options:

Zoom to time range: Refreshes the graph to focus on data for the selected time range.
Export to CSV: Generates and downloads a CSV file with the data for the selected time range.

To create a logging smart alerts quickly, click ADD SMART ALERT.

Graph Interactions — Figure 2 Graph interactions

Logging smart alerts

The Smart Alert tab offers you a quick overview of the configured smart alert for logs in a sortable table. You can sort the table by:

Name
Enabled
Disabled
Date created
Date changed

Change the sorting order or use the search field to find specific alerts.

At the end of each row, use the toggle button to quickly enable or disable a logging smart alert. Click overflow menu to access the following actions:

Edit
Duplicate
Delete

Smart Alerts — Figure 3. Logging smart alerts

Click a row to view an overview of the Logging Smart Alert. The overview displays Alert Configuration on the left and Alerts Created list on the right. Select an item from the Alerts Created list to access the associated issue.

Smart Alerts Overview — Figure 4. Logging smart alert overview

On Logs Smart Alert associated issue provides detailed information to help resolve it, including:

Start and end timestamps
Duration
Severity
Description
Scope
Automation Policies
Recommended Actions - generated with Watsonx AI
Action History

Smart Alerts Issue — Figure 5. Issue related to a Logging smart alert

Deleting logs

If Personal Identifiable Information (PII) is revealed in log messages, you can delete all logs from the Instana database

To delete logs, complete the following steps:

From the navigation menu in the Instana UI, click Logs > Delete logs.

The Settings page displays a brief explanation of this function, an option to start log deletion, a summary table that lists the previous log deletions, and the following information:

The date and time when the deletion was initiated.
The reason for deletion.
The number of logs that were deleted.
The initiator who deleted the logs.
The status of the deletion.

Click Delete Logs to initiate the process.
Enter a reason for deletion in the text field, and type LOGS in the designated field to confirm your decision.
Click Delete logs to confirm the deletion.

You can close the dialog because it does not interrupt the process. When the deletion is complete, the outcome (success or failure) is displayed in the dialog, and a new entry is made in the summary table.

This tab and functionality are only available for users with the permission Log deletion granted.

Configuration

From the navigation menu in the Instana UI, click Logs > Configuration.

For more information about Configuration, see the following topics:

Logs Config Tab — Figure 8. Logs configuration tab

Analyzing logs

You can analyze logs in detail with Unbounded Analytics, where you can see all the log-related information, and slice and dice them to gain valuable insights during troubleshooting.

To start analyzing logs, complete the following steps:

From the navigation menu in the Instana UI, click Analytics.
From the drop-down list, select Logs to view the Analytics logs dashboard.

Filtering and grouping logs

You can filter and group logs by using the following approaches:

Query builder
Directly from within a log
Filter sidebar

You can use each of the approaches individually, but the best results are achieved when you combine these approaches.

Using a query builder

To filter and group logs with the query builder, complete the following steps:

In the Filter field on the Analytics logs dashboard, click Add filter.
From the drop-down list, select the required filter option. A constructor appears on the filtering area with an equals operator as default.

You can filter by using any attribute of a log, such as level, message, stream, custom tags, snapshot, trace IDs, and exceptions. More filters that are related to technologies that are observed by Instana are also available. A quick search function can help to access your filtering options quickly.
Enter a relevant value in the input field of the equals operator. For standard values or related Instana entities, select values from the drop-down list.

Figure 9. Filtering and grouping
To troubleshoot problems, filter by using the is present operator to make a quick query on exceptions. You can change the operator by clicking it.

Figure 10. Filtering and grouping
To add more than one filter, an AND boolean operator appears as default.
To remove any of the filtering or the operators, click the filter or operator and click the x symbol.

Figure 11. Filtering and grouping
To apply grouping, click Add group, and select one of the tags. A common use case for grouping is to find out which services or hosts are generating more logs, which can help scope down the search.

Figure 12. Filtering and grouping

In this example, you can focus on the group with the hostname worker0 by clicking the Focus in the group icon near the Number of logs data. Then, Instana adds this hostname as a filter and remove the host grouping.

Directly from within a log

When the log message contains custom tags, they are highlighted in gray.

To filter directly from within the log message, complete the following steps:

Identify custom tags in the log message. The custom tags are highlighted in gray.
To add the custom tags as filter from within the log message, click the highlighted custom tag, and select Add as filter.

In the following example, you can see the remote address 10.255.201.71 and the remote host 10.255.201.71 are custom tags.

Figure 13. Filtering from log message

The following image shows the result of adding the custom tags as filter from within the log message.

Figure 14. Custom tag in query builder
To view detailed information that is related to the log, expand the log. The log tag table is displayed.
Hover over each row to find contextual actions.
Use the following contextual actions within a log to execute the actions and display a log list with the source log.
- Group by Tag icon: Displays a view where all the logs for the specific time frame and filters are grouped by that specific tag and the different values that it takes. This view is convenient to get hints of log volume for a specific tag.
  
  To add a specific tag and value as a filter and enable endless scrolling, click the focus on this group icon. If you expand a group, only an overview is provided and endless scrolling is not available in this step.
  
  Figure 15. Group by tag
- Add tag as filter icon: Adds the tag and its value as a filter. The default is added as AND.
  
  Figure 16. Add tag as filter
  
  In the following example, the stream is added to the former applied filter WARN.
  
  Figure 17. Result tag added as filter
- Copy to clipboard icon: Copies the tag value to the clipboard. The tag value can be used when you create a troubleshooting ticket for your team.
  
  Figure 18. Copy to clipboard
To get information on an entity's health, check the health indicator before the entity name as shown in the following image. The color codes relate with the Smart Alerts color-code. Hover over it to see the number of issues for that entity, which gives a hint on the magnitude of a specific problem without changing the context.

Figure 19. Entities health

From side filter bar

The side filter bar provides a flexible way of filtering and grouping in combination with the query builder and directly from within a log.

To filter and group with the side filter bar, use the following tags:
- Log levels
- Stream
- Services
To view the number of logs for each different value of that tag even before you apply any filters, look at the number that is displayed near the category. In the following example, you can see that 30.3k logs are provided for the error tag.
To add a value to the filter, select the checkbox for the value.
To remove the value, clear the checkbox for the value.

You can also group by a filter directly from the icon near the main category title.

Handling multiline log messages and stack traces

Instana supports multiline log messages and stack traces from containers (Docker and containerd). Multiline log messages and stack traces are aggregated together as a single event, which simplifies troubleshooting issues.

Multiline log messages

Multiline log messages usually come in as separate single log messages. Instana identifies individual log messages that are part of a single multiline log message based on the timestamp. Starting with the most recent log message with a timestamp, each subsequent log message without a timestamp is considered to be a part of a multiline log message.

The following examples show multiline log messages:

Example 1

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.2.5)

Example 2

2023-10-08T08:49:17.645+00:00 | INFO | nstana-sensor-scheduler-thread-1 | PackageInstaller | com.instana.discovery-python - 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
  	status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
  	return func(self, options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 454, in run
  	options.target_dir, target_temp_dir, options.upgrade
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 461, in _handle_target_dir
	ensure_dir(target_dir)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 117, in ensure_dir
	os.makedirs(path)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
  	makedirs (head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
  	makedirs (head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
  	makedirs (head, mode, exist_ok)
  [Previous line repeated 1 more time]
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
	mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/1710'

2023-10-08T08:49:21.317+00:00 | INFO_ | d8f6c8dac361e3387053486ae4957736 | Docker| com.instana.sensor-docker – 1.1.5

The following image shows a Java™ multi-line log message example in the Instana logs:

Stack traces

Instana supports stack traces from Java™, Python, and Go applications. Instana considers only a subset of log formats that are defined as standard from these languages. Stack traces are identified and grouped by using regular expressions based on standard for each language.

Example for Java™ stack trace

// Example 1:
Exception in thread "main" java.lang.RuntimeException: A test exception
  at com.stackify.stacktrace.StackTraceExample.methodB(StackTraceExample.java:13)
  at com.stackify.stacktrace.StackTraceExample.methodA(StackTraceExample.java:9)
  at com.stackify.stacktrace.StackTraceExample.main(StackTraceExample.java:5)

// Exception name: java.lang.RuntimeException
// Exception message: A test exception
// Stack trace: all lines


// Example 2:
com.instana.backend.common.exception.InstanaException: java.io.InterruptedIOException: Connection has been shut down
	at com.instana.backend.common.client.AbstractHttpClient.execute(AbstractHttpClient.java:217)
	at com.instana.groundskeeper.client.GroundskeeperClient.getHttpEndpointConfigs(GroundskeeperClient.java:360)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache.loadInternal(ReloadingConfigCache.java:79)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache$1.reload(ReloadingConfigCache.java:68)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache$1.reload(ReloadingConfigCache.java:59)
	at com.github.benmanes.caffeine.cache.CacheLoader.lambda$asyncReload$2(CacheLoader.java:190)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.InterruptedIOException: Connection has been shut down
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:342)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at com.instana.backend.common.client.AbstractHttpClient.executeInternalRequest(AbstractHttpClient.java:238)
	at com.instana.backend.common.client.AbstractHttpClient.execute(AbstractHttpClient.java:212)
	... 9 common frames omitted
Caused by: org.apache.http.impl.conn.ConnectionShutdownException: null
	at org.apache.http.impl.conn.CPoolProxy.getValidConnection(CPoolProxy.java:77)
	at org.apache.http.impl.conn.CPoolProxy.getSSLSession(CPoolProxy.java:137)
	at org.apache.http.impl.client.DefaultUserTokenHandler.getUserToken(DefaultUserTokenHandler.java:82)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:326)
	... 16 common frames omitted

// Exception name: com.instana.backend.common.exception.InstanaException
// Exception message: java.io.InterruptedIOException: Connection has been shut down
// Stack trace: all lines


// Example 3 (suppressed exceptions):
Exception in thread "main" java.lang.RuntimeException: I wanted to access this resource. Bad luck. Its dirty resource !!!
    at DirtyResource.accessResource(DirtyResource.java:9)
    at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:12)
    Suppressed: java.lang.NullPointerException: Remember me. I am your worst nightmare !! I am Null pointer exception !!
        at DirtyResource.close(DirtyResource.java:19)
        at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:13)
		Caused by: org.apache.http.impl.conn.ConnectionShutdownException: null
				at org.apache.http.impl.conn.CPoolProxy.getValidConnection(CPoolProxy.java:77)

// Exception name: java.lang.RuntimeException
// Exception message: I wanted to access this resource. Bad luck. Its dirty resource !!!
// Stack trace: all lines

The following image shows a Java™ stack trace example from Instana logs:

Example for Python stack trace

# Example 1:
Traceback (most recent call last):
  File "example.py", line 5, in <module>
    say('Micheal')
  File "example.py", line 3, in say
    print('Hello, ' + nam)
NameError: name 'nam' is not defined

# Exception type: NameError
# Exception message: name 'nam' is not defined
# Stack trace: all lines


# Example 2 (handling an exception raises another exception, with logger used to log):
ERROR:root:Everything is broken
Traceback (most recent call last):
  File "/tmp/spam2.py", line 13, in throws
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/tmp/spam2.py", line 18, in <module>
    f1()
  File "/tmp/spam2.py", line 2, in f1
    f2()
  File "/tmp/spam2.py", line 5, in f2
    f3()
  File "/tmp/spam2.py", line 8, in f3
    throws()
  File "/tmp/spam2.py", line 15, in throws
    raise Exception("boom")
Exception: boom

# Exception type: ZeroDivisionError
# Exception message: division by zero
# Stack trace: all lines


# Example 3 (source not available):
Traceback (most recent call last):
  File "example.py", line 5, in <module>
  File "example.py", line 3, in say
NameError: name 'nam' is not defined

# Exception type: NameError
# Exception message: name 'nam' is not defined
# Stack trace: all lines

Example for Golang stack trace

Panic: Want something

goroutine 1 [running]:
main.Example(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
	/Users/bill/Spaces/Go/Projects/src/github.com/goinaction/code/temp/main.go:9 +0x64
main.main()
	/Users/bill/Spaces/Go/Projects/src/github.com/goinaction/code/temp/main.go:5 +0x85

goroutine 2 [runnable]:
runtime.forcegchelper()
	/Users/bill/go/src/runtime/proc.go:90
runtime.goexit()
	/Users/bill/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 3 [runnable]:
runtime.bgsweep()
	/Users/bill/go/src/runtime/mgc0.go:82
runtime.goexit()
	/Users/bill/go/src/runtime/asm_amd64.s:2232 +0x1

# Exception type: null
# Exception message: Want something
# Stack trace: all lines

Managing logs

You can extend the duration for which logs are retained in Instana by configuring the log retention period settings. Further, you can also manage your log volume reports.

Extended log retention

Extended log retention is an add-on feature for Instana logging. With extended log retention, you can retain logs for 30, 60, and 90 days, compared to the 7 days of default retention time of Instana core logging feature.

For more information about changing the log retention period, see Configuring extended log retention.

Log volume reports

Log volume reports are an add-on feature for Instana logging. With log volume reports, you can see the amount of ingested log message volume that is associated with a specific time frame with monthly granularity.

For more information about viewing log volume reports, see Viewing log volume reports.

Using logs in custom dashboards

The Custom dashboards that are infused with log data provide you with a clear overview of system health and enables better collaboration with teams.

To use logs in custom dashboards, complete the following steps:

From the navigation menu in the Instana UI, click Custom Dashboards.
Select an existing dashboard or create a new one by clicking Add Dashboard.
On the dashboard, click Add Widget.
Select one of the supported widgets and click Next:

Pie chart
Time series
Big number

In the widget configuration window, click Data source and select Logs from the drop-down menu.
Configure the remaining input fields to align with your use cases.
Click Create.

Configuring Widget — Figure 25. Configuring a widget Data source Logs

For more information, see Custom dashboards.