Logging

With logging, Instana collects application and service logs automatically, correlate them with metrics and traces, and enhances traces or calls with extra messages.

  • Collect application and service logs automatically. For more information, see Log messages.
  • Correlate logs with metrics and traces by using the logs within the context of Kubernetes and containers. For more information, see Analyze Kubernetes logs.
  • Enrich traces or calls with extra messages. For more information, see Instana tracing SDK.
  • Integrate with a dedicated logging product. For more information, see Logging integrations.

Viewing logs

To view logging information, complete the following steps:

  1. From the navigation menu in the Instana UI, click Analytics.

  2. From the drop-down list, select Logs.

You can view the Analytics logs dashboard.

Analyzing logs

You can examine logs in detail with Unbounded analytics, where you can see all the log-related information, and slice and dice them to help you on your troubleshooting journey.

Filtering and grouping logs

You can filter and group logs by using the following approaches:

  • Query builder
  • Directly from within a log
  • Filter sidebar

You can use each of the approaches individually, but the best results are achieved when you combine these approaches.

Using a query builder

To filter and group logs with the query builder, complete the following steps:

  1. In the Filter field on the Analytics logs dashboard, click Add filter.

  2. From the drop-dowm list, select the required filter option. A constructor appears on the filtering area with an equals operator as default.

    You can filter by using any attribute of a log, such as level, message, stream, custom tags, snapshot, trace IDs, and exceptions. Additional filters that are related to technologies that are observed by Instana are also available. A quick search function can help to access your filtering options quickly.

  3. Enter a relevant value in the input field of the equals operator. For standard values or related Instana entities, select values from the drop-down list.

  4. To troubleshoot problems, filter by using the is present operator to make a quick query on exceptions. You can change the operator by clicking it.

  5. To add more than one filter, an AND boolean operator appears as default.

  6. To remove any of the filtering or the operators, click the filter or operator, and then click the x symbol.

  7. To apply grouping, click Add group, and select one of the tags. A common use case for grouping is to find out which services or hosts are generating more logs, which can help scope down the search.

    In this example, you can focus on the group with the hostname worker0 by clicking the Focus in the group icon near the Number of logs data. Then, Instana adds this hostname as a filter and remove the host grouping.

Directly from within a log

When the log message contains custom tags, they are highlighted in gray.

To filter directly from within the log message, complete the following steps:

  1. Identify custom tags in the log message. The custom tags are highlighted in gray.

  2. To add the custom tags as filter from within the log message, click the highlighted custom tag, and select Add as filter.

    In the following example, you can see the remote address 10.255.201.71 and the remote host 10.255.201.71 are custom tags.

    The following image shows the result of adding the custom tags as filter from within the log message.

  3. To view detailed information that is related to the log, expand the log. The log tag table is displayed.

  4. Hover over each row to find contextual actions.

  5. Use the following contextual actions within a log to execute the actions and display a log list with the source log.

    • Group by Tag icon: Displays a view where all the logs for the specific time frame and filters are grouped by that specific tag and the different values that it takes. This view is convenient to get hints of log volume for a specific tag.

      To add a specific tag and value as a filter and enable endless scrolling, click the focus on this group icon. If you expand a group, only an overview is provided and endless scrolling is not available in this step.

    • Add tag as filter icon: Adds the tag and its value as a filter. The default is added as AND.

      In the following example, the stream is added to the former applied filter WARN.

    • Copy to clipboard icon: Copies the tag value to the clipboard. The tag value can be used when you create a troubleshooting ticket for your team.

  6. To get information on an entity's health, check the health indicator before the entity name as shown in the following image. The color codes relate with the Smart Alerts color-code. Hover over it to see the number of issues for that entity, which gives a hint on the magnitude of a specific problem without changing the context.

From side filter bar

The side filter bar provides a flexible way of filtering and grouping in combination with the query builder and directly from within a log.

  1. To filter and group with the side filter bar, use the following tags:

    • Log levels
    • Stream
    • Services
  2. To view the number of logs for each different value of that tag even before you apply any filters, look at the number that is displayed near the category. In the following example, you can see that 30.3k logs are provided for the error tag.

  3. To add a value to the filter, select the checkbox for the value.

  4. To remove the value, clear the checkbox for the value.

You can also group by a filter directly from the icon near the main category title.

Sharing information with your team

You can work with logs that are needed to share information with other members of your team. By clicking the link icon on a specific log, you can share a short link with your team. When anyone else uses the link, Instana shows the same screen with the time frame, filters, and source log that are highlighted, open, and centered in the screen, to facilitate collaboration.

Handling multiline log messages and stack traces

Instana supports multiline log messages and stack traces from containers (Docker and containerd). Multiline log messages and stack traces are aggregated together as a single event, which simplifies troubleshooting issues.

Multiline log messages

Multiline log messages usually come in as separate single log messages. Instana identifies individual log messages that are part of a single multiline log message based on the timestamp. Starting with the most recent log message with a timestamp, each subsequent log message without a timestamp is considered to be a part of a multiline log message.

The following examples show multiline log messages:

Example 1
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.2.5)
Example 2
2023-10-08T08:49:17.645+00:00 | INFO | nstana-sensor-scheduler-thread-1 | PackageInstaller | com.instana.discovery-python - 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 223, in _main 
  	status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper 
  	return func(self, options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 454, in run 
  	options.target_dir, target_temp_dir, options.upgrade
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 461, in _handle_target_dir
	ensure_dir(target_dir)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 117, in ensure_dir
	os.makedirs(path)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs 
  	makedirs (head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs 
  	makedirs (head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs 
  	makedirs (head, mode, exist_ok)
  [Previous line repeated 1 more time]
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
	mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/1710'

2023-10-08T08:49:21.317+00:00 | INFO_ | d8f6c8dac361e3387053486ae4957736 | Docker| com.instana.sensor-docker – 1.1.5

The following image shows a Java™ multi-line log message example in the Instana logs:

Stack traces

Instana supports stack traces from Java™, Python, and Go applications. Instana considers only a subset of log formats that are defined as standard from these languages. Stack traces are identified and grouped using regular expressions based on standard for each language.

Example for Java™ stack trace
// Example 1:
Exception in thread "main" java.lang.RuntimeException: A test exception
  at com.stackify.stacktrace.StackTraceExample.methodB(StackTraceExample.java:13)
  at com.stackify.stacktrace.StackTraceExample.methodA(StackTraceExample.java:9)
  at com.stackify.stacktrace.StackTraceExample.main(StackTraceExample.java:5)

// Exception name: java.lang.RuntimeException
// Exception message: A test exception
// Stack trace: all lines


// Example 2:
com.instana.backend.common.exception.InstanaException: java.io.InterruptedIOException: Connection has been shut down
	at com.instana.backend.common.client.AbstractHttpClient.execute(AbstractHttpClient.java:217)
	at com.instana.groundskeeper.client.GroundskeeperClient.getHttpEndpointConfigs(GroundskeeperClient.java:360)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache.loadInternal(ReloadingConfigCache.java:79)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache$1.reload(ReloadingConfigCache.java:68)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache$1.reload(ReloadingConfigCache.java:59)
	at com.github.benmanes.caffeine.cache.CacheLoader.lambda$asyncReload$2(CacheLoader.java:190)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.InterruptedIOException: Connection has been shut down
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:342)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at com.instana.backend.common.client.AbstractHttpClient.executeInternalRequest(AbstractHttpClient.java:238)
	at com.instana.backend.common.client.AbstractHttpClient.execute(AbstractHttpClient.java:212)
	... 9 common frames omitted
Caused by: org.apache.http.impl.conn.ConnectionShutdownException: null
	at org.apache.http.impl.conn.CPoolProxy.getValidConnection(CPoolProxy.java:77)
	at org.apache.http.impl.conn.CPoolProxy.getSSLSession(CPoolProxy.java:137)
	at org.apache.http.impl.client.DefaultUserTokenHandler.getUserToken(DefaultUserTokenHandler.java:82)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:326)
	... 16 common frames omitted

// Exception name: com.instana.backend.common.exception.InstanaException
// Exception message: java.io.InterruptedIOException: Connection has been shut down
// Stack trace: all lines


// Example 3 (suppressed exceptions):
Exception in thread "main" java.lang.RuntimeException: I wanted to access this resource. Bad luck. Its dirty resource !!!
    at DirtyResource.accessResource(DirtyResource.java:9)
    at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:12)
    Suppressed: java.lang.NullPointerException: Remember me. I am your worst nightmare !! I am Null pointer exception !!
        at DirtyResource.close(DirtyResource.java:19)
        at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:13)
		Caused by: org.apache.http.impl.conn.ConnectionShutdownException: null
				at org.apache.http.impl.conn.CPoolProxy.getValidConnection(CPoolProxy.java:77)

// Exception name: java.lang.RuntimeException
// Exception message: I wanted to access this resource. Bad luck. Its dirty resource !!!
// Stack trace: all lines

The following image shows a Java™ stack trace example from Instana logs:

Example for Python stack trace
# Example 1:
Traceback (most recent call last):
  File "example.py", line 5, in <module>
    say('Micheal')
  File "example.py", line 3, in say
    print('Hello, ' + nam)
NameError: name 'nam' is not defined

# Exception type: NameError
# Exception message: name 'nam' is not defined
# Stack trace: all lines


# Example 2 (handling an exception raises another exception, with logger used to log):
ERROR:root:Everything is broken
Traceback (most recent call last):
  File "/tmp/spam2.py", line 13, in throws
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/tmp/spam2.py", line 18, in <module>
    f1()
  File "/tmp/spam2.py", line 2, in f1
    f2()
  File "/tmp/spam2.py", line 5, in f2
    f3()
  File "/tmp/spam2.py", line 8, in f3
    throws()
  File "/tmp/spam2.py", line 15, in throws
    raise Exception("boom")
Exception: boom

# Exception type: ZeroDivisionError
# Exception message: division by zero
# Stack trace: all lines


# Example 3 (source not available):
Traceback (most recent call last):
  File "example.py", line 5, in <module>
  File "example.py", line 3, in say
NameError: name 'nam' is not defined

# Exception type: NameError
# Exception message: name 'nam' is not defined
# Stack trace: all lines
Example for Golang stack trace
Panic: Want something

goroutine 1 [running]:
main.Example(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
	/Users/bill/Spaces/Go/Projects/src/github.com/goinaction/code/temp/main.go:9 +0x64
main.main()
	/Users/bill/Spaces/Go/Projects/src/github.com/goinaction/code/temp/main.go:5 +0x85

goroutine 2 [runnable]:
runtime.forcegchelper()
	/Users/bill/go/src/runtime/proc.go:90
runtime.goexit()
	/Users/bill/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 3 [runnable]:
runtime.bgsweep()
	/Users/bill/go/src/runtime/mgc0.go:82
runtime.goexit()
	/Users/bill/go/src/runtime/asm_amd64.s:2232 +0x1

# Exception type: null 
# Exception message: Want something
# Stack trace: all lines

Integrating logs

Instana supports integrating logs from the external logging providers to enable a fluid workflow between Instana and these logging providers. For more information, see Integrating logs.

Deleting logs

If Personal Identifiable Information (PII) is revealed in log messages, you can delete all the logs.

To delete all logs, complete the following steps:

  1. In the Instana UI, click Settings > Team settings > Log Management > Delete logs.

    In this Settings page, you can see a brief explanation of this function, a button to initiate log deletion, a summary table that lists previous log deletions, and the following information:

    • The exact date that the deletion was initiated.
    • The reason for deletion.
    • The number of logs that were deleted.
    • The initiator who deleted the logs.
    • The status of the deletion.

  2. Click Delete logs.

  3. Enter a reason for deletion, and type LOGS in the designated field to confirm your decision.

  4. Click Delete logs.

You can close the dialog because it does not interrupt the process. When the deletion is complete, the outcome (success or failure) is displayed in the dialog, and a new entry is made in the summary table.