Logging

To use logs with Instana, you have the following options:

  • Collect application and service logs automatically. For more information, see Log messages.
  • Correlate logs with metrics and traces by using the logs within the context of Kubernetes and containers. For more information, see Analyze Kubernetes logs.
  • Enrich traces or calls with extra messages. For more information, see Instana tracing SDK.
  • Integrate with a dedicated logging product. For more information, see Logging integrations.

Analyzing logs

Examine Logs in detail as in Unbounded Analytics, where you can see all the log-related information, and slice and dice them to help you on your troubleshooting journey.

Viewing Logging Analytics

You can find logging information on the drop-down list of Analytics Applications as follows:

Filtering and grouping

Three approaches are provided to filter data:

  • Query Builder
  • From within a Log
  • Filter Sidebar

You can use each of the approaches individually, but best results are achieved when you combine these approaches.

Query Builder

Use the Query Builder on the Analytics dashboard to filter the initial result set. By clicking Add filter, you can apply different filters from a catalog that is divided by sections. A quick search function can help to access your filtering in a quicker way. You can filter by using any attribute of a Log, such as Level, Message, Stream, Custom Tags, Snapshot, Trace Ids and Exceptions. Additional filters related to technologies that are observed by Instana are also available. After you select one of these filters, a constructor appears on the filtering area, with an “equals” operator as default and an input field for typing the relevant value. In case of standard values or related Instana entities, you can see some suggestions on a drop-down list. When you troubleshoot problems, you can filter by using the “is present” operator to make a quick query on exceptions.

When you add more than one filter, an AND boolean operator appears as default. You can change the operator by clicking it. In any time of your troubleshooting journey, if you need to remove any of the filtering or the operators, click the filter or operator, and then click the “x” symbol.

To apply grouping, click Add group, and select one of the tags. A common use case for grouping is to find out which services or hosts are generating more logs, which can help scope down the search.

In this case, you can Focus in the group with the hostname ip-10-255-219-199 by clicking the Funnel icon near the Number of logs data. Then, Instana adds this hostname as a filter and remove the host grouping.

From within a log

When the log message contains custom tags, they are highlighted in gray, and suitable to be added as a filter to the query builder by just clicking over them. In the following example, you can see the remote address 10.255.201.71 and the remote host 10.255.201.71. The second image shows the result of adding the custom tags as filter from within the log message.

If you expand a log, you can see the log tag table, where additional information that is related to the log is shown in sections. If you hover over each row, contextual actions are provided. If you use contextual actions within a log, you can execute the actions and display a log list with the source log, which is open to facilitate the troubleshooting flow.

Available contextual actions and use examples

Group by Tag: Show a view where all the logs for the specific time frame and filters are grouped by that specific tag and the different values that it takes. This view is convenient to get hints of log volume for a specific tag. The recommended next step is to click focus on this group, so that a specific tag and value are added as a filter and infinite scrolling is available for use. If you expand a group, only an overview is provided and infinite scrolling is not available in this step.

Add as a filter: Add the tag and its value as a filter. The default is added as “and”. In the following example, the stream is added to the former applied filter “warn”.

Copy: Copy the tag value to the clipboard. The tag value can be used when you create a troubleshooting ticket for your team.

Reading Entities spark charts and metrics on live mode

If you activate the Live mode on the time picker area, you can see live information on the Spark charts and Metrics section within a log. In the following example, you can see how to inspect different values on a spark chart and how the metric values change with the time.

From side filter bar

The side bar provides a flexible way of filtering and grouping in combination with the query builder and within a log. The side filter bar provides the possibility to filter and group by 3 main tags: Log levels, Stream and Services. Even before to apply any filters, you can get a hint on the number of logs for each different value of that tag by looking at the number that is displayed near the category. In the following example, you can see that 30.3k logs are provided for the error tag. If you check the box for a value, you add the value to the filter. To remove the value, clear the box for the value. Grouping by a filter can be done directly from the icon near the main category title.

Sharing with your team

Work with logs that are needed to share information with other members of your team. By clicking the link button on a specific log, you can share a short link with your team. When anyone else uses the link, Instana shows the same screen with the time frame, filters, and source log that are highlighted open and centered in the screen, to facilitate collaboration.

Handling multiline log messages and stack traces

Instana supports multiline log messages and stack traces from containers (Docker and containerd). Multiline log messages and stack traces are aggregated together as a single event, which simplifies troubleshooting issues.

Multiline log messages

Multiline log messages usually come in as separate single log messages. Instana identifies individual log messages that are part of a single multiline log message based on the timestamp. Starting with the most recent log message with a timestamp, each subsequent log message without a timestamp is considered to be a part of a multiline log message.

The following examples show multiline log messages:

Example 1
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.2.5)

2024-04-18T17:04:11.851Z  INFO 13524 --- [           main] o.s.b.d.f.logexample.MyApplication       : Starting MyApplication using Java 17.0.11 with PID 13524 (/opt/apps/myapp.jar started by myuser in /opt/apps/)
2024-04-18T17:04:11.905Z  INFO 13524 --- [           main] o.s.b.d.f.logexample.MyApplication       : No active profile set, falling back to 1 default profile: "default"
2024-04-18T17:04:16.256Z  INFO 13524 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2024-04-18T17:04:16.322Z  INFO 13524 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-04-18T17:04:16.323Z  INFO 13524 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.20]
2024-04-18T17:04:16.503Z  INFO 13524 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-04-18T17:04:16.513Z  INFO 13524 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4332 ms
2024-04-18T17:04:18.401Z  INFO 13524 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path ''
2024-04-18T17:04:18.429Z  INFO 13524 --- [           main] o.s.b.d.f.logexample.MyApplication       : Started MyApplication in 8.19 seconds (process running for 9.711)
Example 2
2023-10-08T08:49:17.645+00:00 | INFO | nstana-sensor-scheduler-thread-1 | PackageInstaller | com.instana.discovery-python - 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 223, in _main 
  	status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper 
  	return func(self, options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 454, in run 
  	options.target_dir, target_temp_dir, options.upgrade
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 461, in _handle_target_dir
	ensure_dir(target_dir)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 117, in ensure_dir
	os.makedirs(path)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs 
  	makedirs (head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs 
  	makedirs (head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs 
  	makedirs (head, mode, exist_ok)
  [Previous line repeated 1 more time]
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
	mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/1710'

2023-10-08T08:49:21.317+00:00 | INFO_ | d8f6c8dac361e3387053486ae4957736 | Docker| com.instana.sensor-docker – 1.1.5

The following image shows a Java™ stack trace example in Instana logs:

Stack traces

Instana supports stack traces from Java™, Python, and Go applications. Stack traces are identified and grouped based on regular expressions that are already determined for each language based on the observations from application logs.

Example for Java™ stack trace
// Example 1:
Exception in thread "main" java.lang.RuntimeException: A test exception
  at com.stackify.stacktrace.StackTraceExample.methodB(StackTraceExample.java:13)
  at com.stackify.stacktrace.StackTraceExample.methodA(StackTraceExample.java:9)
  at com.stackify.stacktrace.StackTraceExample.main(StackTraceExample.java:5)

// Exception name: java.lang.RuntimeException
// Exception message: A test exception
// Stack trace: all lines


// Example 2:
com.instana.backend.common.exception.InstanaException: java.io.InterruptedIOException: Connection has been shut down
	at com.instana.backend.common.client.AbstractHttpClient.execute(AbstractHttpClient.java:217)
	at com.instana.groundskeeper.client.GroundskeeperClient.getHttpEndpointConfigs(GroundskeeperClient.java:360)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache.loadInternal(ReloadingConfigCache.java:79)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache$1.reload(ReloadingConfigCache.java:68)
	at com.instana.spanprocessing.stream.common.ReloadingConfigCache$1.reload(ReloadingConfigCache.java:59)
	at com.github.benmanes.caffeine.cache.CacheLoader.lambda$asyncReload$2(CacheLoader.java:190)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.InterruptedIOException: Connection has been shut down
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:342)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at com.instana.backend.common.client.AbstractHttpClient.executeInternalRequest(AbstractHttpClient.java:238)
	at com.instana.backend.common.client.AbstractHttpClient.execute(AbstractHttpClient.java:212)
	... 9 common frames omitted
Caused by: org.apache.http.impl.conn.ConnectionShutdownException: null
	at org.apache.http.impl.conn.CPoolProxy.getValidConnection(CPoolProxy.java:77)
	at org.apache.http.impl.conn.CPoolProxy.getSSLSession(CPoolProxy.java:137)
	at org.apache.http.impl.client.DefaultUserTokenHandler.getUserToken(DefaultUserTokenHandler.java:82)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:326)
	... 16 common frames omitted

// Exception name: com.instana.backend.common.exception.InstanaException
// Exception message: java.io.InterruptedIOException: Connection has been shut down
// Stack trace: all lines


// Example 3 (suppressed exceptions):
Exception in thread "main" java.lang.RuntimeException: I wanted to access this resource. Bad luck. Its dirty resource !!!
    at DirtyResource.accessResource(DirtyResource.java:9)
    at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:12)
    Suppressed: java.lang.NullPointerException: Remember me. I am your worst nightmare !! I am Null pointer exception !!
        at DirtyResource.close(DirtyResource.java:19)
        at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:13)
		Caused by: org.apache.http.impl.conn.ConnectionShutdownException: null
				at org.apache.http.impl.conn.CPoolProxy.getValidConnection(CPoolProxy.java:77)

// Exception name: java.lang.RuntimeException
// Exception message: I wanted to access this resource. Bad luck. Its dirty resource !!!
	// Stack trace: all lines
Example for Python stack trace
# Example 1:
Traceback (most recent call last):
  File "example.py", line 5, in <module>
    say('Micheal')
  File "example.py", line 3, in say
    print('Hello, ' + nam)
NameError: name 'nam' is not defined

# Exception type: NameError
# Exception message: name 'nam' is not defined
# Stack trace: all lines


# Example 2 (handling an exception raises another exception, with logger used to log):
ERROR:root:Everything is broken
Traceback (most recent call last):
  File "/tmp/spam2.py", line 13, in throws
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/tmp/spam2.py", line 18, in <module>
    f1()
  File "/tmp/spam2.py", line 2, in f1
    f2()
  File "/tmp/spam2.py", line 5, in f2
    f3()
  File "/tmp/spam2.py", line 8, in f3
    throws()
  File "/tmp/spam2.py", line 15, in throws
    raise Exception("boom")
Exception: boom

# Exception type: ZeroDivisionError
# Exception message: division by zero
# Stack trace: all lines


# Example 3 (source not available):
Traceback (most recent call last):
  File "example.py", line 5, in <module>
  File "example.py", line 3, in say
NameError: name 'nam' is not defined

# Exception type: NameError
# Exception message: name 'nam' is not defined
# Stack trace: all lines
Example for Golang stack trace
panic: This is the message

goroutine 35 [running]:
main.m2(...)
 /tmp/sandbox1090018372/prog.go:19
main.m1()
 /tmp/sandbox1090018372/prog.go:15 +0x27
created by main.myGoRoutine
 /tmp/sandbox1090018372/prog.go:8 +0x25

The following image shows a Java™ stack trace example from Instana logs:

Logging integrations

Instana supports integrating logs from the external logging providers to enable a fluid workflow between Instana and these logging providers. After integration, you can be redirected to the external logging provider UI from the Instana UI to view logs.

Technology Details
Coralogix Link
ELK Link
Humio Link
Mezmo Link
Splunk Link

Deleting logs

If Personal Identifiable Information (PII) is revealed in log messages, you can delete all the logs by clicking Settings > Log Management > Delete logs in the Instana UI. In this Settings page, you can see a brief explanation of this function, a button that is used to initiate log deletion, a summary table that lists previous log deletions, and the following information:

  • The exact date that the deletion was initiated
  • The deletion reason
  • The number of logs that were deleted
  • The initiator who deleted the logs
  • The status of the deletion

To delete logs, click Delete logs. Enter a reason for deletion, and type LOGS in the designated field to confirm your decision. Then, click Delete logs. You can close the dialog because it does not interrupt the process. When the deletion is complete, the outcome (Success or Failure) is displayed in the dialog, and a new entry is made in the summary table.