Multiline log messages and stack traces

Edit online

Instana supports multiline log messages and stack traces from containers (Docker and containerd). Multiline log messages and stack traces are aggregated together as a single event, which simplifies troubleshooting issues.

Multiline log messages

Multiline log messages usually come in as separate single log messages. Instana identifies individual log messages that are part of a single multiline log message based on the timestamp. Starting with the most recent log message with a timestamp, each subsequent log message without a timestamp is considered to be a part of a multiline log message.

The following examples show multiline log messages:

Example 1

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.2.5)

Example 2

2023-10-08T08:49:17.645+00:00 | INFO | nstana-sensor-scheduler-thread-1 | PackageInstaller | com.instana.discovery-python - 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
        status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
        return func(self, options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 454, in run
        options.target_dir, target_temp_dir, options.upgrade
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 461, in _handle_target_dir
        ensure_dir(target_dir)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 117, in ensure_dir
        os.makedirs(path)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
        makedirs (head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
        makedirs (head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
        makedirs (head, mode, exist_ok)
  [Previous line repeated 1 more time]
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
        mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/1710'

2023-10-08T08:49:21.317+00:00 | INFO_ | d8f6c8dac361e3387053486ae4957736 | Docker| com.instana.sensor-docker – 1.1.5

The following image shows a Java™ multi-line log message example in the Instana logs:

Figure 1. Multi line log
Multi line log

Stack traces

Instana supports stack traces from Java™, Python, and Go applications. Instana considers only a subset of log formats that are defined as standard from these languages. Stack traces are identified and grouped by using regular expressions based on standard for each language.

Example for Java™ stack trace

// Example 1:
Exception in thread "main" java.lang.RuntimeException: A test exception
  at com.stackify.stacktrace.StackTraceExample.methodB(StackTraceExample.java:13)
  at com.stackify.stacktrace.StackTraceExample.methodA(StackTraceExample.java:9)
  at com.stackify.stacktrace.StackTraceExample.main(StackTraceExample.java:5)

// Exception name: java.lang.RuntimeException
// Exception message: A test exception
// Stack trace: all lines


// Example 2:
com.instana.backend.common.exception.InstanaException: java.io.InterruptedIOException: Connection has been shut down
        at com.instana.backend.common.client.AbstractHttpClient.execute(AbstractHttpClient.java:217)
        at com.instana.groundskeeper.client.GroundskeeperClient.getHttpEndpointConfigs(GroundskeeperClient.java:360)
        at com.instana.spanprocessing.stream.common.ReloadingConfigCache.loadInternal(ReloadingConfigCache.java:79)
        at com.instana.spanprocessing.stream.common.ReloadingConfigCache$1.reload(ReloadingConfigCache.java:68)
        at com.instana.spanprocessing.stream.common.ReloadingConfigCache$1.reload(ReloadingConfigCache.java:59)
        at com.github.benmanes.caffeine.cache.CacheLoader.lambda$asyncReload$2(CacheLoader.java:190)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.InterruptedIOException: Connection has been shut down
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:342)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.instana.backend.common.client.AbstractHttpClient.executeInternalRequest(AbstractHttpClient.java:238)
        at com.instana.backend.common.client.AbstractHttpClient.execute(AbstractHttpClient.java:212)
        ... 9 common frames omitted
Caused by: org.apache.http.impl.conn.ConnectionShutdownException: null
        at org.apache.http.impl.conn.CPoolProxy.getValidConnection(CPoolProxy.java:77)
        at org.apache.http.impl.conn.CPoolProxy.getSSLSession(CPoolProxy.java:137)
        at org.apache.http.impl.client.DefaultUserTokenHandler.getUserToken(DefaultUserTokenHandler.java:82)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:326)
        ... 16 common frames omitted

// Exception name: com.instana.backend.common.exception.InstanaException
// Exception message: java.io.InterruptedIOException: Connection has been shut down
// Stack trace: all lines


// Example 3 (suppressed exceptions):
Exception in thread "main" java.lang.RuntimeException: I wanted to access this resource. Bad luck. Its dirty resource !!!
    at DirtyResource.accessResource(DirtyResource.java:9)
    at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:12)
    Suppressed: java.lang.NullPointerException: Remember me. I am your worst nightmare !! I am Null pointer exception !!
        at DirtyResource.close(DirtyResource.java:19)
        at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:13)
                Caused by: org.apache.http.impl.conn.ConnectionShutdownException: null
                                at org.apache.http.impl.conn.CPoolProxy.getValidConnection(CPoolProxy.java:77)

// Exception name: java.lang.RuntimeException
// Exception message: I wanted to access this resource. Bad luck. Its dirty resource !!!
// Stack trace: all lines
The following image shows a Java™ stack trace example from Instana logs:
Figure 2. Java stack trace
Java stack trace

Example for Python stack trace

# Example 1:
Traceback (most recent call last):
  File "example.py", line 5, in <module>
    say('Micheal')
  File "example.py", line 3, in say
    print('Hello, ' + nam)
NameError: name 'nam' is not defined

# Exception type: NameError
# Exception message: name 'nam' is not defined
# Stack trace: all lines


# Example 2 (handling an exception raises another exception, with logger used to log):
ERROR:root:Everything is broken
Traceback (most recent call last):
  File "/tmp/spam2.py", line 13, in throws
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/tmp/spam2.py", line 18, in <module>
    f1()
  File "/tmp/spam2.py", line 2, in f1
    f2()
  File "/tmp/spam2.py", line 5, in f2
    f3()
  File "/tmp/spam2.py", line 8, in f3
    throws()
  File "/tmp/spam2.py", line 15, in throws
    raise Exception("boom")
Exception: boom

# Exception type: ZeroDivisionError
# Exception message: division by zero
# Stack trace: all lines


# Example 3 (source not available):
Traceback (most recent call last):
  File "example.py", line 5, in <module>
  File "example.py", line 3, in say
NameError: name 'nam' is not defined

# Exception type: NameError
# Exception message: name 'nam' is not defined
# Stack trace: all lines

Example for Golang stack trace

Panic: Want something

goroutine 1 [running]:
main.Example(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
        /Users/bill/Spaces/Go/Projects/src/github.com/goinaction/code/temp/main.go:9 +0x64
main.main()
        /Users/bill/Spaces/Go/Projects/src/github.com/goinaction/code/temp/main.go:5 +0x85

goroutine 2 [runnable]:
runtime.forcegchelper()
        /Users/bill/go/src/runtime/proc.go:90
runtime.goexit()
        /Users/bill/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 3 [runnable]:
runtime.bgsweep()
        /Users/bill/go/src/runtime/mgc0.go:82
runtime.goexit()
        /Users/bill/go/src/runtime/asm_amd64.s:2232 +0x1

# Exception type: null
# Exception message: Want something
# Stack trace: all lines