Python Troubleshooting

The Instana package has been designed to be fully automatic but if something goes wrong, file a ticket and let us know! We love to hear feedback (good or bad) so we can continually improve.

The following page outlines some steps you can take to potentially diagnose any issues that may arise.

Supported Components

Before spending too much time diagnosing an issue, first make sure that the component in question is supported. A list of all supported components can be found in the Python Supported Components & Versions page.

Logging & Environment Variables

By default, the package will only log messages that indicate if any problems are encountered. If you want a more verbose logging output, set the INSTANA_DEBUG environment variable. It enables debug level logging in the Instana gem.

There are even more methods to control logging output. See the Python Configuration page for more details.

Monitoring Issues

Host Agent based AutoTrace™ failed

Monitoring issue type: python_autotrace_failed

The deprecated Instana host agent based autotracing tried to automatically instrument the Python process as described in the Automatic Installation section, but the attempts didn't succeed.

Potential Solutions

  1. If your Python processes are running on Kubernetes or a similar PaaS, make sure that you've read and followed the document Configuring Network Access for Monitored Applications.

If the above doesn't resolve the issue, read through the troubleshooting section Python Process doesn't show up in dashboard.

Python sensor not installed

Monitoring issue type: python_sensor_not_installed

The Python process runs on a platform we do not support for automatic instrumentation, and neither the Activating Without Code Changes nor Activating With Code Changes methods have been used to instrument the process.

Potential Solutions

  1. Utilize the manual install instructions to properly monitor your Python applications.

Host Agent based AutoTrace™ prerequisites check failed

Monitoring issue type: python_autotrace_prerequisites_failed

Potential Solutions

  1. For the deprecated host agent based AutoTrace™ functionality to work, several prerequisites needs to be fulfilled. You need to verify the following prerequisites are in place:

    • System-level pip or pip3 available to the Instana host agent
    • The required dependencies could be found and installed by the Agent (see the Agent log file with DEBUG level logging enabled)

General Problem Areas

Python Process doesn't show up in dashboard

  1. Check Instana Host Agent logs to see whether any messages are related to this Python process.

    Many Python processes will be remotely instrumented automatically. In the cases where this is not possible, the Host Agent will may log a message indicating a problem.

  2. Validate that your environment is supported and you are running a supported version of Python

    Before spending too much time diagnosing an issue, first make sure that the component and environment in question is supported. A list of all supported components can be found in the Python Supported Components & Versions page.

  3. Validate the Install steps

    Was this process instrumented via AutoTrace or manual?

    If via AutoTrace, make sure that the configured whitelist matches the candidate process you want instrumented.

    If manual, validate that the Manual Installation steps were properly followed.

  4. Validate that the Instana Python package is installed and available to the application

    This can be validated by seeing instana in the Python application requirements.txt file.

    Alternatively, from the Python application environment, you can run pip list. This will list all of the Python packages available to the Python application. Validate that the Instana Python packages is in that list.

  5. Assure that the Instana package is the latest version.

    The latest version released can be found on Pypi.

    In the requirements.txt, you should see the latest version of the Instana Python package noted alongside the name such as instana==x.x.x. pip list also outputs version numbers in the generated list.

  6. Validate that Instana is mentioned in the application log output.

    Application and/or container logs should always have a boot message similar to Stan is on the scene. Starting Instana instrumentation version x.x.x

    If this isn't there, then it is very likely that the package isn't installed or manual installation steps were not properly followed.

  7. Enable debug output and re-check application or container logs.

    Set the environment variable INSTANA_DEBUG=true for the Python process and revalidate the application or container logs (same as #6).

    Pay special attention to see if there are any Instana related error messages.

  8. Try booting the Python application in verbose mode from a console

    An example on how to do this depends largely on the Python application in question. As an example, if the Python application is contained in app.py you can run the following commands:

    export INSTANA_DEBUG=true
python app.py

    Pay special attention to see if there are any Instana related error messages in the Python application output.

  9. If the deprecated Host Agent based AutoTracing is used, ensure that the TERM environment variable is not set and the process is not present in the list of excluded processes

    The deprecated Host Agent based AutoTracing will skip attempting to trace any process which has a TERM environment variable set or is configured in the excludes section of the agent configuration.

    Generally, this value is only set when running the Python process from a terminal attached to an active user session. To check if TERM is set, run cat /proc/<pid>/environ where <pid> is the process id of the Python process in question.

    If TERM needs to be set and the process needs to be instrumented, consider the alternative instrumentation methods, like the Instana AutoTrace WebHook or Activating Without Code Changes.

  10. File a Support Ticket.

If you validated all of the above and the Python process still doesn't show up in the Instana dashboard, file a Support Ticket so that we can investigate further.

The support ticket should include:

  1. Environment details; platform, Python version, frameworks in use
  2. Summary of all of the above validations.

Missing Traces from a Python Application

  1. Make sure that the frameworks and libraries that are in use are supported by referring to the Supported Versions page.

  2. Make sure that the application process shows up in your Instana dashboard under the host in question.

    If your application is not showing up in the Instana dashboard, see the previous section.

  3. Set the INSTANA_DEBUG=true environment for the Python application process and restart the process.

    With this environment variable set, you should see log output from your Python application process similar to the following:

    DEBUG instana: initializing agent
INFO instana: Stan is on the scene.  Starting Instana instrumentation version: 1.22.1
DEBUG instana: initializing fsm
DEBUG instana: Instrumenting asyncio
DEBUG instana: Instrumenting aiohttp client
DEBUG instana: Instrumenting aiohttp server
DEBUG instana: Instrumenting asynqp
DEBUG instana: Instrumenting cassandra
DEBUG instana: Instrumenting couchbase
DEBUG instana: Instrumenting flask (without blinker support)
DEBUG instana: Instrumenting gevent: gevent not detected or loaded.  Nothing done.
DEBUG instana: Instrumenting grpcio
DEBUG instana: Instrumenting tornado client
DEBUG instana: Instrumenting tornado server
DEBUG instana: Instrumenting logging
DEBUG instana: Instrumenting pymysql
DEBUG instana: Instrumenting psycopg2
DEBUG instana: Instrumenting redis
DEBUG instana: Instrumenting sqlalchemy
DEBUG instana: Instrumenting suds-jurko
DEBUG instana: Instrumenting urllib3
DEBUG instana: Instrumenting django
DEBUG instana: Instana host agent found on localhost:42699
DEBUG instana: Attempting to make an announcement to the agent on localhost:42699
DEBUG instana: Spawning metric & span reporting threads
DEBUG instana:  -> Metric reporting thread is now alive
DEBUG instana:  -> Span reporting thread is now alive
DEBUG instana: Announced pid: 34287 (true pid: 34287).  Waiting for Agent Ready...
DEBUG instana: Instrumenting pymongo
DEBUG instana: uwsgi hooks: decorators not available: likely not running under uWSGI
INFO instana: Instana host agent available. We're in business. Announced pid: 34287 (true pid: 34287)
DEBUG instana: Sending process snapshot data

    Make sure that the expected components are in this list for the correct visibility.

  4. Check for request context headers in response headers

    Make a request to your application with either curl or wget to check response headers returned from the application.

    For example, using the -i option for curl will output response headers:

    curl -i http://127.0.0.1:8000/

HTTP/1.1 200 OK
Date: Wed, 10 Jun 2020 08:55:25 GMT
Server: WSGIServer/0.2 CPython/3.6.9
Content-Type: text/html; charset=utf-8
X-Instana-T: 495b8301e86286c4
X-Instana-S: 495b8301e86286c4
X-Instana-L: 1
Server-Timing: intid;desc=495b8301e86286c4
Connection: close

    In the output you should see the X-Instana-T, X-Instana-S and X-Instana-L headers.

    If you see these headers in the output you can then search for this specific trace in Query Analyze by using the filter trace.id equals 495b8301e86286c4 assuming the time window in use covers the period when this request was made.

  5. File a Support Ticket

If you don't see these headers and the frameworks and components are supported, file a support ticket so we can further investigate the situation.

Attach the following to the support ticket:

  1. Application requirements.txt or the list of Python packages in use (output from pip list)
  2. The dashboard link to the application entity from step #2
  3. The output from step #3
  4. The output from step #4

AutoTrace is Crashing a Python process

  1. File a Support Ticket

    If you haven't done so already, file a support ticket so we can investigate the situation.

  2. Update the Instana Agent Host configuration.yaml to modify AutoTrace behavior

    This documentation page explains how to configure Python AutoTrace. With this, you can exclude the crashing process or disable AutoTrace entirely. This can act as a temporary work-around until the root cause of the crash is identified and fixed.

  3. Use the Manual Installation Method

    In the interim, you can alternatively use the manual installation method for Python visibility.

AUTOWRAPT_BOOTSTRAP, gevent & RecursionError

gevent monkey patches many of the standard libraries in Python and one of gevent's requirements is that this is done as early as possible in the boot process.

If you are using the Activating Without Code Changes method of installation (with the AUTOWRAPT_BOOTSTRAP environment variable) and the gevent package, you may get errors such as:

RecursionError: maximum recursion depth exceeded while calling a Python object
[Previous line repeated 212 more times]
super(SSLContext, SSLContext).options.__set__(self, value)

This is because autowrapt loads the instana Python package before any others (even gevent).

In this case, it's advised to either Migrate to Python AutoTrace or switch to the manual install method (by adding import instana to your code).

"Failed to install the Instana Python package to /tmp/.instana/python"

This can occur, with the deprecated Host Agent based AutoTracing for various reasons. To diagnose, :

  1. Enable DEBUG level agent logging to get specifics about the failure.
  2. If the agent is running directly on the host (not in a container), then make sure that the system pip command is updated with pip install -U pip. Some early versions of pip are known to cause issues under certain circumstances.

Migrating to Instana AutoTrace WebHook

The Instana AutoTrace WebHook is now the supported method of automatic instrumentation for Python applications monitored by Instana.

To fully migrate to AutoTrace WebHook, you must remove the following from your applications:

  1. The Instana Python package

    The Instana AutoTrace WebHook will independently manage the Instana Python package for your application. There is no need for you or your organization to install it manually.

    Make sure to remove the instana Python package from your requirements.txt, Pipfile or virtualenv.

    Note: If you are making OpenTracing API calls, you should explicitly import opentracing instead and make sure that your calls are made against that package (e.g. opentracing.tracer.start_active_span). Instana will still receive resulting traces.

  2. Remove AUTOWRAPT_BOOTSTRAP environment variable

    This environment variable is used to load the Instana Python package without any code changes required. With Instana AutoTrace WebHook, this is no longer needed.

  3. Remove any import instana calls

    Python AutoTrace will automatically apply the Instana Python package to your Python applications. Manual imports of the Instana Python package can be removed.

See Also