Python Troubleshooting

Python sensor not installed

Monitoring issue type: python_sensor_not_installed

The Python process runs on a platform we do not support for automatic instrumentation, and neither the Activating Without Code Changes nor Activating With Code Changes methods have been used to instrument the process.

Python Process doesn't show up in dashboard

1. Check Instana Host Agent logs to see whether any messages are related to this Python process.

   Many Python processes will be remotely instrumented automatically. In the cases where this is not possible, the Host Agent will may log a message indicating a problem.

2. Validate that your environment is supported and you are running a supported version of Python

   Before spending too much time diagnosing an issue, first make sure that the component and environment in question is supported. A list of all supported components can be found in the Python Supported Components & Versions page.

3. Validate the Install steps

   Check if this process was instrumented through AutoTrace webhook or manually.

   If the process is instrumented through AutoTrace webhook, make sure that the prerequisites are met and the webhook works.

   If manual, validate that the Manual Installation steps were properly followed.

4. Validate that the Instana Python package is installed and available to the application

   This can be validated by seeing instana in the Python application requirements.txt file.

   Alternatively, from the Python application environment, you can run pip list. This will list all of the Python packages available to the Python application. Validate that the Instana Python packages is in that list.

5. Assure that the Instana package is the latest version.

   The latest version released can be found on Pypi.

   In the requirements.txt, you should see the latest version of the Instana Python package noted alongside the name such as instana==x.x.x. pip list also outputs version numbers in the generated list.

6. Validate that Instana is mentioned in the application log output.

   Application and/or container logs should always have a boot message similar to Stan is on the scene. Starting Instana instrumentation version x.x.x

   If this isn't there, then it is very likely that the package isn't installed or manual installation steps were not properly followed.

7. Enable debug output and re-check application or container logs.

   Set the environment variable INSTANA_DEBUG=true for the Python process and revalidate the application or container logs (same as #6).

   Pay special attention to see if there are any Instana related error messages.

8. Try booting the Python application in verbose mode from a console

   An example on how to do this depends largely on the Python application in question. As an example, if the Python application is contained in app.py you can run the following commands:

   export INSTANA_DEBUG=true
   python app.py
   

   Pay special attention to see if there are any Instana related error messages in the Python application output.

9. File a support ticket.

If you validated each of the steps and the Python process still doesn't show up in the Instana dashboard, file a ticket.

The support ticket should include:

1. Environment details; platform, Python version, frameworks in use
2. Summary of all of the above validations.

Tracing crashes a Python process with can't start a new thread message

If a Python application is crashed by tracing, and the RuntimeError: can't start new thread message appears in the application logs, then the most likely cause is that permissions need to be granted to the application. For more information about tracing and the thread starting permission, see Tracing notes and limitations.

AutoTrace is Crashing a Python process

1. File a Support Ticket

   If the AutoTrace webhook is crashing a Python process, file a ticket.

2. Stop the AutoTrace webhook instrumentation for the particular process.

   For more information about how to label a resource for AutoTrace webhook, see Instana AutoTrace webhook. You can also configure AutoTrace webhook to ignore an entire namespace of resources. Furthermore, you can disable the Python Tracer in the webhook completely. This step is a temporary workaround until the root cause of the crash is identified and fixed.

3. Use the Manual Installation Method

   In the interim, you can alternatively use the manual installation method for Python visibility.

AUTOWRAPT_BOOTSTRAP, gevent & RecursionError

gevent monkey patches many of the standard libraries in Python and one of gevent's requirements is that this is done as early as possible in the boot process.

If you are using the Activating Without Code Changes method of installation (with the AUTOWRAPT_BOOTSTRAP environment variable) or the AutoTrace webhook and the gevent package with Instana Python Tracer earlier than 2.5.0, you might encounter the following error:

RecursionError: maximum recursion depth exceeded while calling a Python object
[Previous line repeated 212 more times]
super(SSLContext, SSLContext).options.__set__(self, value)

This is because autowrapt loads the instana Python package before any others (even gevent).

In this case, use Instana Python Tracer 2.5.0 or later, or switch to the manual installation method (by adding import instana to your code).

AUTOWRAPT_BOOTSTRAP, Django & gevent ImproperlyConfigured

If you use the Activating Without Code Changes method of installation (with the AUTOWRAPT_BOOTSTRAP environment variable), or the AutoTrace webhook and the Django and gevent packages with Instana Python Tracer 2.5.0 or later, you might encounter the following error:

django.core.exceptions.ImproperlyConfigured: Requested settings, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

Then, make sure that the DJANGO_SETTINGS_MODULE environment variable is set before the autotracing starts as specified in the Django documentation. If setting the DJANGO_SETTINGS_MODULE environment variable before the autotracing starts is still insufficient, then you can use the Activating with code changes method of package activation.

Migrating to Instana AutoTrace webhook

The Instana AutoTrace webhook is now the supported method of automatic instrumentation for Python applications monitored by Instana.

To fully migrate to AutoTrace webhook, you must remove the following from your applications:

1. The Instana Python package

   The Instana AutoTrace webhook will independently manage the Instana Python package for your application. There is no need for you or your organization to install it manually.

   Make sure to remove the instana Python package from your requirements.txt, Pipfile or virtualenv.

   Note: If you are making OpenTracing API calls, you should explicitly import opentracing instead and make sure that your calls are made against that package (e.g. opentracing.tracer.start_active_span). Instana will still receive resulting traces.

2. Remove AUTOWRAPT_BOOTSTRAP environment variable

   This environment variable is used to load the Instana Python package without any code changes required. With Instana AutoTrace webhook, this is no longer needed.

3. Remove any import instana calls

   Python AutoTrace will automatically apply the Instana Python package to your Python applications. Manual imports of the Instana Python package can be removed.