Troubleshooting .NET tracing on AWS Lambda

Symptoms: The Lambda function runs successfully but no traces appear in the Instana UI after configuration.

Troubleshooting steps:

1. Verify that the Lambda function runtime is supported (.NET 8 or later).

2. Verify that the Instana Layer is correctly attached to the Lambda function:

   1. In the AWS Console, go to Lambda > Functions > Your Function.
   2. In the Configuration tab of your Lambda function, click Layers.
   3. Verify that the Instana layer is listed.

3. Verify that the layer ARN matches the correct region and .NET runtime version:

   • ARN format for standard AWS regions: arn:aws:lambda:<region>:410797082306:layer:instana-dotnetcore:<layer-version>
   • ARN format for AWS regions in China: arn:aws:lambda:<region>:107998019096:layer:instana-dotnetcore:<layer-version>

4. Verify that the environment variables are correctly set and that paths are valid, accessible by the process or application, and correct:

   CORECLR_ENABLE_PROFILING=1
   CORECLR_PROFILER={cf0d821e-299b-5307-a3d8-b283c03916dd}
   CORECLR_PROFILER_PATH=/opt/instana_tracing/CoreProfiler.so
   DOTNET_STARTUP_HOOKS=/opt/Instana.Tracing.Core.dll
   INSTANA_AGENT_KEY=<your-agent-key>
   INSTANA_ENDPOINT_URL=<serverless_instana_endpoint>
   INSTANA_AWS_ACCOUNT_ID=<your_aws_account_id>
   LAMBDA_HANDLER=<Assembly::Namespace.ClassName::MethodName>

5. Verify that the endpoint URL is correct:

   • SaaS Instana: Use the correct serverless endpoint from your Instana tenant:

     • Format: https://serverless-<region>.instana.io
     • Example: https://serverless-blue-saas.instana.io

   • Self-Hosted Instana:

     • Format: https://<instana-backend-ip>/serverless
     • Set the environment variable INSTANA_DISABLE_CA_CHECK to true or 1
6. Verify that the Lambda function has internet connectivity or proper VPC configuration to reach Instana endpoints.
7. Check the Lambda function execution role has the necessary permissions.
8. Verify that the Lambda function timeout is sufficient (minimum 30 seconds recommended for initial cold starts).
9. Check CloudWatch logs for any Instana-related errors or warnings.

Symptoms: The Lambda function times out after adding Instana tracing, or execution time significantly increases.

Troubleshooting steps:

1. Review Lambda function timeout configuration (increase if necessary):

   1. In the AWS Console, go to Lambda > Functions > Your Function.
   2. Select the Configuration tab.
   3. Click General configuration > Edit.
   4. Increase the Timeout value (minimum 30 seconds suggested).
2. Check for cold start overhead: The first starting takes longer.

3. Verify that the Lambda function has adequate memory allocation (minimum 512 MB suggested):

   1. In the AWS Console, go to Lambda > Functions > Your Function.
   2. Select the Configuration tab.
   3. Click General configuration > Edit.
   4. Increase the Memory allocation if needed.
4. Review CloudWatch logs for performance metrics.
5. Check whether synchronous calls are blocking Lambda execution.
6. Consider using asynchronous tracing if available.
7. Verify network latency to the Instana endpoint.

Symptoms: Some traces appear but are incomplete, missing downstream calls, or showing gaps.

Troubleshooting steps:

1. Verify that all supported libraries are within the compatible version ranges:
2. Check whether custom instrumentation is needed for specific libraries.

3. Review Lambda function code for:

   • Async/await patterns that might not be properly traced
   • Background tasks that complete after Lambda returns
   • External service calls using unsupported clients
4. Enable debug logging to see what spans are created.
5. Check if Lambda function is stopped before all spans are sent.
6. Verify proper context propagation for distributed tracing.

Symptoms: The Instana Layer cannot be attached to the AWS Lambda function, or the layer is attached but does not load properly during function execution.

Troubleshooting steps:

1. Verify that the Layer ARN matches:

   • Correct AWS region
   • Correct .NET runtime version (.NET 8 or later)
   • Latest available layer version

2. Check that Lambda function runtime matches layer compatibility:

   • .NET 8 or later

   If the runtime is not supported, update the Lambda runtime before attaching the layer.

3. Verify that the AWS account has permissions to access the Instana layer.
4. Check if Lambda function has reached the 5-layer limit.
5. Review CloudWatch Logs for layer initialization errors.

6. Verify that the layer is compatible with Lambda function architecture (x86_64 or arm64):

   • Check the Lambda function architecture setting.
   • Ensure that the Instana layer supports the selected architecture.

Symptoms: Instana environment variables are set but not being recognized by the Lambda function.

Troubleshooting steps:

1. Verify that environment variables are set at the Lambda function level, not just in code:

   1. In the AWS Console, go to Lambda > Functions > Your Function.
   2. Select the Configuration tab.
   3. Click Environment variables.
   4. Verify that all required variables are present.
2. Check for typographical errors in environment variable names (case-sensitive).
3. Verify that INSTANA_AGENT_KEY is valid and not expired.
4. Verify that INSTANA_ENDPOINT_URL is accessible from Lambda execution environment.

5. Check whether environment variables are overridden by:

   • Lambda function code
   • Container image settings (for containerized Lambda)
   • Infrastructure as Code (Terraform or CloudFormation)
6. Review CloudWatch logs for environment variable loading messages.

Symptoms: The Lambda function in VPC cannot send traces to Instana.

Troubleshooting steps:

1. Verify that the Lambda function has internet access through:

   • NAT Gateway (for public Instana endpoints)
   • VPC Endpoints (for AWS services)
   • Internet Gateway (if in public subnet)

2. Check that Security Groups allow outbound HTTPS traffic (port 443):

   1. In the AWS Console, go to Lambda > Functions > Your Function.
   2. Select the Configuration tab.
   3. Click VPC.
   4. Review the associated Security Groups.
   5. Verify that outbound rules allow HTTPS (port 443).

3. Verify that Network ACLs don't block outbound traffic:

   1. In the AWS Console, go to VPC > Network ACLs.
   2. Find the Network ACL associated with your Lambda function's subnets.
   3. Verify that outbound rules allow HTTPS traffic.

4. Test connectivity by using a test Lambda function with curl or wget:

   Create a simple Lambda function to test connectivity:

   using System.Net.Http;
   
   public async Task<string> TestConnectivity()
   {
       using var client = new HttpClient();
       var response = await client.GetAsync("https://serverless-<region>.instana.io");
       return $"Status: {response.StatusCode}";
   }

5. For self-hosted Instana:

   • Verify VPC peering or Transit Gateway configuration.
   • Check route tables for proper routing.
   • Verify that DNS resolution works for Instana backend.

Complete the following steps:

1. Enable debug logging by adding environment variables to the Lambda function:

   INSTANA_DEBUG=true
   INSTANA_LOG_LEVEL=DEBUG

2. Access CloudWatch logs:

   1. Go to AWS Console > CloudWatch > Log Groups.
   2. Locate log group: /aws/lambda/<function-name>.
   3. Review recent log streams for Instana-related messages.

3. Collect Lambda function configuration:

   aws lambda get-function-configuration --function-name <function-name>

4. Collect Lambda function details:

   aws lambda get-function --function-name <function-name>

5. Test Lambda function invocation:

   aws lambda invoke --function-name <function-name> --payload '{}' response.json

6. Check Layer information:

   aws lambda list-layers
   aws lambda get-layer-version --layer-name instana-dotnet --version-number <version>