Troubleshooting

Information about how to troubleshoot a problem with Custom Edition.

Collect diagnostics for troubleshooting

Use the kubectl instana diagnostics command to collect a comprehensive set of Kubernetes, platform, node, and datastore diagnostics for troubleshooting an Instana Self-Hosted deployment (Custom Edition) that runs on Kubernetes or OpenShift.

Use the following syntax to run the command:

kubectl instana diagnostics [flags]
kubectl instana debug [flags]

kubectl instana debug is an alias for kubectl instana diagnostics and behaves identically.

Note:

This command does not manage or provision a cluster. It connects to an already-running Kubernetes cluster using the active kubeconfig context.

Command flags

Table 1. Command flags
Flag	Short	Type	Default	Description
--download-key	-	String	Required	Instana download key used as the container registry password (username: `_`) to pull the node-collector image. You must provide this flag to run the command.
--output-dir	-o	Directory	Current working directory	The directory where the command creates the timestamped diagnostics folder and archive.
--deep-debug	-	Boolean	`false`	Runs an additional set of deep-debug scripts after the standard diagnostics collection completes.
--max-log-lines	-l	Integer	`20000`	Maximum number of log lines to collect per container. Set to `0` to collect all available log lines.
--log-collection-days	-	Integer	`7`	Number of days of historical logs to collect.
--node-collector-image	-	String	Default image bundled with the release	Container image used by the temporary node-collector DaemonSet. Override this flag for air-gapped or private registry deployments.
--node-collector-namespace	-	String	`default`	Namespace where the command creates the temporary node-collector DaemonSet.
--external-script	-	String slice	None	One or more user-provided shell scripts to run during diagnostics collection. Provide multiple values as a comma-separated list or by repeating the flag.
--env-file	-	File	None	Loads environment variables from a .env file to override diagnostics configuration (namespaces, credentials, pod selectors, and similar settings).

After diagnostics collection completes, the command prints the location of the generated package.

Done!
Diagnostics package ->
<output-dir>/diagnostics_<YYYYMMDDHHmmss>

The command creates the timestamped directory inside the directory you specify with --output-dir.

What the engine collects

The diagnostics engine runs the following collectors.

Cluster-scoped Kubernetes resources
The engine collects cluster-scoped Kubernetes resources, including:
- Nodes
- Namespaces
- ClusterRoles
- ClusterRoleBindings
- StorageClasses
- PersistentVolumes
- CustomResourceDefinitions (CRDs)
- Other cluster-scoped resources
Host networking information
The engine collects host networking information, including:
- Routing tables
- Network interfaces
- iptables rules
- sysctl networking parameters
- Connectivity information
Envoy diagnostics
The engine collects Envoy diagnostics from Instana Core load-balancer pods, including:
- Cluster statistics
- Listener configuration
- Routes
- Health information
Namespaced Kubernetes resources
The engine collects namespaced Kubernetes resources across all Instana namespaces, including:
- Pods
- Deployments
- StatefulSets
- Services
- ConfigMaps
- Secrets (values redacted)
- Events
- ReplicaSets
- DaemonSets
- PersistentVolumeClaims
- HorizontalPodAutoscalers
- Custom resources
Node-level information
The engine deploys a temporary DaemonSet to collect node-level information, including:
- CPU
- Memory
- Disk topology
- Kernel and OS version
- Mount points
- Disk utilization
- sysctl configuration
- dmesg output
The temporary node collector DaemonSet applies a wildcard toleration. This allows the engine to collect diagnostics from every node in the cluster, including control-plane, infrastructure, dedicated datastore, GPU, and any custom-tainted nodes. This behavior is automatic, and you do not need any additional flags.
Instana backend diagnostics
The engine collects Instana backend diagnostics, including:
- Health endpoint responses
- Internal metrics
- Pod resource usage
Kafka diagnostics
The engine collects Kafka diagnostics, including:
- Topic inventory
- Consumer group lag
- Broker configuration
- Kafka Custom Resource status
Cassandra diagnostics
The engine collects Cassandra diagnostics, including:
- nodetool status output
- nodetool info output
- Table statistics
- CQL validation results
Elasticsearch diagnostics
The engine collects Elasticsearch diagnostics, including:
- Cluster health
- Index statistics
- Shard allocation
- Elasticsearch Custom Resource status
BeeInstana diagnostics
The engine collects BeeInstana diagnostics, including:
- Aggregator logs
- Ingestor logs
- Monconfig logs
- BeeInstana Custom Resource status
- Internal metrics
Kubernetes node-level service logs
The engine collects node-level service logs for standard Kubernetes distributions, including managed cloud platforms. Examples include:
- kubelet
- containerd
- Docker runtime (where applicable)
OpenShift-specific platform service logs
The engine collects OpenShift-specific platform service logs alongside the Kubernetes node-level service logs. Examples include:
- CRI-O
- OpenShift API Server
- Machine Config Daemon
- Other platform components
Note:
On non-OpenShift clusters, this collector exits without producing output.
Deep-debug scripts (requires --deep-debug)
When you enable --deep-debug, the engine runs an additional set of deep-debug scripts after the standard collection completes. Examples include:
- Extended network diagnostics
- Extended datastore inspection

Default namespaces

The diagnostics engine uses the following default namespaces.

Table 2. Default namespaces
Component	Default namespace
ClickHouse	`instana-clickhouse`
PostgreSQL	`instana-postgres`
BeeInstana	`instana-beeinstana`
Elasticsearch	`instana-elasticsearch`
Cassandra	`instana-cassandra`
Kafka	`instana-kafka`
Instana Agent	`instana-agent`

The engine automatically discovers the Core and Unit namespaces from the cluster.

Environment variable reference

You can supply configuration using environment variables directly or by passing a .env file with the --env-file flag. Where the default value is listed as Auto-discovered, the engine reads the value from Kubernetes Secrets in the relevant namespace.

ClickHouse

Table 3. ClickHouse environment variables
Variable	Default	Description
`CLICKHOUSE_NAMESPACE`	`instana-clickhouse`	ClickHouse namespace
`CLICKHOUSE_POD_NAME`	`chi-clickhouse-local-0-0-0`	Target ClickHouse pod
`CLICKHOUSE_CLUSTER`	`local`	Cluster name
`CLICKHOUSE_WINDOW_HOURS`	`24`	Hours of diagnostic data to collect
`CLICKHOUSE_ADMIN_USER`	Auto-discovered	Admin username
`CLICKHOUSE_ADMIN_PASSWORD`	Auto-discovered	Admin password
`CLICKHOUSE_USER`	Auto-discovered	Application username
`CLICKHOUSE_PASSWORD`	Auto-discovered	Application password

PostgreSQL

Table 4. PostgreSQL environment variables
Variable	Default	Description
`POSTGRES_NAMESPACE`	`instana-postgres`	PostgreSQL namespace
`POSTGRES_ADMIN_USER`	Auto-discovered	Admin username
`POSTGRES_ADMIN_PASSWORD`	Auto-discovered	Admin password
`POSTGRES_USER`	Auto-discovered	Application username
`POSTGRES_PASSWORD`	Auto-discovered	Application password

BeeInstana

Table 5. BeeInstana environment variables
Variable	Default	Description
`BEEINSTANA_NAMESPACE`	`instana-beeinstana`	BeeInstana namespace
`BEEINSTANA_USER`	Auto-discovered	Application username
`BEEINSTANA_PASSWORD`	Auto-discovered	Application password
`BEEINSTANA_POD_SELECTOR`	`app.kubernetes.io/name=beeinstana`	Pod selector for BeeInstana pods

Elasticsearch

Table 6. Elasticsearch environment variables
Variable	Default	Description
`ELASTICSEARCH_NAMESPACE`	`instana-elasticsearch`	Elasticsearch namespace
`ELASTICSEARCH_ADMIN_USER`	Auto-discovered	Admin username
`ELASTICSEARCH_ADMIN_PASSWORD`	Auto-discovered	Admin password
`ELASTICSEARCH_USER`	Auto-discovered	Application username
`ELASTICSEARCH_PASSWORD`	Auto-discovered	Application password
`ELASTICSEARCH_POD_SELECTOR`	`common.k8s.elastic.co/type=elasticsearch`	Pod selector for Elasticsearch pods

Cassandra

Table 7. Cassandra environment variables
Variable	Default	Description
`CASSANDRA_NAMESPACE`	`instana-cassandra`	Cassandra namespace
`CASSANDRA_ADMIN_USER`	Auto-discovered	Admin username
`CASSANDRA_ADMIN_PASSWORD`	Auto-discovered	Admin password
`CASSANDRA_USER`	Auto-discovered	Application username
`CASSANDRA_PASSWORD`	Auto-discovered	Application password
`CASSANDRA_POD_SELECTOR`	`app.kubernetes.io/name=cassandra`	Pod selector for Cassandra pods

Kafka

Table 8. Kafka environment variables
Variable	Default	Description
`KAFKA_NAMESPACE`	`instana-kafka`	Kafka namespace
`KAFKA_ADMIN_USER`	Auto-discovered	Admin username
`KAFKA_ADMIN_PASSWORD`	Auto-discovered	Admin password
`KAFKA_CONSUMER_USER`	Auto-discovered	Consumer username
`KAFKA_CONSUMER_PASSWORD`	Auto-discovered	Consumer password
`KAFKA_PRODUCER_USER`	Auto-discovered	Producer username
`KAFKA_PRODUCER_PASSWORD`	Auto-discovered	Producer password
`KAFKA_SASL_MECHANISM`	`SCRAM-SHA-512`	SASL mechanism
`KAFKA_SASL_PLAINTEXT`	`SASL_PLAINTEXT`	SASL transport protocol
`KAFKA_POD_SELECTOR`	`app.kubernetes.io/name=kafka`	Pod selector for Kafka pods

Core and Agent

Table 9. Core and agent environment variables
Variable	Default	Description
`CORE_NAMESPACE`	Auto-discovered	Namespace for the Instana Core component
`CORE_NAME`	Auto-discovered	Name of the Instana Core instance
`INSTANA_AGENT_NAMESPACE`	`instana-agent`	Instana Agent namespace

Backend API credentials

Table 10. Backend API credentials
Variable	Description
`ADMIN_API_USER`	Admin API username (auto-discovered)
`ADMIN_API_PASSWORD`	Admin API password (auto-discovered)
`SERVICE_API_USER`	Service API username (auto-discovered)
`SERVICE_API_PASSWORD`	Service API password (auto-discovered)

Docker registry

Table 11. Docker registry environment variables
Variable	Default	Description
`DOCKER_REGISTRY_USER`	`_`	Registry username
`DOCKER_REGISTRY_PASSWORD`	Value of --download-key	Registry password

Example .env file

Create a .env file to override default configuration values.

CLICKHOUSE_NAMESPACE=my-clickhouse
KAFKA_NAMESPACE=my-kafka

CLICKHOUSE_ADMIN_USER=clickhouse_operator_user
CLICKHOUSE_ADMIN_PASSWORD=s3cr3t

CLICKHOUSE_WINDOW_HOURS=48

CASSANDRA_POD_SELECTOR=app=instana-cassandra,tier=database

Run the command with your overrides file:

kubectl instana diagnostics \
  --download-key <download-key> \
  --env-file ./my-overrides.env

Adjusting log level for Instana components

To adjust the level for Instana components, complete following steps:

Configure a component’s log level in the CoreSpec. In the following example, you change the log level to DEBUG for the butler component:

apiVersion: instana.io/v1beta2
kind: Core
metadata:
  name: instana-core
  namespace: core
spec:
  ...
  componentConfigs:
    - name: butler
      env:
        - name: COMPONENT_LOGLEVEL
          # Possible values are DEBUG, INFO, WARN, ERROR (not case-sensitive)
          value: DEBUG

View the logs by running the following command:
```
kubectl logs <component name> -n instana-core
 
```
<component name> is the component name that you want to troubleshoot.

Using kubectl-specific command to debug and diagnose

The kubectl cluster-info dump command is a helpful tool for debugging and diagnosing Kubernetes clusters. It provides a detailed report on the current state of the Kubernetes cluster, including different information of resources.

Configure the target namespace and the directory to output debugging information. In the following example, the namespace is instana-units and the directory is temp. When you run this command, kubectl generates YAML files for each resource in the instana-units namespace and saves those YAML files to the temp directory.

kubectl cluster-info dump --namespace instana-units --output-directory temp --output yaml

The following extract shows part of the content in the temp directory. In the instana-units subdirectory, you can find details about the daemonsets, deployments, events, pods, and other resources in .yaml files. Logs on all the pods are in each instana-units\pod name subdirectory.

├── instana-units
│   ├── daemonsets.yaml
│   ├── deployments.yaml
│   ├── events.yaml
│   ├── pods.yaml
│   ├── replicasets.yaml
│   ├── replication-controllers.yaml
│   ├── services.yaml
│   ├── tu-instana-prod-appdata-legacy-converter-755bb474c7-xn4vg
│   │   └── logs.txt
│   ├── tu-instana-prod-appdata-processor-6b8f448584-nmvgl
│   │   └── logs.txt
│   ├── tu-instana-prod-filler-9485b85d-wj7pv
│   │   └── logs.txt
│   ├── tu-instana-prod-issue-tracker-bbd5f5d5f-98zxx
│   │   └── logs.txt
│   ├── tu-instana-prod-processor-fc956c46c-fxs5z
│   │   └── logs.txt
│   └── tu-instana-prod-ui-backend-89bccd9c5-8lp76
│       └── logs.txt
...
└── nodes.yaml

Note:

You can choose another namespace such as instana-core.
If kubectl is not installed on your cluster, you can use the command oc cluster-info dump, which provides the same support as the kubectl cluster-info dump command.

For more troubleshooting commands, see Troubleshooting Clusters through kubectl and Red Hat OpenShift CLI developer command reference.

Using internal backend API

You can use some internal component API endpoints to help you administer a self-hosted Instana backend. You can submit these API calls from the cluster to the corresponding pods using curl. Because the resources require authentication, you must first get valid credentials. You can find the API credentials in the internal-instana secret in the core namespace. The API endpoints contain two types of AdminAPIUser and ServiceAPIUser. To get the credentials that are valid for an Instana installation, run the following commands:

kubectl get secret instana-internal -n instana-core --template='{{ index .data.serviceAPIUser  | base64decode }}'

kubectl get secret instana-internal -n instana-core --template='{{ index .data.serviceAPIPassword  | base64decode }}'

To get the credentials for the admin user, query .data.adminAPIUser and .data.adminAPIPassword.

You can use these credentials for the following procedures.

Resetting Instana user password

To reset the password for an Instana user account, run the following command:

kubectl exec -it -n instana-core deploy/butler -- curl -X PUT http://localhost:8601/admin/authentication/{tenant}/reset/user -u {adminAPIUser}:{adminAPIPassword} -H 'Content-Type: application/json' -d '{"email":"{user}","pass":"{newPassword}"}'

Deactivating SSO provider configurations (LDAP/SAML/OIDC)

If you use an Identity Provider for Instana authentication, deactivate that provider with the following command. Afterward, you can use the internal Instana user accounts for authentication.

kubectl exec -it -n instana-core deploy/butler -- curl -X PUT http://localhost:8601/admin/authentication/{tenant}/idp -u {adminAPIUser}:{adminAPIPassword}

Disabling 2FA on a user account

To deactivate 2FA (two-factor authentication) for an Instana user account, run the following command:

kubectl exec -it -n instana-core deploy/butler -- curl -X DELETE http://localhost:8601/admin/2fa/users/{email} -u {adminAPIUser}:{adminAPIPassword}

Verifying licenses

To verify all stored licenses, run the following command:

kubectl exec -it -n instana-core deploy/groundskeeper -- curl -X GET http://localhost:8600/license/list/{tenant}/{unit} -u {serviceAPIUser}:{serviceAPIPassword}

Some metrics are not displayed on dashboards

If dashboards do not display some metrics, you probably reached the metric limit. The limit is set to 3,000 by default.

Tip: You can see the current limit under maxMetrics in the filler/config.yaml file. The limit is there to prevent some entities from sending too many metrics, which would increase storage and CPU used by filler, and increase the bandwidth to send metrics to the UI.

To increase the limit, add the block with config.max.metrics in the Custom Resource for Unit as follows:

kind: unit
...
spec:
...
  properties:
    - name: config.max.metrics
      value: "6000"
...

Instana backend becomes non‑functional when the Elasticsearch data disk exceeds 85% usage

Elasticsearch automatically switches its data store to read‑only mode when the disk it uses exceeds 85% usage. This causes the Instana backend to stop functioning. Free up space on the Elasticsearch data disk or increase its capacity to restore normal operations. Note: Other Instana disks do not trigger read‑only behavior at similar usage levels (even greater than 95%), which can make this issue appear confusing.

To resolve this issue, do one of the following measures:

Free space on the Elasticsearch data disk
Increase the disk size allocated to Elasticsearch

When sufficient space is available, Elasticsearch resumes normal write operations, and the backend becomes functional again.

Note:

Other Instana disks do not enter read‑only mode at similar usage levels (even above 95%), which might lead to confusion during troubleshooting.

License is invalid or missing

If the license is invalid or missing, the backend prevents agents from connecting.

When this occurs

The imported license is invalid.
The Instana Operator cannot apply the license to the Groundskeeper backend.

How to troubleshoot

Verify that the Sales Key in the core secret matches the license strings in the unit secret. If they differ, re-download the license using the correct Sales Key.

Check the Instana Operator logs for license import errors:

kubectl logs -n instana-operator deployment/instana-operator --tail=100

Check the Groundskeeper backend component, pod status, and logs:
```
kubectl get pods -n instana-core | grep groundskeeper 
```
If the license still shows an invalid state, contact IBM Support.