Instana AutoTrace webhook
The Instana AutoTrace webhook is a Kubernetes and Red Hat OpenShift-compatible admission controller mutating webhook. The webhook automatically configures Instana tracing on Node.js, .NET Core, Ruby, and Python applications that run across the entire Kubernetes or Red Hat OpenShift cluster. In addition, you can enable Instana tracing for IBM MQ and App Connect deployments that run in IBM Cloud Pak for Integration.
Limitations
- The Instana AutoTrace webhook works only on new Kubernetes resources. After the webhook is installed, you must create new resources for the transformation to take effect. Delete your Pods, ReplicaSets, StatefulStes, Deployments, and DeploymentConfigs,
and then create them again for the Instana AutoTrace webhook to complete its configuration (for example, by using
kubectl apply
). - Only
linux/amd64
Kubernetes nodes are supported. - Instana AutoTrace webhook does not automatically update the instrumentation that it installs. Therefore, Instana AutoTrace webhook is not enabled by default.
- If Instana Autotrace webhook is installed on IBM Cloud Paks (Cloud Pak for Business Automation and Cloud Pak for Integration), it requires excluding Zen from instrumentation.
Supported runtimes
The Instana AutoTrace webhook supports for the following technologies:
Prerequisites
Before you install the Instana AutoTrace webhook on a Kubernetes-based cluster, confirm that the following prerequisites are met:
- Kubernetes 1.16+
- Red Hat OpenShift 4.5+
kubectl
1.16+- Helm 3.2+ (Some automation relies on Helm
lookup
functions)
Installing the Instana AutoTrace webhook
Replace <download_key> in the following script with a valid Instana agent key or download key, and then run the script with administrator privileges for your cluster:
helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook \
--repo https://agents.instana.io/helm instana-autotrace-webhook \
--set webhook.imagePullCredentials.password=<download_key>
Important: If you are installing on Red Hat OpenShift, you must specify the --set openshift.enabled=true
option in the script.
Installing the Instana Autotrace webhook on IBM Cloud Paks
For instructions about how to install the Instana Autotrace webhook on IBM Cloud Paks, see How to configure Instana AutoTrace webhook in a Cloud Pak Environment.
Configuring in an air-gapped environment
For air-gapped environments, you need to provide the instana-autotrace-webhook
Helm chart, the instana-autotrace-webhook
container image, and the instrumentation
container image in your environment.
To download the latest release of the Helm chart to your current working directory, run the following command:
helm pull instana-autotrace-webhook --repo https://agents.instana.io/helm instana-autotrace-webhook
To change the destination directory of this command, you can use the option -d <DESTINATION_PATH>
to specify another destination directory.
To download the latest instana-autotrace-webhook
image, run the following command:
docker pull containers.instana.io/instana/release/agent/instana-autotrace-webhook:latest
To download the latest instrumentation image, run the following command:
docker pull icr.io/instana/instrumentation:latest
The two container images need to be available in your container registry. The previously downloaded Helm chart archive must be available on the system that runs the helm install
command.
To install the Instana AutoTrace webhook, run the following command:
helm upgrade --install --create-namespace \
--namespace instana-autotrace-webhook \
--set webhook.image=<INSTANA_AUTOTRACE_WEBHOOK_IMAGE_PATH> \
--set autotrace.instrumentation.image=<INSTRUMENTATION_IMAGE_PATH> \
instana-autotrace-webhook <PATH_TO_HELM_CHART_ARCHIVE>
Container registry authentication
The instrumentation
image in the container registry is used as an initContainer in all application pods. If your container registry requires the imagePullSecret resource, it must be available in all application namespaces.
Verifying that the webhook works
To verify that the webhook works, complete the following steps:
-
To verify that the
instana-autotrace-webhook
in theinstana-autotrace-webhook
namespace is running as expected, run the following command:kubectl get pods -n instana-autotrace-webhook
Example result:
NAME READY STATUS RESTARTS AGE instana-autotrace-webhook-7c5d5bf6df-82w7c 1/1 Running 0 12m
-
Use the Instana AutoTrace webhook. If the Instana AutoTrace webhook is running, deploy a Node.js pod. Instana AutoTrace is automatically enabled in the Node.js pod. In the log for the pod, you can see a label that indicates that the Instana AutoTrace was applied to the Node.js pod:
kubectl get pod test-nodejs -n test-apps -o=jsonpath='{.metadata.labels.instana-autotrace-applied}' true
If you installed the Instana host agent by using the instana/agent
Helm chart, the Node.js process appears in your Instana dashboard. For more information, see the Installing the Host Agent on Kubernetes documentation.
However, if you do not see the instana-autotrace-applied
labels appear on your containers, see Troubleshooting.
Updating Instana AutoTrace webhook and instrumentation
Instana AutoTrace webhook does not have an automated way of updating the instrumentation that it installs. The instrumentation is delivered over the icr.io/instana/instrumentation
image. The instana-autotrace-webhook
Helm chart is regularly updated to use the latest icr.io/instana/instrumentation
image.
To update Instana AutoTrace webhook and instrumentation, complete the following steps:
-
Update the Helm chart repository on your local Helm installation with the following command:
helm repo update
-
Upgrade the
instana-autotrace-webhook
Helm chart deployment with the following command:helm upgrade --namespace instana-autotrace-webhook instana-autotrace-webhook \ --repo https://agents.instana.io/helm instana-autotrace-webhook \ --reuse-values
Uninstalling the Instana AutoTrace webhook
To uninstall the Instana AutoTrace webhook, run the following command:
helm uninstall instana-autotrace-webhook \
--namespace instana-autotrace-webhook \
--no-hooks
After you run the helm uninstall
command, the following output is displayed: release "instana-autotrace-webhook" uninstalled
.
Verify that the Instana AutoTrace webhook was uninstalled correctly. You can verify by clicking Kubernetes > Clusters in the Instana UI or by running the following command:
kubectl get pods --namespace instana-autotrace-webhook
If uninstallation is successful, the instana-autotrace-webhook
pod no longer appears in the namespace.
To remove AutoTrace from deployed applications and prevent AutoTrace from being included in new applications, redeploy all higher-order resources that are formerly mutated by the AutoTrace webhook. This ensures that all the AutoTrace configuration (init-containers and environment variables) gets removed from the resource specs and pod templates.
Configurations
Disabling tracers
You can individually disable tracers by using the following option --set autotrace.[technology].enabled=false
. The following technologies are available:
- .Net Core (netcore)
- Python (python)
- Node.js (nodejs)
- Ruby (ruby)
Example:
helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook \
--repo https://agents.instana.io/helm instana-autotrace-webhook \
--set webhook.imagePullCredentials.password=<download_key> \
--set autotrace.ruby.enabled=false \
Role-based access control
To deploy the AutoTrace webhook into a ServiceAccount
guarded by a ClusterRole
and matching ClusterRoleBinding
, set the rbac.enabled=true
flag when you are deploying the Helm chart.
In addition to the role-based access control, if you use pod security policies, add rbac.psp.enabled=true
to the Helm arguments.
Pod Security Standards can also be enforced through the built-in Pod Security Admission controller. For more information about the Pod Security Admission, see Kubernetes documentation.
If the flags rbac.enabled=false
and webhook.pod.hostNetwork=false
are set in the Helm installation, you can run the AutoTrace webhook with the restrictive Pod Security Standard by running the following command:
kubectl label --overwrite ns instana-autotrace-webhook pod-security.kubernetes.io/enforce=restricted
Container port
To be reachable from Kubernetes' apiserver, the AutoTrace webhook pod must be hosted on the host network, and the deployment must be configured to achieve that transparently. By default, the container is bound to port 42650
.
If port 42650
is in use, the AutoTrace webhook crashes because its port is already bound. You can change the port by using the webhook.pod.port
property.
Opt-in or opt-out
The AutoTrace webhook instruments all containers in all pods. However, you can have more control over which features are instrumented and which are not. By setting the autotrace.opt_in=true
value when you are deploying the Helm
chart, the AutoTrace webhook modifies pods, replica sets, stateful sets, daemon sets, and deployments that carry the instana-autotrace: "true"
label.
Irrespective of the value of the autotrace.opt_in
, the AutoTrace webhook does not touch pods that carry the instana-autotrace: "false"
label.
The instana-autotrace: "false"
label is respected in the metadata of DaemonSets, Deployments, DeploymentConfigs, ReplicaSets, and StatefulSets, as in nested pod templates and in stand-alone pods.
Ignoring namespaces
By using the autotrace.exclude.namespaces
configuration, you can exclude entire namespaces from being auto-instrumented.
Resources that have the instana-autotrace: "true"
label are instrumented regardless of namespace exclusion.
The instana-autotrace
label is respected in the metadata of DaemonSets, Deployments, DeploymentConfigs, ReplicaSets, and StatefulSets, as in nested pod templates and in stand-alone pods.
Ignoring resources
Resources that have the instana-autotrace: "false"
label are ignored regardless of other settings.
The instana-autotrace
label is respected in the metadata of DaemonSets, Deployments, DeploymentConfigs, ReplicaSets, and StatefulSets, as in nested pod templates and in stand-alone pods.
NGINX and ingress-nginx
To activate the NGINX and ingress-nginx auto-instrumentation, you must opt-in by setting autotrace.ingress_nginx.enabled=true
. Make sure to read the troubleshooting section afterward, and make sure that the relevant objects are
updated or re-created.
The AutoTrace webhook supports the ingress-nginx Kubernetes ingress controller 0.34.1 or later with a support policy of 45 days, and it is compatible with Helm chart 2.11.2 or later.
IBM MQ and ACE
To activate the IBM MQ and ACE auto-instrumentation, you must opt-in by setting autotrace.ibmmq.enable=true
and autotrace.ace.enable=true
. The AutoTrace webhook supports only IBM MQ and ACE running in IBM Cloud Pak
for Integration. IBM Cloud Pak for Integration runs on the Red Hat OpenShift cluster, and thus you must also set openshift.enabled=true
. To set up Instana AutoTrace webhook with the IBM MQ and ACE auto-instrumentation that are
enabled, enter the following command:
helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook \
--repo https://agents.instana.io/helm instana-autotrace-webhook \
--set webhook.imagePullCredentials.password=<download_key> \
--set openshift.enabled=true \
--set autotrace.ibmmq.enabled=true \
--set autotrace.ace.enabled=true
Node.js ECMAScript modules
The ECMAScript modules are in the experimental phase.
In Autotrace webhook 1.272.2, a new configuration option autotrace.nodejs.application_type
is added due to the breaking changes introduced in Node.js 18.19.0. To ensure that
you are using the latest version of the AutoTrace webhook, see the updates guide.
-
For Node.js 18.19.0 and later, set
autotrace.nodejs.application_type
tomodule_v2
:helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook \ --repo https://agents.instana.io/helm instana-autotrace-webhook \ --set webhook.imagePullCredentials.password=<download_key> \ --set autotrace.nodejs.application_type=module_v2
-
For versions before Node.js 18.19.0, set
autotrace.nodejs.application_type
tomodule_v1
:helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook \ --repo https://agents.instana.io/helm instana-autotrace-webhook \ --set webhook.imagePullCredentials.password=<download_key> \ --set autotrace.nodejs.application_type=module_v1
-
To revert to the default behavior or remove the configuration option entirely, set
autotrace.nodejs.application_type
tocommonjs
:helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook \ --repo https://agents.instana.io/helm instana-autotrace-webhook \ --set webhook.imagePullCredentials.password=<download_key> \ --set autotrace.nodejs.application_type=commonjs
The previous configuration option autotrace.nodejs.esm
is now deprecated. You must change to the new configuration option autotrace.nodejs.application_type
.
Troubleshooting
If you do not see the Instana AutoTrace webhook impacting your new Kubernetes resources, troubleshoot by using the following steps:
Verifying that the Instana AutoTrace webhook is receiving requests
To verify that the Instana AutoTrace webhook is receiving requests, check the logs of the instana-autotrace-webhook
pod by running the following command:
kubectl logs -l app.kubernetes.io/name=instana-autotrace-webhook -n instana-autotrace-webhook
In a functioning installation, you can see the following logs:
14:41:37.590 INFO |- [AdmissionReview 48556a1a-7d55-497b-aa9c-23634b089cd1] Applied transformation DefaultDeploymentTransformation to the Deployment 'test-netcore-glibc/test-apps'
14:41:37.588 INFO |- [AdmissionReview 1d5877cf-7153-4a95-9bfb-de0af8351195] Applied transformation DefaultDeploymentTransformation to the Deployment 'test-nodejs-12/test-apps'
If you do not see such logs, a problem with Kubernetes setup might exist. Continue troubleshooting by checking the following section.
Checking the kube-apiserver logs
Check the logs of your kube-apiserver
. These logs report on whether the Instana AutoTrace webhook is being started and provide information about the outcome of the execution.
Common issues
No network connectivity between kube-apiserver and the instana-autotrace-webhook pods
The most common issue is that the kube-apiserver
cannot reach the worker nodes that are running the instana-autotrace-webhook
pods due to security policies, which prevents the Instana AutoTrace webhook to work.
In this case, the solution is to change the network settings so that kube-apiserver
can access the instana-autotrace-webhook
pods. Review your network security policies to make sure that kubeapi-server
can initiate connections and receive responses from instana-autotrace-webhook
. Instana cannot provide direct guidance for resolving this issue because it solutions vary based on your policy and enforcement mechanisms.
kube-apiserver and the instana-autotrace-webhook pods cannot negotiate a TLS session
Another sporadic issue that can occur is when cryptography restrictions, specifically in terms of which algorithms can be used for TLS, prevent kube-apiserver
from negotiating a TLS session with the instana-autotrace-webhook
pod. In this case, open a ticket, and inform Instana support about which cryptography algorithms your clusters support.