How to set the Instana Agent DEBUG logs on Kubernetes environment by using Helm Chart

1. delete the existing Instana agent installed on Kubernetes environment by using Helm Chart

Note: You can use "helm upgrade" command instead of "helm install" command without deleting the existing Instana agent

helm del instana-agent -n instana-agent

2. execute the following commands

helm repo add instana-agent-repo https://agents.instana.io/helm
helm repo update
helm pull instana-agent-repo/instana-agent
tar xzvf instana-agent-*.tgz

3. modify the instana-agent/values.yaml file 
from
-----------------
  env: {}
-----------------
to
-----------------
  env:
    INSTANA_LOG_LEVEL: "DEBUG"
-----------------

4. Run helm install to pass the modified values.yaml

helm install instana-agent \
   --repo https://agents.instana.io/helm \
   --namespace instana-agent \
   --create-namespace \
   -f instana-agent/values.yaml \
   --set agent.key=<AGENT_KEY> \
   --set agent.downloadKey=<DOWNLOAD_KEY> \
   --set agent.endpointHost=<ENDPOINT_HOST> \
   --set agent.endpointPort=<ENDPOINT_PORT> \
   --set cluster.name='cluster_name' \
   --set zone.name='zone_name' \
   instana-agent

5. check if the Instana agent is running

kubectl get all -n instana-agent
kubectl logs -l app.kubernetes.io/name=instana-agent -n instana-agent -c instana-agent

6. use the pod name of Instana agent in the output of "kubectl get all -n instana-agent" and confirm if the Agent Log Level was set to "DEBUG"

kubectl exec -it <POD_NAME_OF_INSTANA_AGENT> -c instana-agent -n instana-agent -- grep "log4j2.logger.instana.level" /opt/instana/agent/etc/org.ops4j.pax.logging.cfg
kubectl exec -it <POD_NAME_OF_INSTANA_AGENT> -c instana-agent -n instana-agent -- env | grep "INSTANA_LOG_LEVEL"

<example>
---------------------------------------------
$ helm del instana-agent -n instana-agent
release "instana-agent" uninstalled
$ 
$ helm del instana-agent -n instana-agent
release "instana-agent" uninstalled
$ 
$ helm repo add instana-agent-repo https://agents.instana.io/helm
"instana-agent-repo" has been added to your repositories
$ 
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "instana-agent-repo" chart repository
Update Complete. ⎈Happy Helming!⎈
$ 
$ helm pull instana-agent-repo/instana-agent
$ 
$ tar xzvf instana-agent-*.tgz
instana-agent/Chart.yaml
instana-agent/values.yaml
instana-agent/templates/NOTES.txt
instana-agent/templates/_helpers.tpl
instana-agent/templates/agent-configmap.yaml
instana-agent/templates/agent-daemonset-with-zones.yaml
instana-agent/templates/agent-daemonset.yaml
instana-agent/templates/clusterrole.yaml
instana-agent/templates/clusterrolebinding.yaml
instana-agent/templates/headless-service.yaml
instana-agent/templates/k8s-sensor-configmap.yaml
instana-agent/templates/k8s-sensor-deployment.yaml
instana-agent/templates/k8s-sensor-pod-disruption-budget.yaml
instana-agent/templates/k8s-sensor-podsecuritypolicy.yaml
instana-agent/templates/k8s-sensor-role.yaml
instana-agent/templates/k8s-sensor-rolebinding.yaml
instana-agent/templates/k8s-sensor-serviceaccount.yaml
instana-agent/templates/kubernetes-sensor-configmap.yaml
instana-agent/templates/kubernetes-sensor-deployment.yaml
instana-agent/templates/namespace.yaml
instana-agent/templates/podsecuritypolicy.yaml
instana-agent/templates/secrets.yaml
instana-agent/templates/service.yaml
instana-agent/templates/serviceaccount.yaml
instana-agent/.helmignore
instana-agent/DEPLOYMENT.md
instana-agent/README.md
instana-agent/kubernetes.deployment.enabled.png
instana-agent/kubernetes_deployment_mode_diagram.py
$ 
$ vi instana-agent/values.yaml
$ grep -A 2 "  env:" instana-agent/values.yaml
  env:
    INSTANA_LOG_LEVEL: "DEBUG"

$ 
$ helm install instana-agent \
   --repo https://agents.instana.io/helm \
   --namespace instana-agent \
   --create-namespace \
   -f instana-agent/values.yaml \
   --set agent.key=<AGENT_KEY> \
   --set agent.downloadKey=<DOWNLOAD_KEY> \
   --set agent.endpointHost=<ENDPOINT_HOST> \
   --set agent.endpointPort=<ENDPOINT_PORT> \
   --set cluster.name='cluster_name' \
   --set zone.name='zone_name' \
   instana-agent
NAME: instana-agent
LAST DEPLOYED: Tue Mar  5 01:00:39 2024
NAMESPACE: instana-agent
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
It may take a few moments for the agents to fully deploy. You can see what agents are running by listing resources in the instana-agent namespace:

    kubectl get all -n instana-agent

You can get the logs for all of the agents with `kubectl logs`:

    kubectl logs -l app.kubernetes.io/name=instana-agent -n instana-agent -c instana-agent
$ 
$ kubectl get all -n instana-agent
NAME                            READY   STATUS    RESTARTS   AGE
pod/instana-agent-mgdw4         1/1     Running   0          23s
pod/k8sensor-6b57598746-9qkd9   1/1     Running   0          23s
pod/k8sensor-6b57598746-jqqs6   1/1     Running   0          23s
pod/k8sensor-6b57598746-z2jv8   1/1     Running   0          23s

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/instana-agent            ClusterIP   10.96.247.88   <none>        42699/TCP   23s
service/instana-agent-headless   ClusterIP   None           <none>        42699/TCP   23s

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/instana-agent   1         1         1       1            1           <none>          23s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/k8sensor   3/3     3            3           23s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/k8sensor-6b57598746   3         3         3       23s
$ 
$ kubectl logs -l app.kubernetes.io/name=instana-agent -n instana-agent -c instana-agent
2024-03-05T01:01:06.005+00:00 | INFO  | features-3-thread-1              | turesServiceImpl | org.apache.karaf.features.core - 4.2.16 |   Region: root
2024-03-05T01:01:06.005+00:00 | INFO  | features-3-thread-1              | turesServiceImpl | org.apache.karaf.features.core - 4.2.16 |     Bundles to install:
2024-03-05T01:01:06.005+00:00 | INFO  | features-3-thread-1              | turesServiceImpl | org.apache.karaf.features.core - 4.2.16 |       mvn:com.instana/sensor-process/1.1.48
2024-03-05T01:01:06.005+00:00 | INFO  | features-3-thread-1              | turesServiceImpl | org.apache.karaf.features.core - 4.2.16 | Installing bundles:
2024-03-05T01:01:06.005+00:00 | INFO  | features-3-thread-1              | turesServiceImpl | org.apache.karaf.features.core - 4.2.16 |   mvn:com.instana/sensor-process/1.1.48
2024-03-05T01:01:06.054+00:00 | INFO  | features-3-thread-1              | turesServiceImpl | org.apache.karaf.features.core - 4.2.16 | Starting bundles:
2024-03-05T01:01:06.055+00:00 | INFO  | features-3-thread-1              | turesServiceImpl | org.apache.karaf.features.core - 4.2.16 |   com.instana.sensor-process/1.1.48
2024-03-05T01:01:06.058+00:00 | INFO  | features-3-thread-1              | turesServiceImpl | org.apache.karaf.features.core - 4.2.16 | Done.
2024-03-05T01:01:06.059+00:00 | INFO  | instana-scheduler-thread-3-3     | turesManagerImpl | com.instana.agent-bootstrap - 1.2.31 | Installed instana-process-sensor
2024-03-05T01:01:06.075+00:00 | INFO  | 1e2-a060-41b4-85f8-87a058697c92) | Process          | com.instana.sensor-process - 1.1.48 | Activated Sensor for PID 82645
$ 
$ kubectl exec -it instana-agent-mgdw4 -c instana-agent -n instana-agent -- grep "log4j2.logger.instana.level" /opt/instana/agent/etc/org.ops4j.pax.logging.cfg
log4j2.logger.instana.level = DEBUG
$ 
$ kubectl exec -it instana-agent-mgdw4 -c instana-agent -n instana-agent -- env | grep "INSTANA_LOG_LEVEL"
INSTANA_LOG_LEVEL=DEBUG
$ 
---------------------------------------------