Enable debug logs and identify Kata/QEMU processes for Bare Metal

  1. Enable logs in kata configuration file.
    1. Open the file located at /etc/kata-containers/kata-se/configuration.toml.
    2. Set the following flags to true:
      • enable_debug under [hypervisor.qemu]
      • enable_debug under [runtime]
  2. Enable logs in CRI-O.
    1. Check if CRI-O logs are enabled by running the following command:
      $ crio config | grep 'log_level'

      If log_level is debug, logs are enabled.

    2. If the CRI-O logs are not enabled, apply the following MachineConfig to set log level to debug:
      
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        name: 99-worker-crio-loglevel-debug
        labels:
          machineconfiguration.openshift.io/role: worker
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
              - path: /etc/crio/crio.conf.d/99-loglevel.conf
                mode: 0644
                overwrite: true
                contents:
                  source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZV0KbG9nX2xldmVsID0gImRlYnVnIgo=
      
    3. Apply the configuration by running the following command:
      $ oc apply -f 99-worker-crio-loglevel-debug.yaml
  3. Identify QEMU and kata processes for a specific workload.
    1. Retrieve the pod’s MAC address by running the following command:
      $ oc describe pod <pod-name>
    2. Access kata runtime logs on the node by running the following command:
      $ journalctl -x -t kata -f
    3. Search for the QEMU process using the MAC address:

      • QEMU process name, for example: sandbox-a7cfa03549418c02743b9470baedf91d9e2cfbca045836b0a677710a6610150f
      • Kata process ID, for example: kata[1991251]
    4. Use these details to filter logs for targeted debugging and isolate issues related to the specific pod or VM sandbox.
Important: For Bare Metal instances, monitor logs from the log forwarder service to ensure proper logging pipeline behavior. In the console, these logs display the [Forward from journalctl] tag. Verify that logs appear consistently and without interruption.