Enable core memory dump on a cluster

Core memory dumps are useful in debugging in a wide number of scenarios, such as the following cases:
  • The process exits without a useful stack trace.
  • The process runs out of memory.
  • An application does not behave as expected.
When you enable the core memory dumps, the following things occur.
  • Multiple pods might be running on the clusters. Therefore, the pod restarts and failures might generate a large amount of data.
  • Less RAM can be available because the memory is reserved for the crash kernel.
  • More storage space is being used to store the crash memory dumps.
  • The daemonset that is created by using the privileged user can read the core files by querying the detailed information for a pod.

Enable the core memory dump

Use the following steps to enable and collect the core memory dumps that are generated when any container ends unexpectedly. These steps are not part of the deployment process. You can choose whether to enable the core memory dump or not. The core memory dump files are stored on the persistent volume /opt/ibm/ftm/debug-data/dumps/.
  1. Open a terminal and log in to the cluster as a cluster administrator.
  2. The daemonset can be created only by a privileged user. Create a service account for a privileged user.
  3. Log in to the cluster on the virtual machine. Create the following daemonset.yaml file.
    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: init-node
spec:
  selector:
    matchLabels:
      job: init-node
  template:
    metadata:
      labels:
        job: init-node
    spec:
      hostPID: true
      serviceAccountName: privilegeduser
      restartPolicy: Always
      containers:
      - image: registry.access.redhat.com/ubi8/ubi-init
        name: init-node
        securityContext:
          privileged: true
        command: ["/bin/sh"]
        args: ["-c", "nsenter -t 1 -m -- su -c \"echo \\\"/opt/ibm/ftm/debug-data/dumps/core.%P.%u.%g.%s.%t.%c.%h.%e\\\" > /proc/sys/kernel/core_pattern\" && sleep infinity"]
    In the daemonset.yaml file, ensure that the following fields have the correct values.
    • The restartPolicy field is set to Always.
    • You must run the scripts in the context of the host namespace. The nsenter command is used to run in the current namespace.
  4. Run the following command to create the daemonset from the daemonset.yaml file.
    oc create -f daemonset.yaml

Core memory dump file

When a core memory dump occurs, a file is created on the persistent volume. The name of the pod and the namespace is included in the file name. An example of the file name is core.2517940.1000670000.0.11.1653631281.18446744073709551615.FTMpodname-ftm-demo.sh.

Disable the core memory dump

If you want to disable the core memory dump, run the following command.
  1. Log in to the cluster on the virtual machine.
  2. Delete the daemonset that you created to enable the core memory dump. 
    oc delete daemonset init-node