Manta Dataflow pod start failure

Problem

The Manta Dataflow pod isn't starting, enters the CrashLoopBackOff state, becomes unaccessible, and container initialization fails. The pod is waiting for the Neo4j container to start.

If you inspect the pod logs, you might find a message similar to the following one:

oc logs pod/<manta_podname>

Example:

2024-10-11T09:29:46.797144560Z 2024-10-11 09:29:46.796 [main] INFO  eu.profinit.manta.dataflow.repository.server.repository.neo4jembedded.configuration.EmbeddedNeo4jStorageConfiguration [Context: MANTA Server startup - 2024-10-11T09:29:35.906+0000] Waiting for embedded Neo4j database to start...

If the pod enters the Running state and then crashes, open the debug.log file in /opt/mantaflow/server/manta-dataflow-server-dir/data/neo4j/data/databases/neo4j/logs.

If you can't access this path due to the state of the pod, modify the pod YAML configuration. Before you make any changes, back up the original file configuration.

Solution

1. Ensure that you have the necessary permissions to edit deployment configurations and access pods.
2. Go to the Deployments section in your cluster management tool.
3. Locate Manta Dataflow and click Edit YAML.
4. Create a copy of the YAML file.
5. In the spec field, search for the initContainers section.
6. Find the manta-init-migrate-neo4j-5 init container.

7. Add the following line under the container configuration details:

    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
   

   Example:

    spec:
     restartPolicy: Always
     initContainers:
      - name: manta-init-migrate-neo4j-5
        image: 'cp.icr.io/cp/cpd/manta-init-migrate-neo4j-5@sha256:...'
        command: ['sh', '-c', 'echo The app is running! && sleep 3600']
        resources:
         limits:
          cpu: '0'
          ephemeral-storage: 256Mi
          memory: '0'
         requests:
          cpu: '0'
          ephemeral-storage: 64Mi
          memory: '0'
        volumeMounts:
   

   This change delays the start of the container by one hour, which gives you sufficient time to perform necessary actions.

8. Save the edited YAML file.
   A new pod is automatically started with the updated configuration. Wait for the pod to reach the Running state

9. Access the pod.
   Replace <pod_suffix> with the suffix of your pod.

   oc debug manta-dataflow-<pod_suffix> -c manta-init-migrate-neo4j-5
   

10. Retrieve logs, modify configurations, or perform other necessary troubleshooting steps.

11. Restore the YAML file to its original configuration.
    The pod restarts with the initial settings.

12. Monitor logs and pod status to confirm that the issue was resolved.

13. Remove any temporary files or changes made during troubleshooting and document any fixes applied for future reference.