NetObserv plugin ContainerCreating error

Edit online

The netobserv-plugin pods remain in ContainerCreating state due to missing Certificate Authority (CA) bundle ConfigMaps that are required for secure communication with LokiStack. Inspecting the pod events or running oc describe pod identifies recurring volume mount failures.

The Network Observability plugin requires secure communication with LokiStack. To establish trust and validate Loki’s identity, the plugin relies on specific CA bundle ConfigMaps that must be present in the namespace. These ConfigMaps are automatically generated by the Loki Operator when a LokiStack instance named loki-network is correctly configured and running successfully. If the LokiStack instance is missing, misconfigured, or remains in a pending state due to storage-related issues, the required CA bundles are not created. As a result, the Network Observability pods cannot complete initialization and remain stuck in the ContainerCreating state.

One or more symptoms can occur:

Status is shown as ContainerCreating
Events show volume mount failures:
- MountVolume.SetUp failed for volume "loki-status-certs-ca" : configmap "loki-network-ca-bundle" not found
- MountVolume.SetUp failed for volume "loki-certs-ca" : configmap "loki-network-gateway-ca-bundle" not found

Resolution:

Verify the LokiStack existence by checking if the LokiStack custom resource exists in the namespace.
```
oc get lokistack -n netobserv
```
- If custom resource is missing, proceed with step 2.
- If custom resource exists but not in Ready state, then check for storage or secret issues by running the command:
```
oc describe lokistack loki-network
 
```

Deploy or restore the LokiStack. The LokiStack must be named loki-network and configured.

tenants.mode: openshift-network

Apply the following configuration:

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: loki-network
  namespace: netobserv
spec:
  managementState: Managed
  size: 1x.extra-small
  replicationFactor: 1
  storageClassName: odf-storagecluster-ceph-rbd
  hashRing:
    type: memberlist
  tenants:
    mode: openshift-network # Essential for certificate generation
  storage:
    schemas:
      - effectiveDate: '2022-06-01'
        version: v13
    secret:
      name: loki-network-s3
      type: s3
    tls:
      caName: loki-network-storage-ca-bundle
  limits:
    global:
      retention:
        days: 7

Important: Ensure the S3 secret (loki-network-s3) exists and the storageClassName matches the cluster's available storage providers (for example, gp3-csi for AWS/ROSA or odf-storagecluster-ceph-rbd).

Monitor the ConfigMaps and checking pod status.
1. Wait for the operator to generate the bundles.
```
oc get configmap -n netobserv | grep ca-bundle
```
2. Once the ConfigMaps appear, the Kubelet automatically mounts them. The netobserv-plugin pods should transition to Running within few minutes.