Minio buckets and objects are intermittently not accessible

When Minio is deployed in NAS mode and you connect from a client, the created buckets and objects are intermittently not accessible.

Symptoms

Create a Minio objectstore.

mc config host add myminio http://minio-nas-ibm-minio-objectstore:9000 admin admin1234 S3v4

Following is a sample output:

Added `myminio` successfully.

Create a bucket.

mc mb myminio/test

Following is a sample output:

Bucket created successfully `myminio/test`.

When you run the MinIO Client (mc) command to list the buckets and objects, you might see the following error:

mc ls myminio/test
mc: <ERROR> Unable to stat `myminio/test`. Bucket `test` does not exist.
/ # mc ls myminio/test
mc: <ERROR> Unable to stat `myminio/test`. Bucket `test` does not exist.
/ # mc ls myminio/test
mc: <ERROR> Unable to stat `myminio/test`. Bucket `test` does not exist.
/ # mc ls myminio/test
mc: <ERROR> Unable to stat `myminio/test`. Bucket `test` does not exist.
/ # mc ls myminio/test
mc: <ERROR> Unable to stat `myminio/test`. Bucket `test` does not exist.
/ # mc ls myminio/test
mc: <ERROR> Unable to stat `myminio/test`. Bucket `test` does not exist.
/ # mc ls myminio/test
mc: <ERROR> Unable to stat `myminio/test`. Bucket `test` does not exist.

To identify the cause, describe a Minio pod and verify that it is using the intended persistent volume.

Install kubectl. For more information, see Installing the Kubernetes CLI (kubectl).

Get the Minio pod name.

kubectl get pods | grep minio

Following is a sample output:

minio-nas-ibm-minio-objectstore-974f85dc9-7mlrv   1/1       Running   0          4m

Get the Minio pod information.

kubectl describe po minio-nas-ibm-minio-objectstore-974f85dc9-7mlrv

{codeblock}

Following is a sample output:

Name:               minio-nas-ibm-minio-objectstore-974f85dc9-7mlrv
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               10.41.14.22/10.41.14.22
Start Time:         Thu, 25 Apr 2019 22:13:12 -0700
Labels:             app=ibm-minio-objectstore
                   chart=ibm-minio-objectstore-2.4.7
                   heritage=Tiller
                   pod-template-hash=974f85dc9
                   release=minio-nas
Annotations:        kubernetes.io/psp: 00-rook-ceph-operator
                   productID: Minio_RELEASE.2019-04-09T01-22-30Z_free_00000
                   productName: Minio
                   productVersion: RELEASE.2019-04-09T01-22-30Z
                   scheduler.alpha.kubernetes.io/critical-pod:
Status:             Running
IP:                 10.1.106.199
Controlled By:      ReplicaSet/minio-nas-ibm-minio-objectstore-974f85dc9
Containers:
 ibm-minio-objectstore:
   Container ID:  docker://33a355387f1df4db9e932cf2921f095cac61baf174522eabf779db2d6c16a779
   Image:         minio/minio:RELEASE.2019-04-09T01-22-30Z
   Image ID:      docker-pullable://minio/minio@sha256:b363f54fc5a64d259d760106ad02c8725999c935f7aeae5348abfc0bed3fef0d
   Port:          9000/TCP
   Host Port:     0/TCP
   Command:
     /bin/sh
     -ce
     /usr/bin/docker-entrypoint.sh minio -C /root/.minio/ gateway nas /export
   State:          Running
     Started:      Thu, 25 Apr 2019 22:13:34 -0700
   Ready:          True
   Restart Count:  0
   Requests:
     cpu:      250m
     memory:   256Mi
   Liveness:   http-get http://:service/minio/health/live delay=5s timeout=1s period=30s #success=1 #failure=3
   Readiness:  http-get http://:service/minio/health/ready delay=5s timeout=1s period=15s #success=1 #failure=3
   Environment:
     MINIO_ACCESS_KEY:  <set to the key 'accesskey' in secret 'minio'>  Optional: false
     MINIO_SECRET_KEY:  <set to the key 'secretkey' in secret 'minio'>  Optional: false
     MINIO_BROWSER:     on
   Mounts:
     /root/.minio/ from minio-config-dir (rw)
     /var/run/secrets/kubernetes.io/serviceaccount from default-token-vccnm (ro)
Conditions:
 Type              Status
 Initialized       True
 Ready             True
 ContainersReady   True
 PodScheduled      True
Volumes:
 export:
   Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
   Medium:
 minio-user:
   Type:        Secret (a volume populated by a Secret)
   SecretName:  minio
   Optional:    false
 minio-config-dir:
   Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
   Medium:
 default-token-vccnm:
   Type:        Secret (a volume populated by a Secret)
   SecretName:  default-token-vccnm
   Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                node.kubernetes.io/memory-pressure:NoSchedule
                node.kubernetes.io/not-ready:NoExecute for 300s
                node.kubernetes.io/unreachable:NoExecute for 300s
Events:
 Type    Reason     Age   From                  Message
 ----    ------     ----  ----                  -------
 Normal  Scheduled  13m   default-scheduler     Successfully assigned default/minio-nas-ibm-minio-objectstore-974f85dc9-7mlrv to 10.41.14.22
 Normal  Pulling    13m   kubelet, 10.41.14.22  pulling image "minio/minio:RELEASE.2019-04-09T01-22-30Z"
 Normal  Pulled     12m   kubelet, 10.41.14.22  Successfully pulled image "minio/minio:RELEASE.2019-04-09T01-22-30Z"
 Normal  Created    12m   kubelet, 10.41.14.22  Created container

The command output shows that the Minio pods are using the following volume:

Volumes:
  export:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:

The buckets and objects are intermittently not accessible because of the following reasons:

The Persistent volume does not correspond to ReadWriteMany storage.
The Persistent is not labeled correctly.

Resolving the problem

Uninstall the failed deployment.
Use a persistent volume corresponding to a storage technology that supports ReadWriteMany volume.
Bind the persistent volume claim (PVC) to a specific persistent volume (PV). To ensure that the binding happens, label the PV that you need to bind as \"pv: <pv name>\". See the following example:
```
apiVersion: v1
kind: PersistentVolume
metadata:
 labels:
   pv: shared-pv   
 name: shared-pv
```