Accessing legacy application data from the openshift-storage namespace

When using the Multicloud Object Gateway (MCG) NamespaceStore filesystem (NSFS) feature, you need to have the Persistent Volume Claim (PVC) where the data resides in the openshift-storage namespace. In almost all cases, the data you need to access is not in the openshift-storage namespace, but in the namespace that the legacy application uses. A PVC is used in order to access the data.

About this task

In order to access data stored in another namespace, you need to create a PVC in the openshift-storage namespace that points to the same CephFS volume that the legacy application uses.

Procedure

Display the application namespace with scc, where <application_namespace> is the name of the application namespace.

oc get ns <application_namespace> -o yaml | grep scc

For example:

oc get ns testnamespace -o yaml | grep scc

openshift.io/sa.scc.mcs: s0:c26,c5
openshift.io/sa.scc.supplemental-groups: 1000660000/10000
openshift.io/sa.scc.uid-range: 1000660000/10000

Navigate into the application namespace:.

oc project <application_namespace>

For example:

oc project testnamespace

Ensure that a ReadWriteMany (RWX) PVC is mounted on the pod that you want to consume from the noobaa S3 endpoint using the MCG NSFS feature:.

oc get pvc

NAME                                               STATUS VOLUME
CAPACITY ACCESS MODES STORAGECLASS              AGE
cephfs-write-workload-generator-no-cache-pv-claim  Bound  pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a
10Gi     RWX          ocs-storagecluster-cephfs 12s

oc get pod


NAME                                                READY   STATUS              RESTARTS   AGE
cephfs-write-workload-generator-no-cache-1-cv892    1/1     Running             0          11s

Check the mount point of the Persistent Volume (PV) inside your pod.

Get the volume name of the PV from the pod, where <pod_name> is the name of the pod.

oc get pods <pod_name> -o jsonpath='{.spec.volumes[]}'

For example:

oc get pods cephfs-write-workload-generator-no-cache-1-cv892 -o jsonpath='{.spec.volumes[]}'

{"name":"app-persistent-storage","persistentVolumeClaim":{"claimName":"cephfs-write-workload-generator-no-cache-pv-claim"}}

In this example, the name of the volume for the PVC is cephfs-write-workload-generator-no-cache-pv-claim.

List all the mounts in the pod, and check for the mount point of the volume that were identified in step 4.a..

oc get pods <pod_name> -o jsonpath='{.spec.containers[].volumeMounts}'

For example:

oc get pods cephfs-write-workload-generator-no-cache-1-cv892 -o jsonpath='{.spec.containers[].volumeMounts}'


[{"mountPath":"/mnt/pv","name":"app-persistent-storage"},{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","name":"kube-api-access-8tnc5","readOnly":true}]

Confirm the mount point of the RWX PV in your pod, where <mount_path> is the path to the mount point that was identified in step #accessing_legacy_application_data_from_the_openshift-storage_namespace__check_mountpoint.

oc exec -it <pod_name> -- df <mount_path>

For example:

oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- df /mnt/pv

main
Filesystem
1K-blocks Used Available  Use%  Mounted on
172.30.202.87:6789,172.30.120.254:6789,172.30.77.247:6789:/volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c
10485760  0    10485760   0%    /mnt/pv

Ensure that the UID and SELinux labels are the same as the ones that the legacy namespace uses:.

oc exec -it <pod_name> -- ls -latrZ <mount_path>

For example:

oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- ls -latrZ /mnt/pv/

total 567
drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c26,c5      2 May 25 06:35 .
-rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c26,c5 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log
drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c26,c5     30 May 25 06:35 ..

Get the information of the legacy application RWX PV that you want to make accessible from the openshift-storage namespace, where <pv_name> is the name of the PV.

oc get pv | grep <pv_name>

For example:

oc get pv | grep pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a

pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a   10Gi       RWX            Delete           Bound    testnamespace/cephfs-write-workload-generator-no-cache-pv-claim   ocs-storagecluster-cephfs              47s

Ensure that the PVC from the legacy application is accessible from the openshift-storage namespace so that one or more noobaa-endpoint pods can access the PVC.

Find the values of the subvolumePath and volumeHandle from the volumeAttributes.

You can get these values from the YAML description of the legacy application PV.

oc get pv <pv_name> -o yaml

For example:

oc get pv pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a -o yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: openshift-storage.cephfs.csi.ceph.com
  creationTimestamp: "2022-05-25T06:27:49Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a
  resourceVersion: "177458"
  uid: 683fa87b-5192-4ccf-af2f-68c6bcf8f500
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: cephfs-write-workload-generator-no-cache-pv-claim
    namespace: testnamespace
    resourceVersion: "177453"
    uid: aa58fb91-c3d2-475b-bbee-68452a613e1a
  csi:
    controllerExpandSecretRef:
      name: rook-csi-cephfs-provisioner
      namespace: openshift-storage
    driver: openshift-storage.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-node
      namespace: openshift-storage
    volumeAttributes:
      clusterID: openshift-storage
      fsName: ocs-storagecluster-cephfilesystem
      storage.kubernetes.io/csiProvisionerIdentity: 1653458225664-8081-openshift-storage.cephfs.csi.ceph.com
      subvolumeName: csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213
      subvolumePath: /volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c
                volumeHandle: 0001-0011-openshift-storage-0000000000000001-cc416d9e-dbf3-11ec-b286-0a580a810213
  persistentVolumeReclaimPolicy: Delete
  storageClassName: ocs-storagecluster-cephfs
  volumeMode: Filesystem
status:
  phase: Bound

Use the subvolumePath and volumeHandle values that you identified in step 8.a to create a new PV and PVC object in the openshift-storage namespace that points to the same CephFS volume as the legacy application PV.

Example YAML file:

cat << EOF >> pv-openshift-storage.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cephfs-pv-legacy-openshift-storage
spec:
  storageClassName: ""
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
                (1)
  csi:
    driver: openshift-storage.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-node
      namespace: openshift-storage
    volumeAttributes:
    # Volume Attributes can be copied from the Source testnamespace PV
      "clusterID": "openshift-storage"
      "fsName": "ocs-storagecluster-cephfilesystem"
      "staticVolume": "true"
    # rootpath is the subvolumePath: you copied from the Source testnamespace PV
      "rootPath": /volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c
    volumeHandle: 0001-0011-openshift-storage-0000000000000001-cc416d9e-dbf3-11ec-b286-0a580a810213-clone
                (2)
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc-legacy
  namespace: openshift-storage
spec:
  storageClassName: ""
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
                (3)
  volumeMode: Filesystem
  # volumeName should be same as PV name
  volumeName: cephfs-pv-legacy-openshift-storage
EOF

(1): The storage capacity of the PV that you are creating in the openshift-storage namespace must be the same as the original PV.
(2): The volume handle for the target PV that you create in openshift-storage needs to have a different handle than the original application PV, for example, add -clone at the end of the volume handle.
(3): The storage capacity of the PVC that you are creating in the openshift-storage namespace must be the same as the original PVC.

Create the PV and PVC in the openshift-storage namespace using the YAML file specified in step 8.b, where YAML_file is the name of the YAML file.

oc create -f <YAML_file>

For example:

oc create -f pv-openshift-storage.yaml

persistentvolume/cephfs-pv-legacy-openshift-storage created
persistentvolumeclaim/cephfs-pvc-legacy created

Ensure that the PVC is available in the openshift-storage namespace:

oc get pvc -n openshift-storage


NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
cephfs-pvc-legacy                     Bound    cephfs-pv-legacy-openshift-storage         10Gi       RWX                                          14s

Navigate into the openshift-storage project:

oc project openshift-storage

Now using project "openshift-storage" on server "https://api.cluster-5f6ng.5f6ng.sandbox65.opentlc.com:6443".

Create the NSFS namespacestore:

noobaa namespacestore create nsfs <nsfs_namespacestore> --pvc-name='<cephfs_pvc_name>' --fs-backend='CEPH_FS'

nsfs_namespacestore: A name of the NSFS namespacestore.
cephfs_pvc_name: A name of the CephFS PVC in the openshift-storage namespace.

For example:

 noobaa namespacestore create nsfs legacy-namespace --pvc-name='cephfs-pvc-legacy' --fs-backend='CEPH_FS'

Ensure that the noobaa-endpoint pod restarts and that it successfully mounts the PVC at the NSFS namespacestore, for example, /nsfs/legacy-namespace mountpoint, where noobaa_endpoint_pod_name is the name of the noobaa-endpoint pod.

oc exec -it <noobaa_endpoint_pod_name> -- df -h /nsfs/<nsfs_namespacestore>

For example:

oc exec -it noobaa-endpoint-5875f467f5-546c6 -- df -h /nsfs/legacy-namespace


Filesystem                                                                                                                                                Size  Used Avail Use% Mounted on
172.30.202.87:6789,172.30.120.254:6789,172.30.77.247:6789:/volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c   10G     0   10G   0% /nsfs/legacy-namespace

Create an MCG user account:

noobaa account create <user_account> --full_permission --allow_bucket_create=true --new_buckets_path='/' --nsfs_only=true --nsfs_account_config=true --gid <gid_number> --uid <uid_number> --default_resource='legacy-namespace'

user_account: Specify the name of the MCG user account.
gid_number: Specify the GID number.
uid_number: Specify the UID number.

Important: Use the same UID and GID as that of the legacy application. You can find it from the previous output.

For example:

noobaa account create leguser --full_permission --allow_bucket_create=true --new_buckets_path='/' --nsfs_only=true --nsfs_account_config=true --gid 0 --uid 1000660000 --default_resource='legacy-namespace'

Create an MCG bucket.

Create a dedicated folder for S3 inside the NSFS share on the CephFS PV and PVC of the legacy application pod:

oc exec -it <pod_name> -- mkdir <mount_path>/nsfs

For example:

oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- mkdir /mnt/pv/nsfs

Create the MCG bucket using the nsfs/ path:

noobaa api bucket_api create_bucket '{
  "name": "<bucket_name>",
  "namespace":{
    "write_resource": { "resource": "<nsfs_namespacestore>", "path": "nsfs/" },
    "read_resources": [ { "resource": "<nsfs_namespacestore>", "path": "nsfs/" }]
  }
}'

For example:

noobaa api bucket_api create_bucket '{
  "name": "legacy-bucket",
  "namespace":{
    "write_resource": { "resource": "legacy-namespace", "path": "nsfs/" },
    "read_resources": [ { "resource": "legacy-namespace", "path": "nsfs/" }]
  }
}'

Check the SELinux labels of the folders residing in the PVCs in the legacy application and openshift-storage namespaces:

oc exec -it <noobaa_endpoint_pod_name> -n openshift-storage -- ls -ltraZ /nsfs/<nsfs_namespacstore>

For example:

oc exec -it noobaa-endpoint-5875f467f5-546c6 -n openshift-storage -- ls -ltraZ /nsfs/legacy-namespace


total 567
drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c0,c26      2 May 25 06:35 .
-rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c0,c26 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log
drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c0,c26     30 May 25 06:35 ..

oc exec -it <pod_name> -- ls -latrZ <mount_path>

For example:

oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- ls -latrZ /mnt/pv/

total 567
drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c26,c5      2 May 25 06:35 .
-rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c26,c5 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log
drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c26,c5     30 May 25 06:35 ..

In these examples, you can see that the SELinux labels are not the same which results in permission denied or access issues.

Ensure that the legacy application and openshift-storage pods use the same SELinux labels on the files.
You can do this in one of the following ways:
- Changing the default SELinux label on the legacy application project to match the one in the openshift-storage project
- Modifying the SELinux label only for the deployment config that has the pod which mounts the legacy application PVC

Delete the NSFS namespacestore.

Delete the MCG bucket:

noobaa bucket delete <bucket_name>

For example:

noobaa bucket delete legacy-bucket

Delete the MCG user account.

noobaa account delete <user_account>

For example:

noobaa account delete leguser

Delete the NSFS namespacestore.

noobaa namespacestore delete <nsfs_namespacestore>

For example:

noobaa namespacestore delete legacy-namespace

Delete the PV and PVC.
Important: Before you delete the PV and PVC, ensure that the PV has a retain policy configured.
```
oc delete pv <cephfs_pv_name>
```
```
oc delete pvc <cephfs_pvc_name>
```
cephfs_pv_name

Specify the CephFS PV name of the legacy application.

cephfs_pvc_name

Specify the CephFS PVC name of the legacy application.
For example:
```
oc delete pv cephfs-pv-legacy-openshift-storage
```
For example:
```
oc delete pvc cephfs-pvc-legacy
```