Example configuration of IBM Spectrum Scale DAS

The following steps illustrate an example configuration and key concepts of IBM Spectrum Scale DAS.

Before you can configure IBM Spectrum Scale DAS, the configuration of installation prerequisites and the installation of IBM Spectrum Scale DAS must be completed successfully.

The following steps walk you through an example configuration of the IBM Spectrum Scale DAS S3 service and accessing data stored in IBM Spectrum Scale using the S3 access protocol. Customize the following steps according to your workload requirements.

To create and configure the S3 service, you need to accept the license and provide an IP address range for S3 access and the scaling factor.

From a node configured to work with the OCP cluster, create and configure the IBM Spectrum Scale DAS S3 service.
For example,
```
mmdas service create s3 --acceptLicense --ipRange "192.0.2.12-192.0.2.14" --scaleFactor 1
```
A sample output is as follows:
```
Create request for Spectrum Scale Data Access Service: 's3' is accepted
```
View the status of the IBM Spectrum Scale DAS S3 service.
```
mmdas service list
```
A sample output is as follows:
```
Name   Enable  Phase
 -----  -----   ------
 s3     true    Creating
```
Note:
- As the creation and configuration of the IBM Spectrum Scale DAS S3 service progresses, the status shown in the Phase column varies according to the progress of the S3 service configuration.
- IBM Spectrum Scale DAS endpoint and NooBaa pods also recycle until the Phase column shows the Ready state.
- Before proceeding with the next steps, administrators must wait for the Phase column to show the Ready state and until all fields are populated in the output of the mmdas service list s3 command.
After the successful creation of the IBM Spectrum Scale DAS S3 service, mmdas service list reports the status of the S3 service as Ready and mmdas service list s3 reports status and configuration details.
```
mmdas service list
```
```
Name   Enable  Phase
 -----  -----   ------
 s3     true    Ready
```
```
mmdas service list s3
```
```
Name   AcceptLicense   DbStorageClass                  Enable  EnableMD5
 -----  -----           --------------                  ------  ---------
 s3     true            ibm-spectrum-scale-sample       true    true

 ScaleDataBackend       Phase   S3Endpoints
 -----                  -----   --------------
 [/mnt/fs1]             Ready   [https://192.0.2.12 https://192.0.2.13 https://192.0.2.14]

 IpRange                EnableAutoHA    ScaleFactor
 -----                  -----           --------------
 192.0.2.12-192.0.2.14  true            1
```
The IBM Spectrum Scale DAS S3 service is now ready to use. For information about how IBM Spectrum Scale DAS uses resources in Red Hat OpenShift namespaces, see Understanding Red Hat OpenShift resources used by IBM Spectrum Scale DAS.
Before configuring IBM Spectrum Scale DAS S3 accounts and S3 exports, validate the IBM Spectrum Scale DAS configuration.

The S3 service can be accessed through the S3 endpoints shown in the preceding steps. A curl command can be used to confirm that the S3 endpoints are accessible. The response will show Access Denied that confirms that the S3 service is accessible. Authenticated S3 access is covered in a later step.
From a node that can connect to the IBM Spectrum Scale S3 service IP address, issue an unauthenticated curl command to verify access to the S3 service.
For example,
```
curl 192.0.2.12
```
A sample output is as follows:
```
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message>
<Resource>/</Resource><RequestId>l07cquox-6zmwye-ef9</RequestId></Error>
```
S3 accounts are required to authenticate access attempts to the IBM Spectrum Scale DAS S3 service. To create an S3 account, you need to provide an account name, a UID, and a GID, and optionally a path for new S3 buckets. The account name is used for IBM Spectrum Scale DAS management purposes, and the UID and the GID are used to store S3 objects in the IBM Spectrum Scale file system. S3 account creation generates S3 access keys which are used by S3 applications to authenticate access. The configuration of the path for new S3 buckets is shown in a later step.

From a node configured to work with the OCP cluster, create an S3 account.

For example,

mmdas account create project1 --uid 1602 --gid 1996

A sample output is as follows:

Account is created successfully. The secret and access keys are as follows.
Secret Key                                     Access Key
 ----------                                     -----------
czAjbq8/CzyMHJfKWvGi50nTRrS4/Id3DA/P3Hau       P71YOPyNAYCdfmIjIuv4

mmdas account list

Name           UID     GID     New buckets path
----           ---     ---     ----------------
project1       1602    1996    /mnt/fs1/

mmdas account list project1

Name         UID     GID     Accesskey               Secretkey                                     New buckets path
----         ---     ---     ---------               ---------                                     ----------------
project1     1602    1996    P71YOPyNAYCdfmIjIuv4    czAjbq8/CzyMHJfKWvGi50nTRrS4/Id3DA/P3Hau      /mnt/fs1/

The S3 access keys generated in the preceding step can be used by S3 applications to submit authenticated S3 requests to the S3 service.

For demonstrative purpose, the S3 command of the AWS command line interface is used in the following step. An alias is created for the AWS CLI that uses the S3 access keys for the S3 service endpoint that are configured in the preceding steps.

The listing of buckets and objects does not show any results, because no buckets or objects are created so far. The creation of a new S3 bucket fails. This will be resolved in a later step.

From a node that can connect to the IBM Spectrum Scale DAS S3 service IP address, use the S3 account to access the S3 service with the AWS CLI.
```
alias s3p1='AWS_ACCESS_KEY_ID=P71YOPyNAYCdfmIjIuv4 AWS_SECRET_ACCESS_KEY=czAjbq8/CzyMHJfKWvGi50nTRrS4/Id3DA/P3Hau aws --endpoint https://192.0.2.12 --no-verify-ssl s3'
```
```
s3p1 ls
```
```
s3p1 mb s3://mybucket
```
A sample output is as follows:
```
make_bucket failed: s3://mybucket An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied
```
In a preceding step, the bucket creation command by an S3 application failed with the message: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied. S3 applications use the S3 CreateBucket request to create new S3 Buckets.

The following step illustrates basic usage of S3 buckets and S3 objects.

For using S3 buckets and S3 objects, the directories in the IBM Spectrum Scale file system must be configured with proper owner, group, permissions, and SELinux settings, if SELinux enabled. The owner and the group of the directories must match the UID and the GID of the S3 account that is configured in the preceding steps. The owner and the group must have permissions to read, write, and access the directories. If you have SELinux enabled on the storage cluster, then the SELinux settings must match the settings that are configured on the storage cluster.

Using S3 buckets and S3 objects

From one of the storage cluster nodes, prepare directories in the IBM Spectrum Scale file system for S3 access.
1. Create the directories.
```
mkdir /data/fs1/project1-data /data/fs1/project1-buckets
```
2. Assign read and write access to the owner and the group of the directories.
```
chmod 770 /data/fs1/project1-data /data/fs1/project1-buckets
```
3. Change the owner and the group of the directories to match with the UID and GID of the S3 account that is created in a preceding step.
```
chown 1602:1996 /data/fs1/project1-data /data/fs1/project1-buckets
```
4. If you have enabled SELinux on the storage cluster, then follow this step. Change the SELinux settings for the directories to match with the SELinux settings of the IBM Spectrum Scale file system configured during installation prerequisites.
```
chcon system_u:object_r:container_file_t:s0:c111,c234 /data/fs1/project1-data /data/fs1/project1-buckets
```
  You can list the details of the directories including their security context as follows:
```
ls -ldZ  /data/fs1/project1-*
```
  A sample output is as follows:
```
drwxrwx---. 2 1602 1996 system_u:object_r:container_file_t:s0:c111,c234 4096 Mar 12 08:23 /data/fs1/project1-buckets
drwxrwx---. 2 1602 1996 system_u:object_r:container_file_t:s0:c111,c234 4096 Mar 12 08:23 /data/fs1/project1-data
```
From a node configured to work with the OCP cluster, create an S3 export by making the directory accessible as an S3 bucket.
For example,
```
mmdas export create project1-bucket --filesystemPath /mnt/fs1/project1-data
```
A sample output is as follows:
```
Export is successfully created
```
```
mmdas export list
```
A sample output is as follows:
```
Name
 ------
 project1-bucket
```
An S3 application can access such an exported directory as an S3 bucket and, for instance, upload S3 objects.
From a node that can connect to the IBM Spectrum Scale DAS S3 service IP address, use the S3 bucket.
1. View the AWS CLI alias created in step 4 of previous example.
```
alias s3p1
```
  A sample output is as follows:
```
alias s3p1='AWS_ACCESS_KEY_ID=P71YOPyNAYCdfmIjIuv4 AWS_SECRET_ACCESS_KEY=czAjbq8/CzyMHJfKWvGi50nTRrS4/Id3DA/P3Hau aws --endpoint https://192.0.2.12 --no-verify-ssl s3'
```
2. List the S3 buckets.
```
s3p1 ls
```
  A sample output is as follows:
```
2022-03-12 08:35:23 project1-bucket
```
3. Create a file.
```
echo "IBM Spectrum Scale provides scalable performance." > message
```
```
md5sum message
```
  A sample output is as follows:
```
c927f038344fd0ecfbfa8d69230dc0d4  message
```
4. Copy the file to the S3 bucket.
```
s3p1 cp message s3://project1-bucket
```
  A sample output is as follows:
```
upload: ./message to s3://project1-bucket/message
```
5. List the contents of the S3 bucket.
```
s3p1 ls s3://project1-bucket
```
  A sample output is as follows:
```
2022-03-12 08:39:40         51 message
```
  The uploaded file is listed.
  
  The S3 access protocol has no awareness of the underlying file systems. Therefore, IBM Spectrum Scale DAS needs to define where to create the directories that represent new S3 Buckets. The newBucketPath property of S3 accounts defines for each S3 Account where IBM Spectrum Scale DAS creates the directories for new S3 Buckets.
  
  The default value for newBucketPath is the mount point of the IBM Spectrum Scale file system on the IBM Spectrum Scale container native cluster that is used for IBM Spectrum Scale DAS. The permissions of the root directory are configured in a preceding step and they do not allow users to create new directories. Therefore, the creation of a new S3 bucket failed in a preceding step.
  
  The directory /data/fs1/project1-buckets has the required permissions for the S3 account project1 to create directories. To enable S3 account project1, the value of their newBucketPath must be updated respectively.

From a node configured to work with the OCP cluster, update the value of the newBucketPath parameter of an S3 account.

mmdas account list

A sample output before the update is as follows:

Name           UID     GID     New buckets path
 ----           ---     ---     ----------------
 project1       1602    1996    /mnt/fs1/

mmdas account update project1 --newBucketsPath /mnt/fs1/project1-buckets

A sample output is as follows:

Account is successfully updated

mmdas account  list

A sample output after the update is as follows:

Name           UID     GID     New buckets path
 ----           ---     ---     ----------------
 project1       1602    1996    /mnt/fs1/project1-buckets/

After updating the newBucketPath value for the S3 account project1, the account can create new S3 buckets using the S3 CreateBucket request.

From a node that can connect to the IBM Spectrum Scale DAS S3 service IP address, create S3 buckets by using the S3 CreateBucket request.
```
s3p1 mb s3://mybucket
```
A sample output is as follows:
```
make_bucket: mybucket
```
```
s3p1 ls
```
A sample output is as follows:
```
2022-03-12 08:36:04 mybucket
2022-03-12 08:36:04 project1-bucket
```
From an S3 application's perspective, there is no difference between S3 buckets that are created by using the mmdas command and S3 buckets that are created using the S3 CreateBucket request. For instance, S3 objects can be seamlessly copied between S3 buckets that are created by using different means.
Note: Due to IBM Spectrum Scale CNSA SELinux enablement, the SELinux type parameter has changed as unlabeled_t instead of container_file_t for buckets and files created using the s3 command.

Note: In Red Hat OpenShift Data Foundation (ODF) 4.12, new policies are introduced for sharing buckets across the S3 users that share the same group id (gid). For more information, see Setting bucket policy for user created buckets (using S3 command).
From a node that can connect to the IBM Spectrum Scale DAS S3 service IP address, copy S3 objects between S3 buckets that are created by using different means.
```
s3p1 cp s3://project1-bucket/message s3://mybucket
```
A sample output is as follows:
```
copy: s3://project1-bucket/message to s3://mybucket/message
```
IBM Spectrum Scale DAS stores S3 buckets and S3 objects as files and directories in IBM Spectrum Scale file systems. The following command shows the file in the IBM Spectrum Scale file system for the S3 object that is uploaded to the S3 bucket that is created by using mmdas command.
Note: The owner, the group, the permissions, and the SELinux (if enabled) settings for the file are set by IBM Spectrum Scale DAS.
From one of the storage cluster nodes, list the data in the IBM Spectrum Scale file system that is generated by using the S3 access protocol.
```
tree /data/fs1/project1-data
```
A sample output is as follows:
```
/data/fs1/project1-data
└── message

0 directories, 1 file
```
```
md5sum /data/fs1/project1-data/message
```
A sample output is as follows:
```
c927f038344fd0ecfbfa8d69230dc0d4  /data/fs1/project1-data/message
```
If you have enabled SELinux on the storage cluster, then list the directory with the -Z option.
```
ls -lZR /data/fs1/project1-data
```
A sample output is as follows:
```
/data/fs1/project1-data:
total 1
-rw-rw----. 1 1602 1996 system_u:object_r:unlabeled_t:s0 51 Mar 12 08:40 message
```
The following command shows the directory for the S3 bucket that is created by using the S3 CreateBucket request and the file for the S3 object that is copied into that S3 bucket.
Note: The owner, the group, the permissions, and the SELinux (if enabled) settings for the file are set by IBM Spectrum Scale DAS.
```
tree /data/fs1/project1-buckets/
```
A sample output is as follows:
```
/data/fs1/project1-buckets/
└── mybucket
    └── message

1 directory, 1 file
```
```
md5sum /data/fs1/project1-buckets/mybucket/message
```
A sample output is as follows:
```
c927f038344fd0ecfbfa8d69230dc0d4  /data/fs1/project1-buckets/mybucket/message
```
If you have enabled SELinux on the storage cluster, then list the directory with the -Z option.
```
ls -lZR /data/fs1/project1-buckets/
```
A sample output is as follows:
```
/data/fs1/project1-buckets/:
total 1
drwxrwx---. 3 1602 1996 system_u:object_r:unlabeled_t:s0 4096 Mar 12 08:39 mybucket

/data/fs1/project1-buckets/mybucket:
total 1
-rw-rw-----. 1 1602 1996 system_u:object_r:unlabeled_t:s0 51 Mar 12 08:39 message
```
This step illustrates basic data sharing between S3 accounts. Both the example S3 accounts have different UIDs, but the same GID. This step also shows how different permissions of directories and files in the file system affect the access of S3 objects and S3 buckets by using the S3 access protocol.
Three different directories are used that are configured with varying owner, group, and permissions:
- Directory project1-data is owned by UID 1602 and has permissions 700. The S3 export of this directory will be accessible for S3 account project1 only.
- Directory project2-data is owned by UID 1606 and has permissions 700. The S3 export of this directory will be accessible for S3 account project2 only.
- Directory shared-data is owned by GID 1996 and has permissions 770. The S3 export of this directory will be accessible for both the S3 accounts.

Sharing data between S3 accounts

From one of the storage nodes, view the details of the directories that are prepared for S3 access.
```
ls -ladZ /data/fs1/*data
```
A sample output is as follows:
```
drwx------. 3 1602 1996 system_u:object_r:container_file_t:s0:c111,c234 4096 Mar 12 08:40 /data/fs1/project1-data
drwx------. 2 1606 1996 system_u:object_r:container_file_t:s0:c111,c234 4096 Mar 12 10:24 /data/fs1/project2-data
drwxrwx---. 2 1602 1996 system_u:object_r:container_file_t:s0:c111,c234 4096 Mar 12 10:24 /data/fs1/shared-data
```
Two S3 buckets are already created that can be reported by using the mmdas command. In the following step, create S3 exports for the two additional directories project2-data and shared-data. Three different directories are being used that are configured with varying owner, group, and permissions.
Note: S3 exports and the resulting S3 buckets have no awareness of UID, GID, or permissions.
From a node configured to work with the OCP cluster, create additional S3 exports.
```
mmdas export list
```
A sample output is as follows:
```
 Name
 ------
 mybucket
 project1-bucket
```
```
mmdas export create project2-bucket –-filesystemPath /mnt/fs1/project2-data
```
A sample output is as follows:
```
Export is successfully created
```
```
mmdas export create shared-bucket –-filesystemPath /mnt/fs1/shared-data
```
A sample output is as follows:
```
Export is successfully created
```
```
mmdas export list
```
A sample output is as follows:
```
 Name
 ------
 shared-bucket
 project2-bucket
 mybucket
 project1-bucket
```
Before accessing the new S3 exports as S3 buckets, create a second S3 account project2 that has a different UID than the S3 account project1. Both the S3 accounts have the same GID.
Note: The UIDs and GIDs of both the S3 accounts match the owner and the group of the directories configured in a preceding step.
From a node configured to work with the OCP cluster, create the 2nd S3 account.
```
mmdas account create project2 --uid 1606 --gid 1996
```
A sample output is as follows:
```
Account is created successfully. The secret and access keys are as follows. 

 Secret Key                                     Access Key
 ----------                                     -----------
 6POQr6s03Dzu1qKHeaJ3/C4XYcQX4EMFawiQMA6O       IG8hr2UoQzgGoNOtV151
```
```
mmdas account list
```
A sample output is as follows:
```
 Name           UID     GID     New buckets path
 ----           ---     ---     ----------------
 project2       1606    1996    /mnt/fs1/
 project1       1602    1996    /mnt/fs1/project1-buckets/
```
The owner, the group, and the permissions of the directories that are accessible as S3 buckets determine which S3 accounts can access which S3 buckets and S3 objects. For instance, the S3 account project1 can access the S3 buckets project1-bucket and shared-bucket, and it can copy an S3 object from the S3 bucket project1-bucket to the S3 bucket shared-bucket. The S3 account project2 cannot access the S3 bucket project1-bucket.
From a node that can connect to the IBM Spectrum Scale DAS S3 service IP address, as account project1, access the data that is stored in IBM Spectrum Scale by using the S3 access protocol.
Note: The alias command used in this step is set up in a preceding step.
```
s3p1 ls
```
A sample output is as follows:
```
2022-03-12 14:53:46 shared-bucket
2022-03-12 14:53:46 mybucket
2022-03-12 14:53:46 project1-bucket
```
```
s3p1 cp s3://project1-bucket/message s3://shared-bucket
```
A sample output is as follows:
```
copy: s3://project1-bucket/message to s3://shared-bucket/message
```
The S3 account project2 can access the S3 buckets project2-bucket and shared-bucket, and it can copy an S3 object from the S3 bucket shared-bucket to the S3 bucket project2-bucket. The S3 account project2 cannot access the S3 bucket project1-bucket and it cannot access S3 objects stored in the S3 bucket project1-bucket.
From a node that can connect to the IBM Spectrum Scale DAS S3 service IP address, as account project2, access the data that is stored in IBM Spectrum Scale by using the S3 access protocol.
```
alias s3p2='AWS_ACCESS_KEY_ID=IG8hr2UoQzgGoNOtV151 AWS_SECRET_ACCESS_KEY=6POQr6s03Dzu1qKHeaJ3/C4XYcQX4EMFawiQMA6O aws --endpoint https://192.0.2.156 --no-verify-ssl s3'
```
A sample output is as follows:
```
AWS_SECRET_ACCESS_KEY=6POQr6s03Dzu1qKHeaJ3/C4XYcQX4EMFawiQMA6O aws --endpoint https://192.0.2.156 --no-verify-ssl s3'
```
```
s3p2 ls
```
A sample output is as follows:
```
2022-03-12 14:58:04 shared-bucket
2022-03-12 14:58:04 project2-bucket
```
```
s3p2 cp s3://shared-bucket/message s3://project2-bucket
```
A sample output is as follows:
```
copy: s3://shared-bucket/message to s3://project2-bucket/message
```
```
s3p2 ls s3://project2-bucket/message
```
A sample output is as follows:
```
2022-03-12 14:59:58         51 message
```
```
s3p2 ls s3://project1-bucket/message
```
A sample output is as follows:
```
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
```
In the next step, check owner, group, permissions and SELinux (if enabled) settings which are created by using the S3 access protocol. S3 objects are mapped 1:1 to files in the IBM Spectrum Scale file system. Their owner and their group are derived from the S3 accounts that have created the respective S3 objects. In this way, data can be shared between S3 accounts.

From one of the storage nodes, inspect the data in the IBM Spectrum Scale file system that are created by using the S3 access protocol.

ls -lZ /data/fs1/*data/*

A sample output is as follows:

-rw-rw----. 1 1602 1996 system_u:object_r:unlabeled_t:s0 51 Mar 12 08:40 /data/fs1/project1-data/message
-rw-rw----. 1 1606 1996 system_u:object_r:unlabeled_t:s0 51 Mar 12 14:59 /data/fs1/project2-data/message
-rw-rw----. 1 1602 1996 system_u:object_r:unlabeled_t:s0 51 Mar 12 14:54 /data/fs1/shared-data/message

md5sum /data/fs1/*data/*

A sample output is as follows:

c927f038344fd0ecfbfa8d69230dc0d4  /data/fs1/project1-data/message
c927f038344fd0ecfbfa8d69230dc0d4  /data/fs1/project2-data/message
c927f038344fd0ecfbfa8d69230dc0d4  /data/fs1/shared-data/message

for f in /data/fs1/*data/*; do echo -n "$f - "; cat $f ; done

A sample output is as follows:

/data/fs1/project1-data/message - IBM Spectrum Scale provides scalable performance.
/data/fs1/project2-data/message - IBM Spectrum Scale provides scalable performance.
/data/fs1/shared-data/message - IBM Spectrum Scale provides scalable performance.