An IBM Cloud Object Storage (COS) bucket can be used as the backing store for a PVC.
When you imagine a file system, you are probably thinking of the block storage provided by disk drives. Object storage buckets can also be used for file system volumes on Kubernetes and might fit well into your application. For example, buckets can be managed outside of the application with a variety of tools and the IBM Cloud Console. Getting data in and out is a breeze.
This post starts from scratch and demonstrates the creation of a cluster, buckets, volumes and applications. Off-the-shelf container images for nginx and jekyll will be used to demonstrate, so no application coding is required.
An IBM Cloud Object Storage (COS) bucket can be used as the backing store for a PVC. Buckets might fit your use case better than block storage. Examples include the following:
Import or export data
Share data between pods
Utilize COS buckets using file system APIs
These are a few things to think about when choosing a bucket versus block storage:
Performance: Block storage and buckets have drastically different characteristics. Verify against application requirements.
This post demonstrates how to create everything from scratch. The scripts and Terraform configuration are available on GitHub. Use them to create all of the resources required to see PVC buckets in action. The basic steps are as follows:
Create a VPC and Kubernetes cluster.
Create a COS instance with associated Kubernetes secrets and storage classes.
Create the Kubernetes resources: PVC, deployment, service and ingress.
A security feature of the bucket is also highlighted — the bucket IP “allow list” limits bucket access to the Kubernetes cluster VPC.
OK, lets do it!
Prerequisites
The provision steps are going to be done from the CLI. This will allow you to move these steps to a CI/CD pipeline or into IBM Cloud Schematics, over time. See “Getting started with solution tutorials” for help with the following required tools:
Git
IBM Cloud CLI
Terraform
Docker
Jq
You should be the account owner or have enough privileges to create resources. In the terminal, make your own copy of the local.env file and make the suggested edits. The prereq script will verify the tools are installed. This step is complete when you see: >>> Prerequisites met.
The default template.local.env has IBMCR set to “false”, which makes this step optional. Initialize IBMCR to “true” to use the IBM Container Registry. If pod access to hub.docker.com is disabled, set to “true”. This step will copy the required Docker images into a newly created namespace in the IBM Container Registry. Image names will resemble us.icr.io/$BASENAME/:
./010-container-registry.sh
Scroll to view full table
Cluster
You can use an existing VPC-based cluster or execute this step to create a cluster. Either way, the local.env CLUSTER_NAME is required. The creation is done in the cluster/ directory where you will find the Terraform configuration. The script will create a terraform.tfvars, then execute Terraform.
Take a look at cluster/main.tf to see all of the resources created. For example:
Resource “ibm_is_vpc” — create a VPC
Resource “ibm_container_vpc_cluster” — create a cluster
The rest of the resources are created in the terraform/ directory. The script will create a terraform.tfvars, then execute Terraform.
TLDR; skip down to 025-create-resources.sh.
IBM Cloud Object Storage (COS) bucket storage classes are installed by the resources in cos_storage_class.tf. If you had previously followed the Installing the IBM Cloud Object Storage plug-in, there are comments in cos_storage_class.tf that can be used to avoid this step.
The main.tf file creates the rest of the resources:
COS instance and secret keys that are then used to populate a couple of Kubernetes secrets
PVC configured to create a bucket automatically with limited access
The basics are pretty simple. The annotations allow some configuration (e.g. automatically creating the bucket and setting the access policy), which means setting the allow list of IPs for the bucket (demonstrated later). The full list of annotations and storage classes can be found in the documentation at “Storing data on IBM Cloud Object Storage.”
The command echoes a string to the default site file for nginx (index.html). We can test this later to verify success. The volume_mount adds the volume to a directory within the deployment’s pod. The volume configuration ties the PVC to the deployment.
Two more configuration files demonstrate the ability to write contents to a bucket for reading and also share the PVC with another deployment. jekylblog.tf:
Resource “kubernetes_persistent_volume_claim” “jekyllblog” — PVC and associated bucket
Resource “kubernetes_deployment” “jekyllblog” — Deployment that generates a blog and starts a web server. These commands do the work:
Then, jekyllnginx.tf has a deployment that mounts the same PVC:
“kubernetes_deployment” “jekyllnginx” — Deployment that creates a symlink to the same PVC. These are the commands:
cd /usr/share/nginx
rm -rf html
ln -s /blog/kubernetes-cos-pvc/example/jekyllblog/myblog/_site html
exec nginx -g ‘daemon off;’
Ingress exposes all three of these services with the subdomain nginx.<ingress domain>, jekyllblog,<ingress domain> and jekuyllnginx.<ingress domain>. Here is a cut down:
Navigate to the COS instance and then the bucket with nginx in the name. Look for the following:
The Objects section indicates Access denied —this is because of the Access policies.
The Access policies section in the Authorized IPs panel has a list of IP addresses that can access this bucket. The VPC’s cloud service endpoint source addresses are listed. See “VPC behind the curtain” for more details.
For Kubernetes clusters, click your cluster, open the Kubernetes dashboard to see the rest of the resources created or use the kubectl command line like kubectl get deployments. The names all start with $BASENAME*:
deployments
pods
services
ingresses
persistentvolumeclaims
Test
Run the test script. It will read the simple nginx service by using curl. It is expecting the string success that was put into index.html. Two URLs are displayed — you should open each of these to verify that the blog is being served by the other two deployments. It can take a couple of minutes for the blogs to become available:
./030-test.sh
Scroll to view full table
Clean up
Clean it all up with the following command:
./040-cleanup.sh
Scroll to view full table
Conclusion
A Kubernetes Persistent Volume Claims (PVC) bucket for hosting static content works well, and a production environment could build the static contents in the CI/CD pipeline to create a new release.
A PVC bucket may help with legacy applications that use local volume for storage:
Create a container image for the application.
Upload the local volume to a bucket, and mount a PVC bucket for the application.
Manage backup and restore through bucket operations.
The PVC bucket can be used to export application data for archives, analysis and more.