Enforcing cluster-wide policies for a Kubernetes-based Docker cluster

Pradipta Banerjee
Published on December 21, 2016

For enterprise production deployments of Docker clusters, enforcing cluster-wide policies to restrict what a container is allowed to do is an extremely important requirement. For Kubernetes-based clusters this is where the PodSecurityPolicy object comes into picture. A PodSecurityPolicy object controls the actions that a Kubernetes pod can perform. In other words, if you need to specify what a Docker container is allowed to do in your Kubernetes cluster, PodSecurityPolicy is what you need to look at.

As per the Kubernetes documentation, the PodSecurityPolicy object allows an administrator to control the following tasks:

Running privileged containers
Adding capabilities to a container
Specifying SELinux context of the container
Specifying the user ID that will be used by the container process
Allocating an FSGroup that owns the pod's volumes
Configuring allowable supplemental groups
Requiring the use of a read-only root file system
Controlling the usage of volume types

More details are available at: http://kubernetes.io/docs/user-guide/pod-security-policy/

In this article, we'll look at some of the configuration examples to set cluster-wide pod policies for a Kubernetes cluster. These instructions apply to Kubernetes running on any platform: Intel®, IBM® PowerPC®, ARM and so on.

The following steps will guide you with the setup and usage of PodSecurityPolicy.

Step 1: API server configuration

For PodSecurityPolicy to work, the following configuration needs to be enabled for the API server:

Enable API extensions
The following parameter needs to be added to the API server startup argument:
–runtime-config=extensions/v1beta1/podsecuritypolicy=true
On Red Hat based systems, this parameter needs to be set for the KUBE_API_ARGS variable in the /etc/kubernetes/apiserver file
Enable PodSecurityPolicy admission control policy
The following parameter needs to be added to the API server startup argument:
–admission-control=PodSecurityPolicy
On Red Hat based systems, this parameter needs to be set for the KUBE_ADMISSION_CONTROL variable in the /etc/kubernetes/apiserver file.

Step 2: Applying policies

You can define the policies in the YAML or JSON files and apply them using the kubectl command

            # kubectl –f [policy_file]

You can find the example YAML files used in this article at: https://github.com/bpradipt/examples

Example policy: Don’t allow processes inside the container to run as the ‘root’ user

{
  "kind": "PodSecurityPolicy",
  "apiVersion":"extensions/v1beta1",
  "metadata": {
    "name": "noroot"
  },
  "spec": {
      "privileged": false,
      "seLinux": {
          "rule": "RunAsAny"
      },
      "supplementalGroups": {
          "rule": "RunAsAny"
      },
      "runAsUser": {
          "rule": "MustRunAsNonRoot"
      },
      "fsGroup": {
          "rule": "RunAsAny"
      },
      "volumes": ["*"]
  }
}

For a pod to be successfully deployed with the above policy in force, the following conditions must be met:

The pod container images must have the USER attribute defined (or)
The pod YAML file must explicitly specify the non-root user ID as part of securityContext

Here is an example of a pod YAML file:

apiVersion: "v1"
kind: "Pod"
metadata:
 name: "mysql"
 labels:
   name: "db"
spec:
    containers:
      - name: "mysql"
        image: "ppc64le/mysql"
        imagePullPolicy: "IfNotPresent"
        securityContext:
           runAsUser: 102
        env:
           - name: MYSQL_ROOT_PASSWORD
             value: password
ports:
          - containerPort: 3306

Note that user ID needs to be used and not the user name.

When the above policy is in force, the following behavior is observed for various scenarios.

Scenario 1: Container image doesn’t have USER instruction and pod YAML doesn’t specify a non-root user ID

# kubectl get pods
NAME  READY STATUS             RESTARTS   AGE
mysql 0/1   VerifyNonRootError 0          7s

Following is the error text:

Error syncing pod, skipping: failed to "StartContainer" for "mysql" with VerifyNonRootError: "container has no runAsUser and image will run as root"

Scenario 2: Container image having USER instruction

Following error is noticed:

failed to "StartContainer" for "mysql" with VerifyNonRootError: "can't tell if image runs as root: non-numeric user (mysql) is not allowed"

Scenario 3: Pod YAML having runAsUser: 0 specified under securityContext

Following error is noticed when trying to create the pod:

# kubectl create -f ./poddb.yml

Error from server: error when creating "./poddb.yml": pods "mysql" is forbidden: unable to validate against any pod security policy: [securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden by the pod security policy mysql]

Scenario 4: Pod YAML having privileged: true specified under securityContext

Following error is noticed when trying to create the pod:

# kubectl create -f ./poddb.yml

Error from server: error when creating "./poddb.yml": pods "mysql" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

Example policy: Don’t allow processes inside the container to run as the ‘root’ user and drop specific capabilities

{
  "kind": "PodSecurityPolicy",
  "apiVersion":"extensions/v1beta1",
  "metadata": {
    "name": "dropcapabilities"
  },
  "spec": {
      "privileged": false,
      "requiredDropCapabilities": ["SETFCAP", "DAC_OVERRIDE"],
      "seLinux": {
          "rule": "RunAsAny"
      },
      "supplementalGroups": {
          "rule": "RunAsAny"
      },
      "runAsUser": {
          "rule": "MustRunAsNonRoot"
      },
      "fsGroup": {
          "rule": "RunAsAny"
      },
      "volumes": ["*"]
  }
}

With the above policy in place, if a user tries to deploy a pod with the following YAML, an error will be thrown.

apiVersion: "v1"
kind: "Pod"
metadata:
 name: "mysql"
 labels:
   name: "db"
spec:
    containers:
      - name: "mysql"
        image: "ppc64le/mysql"
        imagePullPolicy: "IfNotPresent"
        securityContext:
           privileged: true
           capabilities:
              add: ["SETFCAP"]
        env:
           - name: MYSQL_ROOT_PASSWORD
             value: password
ports:
          - containerPort: 3306

# kubectl create -f ./poddb.yml
Error from server: error when creating "./poddb.yml": pods "mysql" is 
forbidden: unable to validate against any pod security policy: 
[spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not
allowed capabilities.add: Invalid value: "SETFCAP": capability may not be added]

Conclusion

As you can see from the examples provided in this article, PodSecurityPolicy is a very powerful functionality especially suited for enterprise deployments with strict compliance requirements.