Enforcing cluster-wide policies for a Kubernetes-based Docker cluster
For enterprise production deployments of Docker clusters, enforcing cluster-wide
policies to restrict what a container is allowed to do is an extremely important
requirement. For Kubernetes-based clusters this
is where the PodSecurityPolicy
object comes into picture. A
PodSecurityPolicy
object controls the actions that a
Kubernetes pod can perform. In other words, if you need to specify what a Docker
container is allowed to do in your Kubernetes cluster,
PodSecurityPolicy
is what you need to look at.
As per the Kubernetes documentation, the PodSecurityPolicy
object allows
an administrator to control the following tasks:
- Running privileged containers
- Adding capabilities to a container
- Specifying SELinux context of the container
- Specifying the user ID that will be used by the container process
- Allocating an FSGroup that owns the pod's volumes
- Configuring allowable supplemental groups
- Requiring the use of a read-only root file system
- Controlling the usage of volume types
More details are available at: http://kubernetes.io/docs/user-guide/pod-security-policy/
In this article, we'll look at some of the configuration examples to set cluster-wide pod policies for a Kubernetes cluster. These instructions apply to Kubernetes running on any platform: Intel®, IBM® PowerPC®, ARM and so on.
The following steps will guide you with the setup and usage of
PodSecurityPolicy
.
Step 1: API server configuration
For PodSecurityPolicy
to work, the following configuration needs to be
enabled for the API server:
- Enable API extensions
The following parameter needs to be added to the API server startup argument:–runtime-config=extensions/v1beta1/podsecuritypolicy=true
On Red Hat based systems, this parameter needs to be set for the
KUBE_API_ARGS
variable in the /etc/kubernetes/apiserver file - Enable PodSecurityPolicy admission control policy
The following parameter needs to be added to the API server startup argument:
–admission-control=PodSecurityPolicy
On Red Hat based systems, this parameter needs to be set for the
KUBE_ADMISSION_CONTROL
variable in the /etc/kubernetes/apiserver file.
Step 2: Applying policies
You can define the policies in the YAML or JSON files and apply them using the
kubectl
command
# kubectl –f [policy_file]
You can find the example YAML files used in this article at: https://github.com/bpradipt/examples
Example policy: Don’t allow processes inside the container to run as the ‘root’ user
{ "kind": "PodSecurityPolicy", "apiVersion":"extensions/v1beta1", "metadata": { "name": "noroot" }, "spec": { "privileged": false, "seLinux": { "rule": "RunAsAny" }, "supplementalGroups": { "rule": "RunAsAny" }, "runAsUser": { "rule": "MustRunAsNonRoot" }, "fsGroup": { "rule": "RunAsAny" }, "volumes": ["*"] } }
For a pod to be successfully deployed with the above policy in force, the following conditions must be met:
- The pod container images must have the USER attribute defined (or)
- The pod YAML file must explicitly specify the non-root user ID as part of
securityContext
Here is an example of a pod YAML file:
apiVersion: "v1" kind: "Pod" metadata: name: "mysql" labels: name: "db" spec: containers: - name: "mysql" image: "ppc64le/mysql" imagePullPolicy: "IfNotPresent" securityContext: runAsUser: 102 env: - name: MYSQL_ROOT_PASSWORD value: password ports: - containerPort: 3306
Note that user ID needs to be used and not the user name.
When the above policy is in force, the following behavior is observed for various scenarios.
Scenario 1: Container image doesn’t have USER instruction and pod YAML doesn’t specify a non-root user ID
# kubectl get pods NAME READY STATUS RESTARTS AGE mysql 0/1 VerifyNonRootError 0 7s
Following is the error text:
Error syncing pod, skipping: failed to "StartContainer" for "mysql" with VerifyNonRootError: "container has no runAsUser and image will run as root"
Scenario 2: Container image having USER instruction
Following error is noticed:
failed to "StartContainer" for "mysql" with VerifyNonRootError: "can't tell if image runs as root: non-numeric user (mysql) is not allowed"
Scenario 3: Pod YAML having runAsUser: 0 specified under
securityContext
Following error is noticed when trying to create the pod:
# kubectl create -f ./poddb.yml
Error from server: error when creating "./poddb.yml": pods "mysql" is forbidden: unable to validate against any pod security policy: [securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden by the pod security policy mysql]
Scenario 4: Pod YAML having privileged: true specified under securityContext
Following error is noticed when trying to create the pod:
# kubectl create -f ./poddb.yml
Error from server: error when creating "./poddb.yml": pods "mysql" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]
Example policy: Don’t allow processes inside the container to run as the ‘root’ user and drop specific capabilities
{ "kind": "PodSecurityPolicy", "apiVersion":"extensions/v1beta1", "metadata": { "name": "dropcapabilities" }, "spec": { "privileged": false, "requiredDropCapabilities": ["SETFCAP", "DAC_OVERRIDE"], "seLinux": { "rule": "RunAsAny" }, "supplementalGroups": { "rule": "RunAsAny" }, "runAsUser": { "rule": "MustRunAsNonRoot" }, "fsGroup": { "rule": "RunAsAny" }, "volumes": ["*"] } }
With the above policy in place, if a user tries to deploy a pod with the following YAML, an error will be thrown.
apiVersion: "v1" kind: "Pod" metadata: name: "mysql" labels: name: "db" spec: containers: - name: "mysql" image: "ppc64le/mysql" imagePullPolicy: "IfNotPresent" securityContext: privileged: true capabilities: add: ["SETFCAP"] env: - name: MYSQL_ROOT_PASSWORD value: password ports: - containerPort: 3306 # kubectl create -f ./poddb.yml Error from server: error when creating "./poddb.yml": pods "mysql" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: "SETFCAP": capability may not be added]
Conclusion
As you can see from the examples provided in this article, PodSecurityPolicy
is a
very powerful functionality especially suited for enterprise deployments with strict
compliance requirements.