Kubernetes cron job schedule does not match configured job specification

Kubernetes cron jobs are scheduled to run at a different time that the job is configured within the job specification.

For example, if a cron job is configured to run at 2:00 AM, the actual scheduled job can be set to run at 3:00 AM.

This issue occurs when the default Coordinated Universal Time (UTC) time zone that is used within the kube-controller-manager container is different from the time zone that is used on the master node. Since the kube-controller-manager container uses the UTC time zone, the kube-controller-manager schedules the cron job based on the configured time for the UTC time zone.

Solution

The kube-controller-manager, which schedules the cron job, runs within a container for IBM Cloud Private. This container can use a different time zone than your master node. By default, the time zone within the kube-controller-manager container is the UTC time zone.

Since the kube-controller-manager container uses the UTC time zone, the kube-controller-manager schedules the cron job based on the configured time for the UTC time zone. To correct the container time zone, adjust the time zone for the Kubernetes cron job by updating the container specification:

Copy the manifest file for the k8s-master static pod:
```
 cp /etc/cfc/pods/master.json ~/
```

Update the manifest file to mount your local time zone /etc/localtime into the controller-manager container:

 "volumeMounts": [
    {
      "name": "localtime",
      "mountPath": "/etc/localtime"
    },
 ...
 "volumes": [
   {
     "name": "localtime",
     "hostPath": {
       "path": "/etc/localtime"
     }
   },

Copy the updated manifest file back to the k8s-master static pod:
```
 cp ~/master.json /etc/cfc/pods/
```
The kubelet service helps to restart the static pod to pick up your new changes.

Verify that the time zone is updated. For example:

 # date
 Tue Jun 11 07:59:00 EDT 2019

 # docker ps | grep hyper
 95ef02957a75        a28dcbcae557        "/hyperkube schedule…"   About a minute ago   Up About a minute     k8s_scheduler_k8s-master-9.21.55.15_kube-system_ef90d47d77f8989cd6fd3eac6e6b1dfc_2
 d3228e5c8b74        a28dcbcae557        "/hyperkube controll…"   About a minute ago   Up About a minute     k8s_controller-manager_k8s-master-9.21.55.15_kube-system_ef90d47d77f8989cd6fd3eac6e6b1dfc_2
 06e21ec01d45        a28dcbcae557        "/hyperkube apiserve…"   About a minute ago   Up About a minute     k8s_apiserver_k8s-master-9.21.55.15_kube-system_ef90d47d77f8989cd6fd3eac6e6b1dfc_0
 e3402496a3b2        a28dcbcae557        "/hyperkube proxy --…"   2 weeks ago          Up 2 weeks            k8s_proxy_k8s-proxy-dpjch_kube-system_122a8d9a-8121-11e9-a207-0894ef00adc0_0

 # docker exec -it d3228e5c8b74 bash
 root@user:/# date
 Tue Jun 11 07:59:10 EDT 2019