Version 2.0.2.1 release notes

1. Run:

   oc delete pdb zen-metastoredb-budget -n zen

2. Run:

   oc delete pdb
         zen-metastoredb-budget -n ap-console

   for the node to drain the pod and continue with the MCP update.
3. Run:

   ps fx | grep apupgrade

   to determine whether upgrade is still running or not after both oc delete pdb commands. If it is running, the upgrade continues .
4. If upgrade stopped running, restart it.

1. Reboot the unreachable node.
2. Mount the filesystems. Run:

   mmmount all -a

3. Run:

   mmlsmount all -L

   for example :

   [root@gt03-node1 ~]# mmlsmount all -L
   
   File system ips is mounted on 3 nodes:
     9.0.32.16       e1n1
     9.0.32.17       e1n2
     9.0.32.18       e1n3
   
   File system platform is mounted on 3 nodes:
     9.0.32.18       e1n3
     9.0.32.16       e1n1
     9.0.32.17       e1n2

4. Restart the upgrade.

1. Check the nodes and look for those that are NotReady by running:

   oc get nodes

   for example :

   [root@gt01-node1 ~]# oc get nodes
   NAME                STATUS     ROLES    AGE   VERSION
   e1n1-master.fbond   Ready      master   38h   v1.19.0+d670f74
   e1n2-master.fbond   Ready      master   38h   v1.19.0+d670f74
   e1n3-master.fbond   NotReady   master   38h   v1.19.0+d670f74
   e1n4.fbond          Ready      worker   38h   v1.19.0+d670f74
   e2n1.fbond          Ready      worker   38h   v1.19.0+d670f74
   e2n2.fbond          Ready      worker   38h   v1.19.0+d670f74
   e2n3.fbond          NotReady   worker   38h   v1.19.0+d670f74
   e2n4.fbond          Ready      worker   38h   v1.19.0+d670f74

2. Run:

   oc get mcp

   to check the MCP status. For example:

   
   [root@gt01-node1 ~]# oc get mcp
   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-f17afa51f05e6aed60f2eb1931db6c63   False     True       False      3              2                   3                     0                      38h
   unset    rendered-unset-3e6f7f0f84f4a0796786b102c7679233    False     True       False      2              1                   2                     0                      38h
   worker   rendered-worker-3e6f7f0f84f4a0796786b102c7679233   True      False      False      3              3                   3                     0                      38h

3. Verify the CSR certificates for the nodes that show True in the DEGRADED or UPDATING columns.
4. Run:

   oc get csr

   to check whether there are any pending certificates.
5. Run:

   oc get csr -o name | xargs oc adm certificate approve

   to approve all certificates.
6. Run:

   oc get mcp

   oc get nodes

   to confirm that all nodes are Ready and all MCPs are UPDATED. For example:

   [root@gt01-node1 ~]# oc get nodes
   NAME                STATUS     ROLES    AGE   VERSION
   e1n1-master.fbond   Ready      master   38h   v1.19.0+d670f74
   e1n2-master.fbond   Ready      master   38h   v1.19.0+d670f74
   e1n3-master.fbond   Ready      master   38h   v1.19.0+d670f74
   e1n4.fbond          Ready      worker   38h   v1.19.0+d670f74
   e2n1.fbond          Ready      worker   38h   v1.19.0+d670f74
   e2n2.fbond          Ready      worker   38h   v1.19.0+d670f74
   e2n3.fbond          Ready      worker   38h   v1.19.0+d670f74
   e2n4.fbond          Ready      worker   38h   v1.19.0+d670f74

   [root@gt01-node1 ~]# oc get mcp
   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-f17afa51f05e6aed60f2eb1931db6c63   True      False      False      3              3                   3                     0                      38h
   unset    rendered-unset-3e6f7f0f84f4a0796786b102c7679233    True      False      False      3              3                   3                     0                      38h
   worker   rendered-worker-3e6f7f0f84f4a0796786b102c7679233   True      False      False      3              3                   3                     0                      38h

7. Restart the upgrade.

1. Wait for apupgrade to time out.
2. Run oc login command, which is written in the log. For example:

   oc login --token=$(cat /root/.sa/token) https://api.localcluster.fbond:6443

3. Restart the upgrade.

1. Run the following commands for the node in the NotReady state:

         ssh core@<node>.fbond
         sudo systemctl stop crio-wipe
         sudo systemctl stop crio
         sudo systemctl stop kubelet
         sudo chattr -iR /var/lib/containers/*
         sudo rm -rf /var/lib/containers/*
         sudo systemctl start crio-wipe
         sudo systemctl start crio
         sudo systemctl start kubelet
         exit

2. Restart the upgrade.

1. Run:

   oc get co | grep  image-registry

   to confirm the image-registry shows the following string:

   image-registry                             4.6.32    True        False         True       42h

2. Run:

   oc describe co image-registry

   to confirm that the output shows the following message:

       Message:               ImagePrunerDegraded: Job has reached the specified backoff limit
       Reason:                ImagePrunerJobFailed
       Status:                True
       Type:                  Degraded

1. Check the managementState by running:

   oc get config cluster -o yaml 

2. If the managementState is Removed change it to Managed and save it. Run:

   oc edit config cluster
   managementState : Managed

3. If the issue persists and step 2 did not solve it, run:

   oc patch imagepruner.imageregistry/cluster --patch '{"spec":{"suspend":true}}' --type=merge
   oc -n openshift-image-registry delete jobs --all

2022-08-02 10:20:13 INFO: OcsUpgrader.postinstall: Finished post upgrade checks for OCS upgrade
2022-08-02 10:20:13 INFO: ocs:Done executing OCS post-install checks.
2022-08-02 10:20:13 FATAL ERROR: Errors encountered
2022-08-02 10:20:13 FATAL ERROR:
2022-08-02 10:20:13 FATAL ERROR: check_ocs_state : OcsUpgrader.postinstall:Error encountered during OCS cluster health check!
2022-08-02 10:20:13 FATAL ERROR: This error requires manual intervention to resolve. Please contact IBM Support.

1. Run:

   [root@e1n1 ~]# NAMESPACE=openshift-storage
   [root@e1n1 ~]# ROOK_OPERATOR_POD=$(oc -n ${NAMESPACE} get pod -l app=rook-ceph-operator -o jsonpath='{.items[0].metadata.name}')
   [root@e1n1 ~]# oc exec -it ${ROOK_OPERATOR_POD} -n ${NAMESPACE} -- ceph crash archive-all --cluster=${NAMESPACE} --conf=/var/lib/rook/${NAMESPACE}/${NAMESPACE}.config --keyring=/var/lib/rook/${NAMESPACE}/client.admin.keyring

2. Run:

   [root@e1n1 ~]# oc rsh -n openshift-storage $TOOLS_POD ceph status | grep daemon | grep crashed

   to ensure that there is no output on stdout now.
   Note: If you still see output on stdout, wait for another 30 seconds and try again. If the stdout output persists, contact IBM Support.

1. Run:

   oc get route

   and check whether it fails.
2. Run:

   /opt/ibm/appliance/platform/apos-comms/tools/aposDnsCheck.py

   and check whether base FQDN is resolvable, but wildcards are not, for example:

   Validating /opt/ibm/appliance/platform/apos-comms/customer_network_config/ansible/Customer.yml
   Checking hostname
   [RUNNING] - Trying to query base FQDN
   [PASS] - The base FQDN was resolvable
   [RUNNING] - Trying to query a wildcard entry
   [ERROR] DNS server  <x.y.z.a> appears reachable, but is missing DNS record <xyz>. If you need a different suffix, add a DNS search string or specific explicitly
   [WARN] If you do not specify a wildcard entry, you must specify a DNS entry for each service that is deployed
   [RUNNING] - Trying to query customer defined service specific entries
   [ERROR] Could not run oc get route command. Please make sure you have permissions and access to the oc command and try again.
   
   [ERROR] Issue(s) issues above must be corrected on upstream DNS

3. Apply the following workaround:
   1. Manually edit the /localrepo/2.0.1.1/EXTRACT/platform/upgrade/aposcomms/aposcomms_prechecker.py file by commenting out line 233:

      Results.append(self.run_dns_check_util())

   2. Edit /opt/ibm/appliance/apupgrade/bin/apupgrade file by commenting out line 837:

      self.verify_bundle(self.working_dir)

4. If you run step 2., but get the following output:

   [RUNNING] - Trying to query base FQDN
   [ERROR] Error trying to query base FQDN <customer-fqdn>, please make sure the base FQDN indicated in the house yaml can be resolved

   Then you do not have proper DNS base FQDN forwarding records. Stop the upgrade and fix your base FQDN records.

1. Run:

   oc get zenservice lite-cr -n ap-console -o json | jq .spec.version

   to determine the current version.
2. Depending on the oc get version results, perform the following actions:
   • If you get a version above 4.2.0, contact IBM Cloud Pak for Data Support.
   • If you get 4.2.0 as the current version, no action is required.
   • If you get a version below 4.2.0, run:

     oc patch zenservice lite-cr  --namespace ap-console  --type=merge --patch '{"spec": {"version":"4.2.0"}}'

• Remove the old kernel-modules-extra* RPM corresponding to the old kernel version. For example, run:

  # yum remove kernel-modules-extra-4.18.0-193.13.2.el8_2.x86_64