OpenShift cluster installation

1. Prepare the working directories and download the files required for the installation.

   There are 2 folders created on one of the OpenShift LPARs:

   mkdir '/openshift-files' 
   mkdir '/openshift-installation'

   • /openshift-files: Stores all files needed for the installation.

   • /openshift-installation: Contains temporary files created during installation.

2. Create sub-folders that to be used later on:

   mkdir '/openshift-files/bin' '/openshift-files/rhcos-image' '/openshift-files/ip-override' '/openshift-files/install-config'

3. As the /openshift-installation folder later contains files (ignition files) that are read by libvirt during installation (virt-install) the permissions must fit. The SELinux sVirt confinement used by libvirt only allows libvirt to access correctly labelled files. To allow libvirt to access files in the folder /openshift-installation following regex label rule is added (on the installation OpenShift LPAR):

   semanage fcontext --add -t virt_etc_t -f f '/openshift-installation(/.*)?'
   restorecon -R -v '/openshift-installation'

   Note: For more information, read the SELinux sVirt confinement: https://libvirt.org/drvqemu.html#selinux-svirt-confinement.

4. Download the latest OpenShift installer for Linux s390x, Pull secret and the Openshift command line tool (oc tool) and copy them to /openshift-files/bin/: https://console.redhat.com/openshift/install/ibmz/user-provisioned (login required).

5. Extract the installation program and OpenShift client:

   cd /openshift-files/bin
   tar -xvf openshift-install-linux.tar.gz
   tar -xvf openshift-client-linux.tar.gz

6. Make installation program and OpenShift client executable:

   cd /openshift-files/bin
   chmod +x openshift-install oc

7. Download the latest "rhcos-qemu.s390x.qcow2" for s390x and copy it to /openshift-files/rhcos-image/: https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/latest/rhcos-qemu.s390x.qcow2.gz.

8. Extract the image:

   cd /openshift-files/rhcos-image
   gzip -dk rhcos-qemu.s390x.qcow2.gz

If not otherwise stated, all the following steps are performed on the OpenShift LPAR used in the previous step.

1. Create install-config.yaml under /openshift-files/install-config:

   touch /openshift-files/install-config/install-config.yaml

2. Copy the content of the Example configuration files install-config.yaml to /openshift-files/install-config/install-config.yaml and change the following parts:

   • Replace pullSecret with the content of the Pull Secret file downloaded earlier.

     • Show content of the Pull secret file:

       cat /openshift-files/bin/pull-secret

   • Replace sshKey with an ssh public key that will be used later to connect to the default RHCOS user "core" via SSH.

     • Generate SSH key pair:

       ssh-keygen -f /openshift-files/install-config/id_rsa

     • Show content of the public key of the generated SSH key pair:

       cat /openshift-files/install-config/id_rsa.pub

   • Go through and check each value with the official documentation: Sample install-config.yaml file for IBM zSystems.

   Note: Newer OpenShift versions can have additional or different options that may be needed.

3. The following steps are needed to fix a problem when you define multiple NICs to the virtual machines that run OpenShift (OpenShift guest nodes).

   Explanation: When the OpenShift guest nodes starts during installation (virt-install step) it uses the first NIC it finds that can host the IP address for the API Server. The plan is to have one NIC for internet connection and one NIC for high available network traffic between the OpenShift guest nodes (without exposure to the internet). Because two NICs are in place, OpenShift might choose the wrong interface for the OpenShift communication. The solution is to override the IP address with the IP of the correct NIC, no matter what NIC was selected by OpenShift.

   The configuration file where this IP address is configured must be overwritten at the right time. Therefore, a systemd unit file is created that runs after the network comes up and before kubelet starts. The systemd unit runs a script that determines the correct IP address and puts it in a config file that overwrites the original KUBELET_NODE_IP. As you cannot easily tamper with the OpenShift guest nodes you have to create a machineconfig file that creates the script and systemd unit file.

   In the OpenShift documentation this issue is explained here.

   Advanced note: Also, both interfaces are configured via DHCP which leads to problems when both send a different DNS server. However, this is already taken care of with both DHCP servers sending the same custom DNS server. This has the side effect that internet domains must be resolvable over the custom DNS server that was mainly set up to resolve the internal domains (*.sa.boe). Alternatively, you can try to set up Split-DNS on each OpenShift guest node. That enables the possibility to have different DNS servers configured per interface (via DHCP) and the DNS requests are send to the correct DNS server respectively. On modern linux systems this should be as simple as enabling systemd-resolved that then automatically uses the DNS-Servers configured by the NetworkManager. This option does not seem to be available at the moment. Instead, it looks like you can set up different DNS servers for the OpenShift applications after the installation finished.

   1. Create the following files under /openshift-files/ip-override/. Copy from the sample-configuration-files:

      touch /openshift-files/ip-override/override-node-ip.sh \
            /openshift-files/ip-override/compute-ip-override.yaml \
            /openshift-files/ip-override/control-ip-override.yaml

   2. Copy the content of the sample files from sample configuration files to the three files created in the previous step:

      • override-node-ip.sh: This script will be executed on the system start of each "OpenShift guest node" to fix networking.

      • compute-ip-override.yaml: The machineconfig file that will be included during installation for compute nodes to create the systemd unit file and copy the script to the correct location.

      • control-ip-override.yaml: The machineconfig file that will be included during installation for control nodes to create the systemd unit file and copy the script to the correct location.

4. Skip this step if it's your first try of your installation. When redoing the installation the sub steps are required to clean up the environment properly.

   1. Shutdown or destroy the previous VMs (for an unclean shutdown use "destroy" instead of "shutdown"):

      virsh shutdown control0
      virsh shutdown control1
      virsh shutdown control2
      virsh shutdown compute0
      virsh shutdown compute1
      virsh shutdown bootstrap
      # graceful shutdown may take some time
      virsh undefine --remove-all-storage control0
      virsh undefine --remove-all-storage control1
      virsh undefine --remove-all-storage control2
      virsh undefine --remove-all-storage compute0
      virsh undefine --remove-all-storage compute1
      virsh undefine --remove-all-storage bootstrap

   2. If you need to redo the installation, you need to remove all the previous installation files. The normal rm -rf command do not remove the dot-files (files starting with ".") and this can cause installation errors. For this reason enable "dotglob" before removing the files:

      shopt -s dotglob
      rm -rf /openshift-installation/*
      ls -la /openshift-installation/

5. Copy /openshift-files/install-config/install-config.yaml to the /openshift-installation directory. :

   cp /openshift-files/install-config/install-config.yaml /openshift-installation/

   Note: The temporary directory /openshift-installation is used because files such as install-config.yaml are automatically removed from /openshift-installation when they are used.

6. Make sure that the openshift-install tool is build for the s390x architecture (release architecture s390x):

   cd /openshift-files/bin/
   ./openshift-install version

7. Generate the manifest files from install-config.yaml. Reminder: The file install-config.yaml is consumed/removed in this process:

   cd /openshift-files/bin/
   ./openshift-install create manifests --dir /openshift-installation

   In the /openshift-installation there should be some manifest files that are used to further configure OpenShift.

8. Copy the prepared ip-override machineconfig files under /openshift-files/ip-override/ to /openshift-installation/openshift.

   cp -v /openshift-files/ip-override/*.yaml /openshift-installation/openshift
   ls -l /openshift-installation/openshift

   The machineconfig files copied to this directory will be automatically included when the ignition files are generated from the manifest files.

9. Set the "mastersSchedulable:" option in cluster-scheduler-02-config.yml to false and verify:

   sed -i 's+true+false+g' /openshift-installation/manifests/cluster-scheduler-02-config.yml
   grep false /openshift-installation/manifests/cluster-scheduler-02-config.yml

   When this option is set to false application workload is not allowed to run on the control nodes.

10. Generate the ignition config files from the manifest files:

    ./openshift-install create ignition-configs --dir /openshift-installation

    Note: Use the Ignition configuration files within 12 hours of their creation because the 24-hour certificate rotates 16 to 22 hours after the cluster is installed. Using the Ignition configuration files within 12 hours can prevent installation errors.

11. Set SELinux labels and access rights for the ignition files:

    chown -R qemu:qemu /openshift-installation
    restorecon -R -v '/openshift-installation'

12. Create the qcow2 disk images for the cluster (adjust size according to your plan):

    qemu-img create -f qcow2 -F qcow2 -b /openshift-files/rhcos-image/rhcos-qemu.s390x.qcow2 /mnt/control0-mnt/images/control0-disk.qcow2 100G
    qemu-img create -f qcow2 -F qcow2 -b /openshift-files/rhcos-image/rhcos-qemu.s390x.qcow2 /mnt/control1-mnt/images/control1-disk.qcow2 100G
    qemu-img create -f qcow2 -F qcow2 -b /openshift-files/rhcos-image/rhcos-qemu.s390x.qcow2 /mnt/control2-mnt/images/control2-disk.qcow2 100G
    qemu-img create -f qcow2 -F qcow2 -b /openshift-files/rhcos-image/rhcos-qemu.s390x.qcow2 /mnt/compute0-mnt/images/compute0-disk.qcow2 100G
    qemu-img create -f qcow2 -F qcow2 -b /openshift-files/rhcos-image/rhcos-qemu.s390x.qcow2 /mnt/compute1-mnt/images/compute1-disk.qcow2 100G
    qemu-img create -f qcow2 -F qcow2 -b /openshift-files/rhcos-image/rhcos-qemu.s390x.qcow2 /mnt/bootstrap-mnt/images/bootstrap-disk.qcow2 100G

    Note: 100 GB is just a maximum value. It will just allocate the space its actually needing

13. Set SELinux labels for the qcow2 disk images:

    restorecon -R -v '/mnt'

14. Install the KVM guests. Adjust the values according to your plan and then run all virt-install commands in one go. Don't run them one by one:

    # bootstrap
    virt-install --noautoconsole \
    --connect qemu:///system \
    --name bootstrap \
    --memory 16384 \
    --vcpus 4 \
    --disk /mnt/bootstrap-mnt/images/bootstrap-disk.qcow2 \
    --import \
    --network "network=macvtap-10-bridge,mac=02:9B:17:00:B0:60" \
    --network "network=default,mac=02:00:17:00:B0:60" \
    --qemu-commandline="-drive \
    if=none,id=ignition,format=raw,file=/openshift-installation/bootstrap.ign,readonly=on -device
    virtio-blk,serial=ignition,drive=ignition"
    
    
    # control0
    virt-install --noautoconsole \
    --connect qemu:///system \
    --name control0 \
    --memory 16384 \
    --vcpus 8 \
    --disk /mnt/control0-mnt/images/control0-disk.qcow2 \
    --import \
    --network "network=macvtap-10-bridge,mac=02:9B:17:C0:00:61" \
    --network "network=default,mac=02:00:17:C0:00:61" \
    --qemu-commandline="-drive \
    if=none,id=ignition,format=raw,file=/openshift-installation/master.ign,readonly=on -device
    virtio-blk,serial=ignition,drive=ignition"
    
    
    # control1
    virt-install --noautoconsole \
    --connect qemu:///system \
    --name control1 \
    --memory 16384 \
    --vcpus 8 \
    --disk /mnt/control1-mnt/images/control1-disk.qcow2 \
    --import \
    --network "network=macvtap-10-bridge,mac=02:9B:17:C1:00:62" \
    --network "network=default,mac=02:00:17:C1:00:62" \
    --qemu-commandline="-drive \
    if=none,id=ignition,format=raw,file=/openshift-installation/master.ign,readonly=on -device
    virtio-blk,serial=ignition,drive=ignition"
    
    
    # control2
    virt-install --noautoconsole \
    --connect qemu:///system \
    --name control2 \
    --memory 16384 \
    --vcpus 8 \
    --disk /mnt/control2-mnt/images/control2-disk.qcow2 \
    --import \
    --network "network=macvtap-10-bridge,mac=02:9B:17:C2:00:63" \
    --network "network=default,mac=02:00:17:C2:00:63" \
    --qemu-commandline="-drive \
    if=none,id=ignition,format=raw,file=/openshift-installation/master.ign,readonly=on -device
    virtio-blk,serial=ignition,drive=ignition"
    
    
    # compute0
    virt-install --noautoconsole \
    --connect qemu:///system \
    --name compute0 \
    --memory 8192 \
    --vcpus 6 \
    --disk /mnt/compute0-mnt/images/compute0-disk.qcow2 \
    --import \
    --network "network=macvtap-10-bridge,mac=02:9B:17:00:C0:0A" \
    --network "network=default,mac=02:00:17:00:C0:0A" \
    --qemu-commandline="-drive \
    if=none,id=ignition,format=raw,file=/openshift-installation/worker.ign,readonly=on -device
    virtio-blk,serial=ignition,drive=ignition"
    
    
    # compute1
    virt-install --noautoconsole \
    --connect qemu:///system \
    --name compute1 \
    --memory 8192 \
    --vcpus 6 \
    --disk /mnt/compute1-mnt/images/compute1-disk.qcow2 \
    --import \
    --network "network=macvtap-10-bridge,mac=02:9B:17:00:C1:0B" \
    --network "network=default,mac=02:00:17:00:C1:0B" \
    --qemu-commandline="-drive \
    if=none,id=ignition,format=raw,file=/openshift-installation/worker.ign,readonly=on -device
    virtio-blk,serial=ignition,drive=ignition"

15. Shortly after the installation started you should be able to connect to the control and bootstrap nodes. Connect with the previously created ssh key pair and make sure networking is working correctly (also see troubleshooting 2 steps later). For example (replace bootstrap with the 192.168.122.x ip of the bootstrap guest):

    ssh -i /openshift-files/install-config/id_rsa core@bootstrap

16. Ssh into the bootstrap and check that the installation process is ongoing with:

    journalctl -b -f -u release-image.service -u bootkube.service

17. If you face a problem then check the following list of common problems:

    • Bootstrap cannot connect to quay.io (internet): It's likely that your DHCP or DNS configuration is not working properly. Make sure that you only have one default route (Only send the router option via one DHCP server). Make sure both DHCP servers send the same DNS Server (10.128.0.1) because we don't have a Split-DNS setup. Also make sure the DNS Server also resolves internet domain names e.g. setting recursive to true.

    • Other connection errors: Make sure that the ip-override is in place on all control and compute nodes. cat /etc/systemd/system/kubelet.service.d/98-nodenet-override.conf should show a file with the correct 10.128.0.x ip address set. Check out this link to read more.

    • KVM installation issues: Check the logs under /var/log/libvirt/qemu/*.

    • KVM guest panics OR is in paused state: Check that you have enough disk space with df -h.

    • Your problem is not listed: Consult the official OpenShift documentation and solution articles from Red Hat.

    You must start the installation from scratch if you changed something. Keep in mind that you cannot do manual changes that are kept persistently in the CoreOS OpenShift nodes. For this you have to define special yaml files (machineconfigs).

18. While the installation is ongoing perform the following steps to wait for the bootstrap to finish:

    1. Copy the installation files to the bastion (replace bastion with 192.168.122.x ip of the bastion guest):

       rsync -av -e ssh /openshift-files/bin root@bastion:/openshift-files
       rsync -av -e ssh /openshift-installation/ root@bastion:/openshift-installation

       Note: You might have to install rsync on the LPAR and guest first.

    2. Ssh to the bastion and wait for the bootstrap process to complete (perform on the bastion guest):

       cd /openshift-files/bin
       ./openshift-install --dir /openshift-installation wait-for bootstrap-complete --log-level=debug

       If the result looks similar, the bootstrap process completed:

       INFO Waiting up to 20m0s (until 3:57PM) for the Kubernetes API at https://api.ocp0.sa.boe:6443...
       INFO API v1.23.5+b0357ed up
       INFO Waiting up to 30m0s (until 4:07PM) for bootstrapping to complete...
       INFO It is now safe to remove the bootstrap resources
       INFO Time elapsed: 0s

19. The bootstrap node can now be shut down safely:

    virsh shutdown bootstrap

    Note: The resources used for the bootstrap can be used to bring up another compute node. This seems to be done usually after the installation finished (Post installation).

20. For the next steps we need to authenticate ourselves against the OpenShift API as "system:admin". To archive this the KUBECONFIG environment variable is set to the kubeconfig containing the credentials (perform on the bastion guest):

    export KUBECONFIG=/openshift-installation/auth/kubeconfig

21. Verify that the connection to the OpenShift API works and your authenticated as system:admin:

    oc whoami

22. After a while some pending certificate signing requests (CSRs) should show up when running the following command:

    watch oc get csr -A

    Note: When you don't see any certificates for a while the OpenShift compute nodes might not come up properly. In some cases a virsh shutdown of all OpenShift compute nodes and virsh start afterwards might fix this issue.

23. The pending certificates needs to be approved which can be done in an automated fashion with the following command:

    oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

    These certificate signing requests are related to the OpenShift control nodes.

24. After a while you should see two more CSRs which are related to the OpenShift compute nodes:

    watch oc get csr -A

    Note: When you don't see any certificates for a while the OpenShift compute nodes might not come up properly. In some cases a virsh shutdown of all OpenShift compute nodes and virsh start afterwards might fix this issue.

25. Approve the remaining compute node certificates (with status pending):

    oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

26. After the last step all main OpenShift guest nodes should be accessible via ssh.

27. With the ssh session on the bastion guest watch the operators to become available. As long as the Operator is in state "Progressing" and not "Degraded" the Operator is still doing something (this might take a while):

    watch -n5 oc get clusteroperators

28. Set up persistent storage for the image registry. This is required for OpenShift being able to store container images persistently. Perform following sub-steps on the bastion guest with the system:admin user.

    1. Before starting to set up the image registry. Some helpful commands:

       Get list of pods:

       oc get pod -n openshift-image-registry

       View the current configuration and the project description (project namespace):

       oc describe configs.imageregistry.operator.openshift.io
       oc describe project openshift-image-registry

       See available storage types:

       oc get sc

    2. Read the permissions the cluster image registry operator pod is running with. It is assumed that the same permissions will apply for the image registry pod that will be created to mount the NFS server:

       oc debug pods/$(oc get pods  -n openshift-image-registry | awk '/^cluster-image-registry-operator/ {print $1}') -n openshift-image-registry -- capsh --print

       You should see a line in the output which should look similar to this: uid=1000330000(1000330000) euid=1000330000(1000330000).

       Note: The group ID this container runs under. In this example: 1000330000. With this information we can fix the permission on the NFS share folder to allow the image registry pod to read/write/execute on this folder.

    3. Set the permissions on the /nfs_share folder to allow users running "rwx" with the group id (replace the group id 1000330000 with the group id used in your case). Perform these operations on the NFS KVM guest:

       chown root:1000330000 /nfs_share
       chmod 2770 /nfs_share

       Note: The sticky bit is required to make sure subsequent folders will receive the correct group id permissions as well. This prevents permission issues when pushing custom images to the internal image registry.

    4. Switch to the openshift-image-registry project. (Back on the bastion guest)

       oc project openshift-image-registry

    5. On the bastion guest create a Persistent Volume (PV). Replace the values under "nfs" with appropriate values from you plan.

       oc create -f <(echo '{
          "apiVersion": "v1",
          "kind": "PersistentVolume",
          "metadata": {
            "name": "nfs4registry-pv-0001"
          },
          "spec": {
            "capacity": {
              "storage": "100Gi"
            },
            "accessModes": [ "ReadWriteMany" ],
            "nfs": {
              "path": "/nfs_share",
              "server": "172.18.200.200"
            },
            "persistentVolumeReclaimPolicy": "Retain"
          }
       }')

       Note: No namespace (--namespace) required for PersistentVolumes.

       If you have to delete the persistentvolume use (commented out for safety):

       # oc delete persistentvolume/nfs4registry-pv-0001

    6. Verify that the persistent volume got created:

       oc get pv

    7. Create Persistent Volume Claim

       oc create --namespace openshift-image-registry -f <(echo '{
          "apiVersion": "v1",
          "kind": "PersistentVolumeClaim",
          "metadata": {
            "name": "nfs4registry-pvc-0001"
          },
          "spec": {
            "accessModes": [ "ReadWriteMany" ],
            "resources": {
              "requests": {
                "storage": "100Gi"
              }
            },
            "volumeName": "nfs4registry-pv-0001",
            "storageClassName": ""
          }
       }')

       Note: You can also run oc project openshift-image-registry to work within the openshift-image-registry namespace for this session. In this case you can leave out the --namespace option.

       If required use following command to delete the claim (commented out for safety):

       # oc delete --namespace openshift-image-registry persistentvolumeclaim/nfs4registry-pvc-0001

    8. Verify that the persistent volume claim got created:

       oc get pvc

    9. Look out for any errors and fix them if required:

       oc describe pvc/nfs4registry-pvc-0001

    10. Edit the image registry operator configuration to make it use the persistent volume claim:

        oc edit configs.imageregistry.operator.openshift.io

        Add the following lines after "spec:":

        spec:
          storage:
            pvc:
              claim: nfs4registry-pvc-0001

    11. Disable (if not already) and enable the image registry operator to make the changes active. First disable the Operator by setting the managementState to Removed:

        oc patch configs.imageregistry/cluster --patch '{"spec":{"managementState":"Removed"}}' --type=merge

        Wait a bit until the change is reflected here:

        oc get clusteroperator image-registry

        Enable the image registry operator:

        oc patch configs.imageregistry/cluster --patch '{"spec":{"managementState":"Managed"}}' --type=merge

    12. Now check that the image registry pod is running without any issues:

        oc describe pods/$(oc get pods  -n openshift-image-registry | awk '/^image-registry/ {print $1}')

    13. If any problems occurred in the previous step check common issues:

        • In most cases the following command should give you a good hint on why the image registry operator is failing:

          oc describe pods/$(oc get pods  -n openshift-image-registry | awk '/^image-registry/ {print $1}')

        • permissions issues: It can be helpful to directly look into the pod that should mount the NFS directory to see what is going on:

          oc debug pods/$(oc get pods  -n openshift-image-registry | awk '/^image-registry/ {print $1}') -n openshift-image-registry
          ls -ld /registry

        • image pruner: If you wait for too long you might see an image-pruner pod and job. As soon as you see this you either have to restart the installation or run the following command that are not recommended in production:

          oc -n openshift-image-registry describe jobs
          oc -n openshift-image-registry delete jobs --all

        • check log

          POD=$(oc get pods  -n openshift-image-registry | awk '/cluster-image-registry-operator/ {print $1}')
          oc logs ${POD} -n openshift-image-registry

        • guest crashing: If the guests are crashing you might want to set: on_crash to preserve, coredump-restart or coredump-destroy. This allows you to analyse the crashed guest afterwards.

        • broken machineconfig: Machineconfigs might be broken if you see one of the following error messages: "One or more machine config pools are degraded, please see oc get mcp for further details and resolve before upgrading. OR: Unable to apply 4.xx.x: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool master is not ready, retrying. Status: (pool degraded: true total: 3, ready 0, updated: 0, unavailable: 3)]"

          • Every OpenShift node configures itself with a rendered machineconfig yaml file. The rendered machineconfig that a node uses can be looked up under metadata.annotations."machineconfiguration.openshift.io/currentConfig":

            oc describe node/control0

            The file with a similar name to this: rendered-master-*, should be available in the list of machineconfig files:

            oc get machineconfig

            If the file with the exact name does not exist but another rendered-master-* file then following steps might help.

          • Extract the existing rendered-master-* config file to a file named exactly after the name found under the nodes metadata currentConfig. Replace AAAAAA and BBBBBB respectively:

            oc get machineconfig/rendered-master-AAAAA -o yaml >   rendered-master-BBBBB.yaml

            Edit rendered-master-BBBBB.yaml and also change metadata.name from rendered-master-AAAAA with rendered-master-BBBBB. Then create the machineconfig file.

            oc create -f rendered-master-dfe0359da63096c9f3633af0d844280f.yaml   
            oc get machineconfig

            The error message should now go away.

29. At this point all operators should be available and none of them should have error messages or is in a degraded state.

    watch -n5 oc get clusteroperators

30. Finally check on the bastion guest that the installation is completed with:

    cd /openshift-files/bin
    ./openshift-install --dir /openshift-installation wait-for install-complete 

    In the output you should get the username and password needed to log in to the OpenShift web-console:

    INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/openshift-installation/auth/kubeconfig'
    INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp0.sa.boe
    INFO Login to the console with user: "kubeadmin", and password: "PASSWORD"

Congratulation, the installation of OpenShift is now complete. In the following you get instructions on how to access to the web-console and post installation steps.

1. To connect to the OpenShift web-console from the PC Environment through the ssh Gateway follow the sub-steps (Pc is assumed to be linux like):

   1. From the outside the ip is 172.18.100.100, therefore

      sudo ssh -L 443:forward.example.com:443 USER@gateway01.unsafe.example.com -N

   2. Edit the /etc/hosts file with admin rights and add:

      127.0.0.1 console-openshift-console.apps.ocp0.sa.boe
      127.0.0.1 oauth-openshift.apps.ocp0.sa.boe 

      This is required to be able to type the exact hostname in the web-browser used by OpenShift generated TLS certificate for the web interface.

   3. Now you should be able to access the OpenShift web-console with a web-browser: https://console-openshift-console.apps.ocp0.sa.boe.

   4. Log in with the username and password you got at the last OpenShift installation step.

1. Change the View in the top left corner from "Administrator" to "Developer".

2. Click Create Project:

   Name: httpd-example
   Display-name: httpd-example
   Description: example

3. From "Developer Catalog" select "All services" and select "Apache HTTP Server" (Templates).

4. Leave everything on default and press "Create". If you run into a build error you should check the image registry operator and the NFS server.

5. Add the following lines to the /etc/hosts file on your computer.

   127.0.0.1 httpd-example-apache-webserver-example.apps.ocp0.sa.boe

6. Forward port 80 to be able to access the example webpage:

   sudo ssh -L 80:forward.example.com:80 USER@gateway01.unsafe.example.com -N

7. Access the example webpage under: httpd-example-apache-webserver-example.apps.ocp0.sa.boe.

To make the OpenShift guest nodes high available the current KVM config files are exported to the respective shared file systems.

1. Export the config file of each OpenShift guest node with virsh dumpxml to the shared disks:

   virsh dumpxml control0 > /mnt/control0-mnt/control0-guest.xml
   virsh dumpxml control1 > /mnt/control1-mnt/control1-guest.xml
   virsh dumpxml control2 > /mnt/control2-mnt/control2-guest.xml
   virsh dumpxml compute0 > /mnt/compute0-mnt/compute0-guest.xml
   virsh dumpxml compute1 > /mnt/compute1-mnt/compute1-guest.xml
   virsh dumpxml bootstrap > /mnt/bootstrap-mnt/bootstrap-guest.xml

2. Shut down all instances (except the bastion guest, since it is already in the RHEL HA cluster):

   for i in $(virsh list --name|grep -v "bastion"); do virsh shutdown $i; done

3. Wait until all OCP guests have shut down gracefully:

   watch virsh list

4. Undefine the guests that were previously shut down:

   for i in $(virsh list --inactive --name); do virsh undefine $i; done

5. Relabel the mount folder:

   restorecon -R -v '/mnt'

6. Define all guests as pacemaker resource:

   pcs resource create control0-guest \
   ocf:heartbeat:VirtualDomain \
   config="/mnt/control0-mnt/control0-guest.xml" \
   hypervisor="qemu:///system" \
   migration_transport="ssh" \
   meta allow-migrate=false \
   --group control0-group

   pcs resource create control1-guest \
   ocf:heartbeat:VirtualDomain \
   config="/mnt/control1-mnt/control1-guest.xml" \
   hypervisor="qemu:///system" \
   migration_transport="ssh" \
   meta allow-migrate=false \
   --group control1-group

   pcs resource create control2-guest \
   ocf:heartbeat:VirtualDomain \
   config="/mnt/control2-mnt/control2-guest.xml" \
   hypervisor="qemu:///system" \
   migration_transport="ssh" \
   meta allow-migrate=false \
   --group control2-group

   pcs resource create compute0-guest \
   ocf:heartbeat:VirtualDomain \
   config="/mnt/compute0-mnt/compute0-guest.xml" \
   hypervisor="qemu:///system" \
   migration_transport="ssh" \
   meta allow-migrate=false \
   --group compute0-group

   pcs resource create compute1-guest \
   ocf:heartbeat:VirtualDomain \
   config="/mnt/compute1-mnt/compute1-guest.xml" \
   hypervisor="qemu:///system" \
   migration_transport="ssh" \
   meta allow-migrate=false \
   --group compute1-group

   pcs resource create bootstrap-guest \
   ocf:heartbeat:VirtualDomain \
   config="/mnt/compute1-mnt/bootstrap-guest.xml" \
   hypervisor="qemu:///system" \
   migration_transport="ssh" \
   meta allow-migrate=false \
   --group bootstrap-group

7. Keep the bootstrap disabled:

   pcs resource disable bootstrap-group

8. Finally, the resources can be distributed across the cluster:

   pcs resource move control1-group halp02
   pcs resource move control2-group halp03
   pcs resource move compute1-group halp02

install-config.yaml

apiVersion: v1
baseDomain: sa.boe
compute:
  # pool 1
  - hyperthreading: Enabled
    # THIS VALUE SEEMS TO BE HARDCODED: (we wanted: compute)
    name: worker
    # must be 0 (only more with ipi)
    replicas: 0
    architecture: s390x
  # multiple pools are supported in the future

# pool 2
controlPlane:
  hyperthreading: Enabled
  # THIS VALUE IS NOT HARDCODED BUT WILL BE OVERWRITTEN TO: MASTER
  name: master
  replicas: 3
  architecture: s390x

metadata:
  name: ocp0

networking:
  machineNetwork:
    - cidr: 10.128.0.0/14
  clusterNetwork:
    - cidr: 10.192.0.0/14
      # create /23 subnets for each host
      hostPrefix: 23
  # Either OpenShiftSDN or OVNKubernetes (default)
  networkType: OpenShiftSDN
  #networkType: OVNKubernetes
  serviceNetwork:
    - 172.30.0.0/16

platform:
  none: { }
fips: false
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29jbV9hY2Nlc3NfM2FmMzkzNGkFWVVpUUVlJM0xCHSLORUswSDkxWFRQSkMwZTY3NDhhMjhkMjVlMmZjYTg1YzYzZTU6RjhWR09VOVJLWjZDOVZVTElVWpXMEFJQ1FTSzZTVEsyM01TQ041Ng==","email":"example@example.com"},"quay.io":{"auth":"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29jbV9hY2Nlc3NfM2FmMzkzNGkFWVVpUUVlJM0xCHSLORUswSDkxWFRQSkMwZTY3NDhhMjhkMjVlMmZjYTg1YzYzZTU6RjhWR09VOVJLWjZDOVZVTElVWpXMEFJQ1FTSzZTVEsyM01TQ041Ng==","email":"example@example.com"},"registry.connect.redhat.com":{"auth":"fHVoYy1wb29sLTI3NGU5ZjU3LTk1ODctNDUxNS1hMzllLTYwY2E0MjY5lV4TWlKOS5leUp6ZFdJaU9pSmtZV0V3TkdJMU56azNOREEwWXpSbFlXWTBOMlUzTXpjeU1XUTVZVGt6TkNKOS5UMkhYZmF5MXJxSU9TNjBpX2lhN0JzYMWNkZDpleUpoYkdjaU9pSlNVeWhUTWNrUjlTdXZ5SDFfRUd4WHh0NkI1alpHa0hxXzhxeXFKZUVteWtKSWMwcEZXMF81YWFTV0FYRWtla0tsM0VyYTNiRl9yeWFETld2RWJ1QmxfTVJiZTEyJ2Z3hCWnkySnJZMERPa1djR081UElRS2tTT3BadkJJbzJpTXlSNWJaMEd2SV9KXzRSQ0JTUTZjR0xVckpBdVh4QndfeEZwcWx6MVlxNkUyWlo3dnZBdmxUZWFOQUFNcUNiYnVQVHc2eG05Z2VRaTJkdFY3UWtya29EeHNVd2M0SmIxY1Bfc2tYZnEwRGUtWWx1RFVnNW84dHB2R2g2dHJxMjFBOGUxRnhDUjZwam1JRDJBcHBLSGJEMEt2S3NTTTFOclo1aE1Nd3NkJ3OTdOdUxndGNkWUp0UVhkejF6bE5FUmF0TXRONWV2QmFpT1NTeGFWb2FFYkc2Y1dFa3M4N1JVcTBma05yWlZxbkV4WXBQTmluQnh3UjhzX0ZlRXFsOGdlbW5fUWtSdjNuUm11QlB2MnlzRVlEQm1GQUNGZEtQWEVTeTdpb2VJbnhsRFVUSjZnTFp1MS00RFphODVNSzIxN0RzbWdTUVZmOV9ZamM3ZG1VOC1TNjI2aGtBLWpsZ04xZ0VLRmlnU3RrdVo2cXBHamwwdWlwMzRHRk1YMEdYNHdmc2FVdVRoanN0bGx0NWlHU1JoNk1SWDdyWnU0d29BdTNzUHV5NmI5SWk1ZS1NTWJPdEhTYXZlX3dLNTlpMWptX1drb3pFb0Y5el9WU1FhUVhBbV8tUHJnYXlmZC1WZzZLZmd1enV2dktoM2Flbm4yYlhqTF9Eb1RaU1ZHN2pZMEFja1N6UDNiSW1sVFdRdW83TF83MUhGYw==","email":"example@example.com"},"registry.redhat.io":{"auth":"fHVoYy1wb29sLTI3NGU5ZjU3LTk1ODctNDUxNS1hMzllLTYwY2E0MjY5lV4TWlKOS5leUp6ZFdJaU9pSmtZV0V3TkdJMU56azNOREEwWXpSbFlXWTBOMlUzTXpjeU1XUTVZVGt6TkNKOS5UMkhYZmF5MXJxSU9TNjBpX2lhN0JzYMWNkZDpleUpoYkdjaU9pSlNVeWhUTWNrUjlTdXZ5SDFfRUd4WHh0NkI1alpHa0hxXzhxeXFKZUVteWtKSWMwcEZXMF81YWFTV0FYRWtla0tsM0VyYTNiRl9yeWFETld2RWJ1QmxfTVJiZTEyJ2Z3hCWnkySnJZMERPa1djR081UElRS2tTT3BadkJJbzJpTXlSNWJaMEd2SV9KXzRSQ0JTUTZjR0xVckpBdVh4QndfeEZwcWx6MVlxNkUyWlo3dnZBdmxUZWFOQUFNcUNiYnVQVHc2eG05Z2VRaTJkdFY3UWtya29EeHNVd2M0SmIxY1Bfc2tYZnEwRGUtWWx1RFVnNW84dHB2R2g2dHJxMjFBOGUxRnhDUjZwam1JRDJBcHBLSGJEMEt2S3NTTTFOclo1aE1Nd3NkJ3OTdOdUxndGNkWUp0UVhkejF6bE5FUmF0TXRONWV2QmFpT1NTeGFWb2FFYkc2Y1dFa3M4N1JVcTBma05yWlZxbkV4WXBQTmluQnh3UjhzX0ZlRXFsOGdlbW5fUWtSdjNuUm11QlB2MnlzRVlEQm1GQUNGZEtQWEVTeTdpb2VJbnhsRFVUSjZnTFp1MS00RFphODVNSzIxN0RzbWdTUVZmOV9ZamM3ZG1VOC1TNjI2aGtBLWpsZ04xZ0VLRmlnU3RrdVo2cXBHamwwdWlwMzRHRk1YMEdYNHdmc2FVdVRoanN0bGx0NWlHU1JoNk1SWDdyWnU0d29BdTNzUHV5NmI5SWk1ZS1NTWJPdEhTYXZlX3dLNTlpMWptX1drb3pFb0Y5el9WU1FhUVhBbV8tUHJnYXlmZC1WZzZLZmd1enV2dktoM2Flbm4yYlhqTF9Eb1RaU1ZHN2pZMEFja1N6UDNiSW1sVFdRdW83TF83MUhGYw==","email":"example@example.com"}}}
'
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhki96M2kBkq3fxi54ZgvrteHz16pG3PhIVHFFDovHxDtgFz5Ga4FXSl5vNsPSgZJUWHWUkVV6KSZflkYXW6PvdLxwLA15UEx4spxA/KF5CgZuLuNfALsKuJd0DrqJD5/beyQDAUDDHGOkntUNX0Mht8xZIEgibSVsi3E72MiFYIZ9cHX2L2fG4aHjwAQDNd9AzX1wTazaOcDcIewo/XFSJuYRTcClAthlujZWc1XsbNeIA6rfzta2g4YHIWxzlHJBzdXXvPWoHgHyH48d2HKLwURFvETMJCmQTRfCBtC/GRws3bOMCA2MqwU0p9Rdxx49bVlWAauX7NYFaFbjx6MED6dLw/Nk/+3YjoIff6DhkKXni9H3DJwpAX+fqp/s1Y/jtGYjZSQ/0GOrCRPV9CreYxOBQFO67RRVvYjxWlJvCUkHYDvec3wtzmxUTxwsfZSZb08Kt6M/k1aPq5q0C1SVHK8Oge/gNRVY/aoP/P0q38YirnZAvBudK/71j4Q2wGM='

compute-ip-override.yaml

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
     machineconfiguration.openshift.io/role: worker
  name: 98-nodenet-override
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,IyEvdXNyL2Jpbi9lbnYgYmFzaAoKVEhFX0lQX0FERFJFU1M9JChpcCByb3V0ZSBnZXQgMTAuMTI4LjAuMSB8IHNlZCAtbiAnL3NyYy97cy8uKnNyYyAqXChbXiBdKlwpLiovXDEvcDtxfScpCgojIFN1ZG8gd2F5ClsgLWYgL2V0Yy9zeXN0ZW1kL3N5c3RlbS9rdWJlbGV0LnNlcnZpY2UuZC85OC1ub2RlbmV0LW92ZXJyaWRlLmNvbmYgXSAmJiBcCnN1ZG8gdHJ1bmNhdGUgLXMgMCAvZXRjL3N5c3RlbWQvc3lzdGVtL2t1YmVsZXQuc2VydmljZS5kLzk4LW5vZGVuZXQtb3ZlcnJpZGUuY29uZgoKc3VkbyB0ZWUgLWEgL2V0Yy9zeXN0ZW1kL3N5c3RlbS9rdWJlbGV0LnNlcnZpY2UuZC85OC1ub2RlbmV0LW92ZXJyaWRlLmNvbmYgPi9kZXYvbnVsbCA8PEVPRgpbU2VydmljZV0KRW52aXJvbm1lbnQ9IktVQkVMRVRfTk9ERV9JUD0ke1RIRV9JUF9BRERSRVNTfSIgIktVQkVMRVRfTk9ERV9JUFM9JHtUSEVfSVBfQUREUkVTU30iCgpFT0YK
        mode: 0755
        overwrite: true
        path: /usr/local/bin/override-node-ip.sh
    systemd:
      units:
      - contents: |
          [Unit]
          Description=Override node IP detection
          Wants=network-online.target
          Before=kubelet.service
          After=network-online.target
          [Service]
          Type=oneshot
          ExecStart=/usr/local/bin/override-node-ip.sh
          ExecStart=systemctl daemon-reload
          [Install]
          WantedBy=multi-user.target
        enabled: true
        name: nodenet-override.service

control-ip-override.yaml

---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
     machineconfiguration.openshift.io/role: master
  name: 98-nodenet-override
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,IyEvdXNyL2Jpbi9lbnYgYmFzaAoKVEhFX0lQX0FERFJFU1M9JChpcCByb3V0ZSBnZXQgMTAuMTI4LjAuMSB8IHNlZCAtbiAnL3NyYy97cy8uKnNyYyAqXChbXiBdKlwpLiovXDEvcDtxfScpCgojIFN1ZG8gd2F5ClsgLWYgL2V0Yy9zeXN0ZW1kL3N5c3RlbS9rdWJlbGV0LnNlcnZpY2UuZC85OC1ub2RlbmV0LW92ZXJyaWRlLmNvbmYgXSAmJiBcCnN1ZG8gdHJ1bmNhdGUgLXMgMCAvZXRjL3N5c3RlbWQvc3lzdGVtL2t1YmVsZXQuc2VydmljZS5kLzk4LW5vZGVuZXQtb3ZlcnJpZGUuY29uZgoKc3VkbyB0ZWUgLWEgL2V0Yy9zeXN0ZW1kL3N5c3RlbS9rdWJlbGV0LnNlcnZpY2UuZC85OC1ub2RlbmV0LW92ZXJyaWRlLmNvbmYgPi9kZXYvbnVsbCA8PEVPRgpbU2VydmljZV0KRW52aXJvbm1lbnQ9IktVQkVMRVRfTk9ERV9JUD0ke1RIRV9JUF9BRERSRVNTfSIgIktVQkVMRVRfTk9ERV9JUFM9JHtUSEVfSVBfQUREUkVTU30iCgpFT0YK
        mode: 0755
        overwrite: true
        path: /usr/local/bin/override-node-ip.sh
    systemd:
      units:
      - contents: |
          [Unit]
          Description=Override node IP detection
          Wants=network-online.target
          Before=kubelet.service
          After=network-online.target
          [Service]
          Type=oneshot
          ExecStart=/usr/local/bin/override-node-ip.sh
          ExecStart=systemctl daemon-reload
          [Install]
          WantedBy=multi-user.target
        enabled: true
        name: nodenet-override.service

override-node-ip.sh

#!/usr/bin/env bash

THE_IP_ADDRESS=$(ip route get 10.128.0.1 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}')

# Sudo way
[ -f /etc/systemd/system/kubelet.service.d/98-nodenet-override.conf ] && \
sudo truncate -s 0 /etc/systemd/system/kubelet.service.d/98-nodenet-override.conf

sudo tee -a /etc/systemd/system/kubelet.service.d/98-nodenet-override.conf >/dev/null <<EOF
[Service]
Environment="KUBELET_NODE_IP=${THE_IP_ADDRESS}" "KUBELET_NODE_IPS=${THE_IP_ADDRESS}"

EOF