IBM Support

DB mount point switch procedure

General Page

This guide is to be followed if you initially had an older 7.22.x OVA deployed and have upgraded it to a current version (8.1.x or newer). If you have or will have Grafana report enabled, without switching the mount point, pgres will be mounted under /var partition, and consumes up the disk space there, which could lead to pod eviction issue.

_______________________________________________________________

The way to check if you are on an older OVA, ssh into the VM and run the command below

df -h | egrep -v "overlay|shm"
The output will show something similar to the below and you need to verify that the “Filesystem” and “Mounted on” shows exactly what is highlighed in red below before proceeding, otherwise the steps below will fail and the database services won’t start.
  • If the highlighed “Filesystem” is something different then what is highlighed do not proceed
  • If the highlighed “Mounted on” is “/var/lib/dbs” you can skip the step below and do nothing
image-20240219120410-1
  • If and only if the red highlighed Filesystem and Mounted on match the above screen shot and you have at least 200GB or more of Availalbe space then you can continue.

    • The only exception is if the Mounted on is “/var/lib/dbs” in which case you can skip the step to run the script switch_dbs_mount_point.sh

  • You should now shutdown the Turbo VM using the command “sudo init 0” and wait for the VM to be powered off and then take an offline snapshot. This is needed just in case rollback is required. Once the snapshot is complete you can power on the VM

  • You will now need to wait for all the Turbo pods to come back online as per normal process

  • Once all pods are in a 1/1 Ready state you now need to terminate all of the pods using the command below. This will first scale the operator down so it won’t start any other pods

    kubectl scale deployment t8c-operator --replicas=0
  • Now wait about 30 seconds then run the command below to scale all of the remaining pods. This is needed to ensure that nothing is trying to access the database while we switch the db mount point

    kubectl scale deployment --all --replicas=0
  • Now you need to wait for al pods to be terminated by running the command below and ensure that no pods are running, the output should be blank with a response of No resources found once they are all terminated

    kubectl get pods
image-20240229134615-1
  • Once all pods are terminated you will now need to run the script named “switch_dbs_mount_point.sh which is in the /opt/local/bin directory on the server. When you run it you will see the output of the command as shown below

image-20240229134644-2

  • Now when you run the command below you should see that the Mounted on has changed to /var/lib/dbs

    • df -h | egrep -v "overlay|shm"

image-20240229134718-3

  • Now you need to ensure that the MariaDB and Postgresql-12 DB services are active and running using the commands below

    • service mariadb status | grep active

    • service postgresql-12 status | grep active

image-20240229134756-4

  • As long as both services are running you can now scale up the operator to start up all of the pods using the command below

    • kubectl scale deployment t8c-operator --replicas=1

    • Then run the command to check that all of the pods are starting up now: kubectl get pods

    • If for some reason the operator doesn’t scale up all of the other pods you can always run the command: kubectl scale deployment --all --replicas=1

  • You will now see that the operator has started and is starting up all of the other pods, which will take some time for all of the pods to be started up and become 1/1 Ready

[{"Type":"MASTER","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSFV9Z","label":"IBM Turbonomic Application Resource Management"},"ARM Category":[{"code":"a8m3p000000PCRRAA4","label":"System"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
15 August 2024

UID

ibm17120700