SevOne Data Insight Migrate Operating System & Data Guide

Pre-Migration Checklist

Before proceeding with the migration of the operating system, please check the following.

$ sdi-cli exec mysql -- mysqldump -uroot -pdatainsight \
--databases datainsight reportscheduler | \
grep -v "Using a password" > /home/sevone/sdidata-$(date +%d-%m-%Y).sql

$ sdi-cli backup-securitykeys -okeys.json -P /home/sevone

serviceAccount:
  createKeys:
    enabled: false

$ kubectl get pods
Unable to connect to the server: x509: certificate has expired or is not yet valid

$ sudo mkdir /root/.kube

$ sudo cp /etc/kubernetes/admin.conf /root/.kube/config

$ ip route | grep default

$ ip route add default via <default_gateway>

Migrate Operating System

SevOne Data Insight operating system has migrated from RHEL to CentOS 7.

1. Using ssh, log into SevOne Data Insight as sevone.

   $ ssh sevone@<SevOne Data Insight IP address or hostname>
   

2. Execute the following steps to migrate the operating system.

   $ screen -S unrhel
   
   $ tar xvfz /home/sevone/unrhel-*.tgz -C /home/sevone
   
   $ sudo bash /home/sevone/unrhel/scripts/migrate os 2>&1 | \
   tee -a /home/sevone/unrhel_os.log
   

   If you are using unrhel_v5.1.1.tgz or above, the script will automatically remove configshell from /usr/local/bin directory.
   After the migration completes successfully, nodes will restart.

   Perform the following 3 checks:

   • Did the migration from RHEL to CentOS 7 complete successfully?
   • When you run sudo nmcli dev | grep -i ^eth, does the command return anything?
   • When you run sudo grep 'biosdevname=0 net.ifnames=0' /etc/default/grub, is it empty?

   If the answer is yes to all the 3 checks, please execute the following commands.

       $ sudo -i
       
       $ sed -i 's/GRUB_CMDLINE_LINUX="[^"]*/& biosdevname=0 net.ifnames=0/' \ 
       /etc/default/grub
       
       $ grub2-mkconfig --output=/boot/grub2/grub.cfg
   

   You are now ready to migrate your data.

Migrate Data

After the migration completes successfully and the nodes are backed up, you may proceed to restore your data. You will require the latest SevOne Data Insight build. For example, sdi-v6.6.0-build.139.tgz file. Download the latest file from IBM Passport Advantage via Passport Advantage Online or contact the IBM SevOne Support if you have an active Support contract. Place it in /home/sevone.

Please execute the following steps.

1. Using ssh, log into SevOne Data Insight as sevone.

   $ ssh sevone@<SevOne Data Insight IP address or hostname>
   

2. If you have any agent nodes, please make sure your SSH keys are still valid. If not, run the following command to set the SSH keys.

   $ IFS=$'\n'; for line in $(cat /home/sevone/unrhel/backups/workers); \
   do ip=${line##* }; ssh-copy-id sevone@$ip; \
   ssh sevone@$ip 'sudo cp /home/sevone/.ssh/authorized_keys /root/.ssh/'; \
   done
   

3. You are now ready to migrate your data.

   $ screen -S unrhel
   
   $ sudo bash /home/sevone/unrhel/scripts/migrate \
   kube $(ls -Art sdi-*.tgz | tail -n 1) 2>&1 | \
   tee -a /home/sevone/unrhel_kube.log
   

   You are now on SevOne Data Insight 6.6.

Upgrade SOA

1. Using ssh, log into SevOne Data Insight as sevone.

   $ ssh sevone@<SevOne Data Insight IP address or hostname>
   

2. After unrhel completes migrating your data, you will notice the GraphQL pod is in a crash loop. Upgrade SOA to fix this.

   If you are running a single-node NMS cluster, you may omit the --all-peers flag in the command below.

   $ sevone-cli soa upgrade /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm --all-peers
   

   If a cluster is larger than ~20 peers, it is recommended that you only upgrade the datasource nms and schedule to run the upgrade on all peers at a later time.
   
   Execute the following command to obtain the datasource IP address.

   $ sevone-cli mysql 'SELECT address from datasources WHERE \ 
   dstype = "nms-datasource" OR dstype = "METRICS/FLOW"' | \ 
   grep -oP 'https://\K\S+'
   

   Once you have obtained the datasource IP address, execute the following command.

   $ sevone-cli soa upgrade /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm \ <br />
   <enter SevOne NMS IP address>
   

3. After SOA is updated, restart the GraphQL pod.

   $ kubectl delete pods -l app.kubernetes.io/component=graphql
   

Verify Upgrade

Execute the following command to verify the upgrade.

1. Using ssh, log into SevOne Data Insight as sevone.

   $ ssh sevone@<SevOne Data Insight IP address or hostname>
   

2. Check the status of the upgrade by running the following command. Ensure that everything is in Running status.

   $ kubectl get pods
   NAME                                                      READY   STATUS      RESTARTS      AGE
   di-create-secrets-kxzkm                                   0/1     Completed   0             21h
   di-graphql-5879886c46-zshfr                               1/1     Running     1 (27m ago)   21h
   ingress-ingress-nginx-controller-f5fb79d48-fz7bl          1/1     Running     1 (27m ago)   21h
   ingress-ingress-nginx-defaultbackend-7f5fbcdcdc-jps25     1/1     Running     1 (27m ago)   21h
   di-kube-state-metrics-84f86b867b-r7xrg                    1/1     Running     1 (27m ago)   21h
   di-printer-68f6bddb6f-hkhdt                               1/1     Running     1 (27m ago)   21h
   di-redis-master-0                                         2/2     Running     2 (27m ago)   21h
   di-minio-cd4bc55fb-b649g                                  1/1     Running     1 (27m ago)   21h
   di-ui-58b687f949-x46ps                                    1/1     Running     1 (27m ago)   21h
   di-wdkserver-78c499cc8c-tsjzj                             1/1     Running     1 (27m ago)   21h
   datasource-operator-controller-manager-6c8fc59bb9-c7trt   2/2     Running     2 (27m ago)   21h
   di-prometheus-node-exporter-gwbjh                         1/1     Running     1 (27m ago)   21h
   di-mysql-0                                                2/2     Running     2 (27m ago)   21h
   di-migrator-69bc8dbbdc-jgms2                              1/1     Running     1 (27m ago)   21h
   di-prometheus-server-6797d7c66-sfg8m                      2/2     Running     2 (27m ago)   21h
   di-rabbitmq-0                                             1/1     Running     1 (27m ago)   21h
   di-scheduler-67b79494b9-4j9rr                             1/1     Running     1 (27m ago)   21h
   di-asset-sweeper-27949980-kcrh9                           0/1     Completed   0             15m
   di-user-sync-27949980-5v2dv                               0/1     Completed   0             15m
   di-asset-sweeper-27949985-f75cn                           0/1     Completed   0             10m
   di-user-sync-27949985-rm2rn                               0/1     Completed   0             10m
   di-asset-sweeper-27949990-xrgqq                           0/1     Completed   0             5m4s
   di-user-sync-27949990-p8zqb                               0/1     Completed   0             5m4s
   di-user-sync-27949995-2hj7c                               1/1     Running     0             4s
   di-asset-sweeper-27949995-nmx5b                           0/1     Completed   0             4s
   

Old Installer Archives

SevOne Data Insight archives are approximately 1.5GB in size. Removing these archives can quickly clean up your Hard Disk Drive if you are running low on available disk space. Execute the following command to find the files.

Example

$ sdi-cli cluster exec --all 'find /opt/ -name "ins1ght-installer-*.tgz"'
/opt/datainsight/releases/ins1ght-installer-v1.6.0.tgz
/opt/datainsight/releases/ins1ght-installer-v1.6.1.tgz
/opt/datainsight/releases/ins1ght-installer-v1.6.1.1.tgz
/opt/datainsight/releases/ins1ght-installer-v2.0.0.tgz
/opt/datainsight/releases/ins1ght-installer-v2.1.0.tgz
/opt/datainsight/releases/ins1ght-installer-v2.2.0.tgz
/opt/datainsight/releases/ins1ght-installer-v3.0.0.tgz
/opt/datainsight/releases/ins1ght-installer-v3.1.0-build18.tgz
/opt/datainsight/releases/ins1ght-installer-v3.2.0-build14.tgz
/opt/datainsight/releases/ins1ght-installer-v3.3.0-build.19.tgz
/opt/datainsight/releases/ins1ght-installer-v3.4.0-build.21.tgz      <-- active version

Unused elasticsearch Data

Elasticsearch was removed from SevOne Data Insight >= v1.6.1 and its persisted data can be found in /var/sdi-data. This is not removed automatically to preserve the option to rollback. If you are on newer versions of SevOne Data Insight, it is safe to remove this data. Execute the following command to find where this can be found in your setup.

$ sdi-cli cluster exec --all 'sudo find /var/sdi-data -type d -name elastic'
/var/sdi-data/elastic

Log Files

Kubernetes constantly outputs logs to /var/log/messages. Over time, this folder can accumulate several gigabytes worth of logs. Execute the following command to find the log files which can safely be removed.

Example

$ sdi-cli cluster exec --all 'sudo du -sh /var/log/* | grep messages'
27M   /var/log/messages
3.5G  /var/log/messages-20210430
1.4G  /var/log/messages-20210502
79M   /var/log/messages-20210509

Find Expired Certificates

Kubernetes keeps its certificates in /etc/kubernetes/pki. In this directory, expired certificates can be isolated.

For the example below, let’s assume today’s date is Dec 15 2020.

Example

# log in as the root user
$ sudo -i
  
# find the expired certs
$ for i in $(ls /etc/kubernetes/pki/*.crt); \
do echo $i; openssl x509 -dates -noout -in $i | \
grep After; echo "---"; done

/etc/kubernetes/pki/apiserver.crt
notAfter=Dec 10 21:50:53 2020 GMT
---
/etc/kubernetes/pki/apiserver-kubelet-client.crt
notAfter=Dec 10 21:50:54 2020 GMT
---
/etc/kubernetes/pki/ca.crt
notAfter=Dec  7 21:50:53 2029 GMT
---
/etc/kubernetes/pki/front-proxy-ca.crt
notAfter=Dec  7 21:50:54 2029 GMT
---
/etc/kubernetes/pki/front-proxy-client.crt
notAfter=Dec 10 21:50:54 2020 GMT

Based on this example, you will see that the following certificates expired in December 2020.

• /etc/kubernetes/pki/apiserver.crt
• /etc/kubernetes/pki/apiserver-kubelet-client.crt
• /etc/kubernetes/pki/front-proxy-client.crt

And, the two ca certificates expire in December 2029.

• /etc/kubernetes/pki/ca.crt
• /etc/kubernetes/pki/front-proxy-ca.crt
  

Remove Expired Certificates

As you can see in the example above, a few certificates expired in December. Execute the steps below to remove the expired certificates.

# backup the kubernetes folder
$ cp -r /etc/kubernetes /etc/kubernetes.bak

# remove the expired certs
$ rm -rf /etc/kubernetes/pki/apiserver* /etc/kubernetes/pki/front-proxy-client.*

The following expired certificates have been removed.

• /etc/kubernetes/pki/apiserver.crt
• /etc/kubernetes/pki/apiserver-kubelet-client.crt
• /etc/kubernetes/pki/front-proxy-client.crt

Execute the following command and you will see that now, only the two ca certificates (which are still valid and not expired) are available.

# this should show just the two CA certs
$ for i in $(ls /etc/kubernetes/pki/*.crt); \
do echo $i; openssl x509 -dates -noout -in $i | \
grep After; echo "---"; done

/etc/kubernetes/pki/ca.crt
notAfter=Dec  7 21:50:53 2029 GMT
---
/etc/kubernetes/pki/front-proxy-ca.crt
notAfter=Dec  7 21:50:54 2029 GMT

Generate New Certificates

Execute the following step to generate new certificates for the ones expired and removed.

Backup the Kubernetes folder

$ cp -r /etc/kubernetes /etc/kubernetes.bak

Renew the certificates

$ kubeadm alpha certs renew all

Verify the certificates have been generated successfully

$ for i in $(ls /etc/kubernetes/pki/*.crt); \
do echo $i; openssl x509 -dates -noout -in $i | \
grep After; echo "---"; done
 
/etc/kubernetes/pki/apiserver.crt
notAfter=Dec 11 21:50:53 2021 GMT
---
/etc/kubernetes/pki/apiserver-kubelet-client.crt
notAfter=Dec 11 21:50:54 2021 GMT
---
/etc/kubernetes/pki/ca.crt
notAfter=Dec  7 21:50:53 2029 GMT
---
/etc/kubernetes/pki/front-proxy-ca.crt
notAfter=Dec  7 21:50:54 2029 GMT
---
/etc/kubernetes/pki/front-proxy-client.crt
notAfter=Dec 11 21:50:54 2021 GMT

You will see that the following certificates now expire in December 2021.

• /etc/kubernetes/pki/apiserver.crt
• /etc/kubernetes/pki/apiserver-kubelet-client.crt
• /etc/kubernetes/pki/front-proxy-client.crt

And, the two ca certificates will expire in December 2029.

• /etc/kubernetes/pki/ca.crt
• /etc/kubernetes/pki/front-proxy-ca.crt
  

Generate New Admin Configuration File

kubectl runs commands against the Kubernetes API Server. When sending the requests, it looks for the admin config file which contains the certificate data.

After rotating the certificates, the admin config file must be regenerated along with the config files for the other components.

# generate the conf files
$ rm -rf /etc/kubernetes/{admin.conf,kubelet.conf,controller-manager.conf,scheduler.conf}
$ kubeadm init phase kubeconfig all

# copy over the admin config for the root user
$ cp /etc/kubernetes/admin.conf ~/.kube/config

# copy over the admin config for the sevone user
$ cp /etc/kubernetes/admin.conf /home/sevone/.kube/config

$ chown -R sevone:sevone /home/sevone/.kube

Restart Services

Restart kubelet for the changes to take effect.

$ systemctl restart kubelet

At this point, you can now run Kubernetes commands. This will allow you to backup the important security keys in case you have not done so already.