Backing up and restoring the Watson Machine Learning Accelerator service

Edit online

Use this information to backup or restore the IBM Watson® Machine Learning Accelerator service.

Online backup and restore of Watson Machine Learning Accelerator

Online backup

To complete an online backup, see Cloud Pak for Data online backup and restore.

Online restore

After you restore the Watson Machine Learning Accelerator service using the Cloud Pak for Data restore process, you must run an additional script to restore owner references to all Watson Machine Learning Accelerator resources.
Note: The conda PVC will be synced after a restore is completed.

Before you begin:

Before performing an online restore, make sure that the Watson Machine Learning Accelerator namespace is deleted.

Steps:
  1. Log in to your OpenShift cluster as a project administrator.
    oc login OpenShift_URL:port
  2. Update the IBM NamespaceScope Operator in the IBM Cloud Pak for Data platform operator project to watch the project where Watson Machine Learning Accelerator is installed.
    oc edit namespacescope -n ibm-common-services common-service
  3. Switch to the Watson Machine Learning Accelerator namespace.
    oc project wmla-namespace
  4. Return owner references to Watson Machine Learning Accelerator resources, run the following script depending on your version of Watson Machine Learning Accelerator:
    Version 2.4.1:
    #!/bin/bash
wmla_name=`oc get wmla -o name|awk -F/ '{print $NF}'`
wmla_uid=`oc get wmla $wmla_name -o jsonpath='{.metadata.uid}'`
user_pvc=`oc get wmla $wmla_name -o jsonpath={.spec.usePreCreatedPvcs}`

for r in \
certificate.cert-manager.io/wmla-ca-crt \
certificate.cert-manager.io/wmla-internal-keys \
certificate.cert-manager.io/wmla-nginx-keys \
configmap/cpd-wmla-br-cm \
configmap/cpd-wmla-ckpt-cm \
configmap/cpd-wmla-qu-cm \
configmap/wmla-dlpd-bootstrap \
configmap/wmla-edi \
configmap/wmla-edi-dlim \
configmap/wmla-edi-imd-nginx \
configmap/wmla-edi-isd \
configmap/wmla-edi-isd-ingress \
configmap/wmla-grafana-configmap \
configmap/wmla-grafana-ini \
configmap/wmla-grafana-providers \
configmap/wmla-infoservice \
configmap/wmla-jupyter-hub-config \
configmap/wmla-logstash-conf \
configmap/wmla-mongodb-shells \
configmap/wmla-msd \
configmap/wmla-mss \
configmap/wmla-nginx-conf \
configmap/wmla-nginx-grafana-sidecar-conf \
configmap/wmla-nginx-sidecar-conf \
configmap/wmla-prometheus \
configmap/wmla-version-info \
deployment.apps/wmla-auth-rest \
deployment.apps/wmla-conda \
deployment.apps/wmla-dlpd \
deployment.apps/wmla-edi-imd \
deployment.apps/wmla-edi-lbd \
deployment.apps/wmla-grafana \
deployment.apps/wmla-gui \
deployment.apps/wmla-infoservice \
deployment.apps/wmla-ingress \
deployment.apps/wmla-jupyter-gateway \
deployment.apps/wmla-jupyter-hub \
deployment.apps/wmla-jupyter-proxy \
deployment.apps/wmla-logstash \
deployment.apps/wmla-msd \
deployment.apps/wmla-mss \
deployment.apps/wmla-prometheus \
ingress.networking.k8s.io/wmla-jupyter-ingress \
issuer.cert-manager.io/wmla-ca \
issuer.cert-manager.io/wmla-root-issuer \
networkpolicy.networking.k8s.io/wmla-dlpd-netpol \
networkpolicy.networking.k8s.io/wmla-edi-imd-network-policy \
networkpolicy.networking.k8s.io/wmla-edi-isd-network-policy \
networkpolicy.networking.k8s.io/wmla-infoservice-netpol \
networkpolicy.networking.k8s.io/wmla-ingress-network-policy \
networkpolicy.networking.k8s.io/wmla-logstash-network-policy \
networkpolicy.networking.k8s.io/wmla-msd-netpol \
networkpolicy.networking.k8s.io/wmla-namespace-network-policy \
persistentvolumeclaim/wmla-conda \
persistentvolumeclaim/wmla-cws-share \
persistentvolumeclaim/wmla-edi \
persistentvolumeclaim/wmla-infoservice \
persistentvolumeclaim/wmla-logging \
persistentvolumeclaim/wmla-mygpfs \
persistentvolumeclaim/wmla-grafana \
persistentvolumeclaim/wmla-prometheus \
poddisruptionbudget.policy/wmla-jupyter-hub-pdb \
poddisruptionbudget.policy/wmla-jupyter-proxy-pdb \
role.rbac.authorization.k8s.io/wmla-core-role \
role.rbac.authorization.k8s.io/wmla-edi \
role.rbac.authorization.k8s.io/wmla-msd-mss \
role.rbac.authorization.k8s.io/wmla-notebook-role \
role.rbac.authorization.k8s.io/wmla-role \
rolebinding.rbac.authorization.k8s.io/wmla-core-rb \
rolebinding.rbac.authorization.k8s.io/wmla-edi \
rolebinding.rbac.authorization.k8s.io/wmla-msd-mss \
rolebinding.rbac.authorization.k8s.io/wmla-notebook-rb \
rolebinding.rbac.authorization.k8s.io/wmla-rb \
route.route.openshift.io/wmla-console \
route.route.openshift.io/wmla-grafana \
route.route.openshift.io/wmla-inference \
route.route.openshift.io/wmla-jupyter-notebook \
secret/wmla-dlpd-conf \
secret/wmla-eg-secret \
secret/wmla-grafana-secret \
secret/wmla-jupyter-hub-secret \
secret/wmla-mongodb-secret \
secret/wmla-prometheus-htpasswd \
service/wmla-auth-rest \
service/wmla-dlpd \
service/wmla-edi \
service/wmla-etcd \
service/wmla-grafana \
service/wmla-gui \
service/wmla-inference \
service/wmla-infoservice \
service/wmla-ingress \
service/wmla-jupyter-enterprise-gateway \
service/wmla-jupyter-hub \
service/wmla-jupyter-proxy-api \
service/wmla-jupyter-proxy-public \
service/wmla-logstash-service \
service/wmla-mongodb \
service/wmla-msd \
service/wmla-mss \
service/wmla-prometheus \
serviceaccount/wmla-core-sa \
serviceaccount/wmla-msd-mss \
serviceaccount/wmla-norbac \
serviceaccount/wmla-notebook-sa \
serviceaccount/wmla-sa \
statefulset.apps/wmla-etcd \
statefulset.apps/wmla-mongodb;
do
	oc get $r >& /dev/null
	if [ $? == "0" ]; then
		#skip patch user pvc
		if [ x$user_pvc == 'xtrue' ];then
			resoucetype=`echo $r|awk -F'/' '{print $1}'`
			if [ x$resoucetype == 'xpersistentvolumeclaim' ];then
				echo "skip user defined PVC $r"
				continue
			fi
		fi
		echo "Patch ownerReferences for $r"
		oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
	fi
done

#update ownerReferences for wmla resource plans
wmla_rps=`oc get rp -o name`
ns_rp=`oc get rp platform  -o jsonpath={.spec.parent}`
wmla_fix_rp=`oc get rp platform  -o jsonpath={.spec.children[0].name}`
cpd_fix_rp=`oc get rp platform  -o jsonpath={.spec.children[1].name}`
for r in $wmla_rps;
do
	#skip scheduler created resource plans
	rp_name=`echo $r|awk -F'/' '{print $2}'`
	if [ x$rp_name == "xplatform" -o x$rp_name == "x$ns_rp" -o x$rp_name == "x$wmla_fix_rp" -o x$rp_name == "x$cpd_fix_rp" ];then
		echo "skip resource plan $rp_name"
		continue
	fi
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
done

#update ownerReferences for deploy/isd and isd/service
isds=`oc get deploy -o name|grep wmla-edi-isd`
imd_uid=`oc get deploy wmla-edi-imd -o jsonpath='{.metadata.uid}'`
for r in $isds;
do
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"wmla-edi-imd\",\"uid\":\"$imd_uid\"}]}}"
done

isd_servicess=`oc get services -o name|grep wmla-edi-isd`
for r in $isd_servicess;
do
	isd_name=`echo $r|awk -F/ '{print $NF}'`
	isd_uid=`oc get deploy $isd_name -o jsonpath='{.metadata.uid}'`
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"$isd_name\",\"uid\":\"$isd_uid\"}]}}"
done

#update ownerReferences for wmla-add-on cm
wmla_add_on_name=`oc get wmla-add-on -o name|awk -F/ '{print $NF}'`
if [ x$wmla_add_on_name != x ];then
	wmla_add_on_uid=`oc get wmla-add-on $wmla_add_on_name -o jsonpath='{.metadata.uid}'`
	oc patch configmap/cpd-wmla-add-on-br-cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
	wmla_instance_cm=`oc get cm -o name|grep wml-accelerator-instance-cm`
	oc patch $wmla_instance_cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
	wmla_connection_cm=`oc get cm -o name|grep wml-accelerator-connection-info-extension`
	oc patch $wmla_connection_cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
fi

    Version 2.4.0:

    #!/bin/bash
wmla_name=`oc get wmla -o name|awk -F/ '{print $NF}'`
wmla_uid=`oc get wmla $wmla_name -o jsonpath='{.metadata.uid}'`
user_pvc=`oc get wmla $wmla_name -o jsonpath={.spec.usePreCreatedPvcs}`

for r in \
certificate.cert-manager.io/wmla-ca-crt \
certificate.cert-manager.io/wmla-internal-keys \
certificate.cert-manager.io/wmla-nginx-keys \
configmap/cpd-wmla-br-cm \
configmap/cpd-wmla-ckpt-cm \
configmap/cpd-wmla-qu-cm \
configmap/wmla-dlpd-bootstrap \
configmap/wmla-edi \
configmap/wmla-edi-dlim \
configmap/wmla-edi-imd-nginx \
configmap/wmla-edi-isd \
configmap/wmla-edi-isd-ingress \
configmap/wmla-grafana-configmap \
configmap/wmla-grafana-ini \
configmap/wmla-grafana-providers \
configmap/wmla-infoservice \
configmap/wmla-jupyter-hub-config \
configmap/wmla-logstash-conf \
configmap/wmla-mongodb-shells \
configmap/wmla-msd \
configmap/wmla-mss \
configmap/wmla-nginx-conf \
configmap/wmla-nginx-grafana-sidecar-conf \
configmap/wmla-nginx-sidecar-conf \
configmap/wmla-prometheus \
configmap/wmla-version-info \
deployment.apps/wmla-auth-rest \
deployment.apps/wmla-conda \
deployment.apps/wmla-dlpd \
deployment.apps/wmla-edi-imd \
deployment.apps/wmla-edi-lbd \
deployment.apps/wmla-grafana \
deployment.apps/wmla-gui \
deployment.apps/wmla-infoservice \
deployment.apps/wmla-ingress \
deployment.apps/wmla-jupyter-gateway \
deployment.apps/wmla-jupyter-hub \
deployment.apps/wmla-jupyter-proxy \
deployment.apps/wmla-logstash \
deployment.apps/wmla-msd \
deployment.apps/wmla-mss \
deployment.apps/wmla-prometheus \
ingress.networking.k8s.io/wmla-jupyter-ingress \
issuer.cert-manager.io/wmla-ca \
issuer.cert-manager.io/wmla-root-issuer \
networkpolicy.networking.k8s.io/wmla-dlpd-netpol \
networkpolicy.networking.k8s.io/wmla-edi-imd-network-policy \
networkpolicy.networking.k8s.io/wmla-edi-isd-network-policy \
networkpolicy.networking.k8s.io/wmla-infoservice-netpol \
networkpolicy.networking.k8s.io/wmla-ingress-network-policy \
networkpolicy.networking.k8s.io/wmla-logstash-network-policy \
networkpolicy.networking.k8s.io/wmla-msd-netpol \
networkpolicy.networking.k8s.io/wmla-namespace-network-policy \
persistentvolumeclaim/wmla-conda \
persistentvolumeclaim/wmla-cws-share \
persistentvolumeclaim/wmla-edi \
persistentvolumeclaim/wmla-infoservice \
persistentvolumeclaim/wmla-logging \
persistentvolumeclaim/wmla-mygpfs \
poddisruptionbudget.policy/wmla-jupyter-hub-pdb \
poddisruptionbudget.policy/wmla-jupyter-proxy-pdb \
role.rbac.authorization.k8s.io/wmla-core-role \
role.rbac.authorization.k8s.io/wmla-edi \
role.rbac.authorization.k8s.io/wmla-msd-mss \
role.rbac.authorization.k8s.io/wmla-notebook-role \
role.rbac.authorization.k8s.io/wmla-role \
rolebinding.rbac.authorization.k8s.io/wmla-core-rb \
rolebinding.rbac.authorization.k8s.io/wmla-edi \
rolebinding.rbac.authorization.k8s.io/wmla-msd-mss \
rolebinding.rbac.authorization.k8s.io/wmla-notebook-rb \
rolebinding.rbac.authorization.k8s.io/wmla-rb \
route.route.openshift.io/wmla-console \
route.route.openshift.io/wmla-grafana \
route.route.openshift.io/wmla-inference \
route.route.openshift.io/wmla-jupyter-notebook \
secret/wmla-dlpd-conf \
secret/wmla-eg-secret \
secret/wmla-grafana-secret \
secret/wmla-jupyter-hub-secret \
secret/wmla-mongodb-secret \
secret/wmla-prometheus-htpasswd \
service/wmla-auth-rest \
service/wmla-dlpd \
service/wmla-edi \
service/wmla-etcd \
service/wmla-grafana \
service/wmla-gui \
service/wmla-inference \
service/wmla-infoservice \
service/wmla-ingress \
service/wmla-jupyter-enterprise-gateway \
service/wmla-jupyter-hub \
service/wmla-jupyter-proxy-api \
service/wmla-jupyter-proxy-public \
service/wmla-logstash-service \
service/wmla-mongodb \
service/wmla-msd \
service/wmla-mss \
service/wmla-prometheus \
serviceaccount/wmla-core-sa \
serviceaccount/wmla-msd-mss \
serviceaccount/wmla-norbac \
serviceaccount/wmla-notebook-sa \
serviceaccount/wmla-sa \
statefulset.apps/wmla-etcd \
statefulset.apps/wmla-mongodb;
do
	oc get $r >& /dev/null
	if [ $? == "0" ]; then
		#skip patch user pvc
		if [ x$user_pvc == 'xtrue' ];then
			resoucetype=`echo $r|awk -F'/' '{print $1}'`
			if [ x$resoucetype == 'xpersistentvolumeclaim' ];then
				echo "skip user defined PVC $r"
				continue
			fi
		fi
		echo "Patch ownerReferences for $r"
		oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
	fi
done

#update ownerReferences for wmla resource plans
wmla_rps=`oc get rp -o name`
ns_rp=`oc get rp platform  -o jsonpath={.spec.parent}`
wmla_fix_rp=`oc get rp platform  -o jsonpath={.spec.children[0].name}`
cpd_fix_rp=`oc get rp platform  -o jsonpath={.spec.children[1].name}`
for r in $wmla_rps;
do
	#skip scheduler created resource plans
	rp_name=`echo $r|awk -F'/' '{print $2}'`
	if [ x$rp_name == "xplatform" -o x$rp_name == "x$ns_rp" -o x$rp_name == "x$wmla_fix_rp" -o x$rp_name == "x$cpd_fix_rp" ];then
		echo "skip resource plan $rp_name"
		continue
	fi
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
done

#update ownerReferences for deploy/isd and isd/service
isds=`oc get deploy -o name|grep wmla-edi-isd`
imd_uid=`oc get deploy wmla-edi-imd -o jsonpath='{.metadata.uid}'`
for r in $isds;
do
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"wmla-edi-imd\",\"uid\":\"$imd_uid\"}]}}"
done

isd_servicess=`oc get services -o name|grep wmla-edi-isd`
for r in $isd_servicess;
do
	isd_name=`echo $r|awk -F/ '{print $NF}'`
	isd_uid=`oc get deploy $isd_name -o jsonpath='{.metadata.uid}'`
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"$isd_name\",\"uid\":\"$isd_uid\"}]}}"
done

#update ownerReferences for wmla-add-on cm
wmla_add_on_name=`oc get wmla-add-on -o name|awk -F/ '{print $NF}'`
if [ x$wmla_add_on_name != x ];then
	wmla_add_on_uid=`oc get wmla-add-on $wmla_add_on_name -o jsonpath='{.metadata.uid}'`
	oc patch configmap/cpd-wmla-add-on-br-cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
	wmla_instance_cm=`oc get cm -o name|grep wml-accelerator-instance-cm`
	oc patch $wmla_instance_cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
	wmla_connection_cm=`oc get cm -o name|grep wml-accelerator-connection-info-extension`
	oc patch $wmla_connection_cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
fi

Offline backup and restore of Watson Machine Learning Accelerator

Offline backup

Before you complete an offline backup of the Watson Machine Learning Accelerator service using the standard backup process, you must stop all running workloads and delete your conda content.

Steps:
  1. Stop all running workloads.
    1. As an Watson Machine Learning Accelerator project administrator, from the Watson Machine Learning Accelerator console stop all running jobs.
      1. Log in to the Watson Machine Learning Accelerator console as a project administrator.
      2. Navigate to Monitoring > Applications.
      3. For each running applications, select the menu icon and click Stop.
    2. Stop all running deployed models. Use the WML Accelerator console or the command line interface to stop each running model, see Stop an inference service.
  2. If you are using Portworx or OCS, delete the conda content from the persistent volume (PV):
    1. Log in to your OpenShift cluster as a project administrator.
      oc login OpenShift_URL:port
    2. Switch to the WML Accelerator namespace.
      oc project wmla-namespace
    3. Get the conda pod name:
      oc get po |grep wmla-conda
    4. Delete conda PV data from wmla-conda pod:
      oc exec -it wmla-conda-pod-name bash
bash-4.4$ rm -rfv /opt/conda/*
    5. Delete the conda_synced file from the wmla-conda pod: 
      oc exec -it wmla-conda-pod-name bash
bash-4.4$ rm -rf /var/shareDir/dli/work/conda_synced
  3. Back up the Watson Machine Learning Accelerator service using the standard backup process. See: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=project-backing-up

Offline restore

After you restore the Watson Machine Learning Accelerator service using the Cloud Pak for Data restore process, you must run an additional script to restore owner references to all Watson Machine Learning Accelerator resources.
Note: The conda PVC will be synced after a restore is completed.

Before you begin:

Before performing an online restore, make sure that the Watson Machine Learning Accelerator namespace is deleted.

Steps:
  1. Log in to your OpenShift cluster as a project administrator.
    oc login OpenShift_URL:port
  2. Update the IBM NamespaceScope Operator in the IBM Cloud Pak for Data platform operator project to watch the project where Watson Machine Learning Accelerator is installed.
    oc edit namespacescope -n ibm-common-services common-service
  3. Switch to the Watson Machine Learning Accelerator namespace.
    oc project wmla-namespace
  4. Return owner references to Watson Machine Learning Accelerator resources, run the following script depending on your version of Watson Machine Learning Accelerator:
    Version 2.4.1:
    #!/bin/bash
wmla_name=`oc get wmla -o name|awk -F/ '{print $NF}'`
wmla_uid=`oc get wmla $wmla_name -o jsonpath='{.metadata.uid}'`
user_pvc=`oc get wmla $wmla_name -o jsonpath={.spec.usePreCreatedPvcs}`

for r in \
certificate.cert-manager.io/wmla-ca-crt \
certificate.cert-manager.io/wmla-internal-keys \
certificate.cert-manager.io/wmla-nginx-keys \
configmap/cpd-wmla-br-cm \
configmap/cpd-wmla-ckpt-cm \
configmap/cpd-wmla-qu-cm \
configmap/wmla-dlpd-bootstrap \
configmap/wmla-edi \
configmap/wmla-edi-dlim \
configmap/wmla-edi-imd-nginx \
configmap/wmla-edi-isd \
configmap/wmla-edi-isd-ingress \
configmap/wmla-grafana-configmap \
configmap/wmla-grafana-ini \
configmap/wmla-grafana-providers \
configmap/wmla-infoservice \
configmap/wmla-jupyter-hub-config \
configmap/wmla-logstash-conf \
configmap/wmla-mongodb-shells \
configmap/wmla-msd \
configmap/wmla-mss \
configmap/wmla-nginx-conf \
configmap/wmla-nginx-grafana-sidecar-conf \
configmap/wmla-nginx-sidecar-conf \
configmap/wmla-prometheus \
configmap/wmla-version-info \
deployment.apps/wmla-auth-rest \
deployment.apps/wmla-conda \
deployment.apps/wmla-dlpd \
deployment.apps/wmla-edi-imd \
deployment.apps/wmla-edi-lbd \
deployment.apps/wmla-grafana \
deployment.apps/wmla-gui \
deployment.apps/wmla-infoservice \
deployment.apps/wmla-ingress \
deployment.apps/wmla-jupyter-gateway \
deployment.apps/wmla-jupyter-hub \
deployment.apps/wmla-jupyter-proxy \
deployment.apps/wmla-logstash \
deployment.apps/wmla-msd \
deployment.apps/wmla-mss \
deployment.apps/wmla-prometheus \
ingress.networking.k8s.io/wmla-jupyter-ingress \
issuer.cert-manager.io/wmla-ca \
issuer.cert-manager.io/wmla-root-issuer \
networkpolicy.networking.k8s.io/wmla-dlpd-netpol \
networkpolicy.networking.k8s.io/wmla-edi-imd-network-policy \
networkpolicy.networking.k8s.io/wmla-edi-isd-network-policy \
networkpolicy.networking.k8s.io/wmla-infoservice-netpol \
networkpolicy.networking.k8s.io/wmla-ingress-network-policy \
networkpolicy.networking.k8s.io/wmla-logstash-network-policy \
networkpolicy.networking.k8s.io/wmla-msd-netpol \
networkpolicy.networking.k8s.io/wmla-namespace-network-policy \
persistentvolumeclaim/wmla-conda \
persistentvolumeclaim/wmla-cws-share \
persistentvolumeclaim/wmla-edi \
persistentvolumeclaim/wmla-infoservice \
persistentvolumeclaim/wmla-logging \
persistentvolumeclaim/wmla-mygpfs \
persistentvolumeclaim/wmla-grafana \
persistentvolumeclaim/wmla-prometheus \
poddisruptionbudget.policy/wmla-jupyter-hub-pdb \
poddisruptionbudget.policy/wmla-jupyter-proxy-pdb \
role.rbac.authorization.k8s.io/wmla-core-role \
role.rbac.authorization.k8s.io/wmla-edi \
role.rbac.authorization.k8s.io/wmla-msd-mss \
role.rbac.authorization.k8s.io/wmla-notebook-role \
role.rbac.authorization.k8s.io/wmla-role \
rolebinding.rbac.authorization.k8s.io/wmla-core-rb \
rolebinding.rbac.authorization.k8s.io/wmla-edi \
rolebinding.rbac.authorization.k8s.io/wmla-msd-mss \
rolebinding.rbac.authorization.k8s.io/wmla-notebook-rb \
rolebinding.rbac.authorization.k8s.io/wmla-rb \
route.route.openshift.io/wmla-console \
route.route.openshift.io/wmla-grafana \
route.route.openshift.io/wmla-inference \
route.route.openshift.io/wmla-jupyter-notebook \
secret/wmla-dlpd-conf \
secret/wmla-eg-secret \
secret/wmla-grafana-secret \
secret/wmla-jupyter-hub-secret \
secret/wmla-mongodb-secret \
secret/wmla-prometheus-htpasswd \
service/wmla-auth-rest \
service/wmla-dlpd \
service/wmla-edi \
service/wmla-etcd \
service/wmla-grafana \
service/wmla-gui \
service/wmla-inference \
service/wmla-infoservice \
service/wmla-ingress \
service/wmla-jupyter-enterprise-gateway \
service/wmla-jupyter-hub \
service/wmla-jupyter-proxy-api \
service/wmla-jupyter-proxy-public \
service/wmla-logstash-service \
service/wmla-mongodb \
service/wmla-msd \
service/wmla-mss \
service/wmla-prometheus \
serviceaccount/wmla-core-sa \
serviceaccount/wmla-msd-mss \
serviceaccount/wmla-norbac \
serviceaccount/wmla-notebook-sa \
serviceaccount/wmla-sa \
statefulset.apps/wmla-etcd \
statefulset.apps/wmla-mongodb;
do
	oc get $r >& /dev/null
	if [ $? == "0" ]; then
		#skip patch user pvc
		if [ x$user_pvc == 'xtrue' ];then
			resoucetype=`echo $r|awk -F'/' '{print $1}'`
			if [ x$resoucetype == 'xpersistentvolumeclaim' ];then
				echo "skip user defined PVC $r"
				continue
			fi
		fi
		echo "Patch ownerReferences for $r"
		oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
	fi
done

#update ownerReferences for wmla resource plans
wmla_rps=`oc get rp -o name`
ns_rp=`oc get rp platform  -o jsonpath={.spec.parent}`
wmla_fix_rp=`oc get rp platform  -o jsonpath={.spec.children[0].name}`
cpd_fix_rp=`oc get rp platform  -o jsonpath={.spec.children[1].name}`
for r in $wmla_rps;
do
	#skip scheduler created resource plans
	rp_name=`echo $r|awk -F'/' '{print $2}'`
	if [ x$rp_name == "xplatform" -o x$rp_name == "x$ns_rp" -o x$rp_name == "x$wmla_fix_rp" -o x$rp_name == "x$cpd_fix_rp" ];then
		echo "skip resource plan $rp_name"
		continue
	fi
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
done

#update ownerReferences for deploy/isd and isd/service
isds=`oc get deploy -o name|grep wmla-edi-isd`
imd_uid=`oc get deploy wmla-edi-imd -o jsonpath='{.metadata.uid}'`
for r in $isds;
do
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"wmla-edi-imd\",\"uid\":\"$imd_uid\"}]}}"
done

isd_servicess=`oc get services -o name|grep wmla-edi-isd`
for r in $isd_servicess;
do
	isd_name=`echo $r|awk -F/ '{print $NF}'`
	isd_uid=`oc get deploy $isd_name -o jsonpath='{.metadata.uid}'`
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"$isd_name\",\"uid\":\"$isd_uid\"}]}}"
done

#update ownerReferences for wmla-add-on cm
wmla_add_on_name=`oc get wmla-add-on -o name|awk -F/ '{print $NF}'`
if [ x$wmla_add_on_name != x ];then
	wmla_add_on_uid=`oc get wmla-add-on $wmla_add_on_name -o jsonpath='{.metadata.uid}'`
	oc patch configmap/cpd-wmla-add-on-br-cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
	wmla_instance_cm=`oc get cm -o name|grep wml-accelerator-instance-cm`
	oc patch $wmla_instance_cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
	wmla_connection_cm=`oc get cm -o name|grep wml-accelerator-connection-info-extension`
	oc patch $wmla_connection_cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
fi

    Version 2.4.0:

    #!/bin/bash
wmla_name=`oc get wmla -o name|awk -F/ '{print $NF}'`
wmla_uid=`oc get wmla $wmla_name -o jsonpath='{.metadata.uid}'`
user_pvc=`oc get wmla $wmla_name -o jsonpath={.spec.usePreCreatedPvcs}`

for r in \
certificate.cert-manager.io/wmla-ca-crt \
certificate.cert-manager.io/wmla-internal-keys \
certificate.cert-manager.io/wmla-nginx-keys \
configmap/cpd-wmla-br-cm \
configmap/cpd-wmla-ckpt-cm \
configmap/cpd-wmla-qu-cm \
configmap/wmla-dlpd-bootstrap \
configmap/wmla-edi \
configmap/wmla-edi-dlim \
configmap/wmla-edi-imd-nginx \
configmap/wmla-edi-isd \
configmap/wmla-edi-isd-ingress \
configmap/wmla-grafana-configmap \
configmap/wmla-grafana-ini \
configmap/wmla-grafana-providers \
configmap/wmla-infoservice \
configmap/wmla-jupyter-hub-config \
configmap/wmla-logstash-conf \
configmap/wmla-mongodb-shells \
configmap/wmla-msd \
configmap/wmla-mss \
configmap/wmla-nginx-conf \
configmap/wmla-nginx-grafana-sidecar-conf \
configmap/wmla-nginx-sidecar-conf \
configmap/wmla-prometheus \
configmap/wmla-version-info \
deployment.apps/wmla-auth-rest \
deployment.apps/wmla-conda \
deployment.apps/wmla-dlpd \
deployment.apps/wmla-edi-imd \
deployment.apps/wmla-edi-lbd \
deployment.apps/wmla-grafana \
deployment.apps/wmla-gui \
deployment.apps/wmla-infoservice \
deployment.apps/wmla-ingress \
deployment.apps/wmla-jupyter-gateway \
deployment.apps/wmla-jupyter-hub \
deployment.apps/wmla-jupyter-proxy \
deployment.apps/wmla-logstash \
deployment.apps/wmla-msd \
deployment.apps/wmla-mss \
deployment.apps/wmla-prometheus \
ingress.networking.k8s.io/wmla-jupyter-ingress \
issuer.cert-manager.io/wmla-ca \
issuer.cert-manager.io/wmla-root-issuer \
networkpolicy.networking.k8s.io/wmla-dlpd-netpol \
networkpolicy.networking.k8s.io/wmla-edi-imd-network-policy \
networkpolicy.networking.k8s.io/wmla-edi-isd-network-policy \
networkpolicy.networking.k8s.io/wmla-infoservice-netpol \
networkpolicy.networking.k8s.io/wmla-ingress-network-policy \
networkpolicy.networking.k8s.io/wmla-logstash-network-policy \
networkpolicy.networking.k8s.io/wmla-msd-netpol \
networkpolicy.networking.k8s.io/wmla-namespace-network-policy \
persistentvolumeclaim/wmla-conda \
persistentvolumeclaim/wmla-cws-share \
persistentvolumeclaim/wmla-edi \
persistentvolumeclaim/wmla-infoservice \
persistentvolumeclaim/wmla-logging \
persistentvolumeclaim/wmla-mygpfs \
poddisruptionbudget.policy/wmla-jupyter-hub-pdb \
poddisruptionbudget.policy/wmla-jupyter-proxy-pdb \
role.rbac.authorization.k8s.io/wmla-core-role \
role.rbac.authorization.k8s.io/wmla-edi \
role.rbac.authorization.k8s.io/wmla-msd-mss \
role.rbac.authorization.k8s.io/wmla-notebook-role \
role.rbac.authorization.k8s.io/wmla-role \
rolebinding.rbac.authorization.k8s.io/wmla-core-rb \
rolebinding.rbac.authorization.k8s.io/wmla-edi \
rolebinding.rbac.authorization.k8s.io/wmla-msd-mss \
rolebinding.rbac.authorization.k8s.io/wmla-notebook-rb \
rolebinding.rbac.authorization.k8s.io/wmla-rb \
route.route.openshift.io/wmla-console \
route.route.openshift.io/wmla-grafana \
route.route.openshift.io/wmla-inference \
route.route.openshift.io/wmla-jupyter-notebook \
secret/wmla-dlpd-conf \
secret/wmla-eg-secret \
secret/wmla-grafana-secret \
secret/wmla-jupyter-hub-secret \
secret/wmla-mongodb-secret \
secret/wmla-prometheus-htpasswd \
service/wmla-auth-rest \
service/wmla-dlpd \
service/wmla-edi \
service/wmla-etcd \
service/wmla-grafana \
service/wmla-gui \
service/wmla-inference \
service/wmla-infoservice \
service/wmla-ingress \
service/wmla-jupyter-enterprise-gateway \
service/wmla-jupyter-hub \
service/wmla-jupyter-proxy-api \
service/wmla-jupyter-proxy-public \
service/wmla-logstash-service \
service/wmla-mongodb \
service/wmla-msd \
service/wmla-mss \
service/wmla-prometheus \
serviceaccount/wmla-core-sa \
serviceaccount/wmla-msd-mss \
serviceaccount/wmla-norbac \
serviceaccount/wmla-notebook-sa \
serviceaccount/wmla-sa \
statefulset.apps/wmla-etcd \
statefulset.apps/wmla-mongodb;
do
	oc get $r >& /dev/null
	if [ $? == "0" ]; then
		#skip patch user pvc
		if [ x$user_pvc == 'xtrue' ];then
			resoucetype=`echo $r|awk -F'/' '{print $1}'`
			if [ x$resoucetype == 'xpersistentvolumeclaim' ];then
				echo "skip user defined PVC $r"
				continue
			fi
		fi
		echo "Patch ownerReferences for $r"
		oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
	fi
done

#update ownerReferences for wmla resource plans
wmla_rps=`oc get rp -o name`
ns_rp=`oc get rp platform  -o jsonpath={.spec.parent}`
wmla_fix_rp=`oc get rp platform  -o jsonpath={.spec.children[0].name}`
cpd_fix_rp=`oc get rp platform  -o jsonpath={.spec.children[1].name}`
for r in $wmla_rps;
do
	#skip scheduler created resource plans
	rp_name=`echo $r|awk -F'/' '{print $2}'`
	if [ x$rp_name == "xplatform" -o x$rp_name == "x$ns_rp" -o x$rp_name == "x$wmla_fix_rp" -o x$rp_name == "x$cpd_fix_rp" ];then
		echo "skip resource plan $rp_name"
		continue
	fi
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
done

#update ownerReferences for deploy/isd and isd/service
isds=`oc get deploy -o name|grep wmla-edi-isd`
imd_uid=`oc get deploy wmla-edi-imd -o jsonpath='{.metadata.uid}'`
for r in $isds;
do
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"wmla-edi-imd\",\"uid\":\"$imd_uid\"}]}}"
done

isd_servicess=`oc get services -o name|grep wmla-edi-isd`
for r in $isd_servicess;
do
	isd_name=`echo $r|awk -F/ '{print $NF}'`
	isd_uid=`oc get deploy $isd_name -o jsonpath='{.metadata.uid}'`
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"$isd_name\",\"uid\":\"$isd_uid\"}]}}"
done

#update ownerReferences for wmla-add-on cm
wmla_add_on_name=`oc get wmla-add-on -o name|awk -F/ '{print $NF}'`
if [ x$wmla_add_on_name != x ];then
	wmla_add_on_uid=`oc get wmla-add-on $wmla_add_on_name -o jsonpath='{.metadata.uid}'`
	oc patch configmap/cpd-wmla-add-on-br-cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
	wmla_instance_cm=`oc get cm -o name|grep wml-accelerator-instance-cm`
	oc patch $wmla_instance_cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
	wmla_connection_cm=`oc get cm -o name|grep wml-accelerator-connection-info-extension`
	oc patch $wmla_connection_cm --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla-add-on\",\"name\":\"$wmla_add_on_name\",\"uid\":\"$wmla_add_on_uid\"}]}}"
fi