Installing IBM Data Cataloging
IBM Data Cataloging service is a modern metadata management software that provides data insight for exabyte-scale heterogeneous file, object, backup, and archive storage on premises and in the cloud. It can help you manage your unstructured data by reducing the data storage costs, uncovering hidden data value, and reducing the risk of massive data stores.
Before you begin
- Meet the system requirements to install the IBM Data Cataloging service.
- The following details are a base line for finding the resources that are needed for IBM Fusion IBM Data Cataloging service deployment. Based on the following tables, the resources can be estimated based on the number of approximate files that are required. The following are the resource values that are calculated per compute node: You must have at least two worker nodes, each with the same amount of resources available.
- IBM Fusion
IBM Data Cataloging service must have dedicated compute
resources. Make sure that you have enough to cover the resources limits to perform as
expected:
Table 1. Profile requirements CPU RAM Disk space Network Storage Workload 77 162 GB 120 GB 10 GB 1 TB 1 B - The standard deployment for IBM Data Cataloging service
project requests and limits. Ensure you set the resource quota values for IBM Data Cataloging as follows:
kind: ResourceQuota apiVersion: v1 metadata: name: dcs-resource-quota namespace: ibm-data-cataloging spec: hard: limits.cpu: '100' limits.memory: 180Gi pods: '120' requests.cpu: '20' requests.memory: 40Gi -
Important: For the IBM Data Cataloging service to run successfully on all platforms, ensure that the storage classes have the following attributes:
- ReadWriteMany (RWX) permissions
- volumeBindingMode set to Immediate
- AllowVolumeExpansion set to true
- If you have not configured the IBM operator catalog, then configure it. For the procedure to add IBM operator catalog, see Adding the IBM operator catalog.
- Go through troubleshooting information related to the installation of IBM Data Cataloging. See IBM Data Cataloging service installation and upgrade issues.
- IBM Data Cataloging can be deployed as containers on Red Hat OpenShift Virtualization Engine (OVE) but must run on normal worker nodes.
Procedure
- Go to Services page in IBM Fusion user interface.
- In the Available section, click Data cataloging tile.
- In the Data cataloging window, go through the details of the service and click Install.
- In the Install service message box, select a Storage
class. Important: If you want to use Global Data Platform as the storage provider, then it is recommended to select the default storage class
ibm-spectrum-fusion. Otherwise, if you want to use Fusion Data Foundation, then select theocs-storagecluster-cephfsstorage class. You can also use a custom storage class that matches the requirements. - Click Install. In case of failures, go through the downloaded logs to understand the cause of the failure and fix the issue. For more information about service issues in IBM Fusion, see IBM Data Cataloging service installation and upgrade issues.
- Validate the installation.
- IBM Fusion user interface:
After you enable the IBM Data Cataloging service, you can view the service version and health status. From the ellipsis menu, you can download logs and view documentation. After you successfully collect the logs, a success notification gets displayed. The notification disappears automatically after some time.
Table 2. Health states IBM Data Cataloging service State Description InstallingService installation is in progress UpgradingService upgrade is in progress HealthyService is healthy DegradedService is not healthy - Verify the IBM Data Cataloging installation
from the OpenShift® Container Platform web console:
- Run the following command to verify the installed
operators.
oc -n ibm-data-cataloging get csvExpected output NAME DISPLAY VERSION REPLACES PHASE amqstreams.v2.3.0-1 Red Hat Integration - AMQ Streams 2.3.0-1 amqstreams.v2.3.0-0 Succeeded db2u-operator.v3.1.0 IBM Db2 3.1.0 Succeeded ibm-spectrum-discover-operator.v210.0.0 IBM Storage Discover 210.0.0-1678324770 Succeeded
- Run the following command to verify IBM Data Cataloging
status.
oc -n ibm-data-cataloging get isdExpected output NAME READY SUMMARY ERROR AGE HEALTHSTATUS data-cataloging-service-instance True Awaiting next reconciliation 9d Healthy
- Run the following command to verify that all Persistent Volume Claims are
bounded.
oc -n ibm-data-cataloging get pvcExpected output NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE activelogs-c-isd-db2u-0 Bound pvc-7e98eead-9a38-48b6-a880-16d730120796 150Gi RWO ocs-storagecluster-cephfs 9d activelogs-c-isd-db2u-1 Bound pvc-7af71786-1529-4b3c-bbd1-b5d1ee733d33 150Gi RWO ocs-storagecluster-cephfs 9d c-isd-meta Bound pvc-1ba20699-ddb8-4600-bf9f-4d637b14da49 50Gi RWX ocs-storagecluster-cephfs 9d data-c-isd-db2u-0 Bound pvc-488540db-c951-4f48-b65c-814bf634ca33 300Gi RWO ocs-storagecluster-cephfs 9d data-c-isd-db2u-1 Bound pvc-56a3105a-347e-4b55-b11a-d0a1d351d622 300Gi RWO ocs-storagecluster-cephfs 9d data-isd-sasl-kafka-0 Bound pvc-3924c9cc-173d-4244-90fa-a234325dc04b 100Gi RWO ocs-storagecluster-cephfs 9d data-isd-sasl-zookeeper-0 Bound pvc-3c66397b-ac03-4ed8-841a-e33d83355a23 64Gi RWO ocs-storagecluster-cephfs 9d data-isd-ssl-kafka-0 Bound pvc-ce0ebeaf-d2b6-4e40-8c31-e31d45a208e5 100Gi RWO ocs-storagecluster-cephfs 9d data-isd-ssl-zookeeper-0 Bound pvc-cd3fdd20-343b-49ca-ad1a-690f8eec3e60 64Gi RWO ocs-storagecluster-cephfs 9d isd-backup Bound pvc-2e077c8e-e992-4e1a-8b98-d24bfdbb5343 250Gi RWX ocs-storagecluster-cephfs 9d isd-data Bound pvc-b98e124f-1f98-43e5-9bf6-c8758c681192 250Gi RWX ocs-storagecluster-cephfs 9d tempts-c-isd-db2u-0 Bound pvc-7fdc9ac5-9158-48ac-bad3-3c24590f1f2b 50Gi RWO ocs-storagecluster-cephfs 9d tempts-c-isd-db2u-1 Bound pvc-a2adb668-a362-4f61-8ed3-ae103f9b2cd3 50Gi RWO ocs-storagecluster-cephfs 9d
- Run the following command to verify Db2u cluster detailed
progress.
oc -n ibm-data-cataloging get formations.db2u.databases.ibm.com isd -o go-template='{{range .status.components}}{{printf "%s,%s,%s\n" .kind .name .status.state}}{{end}}' | column -s, -tExpected output account account-ibm-data-cataloging-isd OK PersistentVolumeClaim c-isd-meta OK secret c-isd-sshkeys-db2uhausr OK secret c-isd-sshkeys-db2uadm OK secret c-isd-sshkeys-db2instusr OK secret c-isd-ldappassword OK secret c-isd-ldapblueadminpassword OK secret c-isd-instancepassword OK secret c-isd-db2u-lic OK secret c-isd-certs-wv-rest OK secret c-isd-certs-db2u-api OK configmap c-isd-db2uconfig OK configmap c-isd-db2regconfig OK configmap c-isd-db2dbmconfig OK configmap c-isd-db2dbconfig OK service c-isd-ldap OK Deployment c-isd-ldap OK service c-isd-tools OK Deployment c-isd-tools OK configmap c-isd-db2u-api OK service c-isd-db2u-engn-svc OK service c-isd-db2u-head-engn-svc OK Job c-isd-instdb OK service c-isd-etcd OK StatefulSet c-isd-etcd OK service c-isd-db2u-rest-svc OK networkpolicy c-isd-rest-ext OK Deployment c-isd-rest OK service c-isd-db2u-graph-svc OK networkpolicy c-isd-graph-ext OK Deployment c-isd-graph OK service c-isd-db2u-internal OK service c-isd-db2u OK StatefulSet c-isd-db2u OK networkpolicy c-isd OK networkpolicy c-isd-ext OK Job c-isd-restore-morph OK
- Run the following command to verify the Db2u cluster CR
status.
oc -n ibm-data-cataloging get db2uclusterExpected output NAME STATE MAINTENANCESTATE AGE isd Ready None 9d
- Run the following command to verify that the ports 50000 and 50001 are
listing.
oc exec -n ibm-data-cataloging -it $(oc -n ibm-data-cataloging get po --no-headers --show-labels=true --selector name=dashmpp-head-0 | awk '{print $1}') -- su - db2inst1 -c 'netstat -ntlp'Expected output Defaulted container "db2u" out of: db2u, init-labels (init), init-kernel (init) (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 10.254.4.11:60000 0.0.0.0:* LISTEN 23707/db2sysc 0 tcp 0 0 10.254.4.11:60001 0.0.0.0:* LISTEN 23712/db2sysc 1 tcp 0 0 0.0.0.0:9443 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:50022 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:50000 0.0.0.0:* LISTEN 23707/db2sysc 0 tcp 0 0 0.0.0.0:50001 0.0.0.0:* LISTEN 23707/db2sysc 0 tcp6 0 0 :::50052 :::* LISTEN - tcp6 0 0 :::50022 :::* LISTEN -
- Run the following command to verify that workload is
PUREDATA_OLAP.
oc exec -n ibm-data-cataloging -c db2u -it $(oc -n ibm-data-cataloging get po --no-headers --show-labels=true --selector name=dashmpp-head-0 | awk '{print $1}') -- sudo su - db2inst1 -c "db2set -all | grep DB2_WORKLOAD= | awk '{print \$2}'"Expected output DB2_WORKLOAD=PUREDATA_OLAP
- Run the following command to verify the database encryption
configurations.
oc -n ibm-data-cataloging exec -c db2u $(oc -n ibm-data-cataloging get po --no-headers --show-labels=true --selector name=dashmpp-head-0 | awk '{print $1}') -- su - db2inst1 -c "db2 get db cfg for bludb | grep 'Encrypted database'"Encrypted database = YES
- Run the following command to verify the Kafka CRs
readiness.
oc -n ibm-data-cataloging get kafkaExpected output NAME DESIRED KAFKA REPLICAS DESIRED ZK REPLICAS READY WARNINGS isd-sasl 1 1 True isd-ssl 1 1 True
- Run the following command to verify that the IBM®
Spectrum Discover pods are
running.
oc -n ibm-data-cataloging get pod -l component=discoverExpected output NAME READY STATUS RESTARTS AGE isd-api-7564475d98-cs6nx 1/1 Running 0 9d isd-auth-5fcb8bd95f-jvzwp 1/1 Running 0 9d isd-backup-restore-689fc64f7-pgx22 1/1 Running 0 9d isd-connmgr-7d4cdb4cc7-xnkzm 1/1 Running 1 (9d ago) 9d isd-consumer-ceph-le-5f9b5d4676-4fsgg 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-9k9cs 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-bphqp 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-cc2pz 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-chlvk 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-fchp5 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-jlbrw 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-l78cc 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-lpcsn 1/1 Running 0 9d isd-consumer-ceph-le-5f9b5d4676-n2wqt 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-54qm9 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-6hsq9 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-9p4p5 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-dhlgc 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-hr76h 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-mkvx8 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-tvscz 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-vrrd4 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-w59c2 1/1 Running 0 9d isd-consumer-cos-le-846f5fd97f-xqncl 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-4klcc 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-8r6w7 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-fqs5m 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-fwb8p 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-h78g5 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-lvwl5 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-nhznb 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-q2l8b 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-z6tfj 1/1 Running 0 9d isd-consumer-cos-scan-79fb979585-zv2zw 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-5mrrg 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-8rphj 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-b5dbv 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-b5lv2 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-d9gbm 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-f5rxf 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-m8nw6 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-ntfbb 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-ql55x 1/1 Running 0 9d isd-consumer-file-scan-5c7565bf7b-qvggm 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-6s75x 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-845bx 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-8kmnf 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-gntbt 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-mlnfb 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-p9lpt 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-rckm8 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-vkmpd 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-vmh8x 1/1 Running 0 9d isd-consumer-protect-scan-67d5ffb65-zm4gf 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-6dw9h 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-9nhsz 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-dnwpw 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-dz2mf 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-h2ffj 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-hdrxg 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-nf779 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-pf6n4 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-q8gqf 1/1 Running 0 9d isd-consumer-scale-le-5d56449994-xzmvh 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-49vkb 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-54pb8 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-5sr89 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-98lm9 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-d4zlj 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-fc82j 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-mnsn2 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-r2kc6 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-tglsz 1/1 Running 0 9d isd-consumer-scale-scan-7bdc7957c6-zbhgv 1/1 Running 0 9d isd-contentsearchagent-5d7bfbd887-r487x 1/1 Running 1 (9d ago) 9d isd-db-schema-xm5cn 0/1 Completed 0 9d isd-db2whrest-6c6bbfcbc6-9v6v2 1/1 Running 1 (9d ago) 9d isd-generate-license-6g94v 0/1 Completed 0 9d isd-importtags-64498dbdd8-5kqpb 1/1 Running 1 (9d ago) 9d isd-keystone-7dc84b69f5-sw6nn 1/1 Running 0 9d isd-license-check-27982200-c9gkm 0/1 Completed 0 2d21h isd-license-check-27983640-ldxv2 0/1 Completed 0 45h isd-license-check-27985080-rhzjl 0/1 Completed 0 21h isd-policyengine-598f565867-zg5sn 1/1 Running 1 (9d ago) 9d isd-producer-ceph-le-76b9885854-557b7 1/1 Running 0 9d isd-producer-cos-le-85d446c9d8-zvzmg 1/1 Running 0 9d isd-producer-cos-scan-d584f646d-2t669 1/1 Running 0 9d isd-producer-file-scan-5586b48fd-zkt7q 1/1 Running 0 9d isd-producer-protect-scan-7b66bb7c64-wz6qq 1/1 Running 0 9d isd-producer-scale-le-8d7bc64b4-nvqbl 1/1 Running 0 9d isd-producer-scale-scan-5df647956-qfl6p 1/1 Running 0 9d isd-proxy-5f4d7dd8ff-44fwt 1/1 Running 0 9d isd-reports-c6kmq 0/1 Completed 0 9d isd-scaleafmdatamover-85cc44b566-644vx 1/1 Running 1 (9d ago) 9d isd-scaleilmdatamover-6b86dd7b58-j28sn 1/1 Running 1 (9d ago) 9d isd-sdmonitor-6fffc999b7-s2srd 1/1 Running 0 9d isd-tikaserver-79548bdb4-ssft5 1/1 Running 0 9d isd-ui-backend-f6c4b65d7-dwzrs 1/1 Running 0 9d isd-ui-frontend-9659fc847-vcbmv 1/1 Running 0 9d
- Run the following command to verify the console route
availability.
oc -n ibm-data-cataloging get route consoleExpected output NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-ibm-data-cataloging.apps.ocp2.vmlocal isd-svc <all> reencrypt None
- Run the following command to verify the installed
operators.
- IBM Fusion user interface: