Storing and persisting DataStage metrics with a metrics repository

You can use the metrics repository to store metrics in a database.

Note: ds-metrics work only on non-FIPS clusters.

With the ds-metrics service, you can set up metrics storage at the project level for all your DataStage flows. Under the Manage tab of your Cloud Pak for Data project, go to DataStage > Metrics Repository and enable persisting metrics. Specify a connection type, configure properties and security details, and test the connection to verify that it works.

Database structure

In the metrics database, public.databasechangelog and public.databasechangeloglock are internal tables that are used to keep track of the database schema versions. The other tables are in the ds-metrics schema.

Enabling pod metrics

Remember: While metrics are enabled by default for your flows, you must manually enable persisting metrics at a project level.

You can enable pod metrics in the px-runtime pod for your DataStage flows. To enable pod metrics, you need to configure the database for the pod metrics and update the deployment for the px-runtime pod.

The default px-runtime instance is ds-px-default, and deployment for default instance is ds-px-default-ibm-datastage-px-runtime.

Add a default database for the ds-metrics service to store the pod metrics in with the following command:

oc create secret -n ds generic ibm-datastage-metrics-db-properties --from-literal=user=<user> --from-literal=host=<host> --from-literal=port=<port> --from-literal=database=<db> --from-literal=pass=<pass>

Restart metrics by deleting the pod with the following command:
```
oc delete pod datastage-ibm-datastage-metrics-8469db87bb-xnqwn
```
where datastage-ibm-datastage-metrics-8469db87bb-xnqwn is an example name of the metrics pod. You can find your metrics pod name with the following command:
```
$ oc get pods -n ds | grep metrics
```
Edit the px-runtime instance deployment:
```
oc edit deployment inst1-small-880-ibm-datastage-px-runtime -n ds
```
where inst1-small-880-ibm-datastage-px-runtime is an example deployment name and ds is an example namespace.

Add a ENABLE_DS_POD_METRICS environment variable under the following section of the deployment YAML. The ENABLE_DS_POD_METRICS variable name is for pod-level metrics only:

...
spec:
  ...
  template:
    ...
    spec:
      ...
      containers:
      - args: ...
        command: ...
        env:
        ...
        # add env var
        - name: ENABLE_DS_POD_METRICS
          value: "True"
        ...

Save the changes to the deployment YAML file.
The system adds the environment variable to the deployment and the pod restarts with the newly added environment variable.