SD-Monitor

SD monitor obtains metrics about the Kafka messages that are fetched from the Kafka topics.

About this task

The SD monitor provides information about the Kafka messages that are fetched from the Kafka topics.

In the Data Cataloging service, each Kafka topic represents a type of connection, such as NFS, IBM Storage Scale, COS, and S3, which has multiple consumers fetch the messages that are sent from the producers when they connect to a data source and scan data from it.

You can retrieve the following metrics from the SD monitor:
  • Current committed offset for the set of topics and partitions.
  • Log end offset (highest offset in the partition).
  • Lag, the difference between the current committed offset and log end offset values.

Procedure

Follow the steps to get SD monitor information is as follows:
  1. Run the following command to get the logs of the SD monitor pod.
    (oc logs $( oc get pods -n ibm-data-cataloging | grep sdmonitor | awk '{print $1}' ) -n ibm-data-cataloging) | grep Fetch
  2. Locate the array that contains the metrics about the connection type that you want to debug.
    An example array for NFS connections.
    'file-scan-topic': {0: [30768, 30768, 0], 1: [31050, 31050, 0], 2: [30897, 30897, 0], 3: [30788, 30788, 0], 4 : [30608, 30608, 0], 5: [30525, 30525, 0], 6: [30908, 30908, 0], 7: [30610, 30610, 0], 8: [30726, 30726, 0], 9 : [30959, 30959, 0]}
    Note: Each sub-array 0: [30768, 30768, 0] indicates the metrics that are related to one of the consumers. In this sub-array, the first value represents the current committed offset. The second value is the log end offset, and the third value represents the lag.
    It is important to consider the following points when you read the log information:
    • The SD monitor provides a condensed metric of the Kafka messages that are received by each one of the consumers.

      For example, if you run two different NFS scans, the expectation is to get the metrics of all the Kafka messages sent by the two scans.

    • The SD monitor runs every 30 seconds, which means that you can see a small lag between the progress of the scans and the metrics they are reported in the SD monitor log.