Monitoring the data ingestion

While the ingestion script is running, you can monitor the progress from the CCSIngest Kafka topic by running a Kafka console consumer on the topic.

Procedure

  1. Log in to the Kafka server.
  2. Run the following command to list the Docker containers: 
    docker ps
  3. Open a terminal window for the Kafka docker container: 
    docker exec -it CONTAINERID bash
  4. Go to the /opt/kafkfa directory.
  5. Run the following command to run the Kafka consumer: 
    bin/kafka-console-consumer.sh --bootstrap-server servername:port --topic CCSIngest --from-beginning

    The resulting message format contains the following parameters:

    • date: Current date
    • protocol: The format of data, such as FIX
    • stream_id: The source from where data is coming, such as NETS or Raptor
    • records: The number of records that got ingested
    • tRecords: The total number of records to be ingested
    • status: The status of the ingestion, such as Ingesting and DataReady