Installing the sample analytics pipeline

If you want to set up a dual-server configuration, complete the following steps on both the collection server and the analytics server, unless the step explicitly indicates to do the step on only one of the servers. If you want to set up a single-server configuration, complete the following steps on a single server.

1. Copy the base/tpfrtmc/bin/tpf_sample_analytics_pipeline.tar.gz file in binary format from your z/TPF source repository to the home directory on your Linux machine. Enter the following command to extract the content from the tar file:
   tar -xf tpf_sample_analytics_pipeline.tar.gz
2. Default credentials are specified in the tpf_data_sci/tpf_default_credentials.text file. These credentials are used in various scripts that are provided with the sample analytics pipeline. Change the user name and password to values that are more secure for your environment. When you change passwords, you must make updates to various files in the tpf_data_sci/Docker directory.

   There are many components in use in the sample analytics pipeline. The component versions that are indicated were stable at the time of release. To use the latest versions of these components, modify the version numbers that are specified in the tpf_data_sci/user_files/tpf_prepare_configurations.yml and tpf_data_sci/user_files/tpf_zrtmc_analyzer_files/requirements.txt files with the latest version.

3. For the collection server or a single-server configuration, copy the base/tpfrtmc/bin/tpfrtmc.tar.gz file in binary format from your z/TPF source repository to the tpf_data_sci/Docker/tpf_rtmc_docker_files/ directory. Enter the following command to extract the content from the tar file:
   tar -xf tpfrtmc.tar.gz
4. For the analytics server or a single-server configuration, copy the base/tpfrtmc/bin/tpf_zmatc_analyzer.tar.gz file in binary format from your z/TPF source repository to the tpf_data_sci/Docker/tpf_zmatc_analyzer_docker_files/ directory. Enter the following command to extract the content from the tar file:
   tar -xf tpf_zmatc_analyzer.tar.gz
5. Define your Apache Kafka hosts, encryption settings, topic settings, and programmatic variables in the tpf_data_sci/user_files/kafka_hosts.yml file. For more information about how to configure this file, see the comments in the file.
6. If Python 3.8 and the pyyaml library, which are used by the tpf_prepare_configurations.sh script, are not installed on your system, enter the following commands to install them:
   1. sudo yum install python38
   2. sudo python3 -m pip install --upgrade pyyaml
7. Change your directory to the Docker directory. Enter the following command:
   cd tpf_data_sci/Docker
8. For the collection server, analytics server, or single-server configurations, you must prepare your configuration.
   This configuration determines if the server will use MariaDB or MySQL, Linux on an x86 system or Linux on IBM Z, use trusted dependency repositories, and more.
   1. Define your settings in the tpf_data_sci/user_files/tpf_prepare_configurations.yml file.
   2. Enter the following command to configure your server:
      ./tpf_prepare_configurations.sh

      To view which files are edited and what changes are made to achieve your desired settings, see the tpf_prepare_configurations.sh script.

9. Enter the docker-compose command to start the Docker containers.
   • For the collection server or single-server configurations, take one of the following actions:
     • If you are using a MySQL database, enter the following command:

       docker-compose --file tpf-insights-dashboard-network.yml --file tpf_mysql.yml --file tpf_kafka.yml up -d --build

     • If you are using a MariaDB database, enter the following command:

       docker-compose --file tpf-insights-dashboard-network.yml --file tpf_mariadb.yml --file tpf_kafka.yml up -d --build

       Note: For Kafka configurations on Linux on IBM Z, if you need to rebuild the Kafka container, first remove all files and folders in the tpf_data_sci/Docker/tpf_kafka_docker_files/volumes/kafka-logs directory by issuing the following command:

       rm -rf tpf_data_sci/Docker/tpf_kafka_docker_files/volumes/kafka-logs/\*

       Otherwise, you might receive the following error from the Kafka broker when the tpf-kafka-broker container starts:

       The Cluster ID jw3FiOddStufuL211VzUjQ doesn't match stored clusterId.

   • For the analytics server, take one of the following actions:
     • If you are using a MySQL database, enter the following command:

       docker-compose --file tpf-insights-dashboard-network.yml --file tpf_mysql.yml up -d --build

     • If you are using a MariaDB database, enter the following command:

       docker-compose --file tpf-insights-dashboard-network.yml --file tpf_mariadb.yml up-d --build

   For more information about managing containers and images, see the Docker documentation.
10. Set up the database tables and stored procedures by running the SQL script. For the collection server, analytics server, and single-server configurations, enter the following command:
    ./tpf_setup_db.sh
11. Run the following script for the collection server or single-server configurations:
    ./tpf_create_kafka_topics.sh

    This script creates the Apache Kafka topics.

12. Run the following script for the collection server or single-server configurations:
    ./tpf_modify_kafka_topics.sh hostname:port

    where hostname:port is the hostname and port that is specified in the tpf_data_sci/user_files/kafka_hosts.yml file from step 5.

    This script modifies the Apache Kafka topics based on the modify_script_variables settings that are specified for your host in the tpf_data_sci/user_files/kafka_hosts.yml file.

13. Enter the docker-compose command to start the tpfrtmc Docker containers. For the collection server or single-server configurations, enter the following command:
    docker-compose --file tpf-insights-dashboard-network.yml --file tpf_collection_server.yml up -d --build
14. Optional: Configure the ZRTMC analyzer instances to support multiple z/TPF systems.
15. Enter the docker-compose command to start the remaining Docker containers. Enter the following command for the analytics server or single-server configurations:
    docker-compose --file tpf-insights-dashboard-network.yml --file tpf_analytics_server.yml up -d --build
    Note: The ZRTMC analyzer connects to both Apache Kafka and the database upon startup. Any data that is available on the configured Apache Kafka topics will start being processed. The ZMATC analyzer performs analysis on all available message analysis tool results in the database on the analytics server.
16. Optional: If you have an active firewall, ensure that the ports specified in the YAML (.yml) files are open. For example:
    1. Enter the following command for each port that is exposed by the YAML files:
       sudo firewall-cmd --zone=public --add-port=portID/tcp --permanent
       where portID represents the following ports:

       • For MariaDB or MySQL: 3306
       • For Grafana: 3000
       • For Apache Kafka: 2181, 9092, 9093, 8082, 8000
       • For tpfrtmc: 9090
    2. Reload the firewall by entering the following command:
       sudo firewall-cmd --reload

       You can modify all ports before entering the reload command. Additionally, you can use the tpf_data_sci/Docker/tpf_open_firewall_ports.sh script to process all of these commands for the default ports.

17. Optional: If you plan to process tapes created by the name-value pair collection process with the ZCNVP command, enter the docker-compose command to start the tpfrtmc Docker container. For the collection server or single-server configurations, enter the following command:
    docker-compose --file tpf-insights-dashboard-network.yml --file tpf_zcnvp_tpfrtmc.yml up -d --build