Prometheus setup

Required components

• IBM z16 LPAR

• RHEL 8.5

• Prometheus package

• Web browser

• Disk storage

Architecture

The following figure shows an installed instance of Prometheus on an LPAR.

The installation steps are as follows:

1. Create a user/group for Prometheus:

 # groupadd --system prometheus
 # useradd -s /sbin/nologin --system -g prometheus prometheus

Verification:

 # grep prometheus /etc/passwd
 prometheus:x:987:987::/home/prometheus:/sbin/nologin

2. Create the directories for the product manually:

 # mkdir -p /etc/prometheus/rules
 # mkdir /etc/prometheus/rules.d
 # mkdir /etc/prometheus/files_sd

Verification:

 # find /etc/prometheus -exec ls -l {} \;
 total 12
 drwxr-xr-x. 2 root root 4096 Apr 11 14:01 files_sd
 drwxr-xr-x. 2 root root 4096 Apr 11 14:01 rules
 drwxr-xr-x. 2 root root 4096 Apr 11 14:01 rules.d

3. Download and compile the package from Prometheus Source Repository. Alternatively, check your IT policy to see if you can download the pre-compiled package for the "s390x" architecture from Prometheus GitHub Repository. The pre-compiled package is used for this use case.

Verification:

 # ll
 total 74340
 -rw-r--r--. 1 root root 76121535 Apr 11 12:39 prometheus-x.x.x.linux-s390x.tar.gz

4. Unpack the product:

 # tar xvf prometheus-x.x.x.linux-s390x.tar.gz

Output:

 prometheus-x.x.x.linux-s390x/
 prometheus-x.x.x.linux-s390x/consoles/
 prometheus-x.x.x.linux-s390x/consoles/index.html.example
 prometheus-x.x.x.linux-s390x/consoles/node-cpu.html
 prometheus-x.x.x.linux-s390x/consoles/node-disk.html
 prometheus-x.x.x.linux-s390x/consoles/node-overview.html
 prometheus-x.x.x.linux-s390x/consoles/node.html
 prometheus-x.x.x.linux-s390x/consoles/prometheus-overview.html
 prometheus-x.x.x.linux-s390x/consoles/prometheus.html
 prometheus-x.x.x.linux-s390x/console_libraries/
 prometheus-x.x.x.linux-s390x/console_libraries/menu.lib
 prometheus-x.x.x.linux-s390x/console_libraries/prom.lib
 prometheus-x.x.x.linux-s390x/prometheus.yml
 prometheus-x.x.x.linux-s390x/LICENSE
 prometheus-x.x.x.linux-s390x/NOTICE
 prometheus-x.x.x.linux-s390x/prometheus
 prometheus-x.x.x.linux-s390x/promtool

5. Copy the files prometheus and promtool from the current directory to the system local directory:

 # cp prometheus promtool /usr/local/bin/

Verification:

 # ll /usr/local/bin
 total 200660
 -rwxr-xr-x. 1 root root 106771616 Apr 11 14:51 prometheus
 -rwxr-xr-x. 1 root root  98698218 Apr 11 14:51 promtool

6. Copy the prometheus.yml file. If necessary, modify it to fit your requirements.

 # cp prometheus.yml /etc/prometheus/
 # vi /etc/prometheus/prometheus.yml

 global:
   scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
   evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
   # scrape_timeout is set to the global default (10s).
   # external systems (federation, remote storage, Alertmanager).
   external_labels:
   monitor: 'codelab-monitor'
   scrape_configs:
   # The job name is added as a label `job=&ltjob_name>` to any timeseries scraped from this 
   config.
   - job_name: "prometheus"
   # metrics_path defaults to '/metrics'
   # scheme defaults to 'http'.
   # Override the global default and scrape targets from this job every 5 seconds.
   scrape_interval: 5s
   static_configs:
   targets: ["localhost:9090"]

The following figure is a sample screenshot of prometheus.yml from the Prometheus user interface:

7. If necessary modify the service file to fit your requirements:

 # vi /etc/systemd/system/prometheus.service:
 [Unit]
 Description=Prometheus
 Documentation=https://prometheus.io/docs/introduction/overview/
 Wants=network-online.target
 After=network-online.target

8. Start the Prometheus service:

 # systemctl daemon-reload
 # systemctl start prometheus
 # systemctl enable prometheus
 Created symlink 
  /etc/systemd/system/multi-user.target.wants/prometheus.service/etc/systemd/system/prometheus.service.

Verification:

Check status using systemctl status prometheus command:

 # systemctl status prometheus.service
   prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-04-11 17:01:42 CEST; 2min 32s ago
   Docs: https://prometheus.io/docs/introduction/overview/
   Main PID: 1936575 (prometheus)
   Tasks: 39 (limit: 126455)
   Memory: 38.6M
   CGroup: /system.slice/prometheus.service
        └─1936575 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/>

9. Modify firewall rules to allow access to the web service from any local workstation.

 # firewall-cmd --permanent --add-rich-rule 'rule family="ipv4"  source address="192.168.122.0/24" port protocol="tcp" port="9090" accept'
 success

 # firewall-cmd --add-port=9090/tcp --permanent

 # firewall-cmd --reload
 success

10. Use the IP/Hostname of the Prometheus server and the port 9090 from a Web browser on your work station to access Prometheus.

The next sections guide you through the steps to complete the setup on the LPARs:

• Node Exporter setup

• Performance Co-Pilot (PCP) setup

• Grafana setup