Running Prometheus queries from the OpenShift Console
Run Prometheus queries from the
OpenShift
Console to visualize metric
data.
- Who needs to complete this task?
- Cluster administrator A cluster administrator must perform this task.
- How frequently should you perform this task?
- Repeat as needed You should run Prometheus queries as often as necessary to monitor your IBM® Software Hub deployments. It is recommended that you perform this task at least once per day or once per shift.
Ensure that you enable monitoring for user-defined projects and configure the OpenShift Monitoring stack. For more information about how to complete these tasks, see the documentation in the following table:
| OpenShift Version | Resources |
|---|---|
| Version 4.12 | |
| Version 4.14 | |
| Version 4.15 | |
| Version 4.16 | |
| Version 4.17 | |
| Version 4.18 |
To run the following Prometheus queries, go to in the OpenShift Console.
Cluster resource usage
You can run the following Prometheus
queries to monitor your cluster resource usage:
- CPU usage
- Displays how much CPU each node is using over a 5-minute interval in
seconds.
avg by (instance, nodename)(irate(node_cpu_seconds_total{mode!="idle"}[5m])) * 100 * on (instance) group_left (nodename) node_uname_info - Memory usage
- Displays the percentage of available memory that each node is
using.
100 * ((node_memory_MemTotal_bytes -(node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes))/node_memory_MemTotal_bytes) * on (instance) group_left (nodename) node_uname_info - Network input and output
- Displays the rate of network input and output for each node over a 5-minute interval in
KB.
avg by (instance) ((irate(node_network_receive_bytes_total[5m]) + irate(node_network_transmit_bytes_total[5m])) ) * on (instance) group_left (nodename) node_uname_info / 10^3 - OpenShift API call status
- Displays the number of each type of HTTP response that the OpenShift API server answered with in the last 30
minutes.
sum by (code)(rate(apiserver_request_total{verb=~"POST|PUT|DELETE|PATCH|GET|LIST|WATCH"}[30m])) - Pods using the most memory
- Displays the 10 pods that are using the most memory.
In the following query, replace
{PROJECT_CPD_INST_OPERANDS}with the name of the project where the IBM Software Hub control plane and services are installed:topk(10, max(container_memory_working_set_bytes{namespace="{PROJECT_CPD_INST_OPERANDS}",container!="",pod!=""}) by (pod) ) / 10^9
Disk speed and usage
You can run the following Prometheus
queries to monitor your disk speed and usage:
- Disk input and output
- Displays the rate of disk input and output for each node over a 5-minute
interval.
avg by (instance) (irate(node_disk_io_time_seconds_total[5m])/1000) * on (instance) group_left (nodename) node_uname_info - Disk write speed
- Displays the disk write speed for each node over the last two minutes in
MB.
sum by (instance) (irate(node_disk_written_bytes_total[2m])) / 1024 / 1024 - Free disk space
- Displays the percentage of free disk space available for each
node.
node_filesystem_free_bytes{mountpoint ="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100
IBM Software Hub resource usage
You can run the following Prometheus
queries to monitor your IBM Software Hub resource usage:
- CPU cores used per IBM Software Hub component
- Percentage of CPU used compared to limit
- Percentage of memory used compared to limit
Important: In the queries, replace
{PROJECT_CPD_INST_OPERANDS} with the name of the project where the IBM Software Hub control plane and services are
installed.- CPU cores used per IBM Software Hub component
- Displays the number of CPU cores used by each IBM Software Hub
component.
sort_desc(sum(max(kube_pod_labels{namespace="{PROJECT_CPD_INST_OPERANDS}", label_icpdsupport_add_on_id!="" }) by (label_icpdsupport_add_on_id,pod) * on(pod) group_right(label_icpdsupport_add_on_id)max(kube_pod_container_resource_limits{resource="cpu",unit="core",namespace="{PROJECT_CPD_INST_OPERANDS}"}) by (pod)) by (label_icpdsupport_add_on_id)) - Percentage of CPU used compared to limit
- Displays the 25 highest percentages of CPU used by IBM Software Hub compared to the limit of CPU that your OpenShift container is configured
for.
topk(25, sort_desc(100*sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{namespace="{PROJECT_CPD_INST_OPERANDS}"}) by (pod) / sum(kube_pod_container_resource_limits{resource="cpu",unit="core",namespace="{PROJECT_CPD_INST_OPERANDS}"}) by (pod))) - Percentage of memory used compared to limit
- Displays the 25 highest percentages of memory used by IBM Software Hub compared to the limit of memory that your
OpenShift container is configured
for.
topk(25, sort_desc(100 * sum(container_memory_usage_bytes{namespace="{PROJECT_CPD_INST_OPERANDS}",container!=""}) by (pod) / sum(kube_pod_container_resource_limits{resource="memory",unit="byte",namespace="{PROJECT_CPD_INST_OPERANDS}"}) by (pod)))