Displaying PMI metrics in Prometheus format with the metrics app

You can use the metrics.ear file to create a Prometheus endpoint for your WebSphere® Application Server runtimes to display PMI metrics in Prometheus format.

The metrics.ear performs two operations:

Retrieves the PMI data objects by using the JMX Perf MBean
Renders the data from the PMI data objects into Prometheus format output.

Before you begin

Similar to the PerfServlet, the metrics.ear provides a way to use HTTP requests to query the performance metrics for an entire WebSphere Application Server administrative domain. In contrast to the PerfServlet, which returns PMI data in XML format, the metrics.ear converts PMI data into Prometheus format. The metrics.ear enables the scraping of metrics from your application servers into Prometheus format.

The metrics available on the Prometheus endpoint correspond to the set of metrics enabled in the PMI configuration. For the Prometheus output, some PMI metrics are suppressed or split into two metrics to better follow Prometheus best practices. See Prometheus metrics for a mapping of the original PMI metrics to their corresponding Prometheus metrics.

When the metrics.ear application is running on a WebSphere Application Server (base) server, cell, node, and server labels are not added to metrics. To include these labels, set the following system property to true to include the cell, node, and server name labels in the Prometheus output.

com.ibm.ws.pmi.prometheus.includeCellNodeServerLabels=true

Tuning performance

[8.5.5.20 or later] With a WebSphere Application Server (base) server, the metrics.ear endpoint contacts the server to gather metrics. If the server is CPU-bound, or slow to respond for other reasons, response time of the metrics endpoint is adversely affected. Monitor the response time of metrics endpoint requests to determine whether tuning is needed.

The metrics.ear endpoint response time scales linearly with the number of metrics available at the endpoint. If response time is too slow, reduce the number of metrics that are collected by adjusting the PMI configuration.

The only metrics that are displayed are the ones from the JVM where the metrics.ear application is installed.

PMI settings: Enable only the PMI metrics that are relevant for your business needs. Review the PMI settings and use a custom setting to enable or disable metrics. If possible, avoid the use of the All metrics setting. For servers that do not require metrics collections, set the PMI settings to disable.
URL filtering: You can use the metrics endpoint to query for metrics from a single WebSphere Application Server (base) server. The default endpoint /metrics shows the PMI metrics that are collected from the server. To select a server, use URL /metrics/<node_name>/<server_name>.
Prometheus scrape_duration: The default Prometheus scrape_duration is 15 seconds. If the response time for your Prometheus endpoint is a few seconds, increase the Prometheus scrape_duration value. Alternatively, you can scale down the number of PMI metrics available at the endpoint by modifying the PMI settings or by using URL filtering.
Result caching: The metrics app stores the most recent Prometheus metrics result in a cache for 5 seconds by default. A request that is made within 5 seconds of the previous request is served with the cached result. This default interval time value is configurable with the com.ibm.ws.pmi.prometheus.resultCacheInterval system property.
Servers list refresh: A single entry server list for a WebSphere Application Server (base) server to be scraped for PMI data is refreshed when the metrics endpoint is accessed. To reduce the cost, the metrics app does not refresh this list more often than every 600 seconds by default. This default scrape interval value can be configured with the com.ibm.ws.pmi.prometheus.serverListUpdateInterval system property.
Server metrics scrape response time: For metrics scraping, when Prometheus calls the /metrics endpoint from the metrics.ear application, it makes JMX calls to the server to collect metrics. If the WebSphere Application Server (base) server is slow to respond, the /metrics endpoint response time might be large and Prometheus times out with no response, according to the Prometheus scrape_timeout configuration setting. The default timeout that is set in the metrics.ear application when it communicates with the servers in the cell is 8 seconds. After this timeout is reached, it returns a response back to Prometheus, even if some server data is omitted from the response. This configuration limits the Prometheus endpoint response time. You can configure the default server scrape timeout value with the following system property: com.ibm.ws.pmi.prometheus.serverScrapeTimeout. If the value is set to 0 or a negative value, no timeout value is set for the server scrapes.

Performance improvement recommendations

As you create a Prometheus endpoint for your WebSphere Application Server runtimes to display PMI metrics in Prometheus format, consider these recommendations.

Response time is proportional to the number of metrics returned. You can turn off some of the more verbose metrics, such as URI and EJB metrics, to help improve performance.
Long scrape times occur when CPU usage of the application server is near 100%. When you use Prometheus with Grafana to consume metrics, gaps can occur in your Grafana graphs when scrape time exceeds the Prometheus scrape timeout.

Procedure

Install the metrics.ear file that is located in the app_server_root/installableApps directory, where app_server_root is the installation path for WebSphere Application Server.
1. Deploy metrics.ear on a single WebSphere Application Server instance within the domain.
  For unfederated base servers, deploy the metrics.ear on each server from which you want to collect metrics.
2. After the metrics application deploys, access it with the http://hostname:port/metrics URL.
  Important: The port number is required. Make sure that the port number specified is the port for the JVM where the metrics.ear application is installed.
Enable application security so that only users assigned the monitor role can access the endpoint. You can configure the role assignment to the user in the All Applications > metrics.ear > Security role to user/group mapping page of the administrative console.

After the WebSphere Application Server user is assigned to the monitor role, configure the prometheus yaml to include the login information of the WebSphere Application Server user that has the monitor role.

- job_name: 'was-base'
	     basic_auth:
             username: someUser
             password: somePassword
           metrics_path: /metrics
           static_configs:
          - targets: ['localhost:9080']