Application metrics

Edit online
Monitor the following metrics to analyze OpenSearch health.
  • Cluster health
  • Number of shards
  • GC monitoring
Note: The threshold values, configurations, and severities that are mentioned throughout this section are the guidelines that IBM suggests for an optimal performance of OpenSearch. You can modify these threshold values or define actions based on your operational requirements.

For details about how to generate thread dump and heap dump, see Troubleshooting: Monitoring OpenSearch.

If the metrics return an exceeded threshold value, consider the severity as mentioned and perform the recommended possible actions to identify and debug the problem. Contact IBM for further support.

OpenSearch cluster health

To ensure optimal health and performance of OpenSearch, it is recommended to monitor the OpenSearch cluster health regularly.

Table 1. OpenSearch cluster health details
Command Description
curl -X GET http://localhost:9240/_cluster​/health?pretty This command retrieves OpenSearch cluster health status.
$.status This JSON path expression retrieves the cluster health status from the response.
$.number_of_nodes This JSON path expression retrieves the number of nodes in the cluster from the response.

The response JSON of the health check request displays a status field in the response. The status can have the values green, yellow or red. The cluster health status is displayed based on the following color codes:

Table 2. Cluster health status
Status Description
green Indicates that the cluster is in a healthy state. When OpenSearch is handling huge data, it takes some time to display the cluster health status.
yellow Indicates that the cluster is not in a healthy state. Identify the cause and rectify it. During this time, OpenSearch processes the requests for the index that is available. If there are unassigned shards, then identify the unassigned shards, check the reason for the unallocation and resolve the issue.
  • Run the following command to retrieve the list of unassigned shards. 
    curl -X GET “http://localhost:9240/_cat/
shards?h=index,shard,primaryOrReplica,state,docs,store,ip,​node,segments.count,unassigned​.at,unassigned.details,unassigned.for,​unassigned.reason,help,s=index&v”
  • Run the following command to check the unallocated reason for specific shards. 
    curl -X GET "http://localhost:9240/_cluster/allocation/
explain" -d ‘{ "index" :"index name","primary" : "true|false","shard": "shardnumber"}’reason,help,s=index&v”
red Indicates that OpenSearch nodes are down or not reachable or the OpenSearch master is not discovered. If the number of nodes does not match the number of OpenSearch nodes configured, identify the node that did not join the cluster and identify the root cause for the node to not join the cluster. Based on the root cause, identify if your OpenSearch is down. If your OpenSearch is down and not reachable, check the connectivity.
A sample HTTP response is as follows: 

{
"cluster_name": "apidatastore_cluster",
"status": "green",
"timed_out": false,
"number_of_nodes": 3,
"number_of_data_nodes": 3,
"active_primary_shards": 101,
"active_shards": 202,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
The overall cluster status is green since all OpenSearch nodes work as expected.

Number of shards

To ensure proper allocation of shards to nodes, it is recommended to monitor the number of shards regularly.

Table 3. Shard details
Command Description
curl -X GET "http://localhost:9240/_cluster​/health?pretty" This command retrieves the number of shards on OpenSearch.

If the total number of active shards from the response exceeds the heap space * nodes * 20 count, then increase the heap space of OpenSearch nodes or add a new OpenSearch node.

OpenSearch considers a maximum of 20 active shards per GB of heap space as healthy. Perform any of the following actions to maintain the total number of active shards:

  • Scale up theOpenSearch node.
  • If you are not able to scale up the OpenSearch node, then increase the heap size as the last option. The heap space should not be more than half of system memory (RAM). For example, if the system memory is 16 GB, you can allocate a maximum of 8 GB for OpenSearch.

To increase the heap space, modify the parameters Xms2g and Xmx2g in the jvm.options file located at InstallDir\config.

Garbage collection (GC) monitoring

The GC metric provides the GC run-time in seconds. You must check GC run-time once every five minutes. The average GC run-time should not exceed one second.

Table 4. GC metric details
Metric Description
  • opensearch_jvm_gc_​collection_seconds_sum
  • opensearch_jvm_gc_​collection_seconds_count

The quotient of both the metrics gives the GC run time.

If the quotient is more than 1 second, it implies that GC is taking longer time to run and this slows down OpenSearch request processing. You must collect the logs and get the mapping of API index and transaction index.