Sample Prometheus queries to monitor the load balancer

The CICS TX load balancer and the statistics of load distribution can be monitored by using Prometheus. The following samples show the use of Prometheus queries for monitoring. The metrics are returned in JSON format from Prometheus.

The samples query the following information:
The samples assume the following:
  • Prometheus metrics are available at the end point http://10.10.10.10:9090/api/v1
  • The front-end pod name is 'cicslb-txtestwlm-ibm-cic-1’
  • The load-balancing region (LBR) on the front-end pod is ‘LBRREGN’
  • The back-end pod’s primary name is ‘txtestwlm-ibm-cics-tx-on’
  • The maximum number of back-end pod replicas is set as ‘4’
  • The application-owning region (AOR) on the back-end pod is ‘TXREGION’

Health of the AOR on one back-end pod as seen by the LBR

The back-end pod's primary name is ‘txtestwlm-ibm-cics-tx-on-0’ .

The Prometheus query is: 

http://10.10.10.10:9090/api/v1/query?query=cics_LBR_health{pod="cicslb-txtestwlm-ibm-cic-1",
backend="txtestwlm-ibm-cics-tx-on-0"}
.
The output is shown in Figure 1.
Figure 1. Prometheus query
Prometheus query

Health of the AOR on all back-end pod replicas as seen by the LBR

The primary names of the back-end pods are ‘txtestwlm-ibm-cics-tx-on-0’ to ‘txtestwlm-ibm-cics-tx-on-3’.

The Prometheus query is: 
http://10.10.10.10:9090/api/v1/query?query=cics_LBR_health{pod="cicslb-txtestwlm-ibm-cic-1",
backend=~"txtestwlm-ibm-cics-tx-on-.*"}
The output is: 

{"status":"success",
"data":
  {"resultType":"vector",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_health","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562927547.666,"2"]},
	{"metric":{"__name__":"cics_LBR_health","backend":"txtestwlm-ibm-cics-tx-on-1","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562927547.666,"2"]},
	{"metric":{"__name__":"cics_LBR_health","backend":"txtestwlm-ibm-cics-tx-on-2","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562927547.666,"2"]},
	{"metric":{"__name__":"cics_LBR_health","backend":"txtestwlm-ibm-cics-tx-on-3","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562927547.666,"2"]}
   ]
  }
}

Number of transactions routed to the AOR on all back-end pod replicas

The Prometheus query is: 

http://10.10.10.10:9090/api/v1/query?query=cics_LBR_transactions{pod="cicslb-txtestwlm-ibm-cic-1",
backend=~"txtestwlm-ibm-cics-tx-on-.*"}
The output is: 

{"status":"success",
 "data":
  {"resultType":"vector",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928123.978,"1001"]},
	{"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-1","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928123.978,"989"]},
	{"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-2","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928123.978,"959"]},
	{"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-3","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928123.978,"970"]}
   ]
  }
}

Number of transactions routed to an AOR on a single back-end pod

The primary name of the back-end pod is ‘txtestwlm-ibm-cics-tx-on-0’.

The Prometheus query is: 
http://10.10.10.10:9090/api/v1/query?query=cics_LBR_transactions%7Bpod=%22cicslb-txtestwlm-ibm-cic-1%22,%20backend=%22txtestwlm-ibm-cics-tx-on-0%22%7D
The output is: 

{"status":"success",
 "data":
  {"resultType":"vector",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928502.598,"1010"]}
   ]
  }
}

Number of transactions routed to the AOR on a single back-end pod in the last 2 minutes

The primary name of the back-end pod is ‘txtestwlm-ibm-cics-tx-on-0’.

The Prometheus query is: 

http://10.10.10.10:9090/api/v1/query?query=cics_LBR_transactions%7Bpod=%22cicslb-txtestwlm-ibm-cic-1%22,%20backend=%22txtestwlm-ibm-cics-tx-on-0%22%7D%5B2m%5D
The output is: 

{"status":"success",
 "data":
  {"resultType":"matrix",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},
	 "values":
	  [
	   [1562928633.171,"55"],
	   [1562928663.171,"51"],
	   [1562928693.171,"46"],
	   [1562928723.171,"43"]
	  ]
	 }
    ]
  }
}

Number of abends from the AOR on a back-end pod in the last 2 minutes

The primary name of the back-end pod is ‘txtestwlm-ibm-cics-tx-on-0’.

The Prometheus query is: 

http://10.10.10.10:9090/api/v1/query?query=cics_LBR_abends%7Bpod=%22cicslb-txtestwlm-ibm-cic-1%22,%20backend=%22txtestwlm-ibm-cics-tx-on-0%22%7D%5B2m%5D
The output is: 

{"status":"success",
 "data":
  {"resultType":"matrix",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_abends","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},
	 "values":
	 [
	  [1562928963.171,"2"],
	  [1562928993.171,"1"],
	  [1562929023.171,"0"],
	  [1562929053.171,"2"]
	 ]
	}
  ]
 }
}

Weights assigned by the LBR to the AOR on all back-end pod replicas

The Prometheus query is: 

http://10.10.10.10:9090/api/v1/query?query=cics_LBR_weight%7Bpod=%22cicslb-txtestwlm-ibm-cic-1%22%7D
The output is: 

{"status":"success",
 "data":
  {"resultType":"vector",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_weight","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562930435.901,"99991"]},
	{"metric":{"__name__":"cics_LBR_weight","backend":"txtestwlm-ibm-cics-tx-on-1","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562930435.901,"99992"]},
	{"metric":{"__name__":"cics_LBR_weight","backend":"txtestwlm-ibm-cics-tx-on-2","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562930435.901,"99992"]},
	{"metric":{"__name__":"cics_LBR_weight","backend":"txtestwlm-ibm-cics-tx-on-3","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562930435.901,"99992"]}
   ]
  }
}