Sample Prometheus Queries

IBM® CICS® TX on Cloud load balancer can be monitored by using Prometheus. As also the statistics of load distribution. Following are few samples of Prometheus Queries for monitoring. The metrics are returned in JSON format from Prometheus.

Assuming

Prometheus metrics are available at end point http://10.10.10.10:9090/api/v1
Frontend pod name is ‘cicslb-txtestwlm-ibm-cic-1’
LBR on frontend pod is ‘LBRREGN’
Backend pod’s primary name is ‘txtestwlm-ibm-cics-tx-on’
Maximum number of backend pod replicas set as ‘4’
AOR on backend pod is ‘TXREGION’

Health of AOR on backend pod ‘txtestwlm-ibm-cics-tx-on-0’ as seen by LBR.

Prometheus Query: http://10.10.10.10:9090/api/v1/query?query=cics_LBR_health{pod="cicslb-txtestwlm-ibm-cic-1", backend="txtestwlm-ibm-cics-tx-on-0"}.

Output

Figure 1. Prometheus query

Health of AOR on all backend pod replicas as seen by LBR. That is, health of AOR on ‘txtestwlm-ibm-cics-tx-on-0’ to ‘txtestwlm-ibm-cics-tx-on-3’

Prometheus Query: http://10.10.10.10:9090/api/v1/query?query=cics_LBR_health{pod="cicslb-txtestwlm-ibm-cic-1", backend=~"txtestwlm-ibm-cics-tx-on-.*"}.

Output

{"status":"success",
"data":
  {"resultType":"vector",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_health","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562927547.666,"2"]},
	{"metric":{"__name__":"cics_LBR_health","backend":"txtestwlm-ibm-cics-tx-on-1","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562927547.666,"2"]},
	{"metric":{"__name__":"cics_LBR_health","backend":"txtestwlm-ibm-cics-tx-on-2","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562927547.666,"2"]},
	{"metric":{"__name__":"cics_LBR_health","backend":"txtestwlm-ibm-cics-tx-on-3","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562927547.666,"2"]}
   ]
  }
}

Number of transactions routed to AOR on all backend pod replicas

Prometheus Query: http://10.10.10.10:9090/api/v1/query?query=cics_LBR_transactions{pod="cicslb-txtestwlm-ibm-cic-1", backend=~"txtestwlm-ibm-cics-tx-on-.*"}.

Output

{"status":"success",
 "data":
  {"resultType":"vector",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928123.978,"1001"]},
	{"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-1","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928123.978,"989"]},
	{"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-2","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928123.978,"959"]},
	{"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-3","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928123.978,"970"]}
   ]
  }
}

Number of transactions routed to AOR on a single backend pod ‘txtestwlm-ibm-cics-tx-on-0’

Prometheus Query: http://10.10.10.10:9090/api/v1/query?query=cics_LBR_transactions%7Bpod=%22cicslb-txtestwlm-ibm-cic-1%22,%20backend=%22txtestwlm-ibm-cics-tx-on-0%22%7D.

Output

{"status":"success",
 "data":
  {"resultType":"vector",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562928502.598,"1010"]}
   ]
  }
}

Number of transactions routed to AOR on a single backend pod ‘txtestwlm-ibm-cics-tx-on-0’ in last 2 minutes

Prometheus Query: http://10.10.10.10:9090/api/v1/query?query=cics_LBR_transactions%7Bpod=%22cicslb-txtestwlm-ibm-cic-1%22,%20backend=%22txtestwlm-ibm-cics-tx-on-0%22%7D%5B2m%5D.

Output

{"status":"success",
 "data":
  {"resultType":"matrix",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_transactions","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},
	 "values":
	  [
	   [1562928633.171,"55"],
	   [1562928663.171,"51"],
	   [1562928693.171,"46"],
	   [1562928723.171,"43"]
	  ]
	 }
    ]
  }
}

Number of abends from AOR on backend pod txtestwlm-ibm-cics-tx-on-0 in last 2 minutes

Prometheus Query: http://10.10.10.10:9090/api/v1/query?query=cics_LBR_abends%7Bpod=%22cicslb-txtestwlm-ibm-cic-1%22,%20backend=%22txtestwlm-ibm-cics-tx-on-0%22%7D%5B2m%5D.

Output

{"status":"success",
 "data":
  {"resultType":"matrix",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_abends","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},
	 "values":
	 [
	  [1562928963.171,"2"],
	  [1562928993.171,"1"],
	  [1562929023.171,"0"],
	  [1562929053.171,"2"]
	 ]
	}
  ]
 }
}

Weights assigned by LBR to AOR on all backend pod replicas

Prometheus Query: http://10.10.10.10:9090/api/v1/query?query=cics_LBR_weight%7Bpod=%22cicslb-txtestwlm-ibm-cic-1%22%7D.

Output

{"status":"success",
 "data":
  {"resultType":"vector",
   "result":
   [
    {"metric":{"__name__":"cics_LBR_weight","backend":"txtestwlm-ibm-cics-tx-on-0","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562930435.901,"99991"]},
	{"metric":{"__name__":"cics_LBR_weight","backend":"txtestwlm-ibm-cics-tx-on-1","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562930435.901,"99992"]},
	{"metric":{"__name__":"cics_LBR_weight","backend":"txtestwlm-ibm-cics-tx-on-2","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562930435.901,"99992"]},
	{"metric":{"__name__":"cics_LBR_weight","backend":"txtestwlm-ibm-cics-tx-on-3","endpoint":"cicsmetrics","instance":"10.1.93.12:9087","job":"txtestwlm-cicsapploadbalancer-metrics","namespace":"monitoring","pod":"cicslb-txtestwlm-ibm-cic-1","region":"TXREGION","service":"txtestwlm-cicsapploadbalancer-metrics"},"value":[1562930435.901,"99992"]}
   ]
  }
}