Retrieving platform state information with SNMP

This task describes an example of retrieving Cloud Pak for Data System state via SNMP AgentX extension using snmpget, snmptable, and snmpwalk applications. The MIB file dedicated for the platform manager defines SNMP tables, mirroring the information the user can get running such CLI commands as, for example, ap hw, or ap sw.

1. The SNMP sub-agent (delegated by SNMP engine to provide the data) requires snmpd.service to be running. Verify that the snmpd.service is up and running on Cloud Pak for Data System:

   systemctl status snmpd
   

   [root@e1n1 ~]# systemctl status snmpd
   ● snmpd.service - Simple Network Management Protocol (SNMP) Daemon.
      Loaded: loaded (/usr/lib/systemd/system/snmpd.service; enabled; vendor preset: disabled)
      Active: active (running) since Mon 2021-08-16 03:04:44 CEST; 10h ago
    Main PID: 17783 (snmpd)
       Tasks: 1
      Memory: 32.0K
      CGroup: /system.slice/snmpd.service
              └─17783 /usr/sbin/snmpd -LS0-6d -f
   
   Aug 16 03:04:43 e1n1 systemd[1]: Starting Simple Network Management Protocol (SNMP) Daemon....
   Aug 16 03:04:43 e1n1 snmpd[17783]: Duplicate IPv4 address detected, some interfaces may not be visible in IP-MIB
   Aug 16 03:04:44 e1n1 snmpd[17783]: Turning on AgentX master support.
   Aug 16 03:04:44 e1n1 snmpd[17783]: NET-SNMP version 5.7.2
   Aug 16 03:04:44 e1n1 systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon..
   Aug 16 11:06:43 e1n1 snmpd[17783]: IfIndex of an interface changed. Such interfaces will appear multiple times in IF-MIB.
   [root@e1n1 ~]# 

   If the service is running, you can send a request which would be accepted and processed by snmpd.service.

   Note: It's not advisable to use SNMPv1 or SNMPv2c protocols, because the community string used for authentication traverses the network in plain text. Instead, define custom user record using SNMPv3 Usrbased-Security-Model (USM). Only this approach can keep credentials used to poll data secure. Also, super user privileges are needed in order to adjust snmpd.service configuration. In the following examples SNMPv2 and community string are used because all requests are addressed to the local host, so there is no risk of community string leak. Moreover, the command which is used in order to grab rocommunity string can only be run with super user privilege.
2. Verify that the sub-agent is running:

   systemctl status magneto-snmp-agent

   [root@e1n1 ~]# systemctl status magneto-snmp-agent
   ● magneto-snmp-agent.service - SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service
      Loaded: loaded (/usr/lib/systemd/system/magneto-snmp-agent.service; enabled; vendor preset: disabled)
      Active: active (running) since Mon 2021-08-16 11:40:57 CEST; 2h 0min ago
    Main PID: 78845 (python)
       Tasks: 1
      Memory: 33.5M
      CGroup: /system.slice/magneto-snmp-agent.service
              └─78845 /usr/bin/python /opt/ibm/appliance/platform/management/snmp/magneto_snmp_agent.py
   
   Aug 16 11:40:57 e1n1 systemd[1]: Started SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service.
   [root@e1n1 ~]# 

   If it is not running, run:

   apsnmpagent on

   [root@e1n1 ~]# systemctl status magneto-snmp-agent
   ● magneto-snmp-agent.service - SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service
      Loaded: loaded (/usr/lib/systemd/system/magneto-snmp-agent.service; disabled; vendor preset: disabled)
      Active: inactive (dead)
   
   Aug 16 11:40:46 e1n1 systemd[1]: Stopped SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service.
   Aug 16 11:40:46 e1n1 systemd[1]: Started SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service.
   Aug 16 11:40:47 e1n1 systemd[1]: magneto-snmp-agent.service: main process exited, code=exited, status=1/FAILURE
   Aug 16 11:40:47 e1n1 systemd[1]: Unit magneto-snmp-agent.service entered failed state.
   Aug 16 11:40:47 e1n1 systemd[1]: magneto-snmp-agent.service failed.
   Aug 16 11:40:57 e1n1 systemd[1]: magneto-snmp-agent.service holdoff time over, scheduling restart.
   Aug 16 11:40:57 e1n1 systemd[1]: Stopped SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service.
   Aug 16 11:40:57 e1n1 systemd[1]: Started SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service.
   Aug 16 13:42:45 e1n1 systemd[1]: Stopping SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service...
   Aug 16 13:42:45 e1n1 systemd[1]: Stopped SNMP Subagent for IIAS-MIB/IBM-GT-MIB as a Service.
   [root@e1n1 ~]# apsnmpagent on 
   /usr/bin/apsnmpagent: action `systemctl enable snmpd` succeeded on node e1n3.fbond
   /usr/bin/apsnmpagent: action `systemctl start snmpd` succeeded on node e1n3.fbond
   /usr/bin/apsnmpagent: action `systemctl enable magneto-snmp-agent` succeeded on node e1n3.fbond
   /usr/bin/apsnmpagent: action `systemctl start magneto-snmp-agent` succeeded on node e1n3.fbond
   /usr/bin/apsnmpagent: action `systemctl enable snmpd` succeeded on node e1n2.fbond
   /usr/bin/apsnmpagent: action `systemctl start snmpd` succeeded on node e1n2.fbond
   /usr/bin/apsnmpagent: action `systemctl enable magneto-snmp-agent` succeeded on node e1n2.fbond
   /usr/bin/apsnmpagent: action `systemctl start magneto-snmp-agent` succeeded on node e1n2.fbond
   /usr/bin/apsnmpagent: action `systemctl enable snmpd` succeeded on node e1n1.fbond
   /usr/bin/apsnmpagent: action `systemctl start snmpd` succeeded on node e1n1.fbond
   /usr/bin/apsnmpagent: action `systemctl enable magneto-snmp-agent` succeeded on node e1n1.fbond
   /usr/bin/apsnmpagent: action `systemctl start magneto-snmp-agent` succeeded on node e1n1.fbond
   snmpd.service state is consistent on accessible nodes. Current state is 'enabled'.
   magneto-snmp-agent.service state is consistent on accessible nodes. Current state is 'enabled'.
   [root@e1n1 ~]# 

3. Verify the state of snmpd.service and magneto-snmp-agent.service by running:

   apsnmpagent state

   [root@e1n1 ~]# apsnmpagent state
   snmpd.service state is consistent on accessible nodes. Current state is 'enabled'.
   magneto-snmp-agent.service state is consistent on accessible nodes. Current state is 'enabled'.
   [root@e1n1 ~]# 

   Now the system is ready to provide its state via SNMP.
4. Examine the nodes state:

   [root@e1n1 ~]# COMMUNITY="$(grep -m 1 rocommunity /etc/snmp/snmpd.conf | awk '{ print $2 }')"; snmptable -v2c -c$COMMUNITY -Cil localhost IBM-GT-MIB::nodesTable 
   SNMP table: IBM-GT-MIB::nodesTable
   
   index               nodeName         nodeState nodePersonality nodeIsMonitored nodeIsHub nodeIsMaster 
   "enclosure1"."node1"enclosure1.node1 ENABLED   CONTROL         YES             YES       YES          
   "enclosure1"."node2"enclosure1.node2 ENABLED   CONTROL         YES             NO        NO           
   "enclosure1"."node3"enclosure1.node3 ENABLED   CONTROL         YES             NO        NO           
   "enclosure1"."node4"enclosure1.node4 ENABLED   WORKER          YES             NO        NO           
   "enclosure2"."node1"enclosure2.node1 ENABLED   WORKER          YES             NO        NO           
   "enclosure2"."node2"enclosure2.node2 ENABLED   WORKER          YES             NO        NO           
   "enclosure2"."node3"enclosure2.node3 ENABLED   UNSET           YES             NO        NO           
   "enclosure2"."node4"enclosure2.node4 ENABLED   UNSET           YES             NO        NO           
   [root@e1n1 ~]# ap node
   +------------------+---------+-------------+-----------+-----------+
   | Node             |   State | Personality | Monitored | Is Master |
   +------------------+---------+-------------+-----------+-----------+
   | enclosure1.node1 | ENABLED | CONTROL     |       YES |       YES |
   | enclosure1.node2 | ENABLED | CONTROL     |       YES |        NO |
   | enclosure1.node3 | ENABLED | CONTROL     |       YES |        NO |
   | enclosure1.node4 | ENABLED | WORKER      |       YES |        NO |
   | enclosure2.node1 | ENABLED | WORKER      |       YES |        NO |
   | enclosure2.node2 | ENABLED | WORKER      |       YES |        NO |
   | enclosure2.node3 | ENABLED | UNSET       |       YES |        NO |
   | enclosure2.node4 | ENABLED | UNSET       |       YES |        NO |
   +------------------+---------+-------------+-----------+-----------+
   
   Generated: 2021-08-16 13:49:38
   
   [root@e1n1 ~]# 

5. You can also request a particular node's state as follows:

   [root@e1n1 ~]# COMMUNITY="$(grep -m 1 rocommunity /etc/snmp/snmpd.conf | awk '{ print $2 }')"; snmpget -v2c -c$COMMUNITY localhost IBM-GT-MIB::nodePersonality.\"enclosure1\".\"node4\"
   IBM-GT-MIB::nodePersonality."enclosure1"."node4" = STRING: WORKER
   [root@e1n1 ~]#

6. With the following command, you can get the list of software inventory elements whose state is not OK or DISABLED:

   [root@e1n1 ~]# COMMUNITY="$(grep -m 1 rocommunity /etc/snmp/snmpd.conf | awk '{ print $2 }')"; snmpwalk -v2c -c$COMMUNITY localhost IBM-GT-MIB::swStatus | grep -v "OK\|DISABLED"$
   IBM-GT-MIB::swStatus."icp4d"."" = STRING: WARNING
   IBM-GT-MIB::swStatus."rpcbind"."enclosure1.node4" = STRING: STOPPED
   IBM-GT-MIB::swStatus."rpcbind"."enclosure2.node1" = STRING: STOPPED
   IBM-GT-MIB::swStatus."rpcbind"."enclosure2.node2" = STRING: STOPPED
   IBM-GT-MIB::swStatus."rpcbind"."enclosure2.node3" = STRING: STOPPED
   IBM-GT-MIB::swStatus."rpcbind"."enclosure2.node4" = STRING: STOPPED
   IBM-GT-MIB::swStatus."firewalld"."enclosure1.node1" = STRING: FAILED
   IBM-GT-MIB::swStatus."mdraid.md0"."enclosure1.node4" = STRING: WARNING
   IBM-GT-MIB::swStatus."apglobalusrmgmt"."enclosure1.node2" = STRING: STOPPED
   IBM-GT-MIB::swStatus."apglobalusrmgmt"."enclosure1.node3" = STRING: STOPPED
   IBM-GT-MIB::swStatus."container.callhome"."enclosure1.node2" = STRING: STOPPED
   IBM-GT-MIB::swStatus."container.callhome"."enclosure1.node3" = STRING: STOPPED
   [root@e1n1 ~]#