SNMP-based status commands issues

Edit online

This section describes solutions for problems that can cause SNMP-based status commands such as clstat, cldump, and cldisp to fail.

The Simple Network Management Protocol (SNMP) provides access to a database of status and configuration variables that are referred as the Management Information Base (MIB). The SNMP subsystem included with base AIX® provides a subset of the overall MIB, and can also work with peer daemons that provide access to other portions of the MIB. The PowerHA® SystemMirror® cluster manager daemon acts as such a peer and provides access to the PowerHA SystemMirror specific variables in the MIB.

When you experience problems with SNMP or the utilities that rely on it, you must first verify that the basic SNMP configuration is functioning, then proceed to check the SystemMirror specific function.

You can check for the basic function of SNMP by using the snmpinfo command. Use the snmpinfo -m dump command to display the default part of the MIB. If this command does not produce any output, there is a problem with the base setup of SNMP and the snmpd subsystem itself. Check to ensure that the snmpd subsystem is running and follow the steps in the following sections to make sure that the basic snmpinfo command is working.

After you verify the basic function, you can query the PowerHA SystemMirror specific portion of the MIB with the following command:

snmpinfo -m dump -v -o /usr/sbin/cluster/hacmp.defs risc6000clsmuxpd

If the preceding command does not produce an output (and snmpinfo -m dump does), the problem is specific to the SystemMirror portion of the MIB.

Complete the steps in the Troubleshooting common SNMP problems section to resolve the following common issues with the snmpdv3.conf file that is included with the AIX operating system:

Access to the internet portion of the SNMP Management Information Base (MIB) is commented out.
In PowerHA SystemMirror Version 7.1.2, there is no COMMUNITY entry for the IPv6 loopback address.

However, even after these two issues are fixed, other issues might still interfere with the proper working of the SNMP-based status commands. Complete the steps in the Troubleshooting SNMP status commands section to resolve these issues. If the status commands still fail, complete the steps in the Troubleshooting snmpdv3.conf file section to resolve the rest of the issues.

Troubleshooting common SNMP problems
Troubleshooting SNMP status commands
Troubleshooting snmpdv3.con file

Troubleshooting common SNMP problems

Check for access permission to the PowerHA SystemMirror portion of the SNMP Management Information Base (MIB) in the SNMP configuration file. Find the defaultView entries in the /etc/snmpdv3.conf file:
```
# grep defaultView /etc/snmpdv3.conf 
#VACM_VIEW defaultView       internet                   - included -
VACM_VIEW defaultView        1.3.6.1.4.1.2.2.1.1.1.0    - included -
VACM_VIEW defaultView        1.3.6.1.4.1.2.6.191.1.6    - included -
VACM_VIEW defaultView        snmpModules                - excluded -
VACM_VIEW defaultView        1.3.6.1.6.3.1.1.4          - included -
VACM_VIEW defaultView        1.3.6.1.6.3.1.1.5          - included -
VACM_VIEW defaultView        1.3.6.1.4.1.2.6.191        - excluded -
VACM_ACCESS  group1 - - noAuthNoPriv SNMPv1  defaultView - defaultView -
```
Beginning with AIX 7.1, as a security precaution, the snmpdv3.conf file is included with the internet access commented out. The preceding example shows the unmodified configuration file: the internet descriptor is commented out, which means that there is no access to most of the MIB, including the PowerHA SystemMirror information. (Other included entries provide access to other limited parts of the MIB.) By default in AIX 7.1 and later, the PowerHA SystemMirror SNMP-based status commands do not work, unless you edit the snmpdv3.conf file. There are two ways to provide access to the PowerHA SystemMirror MIB:
- Uncomment the following internet line in the snmpdv3.conf file to access the entire MIB:
```
VACM_VIEW defaultView  internet  -  included   -
```
- If you do not want access to the entire MIB, add the following line to the snmpdv3.conf file, which gives you access to the PowerHA SystemMirror MIB only:
```
VACM_VIEW defaultView   risc6000clsmuxpd   -   included   -
```
Note: After you edit the SNMP configuration file, you must stop and restart snmpd, and then refresh the cluster manager, by using the following commands:
```
stopsrc -s snmpd
startsrc -s snmpd
refresh -s clstrmgrES
```
Try the SNMP-based status commands again. If the commands work, you do not need to go through the rest of the section.
If you use PowerHA SystemMirror Version 7.1.2 or later, check for the correct IPv6 entries in the configuration files for clinfoES and snmpd. In PowerHA SystemMirror 7.1.2, an entry is added to the /usr/es/sbin/cluster/etc/clhosts file to support IPv6. However, the required corresponding entry is not added to the /etc/snmpdv3.conf file, which causes intermittent problems with the clstat command. To resolve the issue complete one of the following steps:
- If you do not plan to use IPv6, comment the line in the /usr/es/sbin/cluster/etc/clhosts file and restart clinfoES, by using the following commands:
```
# ::1     # PowerHA SystemMirror
stopsrc -s clinfoES
startsrc -s clinfoES
```
  Try the SNMP-based status commands again. If the commands work, you do not need to go through the rest of the section.
- If you plan to use IPv6 in the future, add the following line to the /snmpdv3.conf file:
```
COMMUNITY public    public     noAuthNoPriv ::  0       -
```
  If you are using a different community (other than public), substitute the name of that community for the word public.
Note: After you edit the SNMP configuration file, you must stop and restart snmpd, and then refresh the cluster manager, by using the following commands:
```
stopsrc -s snmpd
startsrc -s snmpd
refresh -s clstrmgrES
```
Try the SNMP-based status commands again. If the commands work, you do not need to go through the next section.

Troubleshooting SNMP status commands

Run the following command to check whether snmpd is running:
```
lssrc -s snmpd
```
Run the following command to check whether cluster services are running:
```
lssrc -ls clstrmgrES | grep state (looking for a state of ST_STABLE)
```
If not, start the cluster services. None of the SNMP status commands work if the cluster services are not running.
If you are using the clstat command, check if the /usr/es/sbin/cluster/etc/clhosts file is correct. The clhosts file must contain a list of IP addresses of the PowerHA SystemMirror nodes with which the clinfoES daemon can communicate. (Persistent addresses are preferred. If the file contains addresses that do not belong to a cluster node, it might cause further problems.) If you edit the file on a system, you must restart clinfoES on that system.
Cluster node
- By default, the clhosts file is populated with the local host address. You can add entries for all the nodes in the cluster so that the clstat command works while the cluster services are running on the node.
- Beginning with PowerHA SystemMirror Version 7.1.2, an entry for the IPv6 loopback address is added to the default clhosts file. As described in the Troubleshooting common SNMP problems section, you can either comment this line or add a line for the IPv6 loopback address to the SNMP configuration file.
Client system

By default the clhosts file is empty. You must add addresses for the cluster nodes.
If you are using the clstat command, run the following command to check whether clinfoES is running:
```
lssrc -s clinfoES
```
If not, run the following command to start it:
```
startsrc -s clinfoES
```
Tip: Start clinfoES every time you start cluster services to avoid this issue.

Check whether snmpd is listening at the smux port and if the cluster manager is connected. Run the following netstat command to list active sockets that use the smux port:

# netstat -Aa | grep smux
f1000e0002988bb8 tcp 0  *.smux *.* LISTEN
f1000e00029d8bb8 tcp4 0 0 loopback.smux loopback.32776 ESTABLISHED
f1000e00029d4bb8 tcp4 0 0 loopback.32776 loopback.smux ESTABLISHED
f1000e000323fbb8 tcp4 0 0 loopback.smux loopback.34266 ESTABLISHED
f1000e0001b86bb8 tcp4 0 0 loopback.34266 loopback.smux ESTABLISHED

If you do not see a socket in the LISTEN state, use the following commands to stop and start snmpd:

stopsrc -s snmpd; startsrc -s snmpd

After you have an smux socket in the LISTEN state, look for a socket pair in the ESTABLISHED state, with one of the sockets owned by the cluster manager. You can use the rmsock command to find which process owns the sockets. If you just restarted snmpd, ensure that there is a LISTEN socket at the smux port. If you do not see any smux socket in the ESTABLISHED state, you can either refresh the cluster manager (refresh -s clstrmgrES), or you can wait for a couple of minutes. Then, try the netstat -Aa command again. The cluster manager tries to connect to snmpd when services are started and then every few minutes after the services are started. The refresh command causes the cluster manager to try to connect to snmpd immediately. Do not use stopsrc and startsrc on the cluster manager.
Use rmsock to find the owners of the smux sockets in the ESTABLISHED state. Use the first field in the netstat output, which is the memory address of the socket, as an argument to rmsock. For example,
```
# rmsock f1000e00029d4bb8 tcpcb
The socket 0xf1000e00029d4808 is being held by proccess 4063356 (muxatmd).
# rmsock f1000e0001b86bb8 tcpcb
The socket 0xf1000e0001b86808 is being held by proccess 18546850 (clstrmgr).
```
In this example, there are two ESTABLISHED socket pairs. One between snmpd and muxatmd and one between snmpd and the cluster manager.
Try the SNMP-based status commands again. If the commands work, you do not need to go through the next section.

Troubleshooting snmpdv3.con file

Determine which version of snmpd is running, by using the following command:
```
#  ls -l /usr/sbin/snmpd
lrwxrwxrwx    1 root system 9 May 14 22:19 /usr/sbin/snmpd -> snmpdv3ne
```
snmpdv1 uses the /etc/snmpd.conf file and snmpdv3 uses the /etc/snmpdv3.conf file.
Note: In the rest of these instructions, it is assumed that snmpdv3 daemon, which is the default version, is running.
Check authentication and access control (authorization) settings for snmpdv3.conf file. clinfoES, cldump, and cldisp use community-based authentication. They use the first community that is listed in the configuration file. Although rare, it is possible to specify the community to clinfoES. To check this setting, use the following command:
```
odmget SRCsubsys | grep -p clinfo
```
Look for the value of the cmdargs field.
- If the field is empty, clinfoES uses the first COMMUNITY entry in the configuration file.
- If the field is set to -c community_name, clinfoES uses community_name.
  Note: If you want to change the community that is used by clinfoES, use the chssys command. After you change the community that is used by clinfoES, you must restart clinfoES.
Find the first SNMP community in the snmpdv3.conf file.
```
# grep -i comm /etc/snmpdv3.conf | grep -v ^#
COMMUNITY powerha powerha noAuthNoPriv 0.0.0.0   0.0.0.0         -
COMMUNITY test test noAuthNoPriv 0.0.0.0     0.0.0.0         -
```
In this example, the first community is powerha.
- If there are no uncommented community entries, you must add an entry in the snmpdv3.conf file. You can use these entries as a template. Use any text string as the community name (although public is not considered a good choice because it is common). The community name must be the second and third fields in the line.
- For the changes to take effect, you must restart snmpd after you edit the file.
The snmpdv3 daemon uses view-based access control model (VACM) for access control. Find the VACM_GROUP, the VACM_ACCESS, and the VACM_VIEW entries that are associated with the community you are using.
1. Find the group that is associated with the first community. Search in the configuration file for the community name. For example,
```
# grep powerha /etc/snmpdv3.conf
VACM_GROUP group1 SNMPv1  powerha  -
TARGET_PARAMETERS trapparms1 SNMPv1  SNMPv1  powerha  noAuthNoPriv -
COMMUNITY powerha    powerha     noAuthNoPriv 0.0.0.0   0.0.0.0         -
```
  In this example the VACM_GROUP is group1.
2. Find the view that is associated with this group by searching for the group you identified. The view is listed in a VACM_ACCESS entry.
```
# grep group1 /etc/snmpdv3.conf 
VACM_GROUP group1 SNMPv1  powerha  -
VACM_ACCESS  group1 - - noAuthNoPriv SNMPv1  defaultView - defaultView -
```
  The syntax of a VACM_ACCESS entry is as follows:
```
VACM_ACCESS groupName contextPrefix contextMatch securityLevel 
securityModel readView writeView notifyView storageType
```
  Look for the name of the view for readView access. In this example, defaultView is used for readView and notifyView access for group group1. No access is provided for writeView and storageType.
3. Find the VACM_VIEW entries that are associated with this community by searching for the view you identified:
```
# grep defaultView /etc/snmpdv3.conf 
#VACM_VIEW defaultView       internet                   - included -
VACM_VIEW defaultView        1.3.6.1.4.1.2.2.1.1.1.0    - included -
VACM_VIEW defaultView        1.3.6.1.4.1.2.6.191.1.6    - included -
VACM_VIEW defaultView        snmpModules                - excluded -
VACM_VIEW defaultView        1.3.6.1.6.3.1.1.4          - included -
VACM_VIEW defaultView        1.3.6.1.6.3.1.1.5          - included -
VACM_VIEW defaultView        1.3.6.1.4.1.2.6.191        - excluded -
VACM_ACCESS  group1 - - noAuthNoPriv SNMPv1  defaultView - defaultView -
```
  1. Look for a VACM_VIEW entry that gives access to the PowerHA SystemMirror MIB. Locations in the MIB are identified either by a string of numbers (object identifier (OID)) or by a name (object descriptor). In this example, the first entry uses the object descriptor internet. That corresponds to the OID 1.3.6.1. If this line is uncommented, it allows access to the entire MIB, that is 1.3.6.1 and everything that starts with 1.3.6.1, which is effectively the entire SNMP MIB.
  2. However, in this example, the internet descriptor is commented out, which means that there is no access at that level. Beginning with AIX 7.1, as a security precaution, the snmpdv3.conf file is included with the internet access commented out. By default in AIX 7.1 or later, the PowerHA SystemMirror SNMP-based status commands do not work, unless you edit the snmpdv3.conf file. Also, ensure that the relevant VACM_VIEW entry has the word included in the second last field and not excluded.
  3. As described in the Troubleshooting common SNMP problems section, there are two ways to provide access to the PowerHA SystemMirror MIB:
    - Uncomment the internet line in snmpdv3.conf file to access the entire MIB.
    - Add a line that provides access to the PowerHA SystemMirror MIB only. The PowerHA SystemMirror MIB can be identified by the object descriptor or by the OID.
Edit the snmpdv3.conf file to ensure that the PowerHA SystemMirror MIB is accessible for the first community. You must make sure that the first COMMUNITY entry in the file maps to a VACM_GROUP entry that maps to a VACM_ACCESS entry that maps to a VACM_VIEW that includes the PowerHA SystemMirror MIB. In this example, the only change that is needed is to add a VACM_VIEW entry for the risc6000clsmuxpd object descriptor:
```
VACM_VIEW defaultView     risc6000clsmuxpd     - included -
```
If you edited the snnmpdv3.conf file, restart snmpd.
Note: You must use the stopsrc and startsrc commands, instead of the refresh command for snmpd.
```
stopsrc -s snmpd; startsrc -s snmpd
```
Repeat steps 5, 6, 7 as described in the Troubleshooting SNMP status commands section to ensure that the cluster manager is connected to snmpd.
Try the SNMP-based status commands again.