Event not in the CSM/MS /var/log/csm/syslog.fabric.notices file

Use this procedure if an expected event is not in the Cluster Systems Management/Management Server (CSM/MS) /var/log/csm/syslog.fabric.notices log file.

1. Log on to the CSM/MS.
2. Verify that you can ping the source, which can be either the fabric management service VLAN IP address for the server or the switch.
   Note: If you cannot ping the source device, then you must use standard network debug techniques to isolate the problem on the service VLAN. Consider, the CSM/MS connection, the fabric management server connection, the switch connection, and any Ethernet devices on the network. Also, ensure that the addressing has been set up correctly.
3. Select from the following options:
   • If you are using CSM on Linux, continue with step 4.
   • If you are using CSM on AIX, continue with step 5.
4. If you are using CSM on Linux®, check the configuration of the AppArmor application with the syslog-ng command to ensure that the /var/log/csm/syslog.fabric.notices wr, variable is in the /etc/apparmor.d/sbin.syslog-ng file.
   • If the /var/log/csm/syslog.fabric.notices wr, variable is in the log file, continue with step 5.
   • If the /var/log/csm/syslog.fabric.notices wr, variable is not in the log file, perform the following steps:
     1. Add the line /var/log/csm/syslog.fabric.notices wr, to the /etc/apparmor.d/sbin.syslog-ng file before the right bracket (}). You must put the comma at the end of the line.
     2. Restart the AppArmor application by using the /etc/init.d/boot.apparmor restart file.
     3. Restart the syslog-ng command by using the /etc/init.d/syslog restart file.
     4. If this action resolves the problem, this ends the procedure. Otherwise, continue with step 5.
5. Check the syslog configuration file and verify that the following entry is in there.
   1. If the CSM/MS is running AIX®, it is using syslog (not syslog-ng) and the following line should be in the /etc/syslog.conf file. If /etc/syslog.conf does not exist, go to step 4b. Otherwise, after finishing this step, go to step 6.

      # all local6 notice and above priorities go to the following file
      local6.notice  /var/log/csm/syslog.fabric.notices

   2. If the CSM/MS is running SLES 10 SP1 or higher, it is using syslog-ng and the following lines will be in the /etc/syslog-ng/syslog-ng.conf file:
      Note: The actual destination and log entries might vary slightly if they were set up by using the monerrorlog command. Because it is a generic CSM command being used for InfiniBand, the monerrorlog command uses a different name from fabnotices_fifo in the destination and log entries. It is a pseudo random name that is like fifonfJGQsBw.

      filter f_fabnotices        { facility(local6) and level(notice, alert, warn,
          err, crit)  and not filter(f_iptables); };
      destination fabnotices_fifo { pipe("/var/log/csm/syslog.fabric.notices"
          group(root) perm(0644)); };
      log { source(src); filter(f_fabnotices); destination(fabnotices_fifo); };
      
       The following udp and tcp definitions are in the src stanza.
      
      udp(ip("0.0.0.0") port(514));
      tcp(ip("0.0.0.0") port(514));

      Note: If the fabric management server is only using User Datagram Protocol (UDP) as the transfer protocol for log entries, then the Transmission Control Protocol (TCP) line is not needed. Step 7 indicates how to check this. In either case, make note of the protocols and ports and IP addresses in these lines. Using the 0.0.0.0 address accepts logs from any address. If you want more security, you might have a line for each switch and fabric management server from which you want to receive logs. If you have a specific address named, ensure that the source of the log has an entry with its address. Switches use UDP, fabric management servers are configurable for TCP or UDP. In any case, ensure that the UDP line is always used.
6. If the entries are not there, perform the procedure in Reconfiguring the CSM event management. If the procedure fixes the problem, this ends the procedure. If the entries are there, go to the next step.
7. Look at the log on the device that is logging the problem and make sure that it is there.
   1. For the fabric management server, look at the /var/log/messages file
   2. For switches, log on to the switch and look at the log. If necessary use the switch command-line help, or the switch Users Guide for instructions.
8. If the setup on the CSM/MS is valid and the log entry is in the log for the source, check to see that the source is set up for remote logging:
   1. For a fabric management server running the syslog command (not the syslog-ng command), check the /etc/syslog/syslog.conf file for the following line:
      Note: If the /etc/syslog.conf file does not exist, go to step 7c.

      local6.*   @CSM/MS IPp-address

      Note: If you make a change, restart the syslogd command by using the /etc/init.d/syslog restart command.
   2. Continue with step 9.
   3. For a fabric management server running the syslog-ng command, check the /etc/syslog-ng/syslog-ng.conf file for the following lines: Ensure that the destination definition uses the same protocol and port as is expected on the CSM/MS; the definition shown here is UDP on port 514. The CSM/MS information should have been noted in step 4. The standard syslogd uses UDP.

      filter f_fabinfo        { facility(local6) and level(info, notice, alert, warn,
          err, crit)  and not filter(f_iptables); };
          destination fabinfo_csm { udp("[CSM/MS IP-address]" port(514)); };
      log { source(src); filter(f_fabinfo); destination(fabinfo_csm); };

      Note: If you make a change, you will have to restart the syslogd command by using the /etc/init.d/syslog restart file.
   4. For a switch, check that it is configured to log to the CSM/MS by using the logSyslogConfig command on the switch command line. Check that the following information is correct. If it is not, update it by using:

      logSyslogConfig –h [host] –p 514 –f 22 –m 1

      • The CSM/MS is the host IP address.
      • The port is 514 (or another one you have chosen to use).
      • The facility is local6.
9. If the problem persists, then try restarting the syslogd command on the CSM/MS and also resetting the source's logging:
   1. Log on to the CSM/MS.
   2. For AIX CSM, run the refresh -s syslogd command.
   3. For Linux CSM, run the /etc/init.d/syslog restart command.
   4. If the source is Subnet Manger running on a fabric management server, log on to the fabric management server and run the /etc/init.d/syslog restart command.
   5. If the source is a switch, reboot the switch by using the instructions in the Switch Users Guide (by using the reboot command on the switch CLI), or the Fast Fabric Users Guide (by using the ibtest command on the fabric management server).
10. If the problem has not been fixed, contact your next level of support