IBM Support

Syslog / Syslogd in PASE for IBM i

Troubleshooting


Problem

This document provides basic information about syslog and syslogd as it pertains to PASE on IBM i OS.

Environment

IBM i OS; Portable Application Solutions Environment (PASE)

Resolving The Problem

Syslog is a very popular reporting system that runs on many devices and OSes. It uses various parts and programs to encode, transmit, consolidate, and analyze messages from a wide range of devices. Everything from Windows and UNIX to firewalls and IoT devices participate in sending and centralizing messages from across the enterprise.  That is, except for IBM i.  Of course, we always must be different (usually for good reasons).  However, when a security team wants to have an enterprise-wide view of events with a tool such as a SIEM (Security Information and Event Management), we sometimes feel left out.  Until now!

With the release of IBM i 7.3 Db2 for i PTF Group level 7 and IBM i 7.2 Db2 for i PTF Group level 19 on October 27, 2017, IBM i now has methods to generate Syslog formatted messages for both the Audit Journal (QAUDJRN) and the History Log (QHST).  With the original PTF release, U (User Generated) Audit Journal Code entries were not being converted.  Beginning in SF99703 level 22 or SF99704 level 10 IBM i Db2 Group PTFs, all U (User Generated) Audit Journal Code entries are now converted.

image 7296

image 7317

With the assistance of Db2 for i Table Functions (DISPLAY_JOURNAL & HISTORY_LOG_INFO), these sources of information can be translated from their native IBM i format to that of either RFC3164 (older) or RFC5424 (newer, preferred) formats. Refer to the following URLs to find out more about these Db2 for i Table Functions and the generated syslog history and audit journal data.

DISPLAY_JOURNAL Table Function
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/rzajq/rzajqudfdisplayjournal.htm

HISTORY_LOG_INFO Table Function
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/rzajq/rzajqudfhistoryloginfo.htm

Each Syslog message contains a header and an event message. The header contains information such as the priority (the lower the number, the more urgent or severe), date, time, and originating system.  Keep in mind most Syslog messages are sent from all end points to one, centralized repository (SIEM) for analysis and notification. The event message itself uses another format called CEF, or Common Event Format.  This message contains the OS level, source (Audit Journal or History Log), message name (Journal Code and Type or Message ID), severity, and detailed key-value pairs. These key-value pairs are items such as “msg=User BADUSR name not valid” or “sproc=123456/QTCP/QTFTP00001” (where sproc means source process—aka IBM i three-part job name).

Examples:

  • RFC5424
    • <11>1 2017-01-01T01:02:03.123456-06:00 MYSYSTEM.MYDOMAIN.COM - - - - CEF:0|IBM|IBM i|7.3|QSYS-AUDJRN|T=PW|Low|reason=Invalid password msg=User BADUSR name not valid suser=QTCP sproc=123456/QTCP/QTFTP00001 shost=SYSTEMX src=1.2.3.4 spt=12345
  • RFC3164
    • <18> Jan 01 01:02:03 MYSYSTEM CEF"0|IBM|IBM i|7.3|QSYS-QHST|CPC3723|Low|reason=CPC3723 msg=1 objects saved from library MYLIB. 0 objects not included. suser=MYUSER sproc=123456/QTCP/QTFTP00001

By default, the IBM i operating system does not log system operation messages (History or Audit Journal Entries) using the PASE logger or syslog API. 5250 green screen, ftp, IBM Navigator for i, and other native IBM i connections will not be logged to the syslogd Daemon. IBM i object authorization errors will not be logged to the syslogd Daemon either. The IBM i OS logs these types of messages using facilities such as the History Log and the QAUDJRN security audit journal. Some programs ported to IBM i (such as OpenSSH) will log messages to both the syslogd Daemon and an IBM i facility.

However, IBM Lab Services does provide the IBM i Syslog Reporting Manager to fill this gap.  Please see the following URL for more information.

If you want to try generating these messages yourself, first make sure you have the IBM i Db2 Group PTF level listed above applied to your system. Then, from any SQL Interface (have you tried the latest Run SQL Scripts inside IBM i Access Client Solutions yet?), run a statement such as:

  • Select entries from the audit journal that return syslog informiton and format them with an RFC5424 header:
    • SELECT syslog_facility, syslog_severity, syslog_event FROM TABLE (QSYS2.DISPLAY_JOURNAL('QSYS','QAUDJRN',GENERATE_SYSLOG => 'RFC5424')) AS X WHERE syslog_event IS NOT NULL;
  • Return syslog information formatted with an RFC5424 header for all history log messages from the start of today forward into the future:
    • SELECT syslog_facility, syslog_severity, syslog_event FROM TABLE (QSYS2.HISTORY_LOG_INFO(START_TIME => CURRENT DATE,GENERATE_SYSLOG => 'RFC5424')) AS X'
  • Note that SYSLOG_EVENT field comes back in a format that is not consumable in ILE apps.  The default format is CCSID 1200.  As a result, you will need to cast as CCSID 37 to work with this data:
    • SELECT syslog_facility, syslog_severity, cast(syslog_event as varchar(2048) CCSID 37) FROM TABLE (QSYS2.DISPLAY_JOURNAL('QSYS','QAUDJRN',GENERATE_SYSLOG => 'RFC5424')) AS X WHERE syslog_event IS NOT NULL;
Now, not every Audit Journal entry translates to a Syslog message. Hence, we added the WHERE clause to only get those entries that generate a Syslog event. You should see one row per message in your Audit Journal. Use the information above to try to understand the message. Keep in mind it is the centralized Syslog repositories job to read these messages, not you, so don’t feel bad if you don’t understand it.
Getting information from the History Log is just as simple, but uses a different Db2 Table Function. For the History Log, all messages are translated to Syslog events. Again, using the WHERE clause, we can limit what entries from the History Log are converted.
 
Finally, now that we have the information from IBM i in the proper Syslog format, we need to send those messages to our centralized Syslog repository. Your security team should be able to give you the information about what system to send the messages to. You can use the built-in PASE command “logger” to send a message or you can also use the syslog() or syslog_r() APIs in PASE to send messages to the syslogd daemon, which will forward the message to the log or SEIM system, i.e. IBM QRadar, of your choosing.


The PASE syslog client API is used on UNIX systems to log messages regarding system operation. Messages sent using the syslog client API are received and processed by a syslogd daemon running in the background on a local system, such as the IBM i OS. The syslogd daemon routes the messages to a destination described by a configuration file. Messages can be logged locally or routed to another system across the network.

The syslog client API and the associated syslogd server daemon are provided on IBM i OS using the PASE for i runtime installed with 5770SS1 Option 33. To get started with syslog messaging, the syslogd daemon needs to be manually configured and started first before it can process messages from syslog API clients.

The syslog API is described in the AIX Information Center at the following URL:
https://www.ibm.com/support/knowledgecenter/ssw_aix_72/s_bostechref/syslog.html

The configuration of the syslogd daemon is described in the AIX Information Center at the following URL:
https://www.ibm.com/support/knowledgecenter/ssw_aix_72/s_commands/syslogd.html 

One deviation from the above AIX documentation is that the syslogd configuration file on IBM i is named /QOpenSys/etc/syslog.conf, versus /etc/syslog.conf and must be manually created with *LF end of line (EOL) only (it does not exist by default).

How to configure syslogd

1) We need to create both a syslog log file, and the syslog config file:
  • STRQSH
  • touch -C 819 /home/syslog.log   (Creates the log file)
You can choose the log file name and location. This log file will need to be referenced in the syslog configuration file during the next configuration steps.
  • touch -C 819 /QOpenSys/etc/syslog.conf   (Creates conf file in proper CCSID 819)
  • echo "" >> /QOpenSys/etc/syslog.conf  (This will set the *LF stream file EOL option, rather than *CRLF, which is not compatible, and would result in errno=2 when reading)
2) Edit the /QOpenSys/etc/syslog.conf file to add your specific configuration and inform the syslogd daemon where to send a system message, depending on the message's priority level and the facility that generated it.
3) Start the syslogd daemon in batch.
  • SBMJOB CMD(STRQSH CMD('/QOpenSys/usr/sbin/syslogd')) JOB(SYSLOGD) JOBQ(QSYSNOMAX)
  • To verify the syslogd daemon is started, you can...
    1. Look at the date changed attribute on the /QOpenSys/etc/syslog.pid file.
    2. Execute the Qshell command, "ps -ef | grep syslogd".
  • If you encounter an error with syslogd daemon, you can locate the mean of the errno in the /QOpenSys/usr/include/errno.h file.
    • i.e. "syslogd: bind: errno = 67"  = EADDRINUSE or "Address already in use."  This indicates the syslogd process is already running on the IBM i server. Use the Qshell command, "ps -ef | grep syslogd" to identify the user running the process and then the CL command, "WRKUSRJOB <user>" to identify the job tied to the process.  End the previous syslogd process to resolve the 67 error number.

Putting this all together and piping the information to SEIM.

  • Configure the Syslog daemon on IBM i to send entries to a central collector/SIEM
  • Create a shell script to pass parameters to the Syslog logger command or use syslog() and syslog_r() client APIs in PASE or Java
  • Create a program using the IBM i Db2 SQL Table Functions (DISPLAY_JOURNAL & HISTORY_LOG_INFO) to retrieve the IBM i Audit Journal and History entries in Syslog format and pass to your shell script
    • Can limit entries created by adding WHERE clauses to SQL statement
    • Use EOF_DELAY to keep running indefinitely


If the syslogd daemon is not running when the syslog API is used, the system routes message priorities LOG_EMERG, LOG_ALERT and LOG_CRIT to the *REQUESTER message queue (in other words, QSYSOPR for batch jobs, or the user's message queue for an interactive job). LOG_ERR and LOG_WARNING priorities are sent to the joblog as *DIAG messages. LOG_NOTICE, LOG_INFO, and LOG_DEBUG are sent to the joblog as *INFO messages.

The logging priorities used by syslog can be found in the following include file:

/usr/include/sys/syslog.h

and are noted below:

/* Priorities (these are ordered - smaller number is a higher priority) */
#define LOG_EMERG   0    /* system is unusable */
#define LOG_ALERT   1    /* action must be taken immediately */
#define LOG_CRIT    2    /* critical conditions */
#define LOG_ERR     3    /* error conditions */
#define LOG_WARNING 4    /* warning conditions */
#define LOG_NOTICE  5    /* normal but signification condition */
#define LOG_INFO    6    /* informational */
#define LOG_DEBUG   7    /* debug-level messages */

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000000CJRAA2","label":"PASE"}],"ARM Case Number":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"7.1;7.2;7.3;7.4","Line of Business":{"code":"LOB57","label":"Power"}}]

Historical Number

518681384

Document Information

Modified date:
14 March 2022

UID

nas8N1013082

Page Feedback