IBM Support

AIX Cron: Parse the failures from the cron log

How To


Summary

This document demonstrates a method to parse the cron log for failed jobs.

Objective

Users often need to get quick details about failed jobs. The cron log might be very verbose, and reports the failure on a different line than the command details. An unsupported script is provided in this document, to show a method to get a simple list of failed commands, with the PID.

Steps

The following bash script wil generate a list of users who have not logged in since a specified number of days.
Note that the bash.rte file set is installed by default on AIX 7.3.  If you are running older versions of AIX, you can install bash from the AIX Toolbox for Open Source Software. The open source package is not supported by AIX Support, however, an "AIX Open Source Software Forum" is available on the IBM Community. 
The AIX Toolbox team recommends using DNF to install and manage Open Source software packages and dependencies.
1) Create the bash script
# cat cronCheck.sh
#!/bin/bash
if [[ $# -eq 0 ]]; then
  echo "Usage: $0 <path to log file>"
  exit 1
fi
log_file="$1"
# Validate input
if [[ ! -f "$log_file" ]]; then
  echo "$log_file does not exist!"
  exit 1
fi
awk '
/^root.*CMD/ {
  # Extract the command line and PID manually
  cmd = $0;
  pid_start = index($0, "PID (");
  if (pid_start > 0) {
    pid = "";
    for (i = pid_start + 5; i <= length($0); i++) {
      char = substr($0, i, 1);
      if (char == ")") break;
      if (char ~ /[0-9]/) pid = pid char;
    }
    # Remove extra spaces from the command
    gsub(/  +/, " ", cmd)
    seen[pid] = cmd; # Store the command line using PID as key
  }
}
/Cron Job with pid:/ {
  # Extract the PID from the "Failed" line manually
  pid_start = index($0, "pid: ");
  if (pid_start > 0) {
    pid = "";
    for (i = pid_start + 5; i <= length($0); i++) {
      char = substr($0, i, 1);
      if (char ~ /[^0-9]/) break;
      pid = pid char;
    }
    if (pid in seen) {
      if ($0 !~ /Successful/) { # Check for the word "Successful"
        print seen[pid] "\n" $0 "\n"; # Print the command and the status
        delete seen[pid]; # Remove the PID after processing
      }
    }
  }
}' "$log_file"

 
Sample /var/adm/cron/log data:
root      : CMD ( /usr/sbin/acct/fwtmp < /var/adm/wtmp > /tmp/"$(/usr/bin/date +%Y%m%d)".wtmp.backup   2>> /tmp/wtmp.backup.log ) : PID ( 17432832 ) : Wed Dec 11 20:33:00 2024
root      : CMD ( /usr/local/cronscripts/wtmpRotate.sh #  Error logging in script (ROTATE_LOG) ) : PID ( 18350362 ) : Wed Dec 11 20:33:00 2024
root      : CMD ( /tmp/mybash.sh >> /tmp/bash4.out 2>&1 ) : PID ( 17367388 ) : Wed Dec 11 20:33:00 2024
Cron Job with pid: 17760636 Failed
Cron Job with pid: 17432832 Successful
Cron Job with pid: 17367388 Failed
Cron Job with pid: 18350362 Successful
root      : CMD ( /tmp/non_existing_script >> /tmp/noscript.out  ) : PID ( 18481482 ) : Wed Dec 11 20:34:00 2024
root      : CMD ( /usr/sbin/acct/fwtmp < /var/adm/wtmp > /tmp/"$(/usr/bin/date +%Y%m%d)".wtmp.backup   2>> /tmp/wtmp.backup.log ) : PID ( 17826060 ) : Wed Dec 11 20:34:00 2024
root      : CMD ( /usr/local/cronscripts/wtmpRotate.sh #  Error logging in script (ROTATE_LOG) ) : PID ( 17760656 ) : Wed Dec 11 20:34:00 2024
root      : CMD ( /tmp/mybash.sh >> /tmp/bash4.out 2>&1 ) : PID ( 17367448 ) : Wed Dec 11 20:34:00 2024
Cron Job with pid: 17826060 Successful
Cron Job with pid: 18481482 Failed
Cron Job with pid: 17367448 Failed
Cron Job with pid: 17760656 Successful


 
2) Run the script, passing the full log path.
 
# ./cronCheck.sh /var/adm/cron/log
Cron Job with pid: 17367388 Failed :root : CMD ( /tmp/mybash.sh >> /tmp/bash4.out 2>&1 ) : PID ( 17367388 ) : Wed Dec 11 20:33:00 2024
Cron Job with pid: 18481482 Failed :root : CMD ( /tmp/non_existing_script >> /tmp/noscript.out ) : PID ( 18481482 ) : Wed Dec 11 20:34:00 2024
Cron Job with pid: 17367448 Failed :root : CMD ( /tmp/mybash.sh >> /tmp/bash4.out 2>&1 ) : PID ( 17367448 ) : Wed Dec 11 20:34:00 2024

 

 
TIP: You can modify the script to hard code the file.
log_file="/var/adm/cron/log"
# Validate input
if [[ ! -f "$log_file" ]]; then
    echo "$log_file does not exist!"
    exit 1
fi

Additional Information

SUPPORT

If you require more assistance, use the following step-by-step instructions to contact IBM to open a case for software with an active and valid support contract.  

1. Document (or collect screen captures of) all symptoms, errors, and messages related to your issue.

2. Capture any logs or data relevant to the situation.

3. Contact IBM to open a case:

   -For electronic support, see the IBM Support Community:
     https://www.ibm.com/mysupport
   -If you require telephone support, see the web page:
      https://www.ibm.com/planetwide/

4. Provide a clear, concise description of the issue.

 - See: Working with IBM AIX Support: Describing the problem.

5. Collect a system snap with cron data.

# snap -r (removes previous snap data)
# snap -aZ (excludes system dump)
# tar -cvf /tmp/ibmsupt/testcase/cron.tar -C /var adm/cron spool/cron -C /etc cronlog.conf
# ls -al /usr/sbin/cron /usr/bin/crontab /usr/bin/at > /tmp/ibmsupt/testcase/cronfiles.ls
 
6. Prepare the snap for upload.
# snap -c (Compress snap)
# mv /tmp/ibmsupt/snap.pax.Z /tmp/ibmsupt/yourcase#.snap.pax.Z (Rename file to associate with your case)
 
7. Upload all of the details and data for your case.
RECOMMENDED:
 
a) Attach to your case
 
b) Upload to the Enhanced Customer Data Repository(ECuRep)
 
ADDITIONAL OPTIONS:
 
c) Blue Diamond customer data uploads (US only)

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB08","label":"Cognitive Systems"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z000000cvzgAAA","label":"Commands"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
11 December 2024

UID

ibm17178627