Audit data REST API

For retrieving audit event records with the REST API, Cloud Pak SystemCloud Pak System Software provides sample scripts that you can customize to automate the download process, and, ultimately, the process of deleting the records from the system (to free storage). You can incorporate the scripts into your regularly scheduled jobs to archive audit data according your organization's requirements.

Note: the REST API returns audit data in comma-separated value (CSV) format in a compressed file.

Before you begin

Working with the auditing resources and scripts entails the following prerequisites:

  • You need an environment that supports shell scripts, Python, Java™, Curl, and a file archiving utility (for manipulating compressed files). For example: Cygwin, a compatibility layer, meets all of these requirements.
  • You must have downloaded and configured the command line interface (CLI). All of the scripts are located in CLI directories.
  • You must have one of the following user permission assignments in Cloud Pak SystemCloud Pak System Software:
    • Manage auditing (full permission) role
    • View all auditing reports (Read-only) role
You also might want to review general information about the REST API. For this particular procedure, you need to understand that invocation of the API is accomplished through HTTP requests.
Tip: If you want to create your own scripts for retrieving audit data, consult the auditing section of this information center for the syntax to use in HTTP requests. You can use the procedures on this page and the CLI audit sample scripts to understand how code must integrate the fundamental tasks, authenticating with the REST API, issuing the HTTP requests, and specifying a compressed file for the returned data.

About this task

Table 1 lists the sample scripts that you can use to download the compressed file of audit data and extract it. The following procedure steps provide guidelines for using the scripts.

Table 1. Overview of sample scripts
File name Functional description Location
create_basicauth_header.py This Python script is used by the following Curl script to create an HTTP basic authentication header with your user key information (your Cloud Pak SystemCloud Pak System Software: credentials). ...\deployer.cli-XXX\deployer.cli\lib\XXX\deployer, where XXX is the version number of the CLI
cscurl.sh This shell script performs the following tasks:
  • Runs the create_basicauth_header.py script to create the authentication header.
  • Issues a Curl command to Cloud Pak SystemCloud Pak System Software with the authentication header to authenticate your credentials.
    (The Curl command targets the URL of the Cloud Pak SystemCloud Pak System Software component that provides the audit-related REST API.)
  • Sends the HTTP request for downloading the audit data to the REST API.
    Note: You must specify the HTTP request as a parameter of cscurl.sh.
 ...\deployer.cli-XXX\deployer.cli\lib\XXX\deployer, where XXX is the version number of the CLI
auditFetch.sh This sample script provides a very simple demonstration of how you can automate the entire process, and thus adopt it as another job that you run regularly in your IT organization. ...\deployer.cli-XXX\deployer.cli\samples, where XXX is the version number of the CLI
Steps 1 - 5 are preparatory steps for using these scripts. Steps 6 - 7 describe how to use cscurl.sh to download audit data in a compressed file, and then extract that file. Step 8 describes how to use auditFetch.sh to automate the download and extract operations. Consider auditFetch.sh as a model for code that you can run with a job scheduler to regularly download audit data.

Procedure

  1. Place the scripts in an appropriate working directory.
  2. Download your Cloud Pak SystemCloud Pak System Software user keys from the system and save them in the same directory that contains the scripts.
    Alternatively, use CLI command, deployer.security.getUserKeys() to request user keys.
    Note: Once you request new user keys at https://your_IWD_server/resources/userKeys/ , any saved keys from past requests are no longer valid.
    1. Open a browser window and go to https://your_IWD_server/resources/userKeys/.
    2. Type your user name and password in the authentication fields.
    3. Select the directory and provide a file name for storing your keys.
    4. Save the keys as a .json file.
  3. Optional: For added security in a production environment, you can use the Cloud Pak SystemCloud Pak System Software root certificate to authenticate the scripts to the REST API. Follow these steps to download the certificate and save it in the same directory that contains the scripts and your user keys:
    1. Open a browser window and go to https://your_IWD_server/resources/rootcacertificate/.
    2. Select the directory and provide a file name for storing the root certificate.
      Note that the default file name is cert.pem.
    3. Save the root certificate.
  4. Optional (but necessary if you want to use the root certificate for authentication): In the /etc/hosts file of your workstation, bind the IP address of the Cloud Pak SystemCloud Pak System Software to the name that is used in the root certificate.
  5. Construct the URL for the download request that the cscurl.sh script sends to the REST API; you must provide this URL as a parameter to run cscurl.sh in the next step.
    You can choose between two options for downloading audit data. The more basic option is to specify a maximum number of records to download. Alternatively, you can specify both a maximum number of records and the time frame in which the product logged those records. For either option, the URL must include the location of the REST API code that downloads the data and the resource name of the option that you choose. Use the following models for the URL:
    • To simply specify a maximum number of records in the request, construct a URL for the events resource and use the size parameter:
      https://Cloud_Pak_SystemSystem:9444/audit/archiver/events?size=X
      (For the X variable substitute the number of records that you want to download.)
    • To add a time frame to your request, construct a URL for the filteredEvents resource and specify the start and end times as long values:
      https://Cloud_Pak_SystemSystem:9444/audit/archiver/filteredEvents?size=X&startTime=long_value&endTime=long_value
      Note: If you did not perform Steps 3 - 4, replace the name Cloud_Pak_SystemSystem in the URL with the IP address of your Cloud Pak SystemCloud Pak System Software server.
  6. Run cscurl.sh with parameters that specify the URL that you created in Step 5, as well as the name of the compressed file in which you want the REST API to store the audit data.
    For example: 
    ./cscurl.sh username=user1 password=user1 
keyfile=userKeys.user1.json -v --cacert root_cert.pem -H "Accept:application/octet-stream" 
-H "Content-Type:text/plain" "https://Cloud_Pak_SystemSystem:9444/audit/archiver/events?size=X" > ArchiveFetchTempFile
    Note all of the variables that represent parameter values in the statement:
    • user1 = the user name
    • user1 = the user password
    • userkeys.user1.json = the file that contains the user keys
    • root_cert.pem = the name of the file that contains the Cloud Pak SystemCloud Pak System Software root certificate
    • X = the number of records to be downloaded
    • ArchiveFetchTempFile = the name of the compressed file to which the audit data is written

    Also, be aware of the following usage notes for running cscurl.sh:
    • Invoking this script without parameters triggers display of its help message.
    • To guard against data loss, you must specify a file for the audit data. Otherwise, the script returns it as simple command-line output.
  7. Extract the archive file that is returned from the REST API. (In response to the previous example of running the script, the REST API would return ArchiveFetchTempFile.zip.)
    Consequently you now have four files, including your audit data:
    • audit-events.csv - Contains your audit event records in CSV format.
    • audit-events-signed-events-checksum - Contains the digital signature that verifies both the integrity and authenticity of your audit data.
    • audit-events-record-IDs - Contains the ID of each event record that you downloaded.
    • audit-events-signed-record-IDs - Contains the signatures that verify the authenticity of each record.
    At this point the retrieval process is complete. If you followed Steps 1 - 7, you successfully used the individual scripts and the REST API to write your audit data to a compressed file and download it. Step 8 describes auditFetch.sh , which automates the entire process; the script provides an example of code that you can run with a job scheduler to regularly download audit data.
  8. To run auditFetch.sh, use the following statement as a model: 
    ./auditFetch.sh username=auditor password=auditor 
keyfile=userkeys.auditor.json IWD=IP address size=X startTime=long_value endTime=long_value > ArchiveFetchTempFile
    Tip: If you performed Steps 3 - 4 in order to use the root certificate to authenticate the scripts, change the value of the IWD parameter to the name Cloud_Pak_SystemSystem. Then modify the script's invocation of cscurl.sh; replace the -k option with the -v --cacert options. Remember to specify the name of the file in which the Cloud Pak SystemCloud Pak System Software root certificate is stored on your workstation, as in: -v --cacert cert.pem.
    Based on the values that you supply for the parameters, auditFetch.sh performs these tasks to download your audit event records:
    • Authenticates your credentials with Cloud Pak SystemCloud Pak System Software.
    • Constructs an HTTP request and sends it to the REST API.
    • Stores the records that are retrieved by the REST API in a compressed file with the name that you specified. (If you did not specify a file name, the script uses the default name of ArchiveFetchTempFile.zip.)
    • Extracts that archive file, which contains the following files:
      • audit-events.csv (your audit event records in CSV format)
      • audit-events-signed-events-checksum
      • audit-events-record-IDs
      • audit-events-signed-record-IDs

    Be aware of the following usage notes for running auditFetch.sh:
    • Invoking this script without parameters triggers display of its help message.
    • In some environments, such as Ubuntu, you might need to modify the script's invocation of cscurl.sh; add the period and forward-slash characters (./) as a prefix to the statement that invokes cscurl.sh.

What to do next

Analyze the data that you just downloaded and archive it. (The most important files for archival are audit-events.csv and audit-events-signed-events-checksum.)
Tip: Review the Audit record attributes topic in this information center for a listing of the record attributes and guidelines on how you can best exploit the data.
Then, because Cloud Pak SystemCloud Pak System Software does not automatically delete audit data after you download it, you must run the auditDelete.sh script to delete the data from the system and free storage resources. You can use this script along with your customization of auditFetch.sh as part of a regularly scheduled job to download, archive, and then delete audit data.