Technical Blog Post
Abstract
Using an API to ingest events into Cloud Event Management
Body
Cloud Event Management provides DevOps (& Operations) teams with pre-configured event correlation, incident prioritization, notifications and runbooks (automated and manual) in a seamless experience suitable for any skill level. Cloud Event Management is currently in beta.
This blog outlines how to send events to Cloud Event Management via a REST API, using our example of events generated based on contents of log files. The same pattern could be applied to send events to Cloud Event Management from any source. CEMSendEvent.sh sends the events to Cloud Event Management. This script may be the only example in this blog entry you will need. The CEMLogMon.pl and loggingErrors.txt files, provide an optional log monitoring solution, demonstrating how CEMSendEvent.sh can be invoked or used by other programming.
Solution Flow
This diagram shows how the example is applied, outlining the flow from your log files to Cloud Event Management.
Shell script to send events to Cloud Event Management API
When creating your own event source, you generally can pre-fill many of the event fields (shown in the MYEVENT variable below) as they may be static. Perhaps hostname, ip address, event type etc.
To get started, look at the model and example values for an event POST API call here: https://console.bluemix.net/apidocs/919-event-management-api?&language=node#create-an-event
The following script sets a subset of these fields to known (what we already know about the source) or variable values (the fields that vary based on inputs). All events are set with a severity of Indeterminate. Out of the box the Priority of the associated incident will be set to 5 by the default incident policies. Create your own incident policies to identify specific incidents, assign and notify specific users or groups, and raise priorities as needed.
All files mentioned here, and used for this example, use the default location of /tmp
CEMSendEvent.sh - file name
#!/bin/bash
# Cloud Event Management script to send an event from a known source with variable fields
# Place your API keyname and password and set the URL below for your subscription
USER1="<< provide your API key name here, like:e0420355-933e-452f-9e38-7a48214c772a/nobsobhadkon>>"
PASS1="<< provide your API key password here, like: hL9i+GVLqxv3NASuvdgW4mYrYv6F6E+s>>"
URL="https://ibmeventmgt-bm-eventpreprocessor.mybluemix.net/api/events/v1"
usage="usage: $(basename "$0") [-r true/false] [-m summary] [-s source] [-o optdetails]
-r Provide a value of true/false. This sets the events resolution flag. (default: false - indicating its a problem and not a resolution)
-m summary Use 'summary' as the event summary
-s source Use 'source' as the event source
-o optdetails Use 'optdetails' as the optional details
-h Display this message"
# Set a default value for the Other Detail
OTHER="N/A"
RESOLUTION=false
while getopts ":m:s:o:r:h:" option; do
case "${option}" in
m) SMSG="${OPTARG}";;
s) SSRC="${OPTARG}";;
o) OTHER="${OPTARG}";;
r) RESOLUTION=$OPTARG;;
h | *) echo "$usage"; exit 0;;
esac
done
mtime=`date +%s`
isoutctime=`date -u +"%Y-%m-%dT%H:%M:%SZ"`
# Remove isoutctime here if you want to send resolution=false and true events. CEMLOGMON is the given name for this Application.
application="CEMLOGMON:$SMSG:($isoutctime)"
# Event Template (Subset of all possible event fields - Add cluster if you want to correlate all events into single incident)
# You can copy the event template and view the full event model using this URL: https://ibmeventmgt-bm-eventpreprocessor.mybluemix.net/docs/events/v1/#!/%2Fevents/createEvent
MYEVENT='{
"resource": {
"type": "CEMOpsEvent",
"name": "'$SMSG'",
"sourceId": "OpsLogMonitoring",
"application": "'$application'",
"component": "'$SSRC'",
"location": "'$SSRC'",
"interface": "RestAPI"
},
"summary": "'$SMSG'",
"severity": "Indeterminate",
"type": {
"statusOrThreshold": "log msgs exceeded percentage",
"eventType": "Log Threshold Breached"
},
"urls": [
{
"url": " https://yourslackteam.slack.com/messages/C6WFV0D2M/",
"description": "Your Slack Channel"
}
],
"details": {
"OtherDetail": "'$OTHER'"
},
"resolution": '$RESOLUTION'
}'
MYEVENT="$MYEVENT"
curl -X POST -H "Content-Type: application/json" -u "${USER1}:${PASS1}" -d "$MYEVENT" "$URL" --tlsv1
exit 0
Correlation
The sample above would attempt correlation on event.resource.application name, however we elected to insert timestamp into this field so each log event will generate a unique uncorrelated incident. If you want all log events associated with each product, to be correlated into single incidents, make the following change. Replace the MYEVENT variable shown above with the following version:
MYEVENT='{
"resource": {
"type": "CEMOpsEvent",
"name": "'$SMSG'",
"sourceId": "OpsLogMonitoring",
"application": "'$application'",
"component": "'$SSRC'",
"location": "'$SSRC'",
"cluster": "'$SSRC'",
"interface": "RestAPI"
},
"summary": "'$SMSG'",
"severity": "Indeterminate",
"type": {
"statusOrThreshold": "log msgs exceeded perscentage",
"eventType": "Log Threshold Breached"
},
"urls": [
{
"url": "https://yourteam.slack.com/messages/C6WFV0D2M/",
"description": "Your Slack Channel"
}
],
"details": {
"OtherDetail": "'$OTHER'"
},
"resolution": '$RESOLUTION'
}'
Adding cluster to the resource bundle, and setting it with the product name in loggingErrors.txt file, ensures all related product log messages detected are correlated into one incident.
Log Monitoring
Using the simple script above, the following log monitoring example provides a control script to forward operations monitored log events to Cloud Event Management. This script can be invoked at 15, 20 or 30 minute intervals using cron to send events. The script runs against a file you generate, that counts the occurrences of errors found in one or more logs. See the next section for details on the files format.
loggingErrors.txt - file name
Only the lines prefixed with "Event:"are read as input by the tool.
FILE SYNTAX
Event: {followed by the type of error detected in the logs}: {related product name}: {Total occurrence count}: {The delta%, if the new count exceeds the current count by this %, an event is emitted}
loggingErrors.txt|View Details
Bringing the logging data and the event sender together
The following Perl code processes the loggingErrors.txt file, and when thresholds are breached, emits events to Cloud Event Management using CEMSendEvent.sh.
CEMLogMon.pl - file name, invoked with perl CEMLogMon.pl
Sample incidents generated
Using the loggingErrors.txt file initial state provided above, and after running CEMlogMon.pl script at least once.
Find these 2 lines:
Event: Nexus middleware Errors: MyProduct1: 10: 10%
Event: 400 RC EventProcessor Timeout: MyProduct3: 0: 5%
Make the changes that follow, prior to rerunning CEMLogMon.pl.
Event: Nexus middleware Errors: MyProduct1: 12: 10% -- occurrence increased from 10->12 which is an increase greater than 10%
Event: 400 RC EventProcessor Timeout: MyProduct3: 1: 5% -- occurrence increased from 0->1 which is the first time this log entry was detected
The above changes generate these 2 incidents in Cloud Event Management, as seen from this URL: https://ibmeventmgt-bm.mybluemix.net/incident-queue/all
What this example demonstrated
The shell script for sending in events to Cloud Event Management is simple, short and easy to understand. You could provide more event details in the json shown, but only if those fields provide additional information needed for problem resolution. For our log message detection example, just a few fields fully describe what was detected and where to begin analysis. The loggingErrors file and Perl example show how our simple event script can be called by a broader monitoring solution. Use these scripts as a guide as you prepare to send your critical events to Cloud Event Management.
Authors:
Phil Riedel
Sudhakar Chellam
UID
ibm11080039