Netcool integration with BMC Remedy via REST

Body

Overview

Many IBM Netcool clients use BMC Remedy as IT Service Management system for Incident/Problem Management. IBM Tivoli Netcool/Omnibus Java Gateway for BMC Remedy ARS can be used to integrate Netcool/Omnibus with BMC Remedy. It uses BMC Remedy ARS API to communicate with BMC Remedy. Gatweway support bi-directional communication and provides capabilities to create and update incidents as well as add journal records to incident work log.

Many companies move to REST architecture to simplify integration between components and systems. BMC 9 now provides REST API as integration option as well.

This document provides information on how to integrate Netcool/Omnibus with BMC 9 via REST API using Netcool/Impact.

BMC REST API Overview

As BMC Remedy ARS API, BMC REST API is essentially providing interface to BMC Remedy forms. BMC Remedy is shipped with a set of pre-defined forms to create/update incidents/problems, etc. E.g. HPD:Help Desk form is a default incident form.

BMC Remedy Administrators have capabilities to manage forms (modify fields, add/delete fields, create new forms) so any effort to integrate Netcool with BMC Remedy requires assistance from BMC Remedy Administrators to identify forms to use, fields to map data from Netcool events, etc.  This effort is required both for Netcool/Omnibus Gateway for Remedy ARS and BMC REST integration options. In case of Netcool/Omnibus Gateway for Remedy ARS data mapping is done in gateway map files using BMC Field ID's mapped to Netcool event fields. In case of REST integration mapping is done in json objects assigning BMC fields (by name) to Netcool event fields.

BMC REST API is very powerful and essentially provides capabilities to create/update or retrieve any entry in BMC Remedy AR system. From Netcool integration point of view the following capabilities will be reviewed in this document: create/update incidents/problems, retrieve incident data for specific incident or for a set of incidents using qualification, add incident work log, query CI data from BMC Atrium CMDB.

Certificates

BMC REST API supports https only communication, so BMC REST API endpoint certificates should be loaded to Netcool/Impact server (and optionally UI server).

Certificate can be extracted to a file from BMC REST API UI using Mozilla Browser.

Then it can be loaded to Netcool/Impact using the following command:

$IMPACT_HOME/sdk/bin/keytool -importcert -alias bmc_prod -file bmcprod.crt -keystore $IMPACT_HOME/wlp/usr/servers/NCI/resources/security/trust.jks -storepass Netcool

Certificate should be loaded on each Netcool/Impact instance in a cluster.

The following command can be used to verify that certificate is added to a trust store.

$IMPACT_HOME/sdk/bin/keytool -list -keystore $IMPACT_HOME/wlp/usr/servers/NCI/resources/security/trust.jks -storepass Netcool

It should produce an output similar to the following:

    Keystore type: jks  Keystore provider: IBMJCE    Your keystore contains 5 entries    default, Apr 20, 2018, trustedCertEntry,  Certificate fingerprint (SHA1): 46:10:BC:AA:..............;:2F:3E  .....................  bmc_prod, Sep 27, 2018, trustedCertEntry,  Certificate fingerprint (SHA1): 0B:93:EE:C6:................;.:97:71  .....................    

Restful API DSA vs GetHTTP function

Netcool/Impact provides two standard ways to interact with REST API endpoints: ResfulAPI DSA and GetHTTP function.

RestfulAPI DSA works as any other DSA providing capabilities to query data from REST endpoint.

GetHTTP function provides more flexibility in querying and posting data to REST API endpoint.

BMC REST API authentication specifics along with other complexities described later make use of GetHTTP function a better choice for Netcool integration with BMC REST API.

BMC REST API Authentication

As other REST providers BMC requires all request to be authorized. BMC REST API has some specifics that make it somewhat not easy to use.

BMC REST API does not support basic authentication using username and password for queries.
Instead it has a specialized login method which expects username and password provided in request body in x-www-form-urlencoded type.
This method returns authorization token in the response body. Token should be used in Authorization HTTP header for all subsequent calls to other methods.

   BMCUser="user";   BMCPwd="password";   HTTPHost="host.domain.com";   HTTPPort=8443;   Protocol="https";   Path="/api/jwt/login";   ChannelKey="";    Method="POST";   AuthHandlerActionTreeName="";   FormParameters=null;   FilesToSend=null;   HeadersToSend=newobject();     HttpProperties=newobject();;   HttpProperties.ContentType = "application/x-www-form-urlencoded";   HttpProperties.Content = 'username='+BMCUser+'&password='+BMCPwd;   HttpProperties.UseProxy=false;   HttpProperties.ConnectionTimeout=0;   HttpProperties.Protocol = "https";      x=GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method,   AuthHandlerActionTreeName,   FormParameters, FilesToSend, HeadersToSend, HttpProperties);   if(ResultCode==200) {    token="AR-JWT " + ThePage;   } else {    log("LOGIN API ERROR: "+ErrorReason);   }  

BMC REST API authentication/authorization has the following restrictions that introduce additional complexity:

• Authorization token expires within one hour. Other REST providers (e.g. Splunk) can be configured so authorization token never expires. Unfortunately in BMC REST API it's not possible.
  This requires to either pre-pend each call to BMC REST API methods with a call to login method or create a specialized "token persistence" logic to periodically call login method and save authorization token in some permanent storage (e.g. database or Netcool/Impact server variable)
• BMC Remedy Server restricts all non-administrative users to have active login session only from one host/ip address. This can result in all login calls from Netcool/Impact to BMC REST API to fail with "Error: 9093: User is currently connected from another machine or incompatible session." if user already logged in from other tool.
  This issue is particularly troublesome in clustered Netcool/Impact environment, where calls to BMC REST API will fail on secondary Netcool/Impact instances, when Netcool/Impact cluster decides to offload some processing from primary instance. Solution for this issue is to configure separate users for each Netcool/Impact instance and make policies aware on which instance they run (GetServerName function).
  If "token persistence" logic is implemented - it is important to know that Impact Server Variables are instance specific and server variable values are not synchronized across cluster which makes it useful in this case.

BMC REST API also has logout method (/api/jwt/logout) which requires only HTTP Authorization header containing BMC REST token.
It is noted, however, that calls to logout method will not reset host binding for account.

Creating Incident

Incidents can be created in BMC Remedy via REST API using REST method that exposes HPD:IncidentInterface_Create form.

As any other methods it requires HTTP Authorization Header to contain authorization token.
POST method should be used when accessing the API URI.
Payload should be send in a json format in the request body.
Note that content of json object will differ between BMC Remedy deployment, as BMC Administrators can make changes in HPD:IncidentInterface_Create form as well as change incident creation processes and/or implement templates for incident creation. It is important to communicate with BMC Administrators to find out a set of required fields, validate Netcool/Omnibus event data mapping and determine what additional requests might be required against BMC REST API before attempting to create an incident. The following code provides a template that can be used to create a working logic.

   request_json = '{'+ '"values" : {';   request_json = request_json + '  "First_Name" : "' + FirstName + '" ,';   request_json = request_json + '  "Last_Name" : "' + LastName + '" ,';   request_json = request_json + '  "Description" :  "' + Summary + '" ,';   request_json = request_json + '  "Detailed_Decription" : "' + DetailedDescription + '" ,';   request_json = request_json + '  "Impact" : "' + Impact + '" ,';   request_json = request_json + '  "Urgency" : "' + Urgency + '" ,';   request_json = request_json + '  "Assigned Support Company" : "' + SvdCompany + '" ,';   request_json = request_json + '  "Assigned Support Organization" : "' + SvdOrg + '" ,';   request_json = request_json + '  "Assigned Group" : "' + SvdGroup + '" ,';   request_json = request_json + '  "z1D_Action" : "CREATE" ,';   request_json = request_json + '  "Flag_Create_Request" : "No" ';   request_json = request_json + ' } '+'}';     HTTPHost="host.domain.com";   HTTPPort=8443;   Protocol="https";   Path="/api/arsys/v1/entry/HPD:IncidentInterface_Create";   ChannelKey="";    Method="POST";   AuthHandlerActionTreeName="";   FormParameters=null;   FilesToSend=null;     HeadersToSend=newobject();   bmc_token=GetServerVar("BMC_Rest_Token");   HeadersToSend.Authorization=bmc_token;     HttpProperties=newobject();;   HttpProperties.ContentType = "application/json; charset=utf-8";   HttpProperties.Content = request_json;   HttpProperties.UseProxy=false;   HttpProperties.ConnectionTimeout=0;   HttpProperties.Protocol = "https";      x=GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method,   AuthHandlerActionTreeName,   FormParameters, FilesToSend, HeadersToSend, HttpProperties);   if(ResultCode == 201) {  // Success     incidenturl=HeadersReceived;   } else {     log("INCIDENT CREATE API ERROR: "+ErrorReason);   }  // if(ResultCode == 401) {  // Unauthorised  // if(ResultCode == 500) {  // Server Error  

Note that call to HPD:IncidentInterface_Create will not return Incident number in the response. Instead it will return a direct URL to BMC REST API which can be used to retrieve incident data including Incident Number.

E.g.:

  https://host.domain.com:8443/api/arsys/v1/entry/HPD:IncidentInterface_Create/000000000002807  

Adding Incident work logs

Work logs can be added to incidents in BMC Service Desk using REST API that exposes HPD:WorkLog form.

As any other methods it requires HTTP Authorization Header to contain authorization token.
POST method should be used when accessing the API URI.
Payload should be send in a json format in the request body.
Note that Incident number to add work log to should be provided in 'Incident Number' json field.

   request_json = '{'+ '"values" : {';   request_json = request_json + '  "Work Log Submitter" : "user" ,';   request_json = request_json + '  "Status" : "Enabled" ,';   request_json = request_json + '  "Description" :  "' + Summary + '" ,';   request_json = request_json + '  "Detailed Description" :  "' + LogText + '" ,';   request_json = request_json + '  "Incident Number" : "' + IncidentNo + '" ,';   request_json = request_json + '  "Work Log Type" : "Incident Task / Action" ,';   request_json = request_json + '  "View Access" : "Public" ,';   request_json = request_json + '  "Secure Work Log" : "No"';   request_json = request_json + ' } '+'}';     HTTPHost="host.domain.com";   HTTPPort=8443;   Protocol="https";    Path="/api/arsys/v1/entry/HPD:WorkLog";   ChannelKey="";    Method="POST";   AuthHandlerActionTreeName="";   FormParameters=null;   FilesToSend=null;     HeadersToSend=newobject();   bmc_token=GetServerVar("BMC_Rest_Token");   HeadersToSend.Authorization=bmc_token;     HttpProperties=newobject();;   HttpProperties.ContentType = "application/json; charset=utf-8";   HttpProperties.Content = request_json;   HttpProperties.UseProxy=false;   HttpProperties.ConnectionTimeout=0;   HttpProperties.Protocol = "https";      x=GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method,   AuthHandlerActionTreeName,   FormParameters, FilesToSend, HeadersToSend, HttpProperties);   if(ResultCode == 201) {  // Success     result=0;   } else {     log("WORKLOG ADD ERROR: "+ErrorReason);   }  // if(ResultCode == 401) {  // Unauthorised  // if(ResultCode == 500) {  // Server Error  

Retrieving incident Data

Incident data can be retrieved from BMC using REST API methods exposing either HPD:Help Desk or HPD:IncidentInterface forms.

Both forms can provide Incident number and other generic incident information, but there are some differences. E.g. only HPD:IncidentInterface form provides incident status (Assigned, Resolved, Closed) information.

As any other methods it requires HTTP Authorization Header to contain authorization token.
GET method should be used when accessing the API URI.
Incident number (one or multiple) and (optionally) list of fields to be retrieved should be provided as URL parameters.

Output data will be sent as json payload in response body. It needs to be parsed using ParseJSON function. Note that resulting object will have a multi-level structure and accessing data might require a rather complex references.

   fields="values(Incident+Number,Status,Request+ID)";   query="('Incident+Number'=\"INC00000000123\"+OR+'Incident+Number'=\"INC00000000124\"+OR+'Incident+Number'=\"INC00000000123\")";     HTTPHost="host.domain.com";   HTTPPort=8443;   Protocol="https";  // Path="/api/arsys/v1/entry/HPD:Help%20Desk";   Path="/api/arsys/v1/entry/HPD:IncidentInterface";   ChannelKey="";    Method="GET";   AuthHandlerActionTreeName="";   Path=Path+"?q="+query;   if(fields!=null and fields!=''){     Path=Path+"&fields="+fields;   }     FilesToSend=null;   HeadersToSend=newobject();   bmc_token=GetServerVar("BMC_Rest_Token");   HeadersToSend.Authorization=bmc_token;     HttpProperties=newobject();;   HttpProperties.UseProxy=false;   HttpProperties.ConnectionTimeout=0;   HttpProperties.Protocol = "https";      x=GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method,   AuthHandlerActionTreeName,   FormParameters, FilesToSend, HeadersToSend, HttpProperties);   if(ResultCode == 200) {  // Success     result=0;     resultdata=ThePage;   } else {     log("INCIDENT GET ERROR: "+ErrorReason);     resultdata="";   }   if(resultdata!=null && resultdatata!="") {     incidentdata=ParseJSON(resultdata);   }     k = 0;   kl = length( incidentdata["entries"] );   Log('Found ' + kl + ' records.');    while ( k < kl) {     retIncNo = incidentdata["entries"][k]['values']["Incident Number"];     retIncStatus = incidentdata["entries"][k]['values']["Status"];     retIncRequestID = incidentdata["entries"][k]['values']["Request ID"];       Log( retIncNo + '=' + retIncStatus);      k = k + 1;    }  

Updating Incident

Incidents can be updated in BMC Remedy via REST API using REST method that exposes HPD:IncidentInterface form.

As any other methods it requires HTTP Authorization Header to contain authorization token.
PUT method should be used when accessing the API URI.
Payload should be send in a json format in the request body.

Incident to update should be addressed by adding "Request ID" to REST API URL. E.g.

  /api/arsys/v1/entry/HPD:IncidentInterface/INC000000000123%7CINC000000000123  

Request ID can be retrieved for incident as described above in Retrieving Incident Data above.

Json object should be populated the same way as for Incident Creation example above.

Querying CMDB

BMC Atrium CMDB data can be queried via REST API using URL's that similar to CI relationship tree. E.g.

  /api/arsys/v1/entry/BMC.CORE:BMC_BaseElement  

Then additional parameters to the URL can be used for additional qualification. E.g:

  /api/arsys/v1/entry/BMC.CORE:BMC_BaseElement?q='DatasetId'="BMC.ASSET"; AND 'ClassId'="BMC_COMPUTERSYSTEM" AND 'Name'="hostname"

Additional URL parameter  'fields' can be used to restrict output by pre-defined set of fields. E.g.:

  /api/arsys/v1/entry/BMC.CORE:BMC_BaseElement?q='DatasetId'="BMC.ASSET"; AND 'ClassId'="BMC_COMPUTERSYSTEM" AND 'Name'="hostname"&fields=values(Name,ReconciliationIdentity,Category,Type,Item,Model,ManufacturerName,SystemEnvironment)

The following code template provides example of CMDB query:

   datasetid="BMC.ASSET";   classid="BMC_COMPUTERSYSTEM";   name=EventContainer.Node;   HTTPHost="host.domain.com";   HTTPPort=8443;   Protocol="https";   Path="/api/arsys/v1/entry/BMC.CORE:BMC_BaseElement";   ChannelKey="";    Method="GET";   AuthHandlerActionTreeName="";   FormParameters=newobject();   FormParameters.q="'DatasetId'=\""+datasetid+"\" AND 'ClassId'=\""+classid+"\" AND 'Name'=\""+name+"\"";     FilesToSend=null;     HeadersToSend=newobject();   bmc_token=GetServerVar("BMC_Rest_Token");   HeadersToSend.Authorization=bmc_token;     HttpProperties=newobject();;   HttpProperties.ContentType = "application/json; charset=utf-8";   HttpProperties.Content = request_json;   HttpProperties.UseProxy=false;   HttpProperties.ConnectionTimeout=0;   HttpProperties.Protocol = "https";      x=GetHTTP(HTTPHost, HTTPPort, Protocol, Path, ChannelKey, Method,   AuthHandlerActionTreeName,   FormParameters, FilesToSend, HeadersToSend, HttpProperties);   if(ResultCode == 200) {  // Success     result=0;     resultdata=ThePage;   } else {     Log("CI GET ERROR: "+ErrorReason);   }     resultobject=newobject();   if(result == 0) {     if(resultdata!="" && resultdata!=NULL){       cidata=ParseJSON(ci_data);     }     if(length(cidata["entries"]) > 0) {       resultobject["Name"]                   = cidata["entries"][0]["values"]["Name"];       resultobject["ReconciliationIdentity"] = cidata["entries"][0]["values"]["ReconciliationIdentity"];       resultobject["Category"]               = cidata["entries"][0]["values"]["Category"];       resultobject["Type"]                   = cidata["entries"][0]["values"]["Type"];       resultobject["Item"]                   = cidata["entries"][0]["values"]["Item"];       resultobject["Model"]                  = cidata["entries"][0]["values"]["Model"];       resultobject["ManufacturerName"]       = cidata["entries"][0]["values"]["ManufacturerName"];       resultobject["SystemEnvironment"]      = cidata["entries"][0]["values"]["SystemEnvironment"];     }  

Error Handling

BMC REST API calls return various return codes to indicate successful query or error. BMC follows HTTP standards for return status codes.

Successful queries return status codes in 2xx range.

Any errors return status codes in 4xx or 5xx range.

I noticed the following error types:

• 401 Unauthorized - usually indicates that authorization token is expired. Logging back to BMC REST and refreshing token fixes the issue
• 500 Internal Server Error - usually indicates some issues with a data or internal processing error. It could be anything from unescaped control characters in json object, missing or malformed data, or some internal data processing errors

ErrorReason variable usually populated by GetHTTP function and in case of a error contains HTTP status message (e.g. "Internal Server Error") and basically useless.

Actual error messages returned by BMC REST API in HTML response body.

These messages can be plain text if processing failed at REST API level. E.g.

  “Illegal unquoted character ((CTRL-CHAR, code 9)): has to be escaped using backslash to be included in string value”  

To avoid issues with json content it is important that data sent to BMC does not contain special characters such as single quote, double quote, tab, backslash, etc.

Fields that might contain these characters, such as Summary, Journal, etc. should be cleaned. E.g.:

    tmptext=Replace(outtext,'\\\\','',1000);    outtext=tmptext;    tmptext=Replace(outtext,"'","",1000);    outtext=tmptext;    tmptext=Replace(outtext,'"','',1000);    outtext=tmptext;    tmptext=Replace(outtext,'\t',' ',1000);    outtext=tmptext;   

Error messages can also be json formatted if processing is failed at BMC Remedy AR server level. E.g.

  [  {    "messageType":"ERROR",    "messageText":null,    "messageNumber":1291053,    "messageAppendedText":"The Assigned Group fields are invalid.  Use the menus for the Support Company, Support Organization, Assigned Group, and Assignee fields or the type ahead return function on the Assigned Group or Assignee fields to select this information."  }  ]  

These error messages can be parsed using ParseJSON function as shown below.

     if(ThePage!="" && ThePage!=NULL){       incdata=ParseJSON(ThePage);     }     errormessage="INCIDENT UPDATE ERROR: "+ErrorReason+": "+                  incdata["arrayItems"][0]["messageNumber"]+": "+                  incdata["arrayItems"][0]["messageText"]+": "+                  incdata["arrayItems"][0]["messageAppendedText"];