Configuring File Observer jobs

Using the File Observer functionality, you can write bespoke data to a file in a specific format, upload this data to the topology service, and then visualize this data as a topology view. You can load data from a single file, or multiple files.

Before you begin

The File Observer reads topology data from specified files, and uploads it. You must create these files manually.

Tip: Suggested practice to update, add to, or delete data

When you load data from your external system via the File Observer, this data is injected into the topology database, which then presents resources to be viewed and managed using your system's functionality.

When you want to add to, remove or update these resources, do the following:

  • Update topology data: To (periodically) update your topology system, for example when your external sources require the addition, change or removal of resources, create a new input file and trigger the observer job. Your previously loaded resources are updated.

  • Delete all topology data: To remove all previously ingested topology data, upload a blank input file and trigger the observer job.
    CAUTION:
    All previously uploaded resources are removed from the topolgy database.

About this task

Topology data in a file is composed of vertices (nodes) and edges. A vertex represents an object (resource), while an edge represents the relationship between two objects.

Each line of the file you create should be in one of the following formats, loading a single resource vertex (including optional relationships in the '_references' field) or a single edge, deleting a single vertex, or pausing execution.

Lines starting with V: (vertex), E: (edge), D: (delete) or W: (wait) are treated as instruction lines to be processed. Other lines, for example lines that are empty or commented out, are ignored.

Line format

V - Load a resource vertex, with a JSON representation as documented for the body of the topology service API method: POST /resources

If specifying the _status element, acceptable state values are open, closed, or clear, and acceptable severity values are clear, indeterminate, warning, minor, major, or critical. The following geolocation example adds point geolocation data as a GeoJSON object inside the vertex JSON, with the coordinates being longitude first, then latitude.

V:{"name":"Smallbrook Junction", "uniqueId":"gb-nr-SAB", "entityTypes":["railStation"], "geolocation":{"geometry":{"coordinates":[-1.15511,50.71141],"type":"Point"}}}

E - Load an edge, with a JSON representation as documented for the _references section of the body of the topology service API method POST /resources.

D - Delete a resource vertex, identified by it's uniqueId.

W - Pause for the duration (for testing purposes only). Takes an integer period followed by a string specifying the units.

G - To load a group via the File Observer, see the following example syntax:

G:{"uniqueId":"EA","entityTypes":["group"],"name":"Event Analytics"}
Note: When creating a group's edge, it must be based on uniqueId.
Important: Ensure that the file is structured correctly. For each line of the file, information included after the closing } that matches an opening { is ignored, and no error is recorded.

Mapping alerts to topologies: When using the File Observer to correlate events from Logs Anomaly, ensure you use the matchTokens property when creating topology templates. The matchTokens property is a list of values you can use to look up a resource from an alert. If a unique matchToken is found that matches an alert's resource fields, such as resource.name or resource.sourceId, the alert will be assigned to that resource.

File Observer syntax

When uploaded to the file observer in a file, say example.txt, the following small topology can be generated by running a File Observer job for that file. The figure also shows an alert matched to a resource.

Figure that shows a small topology generated by the File Observer 

V:{"uniqueId":"Thomas J. Watson","entityTypes":["person"],"name":"Watson"}
V:{"uniqueId":"IBM","entityTypes":["company","domain"],"name":"IBM","_references":[{"_edgeType":"owns","_toUniqueId":"Cloud Pak for AIOps"},{"_edgeType":"contributedTo", "_fromUniqueId":"Thomas J. Watson"}]}
V:{"uniqueId":"Cloud Pak for AIOps","entityTypes":["ai"],"name":"Watson","platform":"OCP","matchTokens":["AIOps","Cloud Pak for AIOps"]}

Line 1: This line has the minimal set of properties for a resource vertex:

  • uniqueId - Uniquely identifies the resource within the provider namespace (the provider is supplied when running the job).
  • entityTypes - Determines the icon displayed in the UI. Some types have a pre-defined mapping to icons, as described in the Entity types topic.
  • name - The label displayed in the UI.

Line 2: This line shows multiple entityTypes. There is no default icon mapped to type company, so the icon is determined from the domain type. In addition to the properties of the resource, this adds an edge to another resource.

  • The edgeType must exist already. Either one of the default Edge types or one created by the topology service's POST /types/edge API (see Accessing APIs)
  • The referenced uniqueId does not have to exist yet.
  • Multiple edges can be added in the _references list.

Line 3: This line shows that more data or properties can be added. It adds a custom platform property (might be any additional property required). See the Properties topic for a list of existing properties. This line also illustrates that any entityType can be supplied, and that the name does not have to be unique. The UI does not show an icon for this resource because no default icon mapping has been created. You can add custom mapping through the UI as described in the Defining custom icons topic. And finally, this line also adds a matchToken property, which is a list of values that can be used to link a resource to an alert.

File Observer validation rules

The validation rules applicable to File Observer input files are outlined in the File Observer Validations topic.

Procedure

Define or edit the following parameters, then click Save to save and run the job.

By default, these jobs are one-off, transient jobs that upload all of the requested topology data as soon as they are triggered.

You can also run these jobs (again) from the Observer UI, or schedule them to run at set times when configuring them.

Encryption requirement: See the Configuring observer jobs security topic for more information.

Required parameters for single file load jobs

Parameter Action Details
Unique ID Enter a unique name for the job Required.
File mode Select either Upload local file, then drag and drop (or browse) to the input file; or Use server file, then specify the name of the file previously uploaded through the POST /files API. Required. If the same file name is used in a second job, or the file is replaced through PUT /files, any resources missing from the second load will be deleted.
Provider Enter the job's provider, that is, a namespace for the resources' uniqueIds. Resources can be loaded from multiple files with the same provider, and linked as if they were all loaded from the same file. To link to resources from another provider (for example, from another observer), the cross-provider semantics are required. Optional. Defaults to FILE.OBSERVER:{fileName} if not supplied.

Required parameters for multi-file load jobs

Parameter Action Details
Unique ID Enter a unique name for the job Required
Provider Enter the name of the data provider for all files. The uniqueId of each loaded resource is considered unique within the context of the data provider. If the same files are loaded a second time with a different provider, the topology resources created will be considered as new resources. Changing the provider therefore leads to the creation of duplicate resources. Required
File names Specify or choose the files to be loaded, without a path. The files all share the same provider and are processed in the order they are defined. Select either Upload local file, then drag and drop (or browse) to the input file; or Use server file, then specify the name of the file previously uploaded through the POST /files API. Required. If the same file name is used in a second job, or the file is replaced through PUT /files, any resources missing from the second load will be deleted.

Optional parameters for both single- and multi-file jobs

Parameter Action Details
Include delays Select true (default) to accept any delays included in the files, or false to skip such delays. Optional
Access scope Enter text to provide a scope for the resources. Access scope can help map alerts to resources when resources in different scopes share the same parameters, such as matchTokens. Optional.
Tip: You can define access scope for locations, project names, namespaces, etc.
Bulk load job Select either true or false to distribute the loading of job data more evenly across partitions. This is useful if some jobs contain substantially more data than others. The default is 'false' for single-file jobs, and 'true' for multi-file jobs. Optional
Generate debug support file Set this parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer. Optional
Observer job description Enter additional information to describe the job Optional
Job schedule Specify when the job runs Optional. Load jobs only.

Swagger API documentation

For the Swagger API documentation, see Topology File Observer API. The documentation explains how to use the File Observer APIs.

File Observer API access

Two public APIs are available, and both APIs can be used to retrieve data.

  • API name

    topology-file-observer

  • Public API URLs 

    /aiops/api/application-manager/topology-file-observer/v1/*

    /aiops/api/app/topology-file-observer/v1/*

For information on how to obtain route and token information, see Accessing the console.

For information on how to obtain your token, see Accessing APIs.

Example API queries

GET /health API 

curl -k -X GET --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" https://$ROUTE/aiops/api/application-manager/topology-file-observer/v1/healthcheck
{"date":"2021-02-05T19:24:00.974+00:00","jobs":0,"threads":10,"items":[{"type":"load"}]}

curl -k -X GET --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" https://$ROUTE/aiops/api/app/topology-file-observer/v1/healthcheck
{"date":"2021-02-05T19:24:00.974+00:00","jobs":0,"threads":10,"items":[{"type":"load"}]}

GET /service/info API 

curl -k -X GET --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://$ROUTE/aiops/api/application-manager/topology-file-observer/v1/service/info"
{"startDate":"2021-02-04T10:30:56.991+00:00","id":"example-noi-topology-file-observer-6f88fb6fc4-n9xm2","name":"file-observer","version":"1.1.0.17 66dd27a","adminConnector":"https://example-noi-topology-file-observer-6f88fb6fc4-n9xm2:9099","appConnector":"https://example-noi-topology-file-observer-6f88fb6fc4-n9xm2:9098","uptimeMillis":118575702}

curl -k -X GET --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://$ROUTE/aiops/api/app/topology-file-observer/v1/service/info"
{"startDate":"2021-02-04T10:30:56.991+00:00","id":"example-noi-topology-file-observer-6f88fb6fc4-n9xm2","name":"file-observer","version":"1.1.0.17 66dd27a","adminConnector":"https://example-noi-topology-file-observer-6f88fb6fc4-n9xm2:9099","appConnector":"https://example-noi-topology-file-observer-6f88fb6fc4-n9xm2:9098","uptimeMillis":118575702}

Upload a file 

curl -k -X POST --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" https://$ROUTE/aiops/api/app/topology-file-observer/v1/files --form 'job_file=@/tmp/file.json.txt' --header 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255'

Creating cross-data origin relationships

Using the File Observer functionality, you can create edges between resources from different observations or providers.

It requires both the uniqueID and the provider of the resources at each end of the link (so typically applies to previously observed resources).

In the following syntax examples, note the _toProvider and _fromProvider edge properties.

You can define an edge from a resource in the current observation to one owned by a different provider:

V:{"uniqueId":"host1.ibm.com","entityTypes":["host"],"name":"host1","_references":[{"_edgeType":"connectedTo","_toProvider":"someOtherProvider","_toUniqueId":"host2.ibm.com"}]}

You can define an edge from a resource between two different providers:

E:{"_fromProvider":"providerOne","_fromUniqueId":"host1-interface1","_edgeType":"connectedTo","_toProvider":"providerTwo","_toUniqueId":"host2-interface1"}

File Observer sample files

You can upload sample file data to the topology service on File Observer, and then view this data in the topology. To download the sample files, go to Sample files.