Configuring File Observer jobs

Using the File Observer functionality, you can write bespoke data to a file in a specific format, upload this data to the topology service, and then visualize this data as a topology view.

Before you begin

The File Observer reads topology data from specified files, and uploads it. You must create these files manually.

About this task

Topology data in a file is composed of vertices (nodes) and edges. A vertex represents an object (resource), while an edge represents the relationship between two objects.

Each line of the file you create should be in one of the following formats, loading a single resource vertex (including optional relationships in the '_references' field) or a single edge, deleting a single vertex, or pausing execution.

Lines starting with V: (vertex), E: (edge), D: (delete) or W: (wait) are treated as instruction lines to be processed. Other lines, for example lines that are empty or commented out, are ignored.

Line format

V - Load a resource vertex, with a JSON representation as documented for the body of the topology service API method: POST /resources

If specifying the '_status' element, acceptable state values are open, closed, or clear, and acceptable severity values are clear, indeterminate, warning, minor, major, or critical.

E - Load an edge, with a JSON representation as documented for the '_references' section of the body of the topology service API method POST /resources

D - Delete a resource vertex, identified by it's uniqueId.

W - Pause for the duration (for testing purposes only). Takes an integer period followed by a string specifying the units.

G - To load a group via the File Observer, see the following example syntax:

G:{"uniqueId":"EA","entityTypes":["group"],"name":"Event Analytics"}

Note: When creating a group's edge, it must be based on uniqueId.

Important: Ensure that the file is structured correctly. For each line of the file, information included after the closing } that matches an opening { is ignored, and no error is recorded.

Mapping alerts to topologies When using the File Observer to correlate events from Logs Anomaly, ensure you use the matchTokens property when creating topology templates. The matchTokens property is a list of values you can use to look up a resource from an alert. If a unique matchToken is found that matches an alert's resource fields, such as resource.name or resource.sourceId, the alert will be assigned to that resource.

File Observer syntax

When uploaded to the file observer in a file, say example.txt, the following small topology can be generated by running a File Observer job for that file. The figure also shows an alert matched to a resource.

Figure that shows a small topology generated by the File Observer — Figure. File Observer topology

V:{"uniqueId":"Thomas J. Watson","entityTypes":["person"],"name":"Watson"}
V:{"uniqueId":"IBM","entityTypes":["company","domain"],"name":"IBM","_references":[{"_edgeType":"owns","_toUniqueId":"Watson AIOps"},{"_edgeType":"contributedTo", "_fromUniqueId":"Thomas J. Watson"}]}
V:{"uniqueId":"Watson AIOps","entityTypes":["ai"],"name":"Watson","platform":"OCP","matchTokens":["AIOps","Watson AIOps"]}

Line 1: This line has the minimal set of properties for a resource vertex:

uniqueId - Uniquely identifies the resource within the provider namespace (the provider is supplied when running the job).
entityTypes - Determines the icon displayed in the UI. Some types have a pre-defined mapping to icons, as described in the Entity types topic.
name - The label displayed in the UI.

Line 2: This line shows multiple entityTypes. There is no default icon mapped to type company, so the icon is determined from the domain type. In addition to the properties of the resource, this adds an edge to another resource.

The edgeType must exist already. Either one of the default Edge types or one created by the topology service's POST /types/edge API (see Accessing APIs)
The referenced uniqueId does not have to exist yet.
Multiple edges can be added in the _references list.

Line 3: This line shows that more data or properties can be added. It adds a custom platform property (might be any additional property required). See the Properties topic for a list of existing properties. This line also illustrates that any entityType can be supplied, and that the name does not have to be unique. The UI does not show an icon for this resource because no default icon mapping has been created. You can add custom mapping through the UI as described in the Defining custom icons topic. And finally, this line also adds a matchToken property, which is a list of values that can be used to link a resource to an alert.

File Load job

By default, Load jobs are one-off, transient jobs that upload all of the requested topology data as soon as they are triggered.

You can also run these jobs (again) from the Observer UI, or schedule them to run at set times when configuring them.

Procedure

Define or edit the following parameters, then click Save to save and run the job.

Encryption requirement: See the Configuring observer jobs security topic for more information.

Parameter	Action	Details
Unique ID	Enter a unique name for the job	Required.
File Mode	Select either Upload local file, then drag and drop (or browse) to the input file; or Use server file, then specify the name of the file previously uploaded through the `POST /files` API.	Required. Note: If the same file name is used in a second job, or the file is replaced through `PUT /files`, any resources missing from the second load will be deleted.
Provider	Enter the job's provider, that is, a namespace for the resources' uniqueIds. Resources can be loaded from multiple files with the same provider, and linked as if they were all loaded from the same file. To link to resources from another provider (for example, from another observer), the cross-provider semantics are required.	Optional. Defaults to `FILE.OBSERVER:{fileName}` if not supplied.
Generate debug support file	Set this parameter to 'True' in order to capture the output of the next scheduled job run as a file. This file will be stored with an observer's log files and can be used to debug observer issues, for example at the request of your designated Support team, or while using a test environment. For one-off jobs (that is, Load jobs), this parameter reverts to 'False' after the next completed run. To examine the output produced, you can load the generated debug file using the File Observer.	Optional
Job schedule	Specify when the job runs	Optional. Load jobs only.
Observer job description	Enter additional information to describe the job	Optional

Swagger API documentation

For the Swagger API documentation, see Topology File Observer API. The documentation explains how to use the File Observer APIs.

File Observer API access

Two public APIs are available, and both APIs can be used to retrieve data.

API name

topology-file-observer

Public API URLs

/aiops/api/application-manager/topology-file-observer/v1/*

/aiops/api/app/topology-file-observer/v1/*

For information on how to obtain route and token information, see Accessing the console.

For information on how to obtain your token, see Accessing APIs.

Example API queries

GET /health API

curl -k -X GET --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" https://$ROUTE/aiops/api/application-manager/topology-file-observer/v1/healthcheck
{"date":"2021-02-05T19:24:00.974+00:00","jobs":0,"threads":10,"items":[{"type":"load"}]}

curl -k -X GET --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" https://$ROUTE/aiops/api/app/topology-file-observer/v1/healthcheck
{"date":"2021-02-05T19:24:00.974+00:00","jobs":0,"threads":10,"items":[{"type":"load"}]}

GET /service/info API

curl -k -X GET --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://$ROUTE/aiops/api/application-manager/topology-file-observer/v1/service/info"
{"startDate":"2021-02-04T10:30:56.991+00:00","id":"example-noi-topology-file-observer-6f88fb6fc4-n9xm2","name":"file-observer","version":"1.1.0.17 66dd27a","adminConnector":"https://example-noi-topology-file-observer-6f88fb6fc4-n9xm2:9099","appConnector":"https://example-noi-topology-file-observer-6f88fb6fc4-n9xm2:9098","uptimeMillis":118575702}

curl -k -X GET --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://$ROUTE/aiops/api/app/topology-file-observer/v1/service/info"
{"startDate":"2021-02-04T10:30:56.991+00:00","id":"example-noi-topology-file-observer-6f88fb6fc4-n9xm2","name":"file-observer","version":"1.1.0.17 66dd27a","adminConnector":"https://example-noi-topology-file-observer-6f88fb6fc4-n9xm2:9099","appConnector":"https://example-noi-topology-file-observer-6f88fb6fc4-n9xm2:9098","uptimeMillis":118575702}

Upload a file

curl -k -X POST --header 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" https://$ROUTE/aiops/api/app/topology-file-observer/v1/files --form 'job_file=@/tmp/file.json.txt' --header 'X-TenantID: cfd95b7e-3bc7-4006-a4a8-a73a79c71255'

Creating cross-data origin relationships

Using the File Observer functionality, you can create edges between resources from different observations or providers.

It requires both the uniqueID and the provider of the resources at each end of the link (so typically applies to previously observed resources).

In the following syntax examples, note the _toProvider and _fromProvider edge properties.

You can define an edge from a resource in the current observation to one owned by a different provider:

V:{"uniqueId":"host1.ibm.com","entityTypes":["host"],"name":"host1","_references":[{"_edgeType":"connectedTo","_toProvider":"someOtherProvider","_toUniqueId":"host2.ibm.com"}]}

You can define an edge from a resource between two different providers:

E:{"_fromProvider":"providerOne","_fromUniqueId":"host1-interface1","_edgeType":"connectedTo","_toProvider":"providerTwo","_toUniqueId":"host2-interface1"}

File Observer sample files

You can upload sample file data to the topology service on File Observer, and then view this data in the topology. To download the sample files, go to Sample files.