Monitoring Agent for InfoSphere DataStage metrics

The metrics for Monitoring Agent for InfoSphere DataStage resource types collect data for monitoring with IBM Cloud Pak for Multicloud Management. Every Monitoring Agent for InfoSphere DataStage resource type defines a set of dimensions and metrics. The descriptions provide such information as data type, dimension key, and metric unit.

Resource dataStageEngine

Component where actual job run takes place. The following section lists the metrics, dimensions and components of Resource dataStageEngine.

Metrics

The following metrics are available for the resource.

Engine Status

• Status of the engine on the basis of services.If any service is down status is critical otherwise it is normal.
• The type is string.
• The unit is status.

Service Disabled Count

• Count of services with Disabled status.
• The type is int.
• The unit is count.

Service Down Status

• Count of services with Not running status.
• The type is string.
• The unit is status.

Service Up Count

• Count of services which are running.
• The type is int.
• The unit is count.

timestamp

• This is the local time when the data was collected. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is timestamp.
• The unit is timestamp.

Dimensions

The following dimensions are available for the resource.

Agent Hostname

• The hostname of the system where agent is installed.
• The type is string.

Agent Instance Name

• It shows the name of agent instance.
• The type is string.

DatStage Engine Rest Hostname

• The hostname of system where service tier is installed.
• The type is string.

Engine Name

• Name of the engine.
• The type is string.

Engine Origin Node

• Engine Origin Node.
• The type is string.

Component: engineGeneralProperties

it shows properties of the engine nodes, such as operating system, CPU configuration, and memory configuration. The following section lists the metrics and dimensions of Component engineGeneralProperties.

Metrics

The following metrics are available for the component.

Engine General Properties Number Of Cpus

• The number of CPUs connected to the host system.
• The type is int.
• The unit is count.

EngineGeneralProperties HostId

• The ID of the host system on which project is created for this job. This is available for thresholds, but not in the UI.
• The type is int.
• The unit is Id.

Physical Memory

• The size, in gigabytes, of the physical memory of the host system.
• The type is double.
• The unit is gigabytes.

Dimensions

The following dimensions are available for the component.

CPU Model

• The descriptive name of the hardware.
• The type is string.

Engine General Properties HostName

• The name of the DataStage engine or the remote system.
• The type is string.

Engine General Properties Node

• nodename.
• The type is string.

EngineGeneralProperties PlatformName

• The descriptive name of the engine node. This is available for thresholds, but not in the UI.
• The type is string.

EngineGeneralProperties Timestamp

• This is the local time when the data was collected. This is available for thresholds, but not in the UI.
• The type is timestamp.

Platform Version

• The version of the operating system.
• The type is string.

Component: engineJobActivity

It contains information about the historical trend of number of jobs that were in running state at a particular time across all engines, for a given time interval.For more details about jobs on a particular engine, click on this GW to navigate to Job Run Summary by Engine page. The following section lists the metrics and dimensions of Component engineJobActivity.

Metrics

The following metrics are available for the component.

Engine Job Activity Job Count

• Count of jobs.
• The type is int.
• The unit is count.

EngineJobActivity Timestamp

• This is the local time when the data was collected. This is available for thresholds, but not in the UI.
• The type is timestamp.
• The unit is milliseconds.

Dimensions

The following dimensions are available for the component.

Engine Job Activity HostName

• The host name of the system where DataStage engine tier is installed..
• The type is string.

Engine Job Activity Node

• engineJobActivity_node.
• The type is string.

Component: engineJobConfiguration

It contains an configuration information about the job such as the name, project that name that the job was designed in, type of job(server, parallel, sequence)type,whether the job is multi instance or not,latest compilation time of the job in case of jobs that have been compiled multiple times. The following section lists the metrics and dimensions of Component engineJobConfiguration.

Metrics

The following metrics are available for the component.

Engine Job Configuration Compilation Timestamp

• The compilation time for the version of the job executable that is running.
• The type is string.
• The unit is timestamp.

Dimensions

The following dimensions are available for the component.

Engine Job Configuration HostName

• The host name of the system where DataStage engine tier is installed.
• The type is string.

Engine Job Configuration Is Multiinstance

• It is set to Yes if the job is a multi-instance job. It is set to No if the job is not a multi-instance job.
• The type is string.

Engine Job Configuration Job Type

• The type of the job.
• The type is string.

Engine Job Configuration JobId

• The ID used while creating the job.
• The type is string.

Engine Job Configuration JobName

• The name of the job, which is unique within the project.
• The type is string.

Engine Job Configuration Node

• engineJobConfiguration_node.
• The type is string.

Engine Job Configuration Project Name

• The project that the executable was run from.
• The type is string.

engineJobConfiguration hostId

• The ID of the host system on which project is created for this job. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

engineJobConfiguration timestamp

• This is the local time when the data was collected. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is timestamp.

Component: engineJobRunDetails

It contains information related to JobRuns. The following section lists the metrics and dimensions of Component engineJobRunDetails.

Metrics

The following metrics are available for the component.

Engine Job Run Details Elapsed Run Secs

• The length of time in seconds that the job ran for.
• The type is int.
• The unit is seconds.

Engine Job Run Details Minor Status

• The detailed status of the job run. The column is updated when the first warning or fatal message appears.
• The type is string.
• The unit is status.

Engine Job Run Details Total Cpu

• The total usage of CPU, in milliseconds, for all stages and control processes.
• The type is int.
• The unit is milliseconds.

Engine Job Run Details Total Logical Nodes

• For parallel jobs, the number of different logical node names found in the configuration file.
• The type is int.
• The unit is names.

Engine Job Run Details Total Physical Nodes

• For parallel jobs, the number of different physical node names (fastnames) found in the configuration file.
• The type is int.
• The unit is names.

Engine Job Run Details Total Rows Consumed

• The total row counts of data from the source stage links that needs to be processed.
• The type is int.
• The unit is rows.

Engine Job Run Details Total Rows Produced

• The total row counts of data from the target stage links that was processed successfully.
• The type is int.
• The unit is rows.

Dimensions

The following dimensions are available for the component.

Configuration ID

• It is used in the case of parallel job runs to get information about parallel configuration. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is int.

DataStageEngine Major Status

• The status of job run : Submitted, Started or Finished. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

DataStageEngine Run Type

• The type of job run: Run,Validate or Reset. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

DS User Name

• The credential-mapped InfoSphere DataStage user. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

Engine Job Run Details HostName

• The host name of the system where DataStage engine tier is installed.
• The type is string.

Engine Job Run Details Job Name

• The name of the job, which is unique within the project.
• The type is string.

Engine Job Run Details JobId

• The job ID that was used while creating the job.
• The type is string.

Engine Job Run Details Node

• Node.
• The type is string.

Engine Job Run Details Queue Name

• The name of the workload management queue that the job run was submitted to.
• The type is string.

Engine Job Run Details Run End Timestamp

• The time when the job execution was completed. If job is running or in waiting queue, the value is null.
• The type is string.

Engine Job Run Details Run Start Timestamp

• The time that the job execution started. Otherwise, the value is null.
• The type is string.

Engine Job Run Details RunId

• The run ID of the job that is running or that completed execution with some return code.
• The type is string.

Engine Job Run Details Timestamp

• This is the local time when the data was collected.
• The type is timestamp.

engineJobRunDetails controllingRunid

• The job ID of the parent job that started the execution of the current job. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

engineJobRunDetails creationTimestamp

• The time that is set when the run event for the job is created by the engine. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

EngineJobRunDetails Last Update Timestamp

• The time that this JobRun details were captured by Datastage. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

engineJobRunDetails numMessagesWarning

• The number of warnings that are logged by the job run. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

Invocation ID

• For a multi-instanced job, it is a string that identifies the instance. For a job that is not multi-instanced, it is N/A. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

IS User Name

• The Information Server user that started the job run. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

Number Messages Fatal

• The number of fatal messages that are logged by the job run. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

Number Messages Total

• The number of messages that are logged by the job run. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

Component: ENGINESERVICESTATUS

The Engine Service Status data set provides information about the status of the service. The following section lists the metrics and dimensions of Component ENGINESERVICESTATUS.

Metrics

The following metrics are available for the component.

Engine Service Status Error Description

• It describes about the 'Not running ' status of the service.
• The type is string.
• The unit is description.

Engine Service Status Status Message

• The status of the service, such as OK or Not Running.
• The type is string.
• The unit is status.

Dimensions

The following dimensions are available for the component.

Engine Service Status node

• engineServiceStatus node.
• The type is string.

Engine Service Status Service Name

• The name of the service.
• The type is string.

engineServiceStatus timestamp

• This is the local time when the data was collected. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is timestamp.

Component: engineSystemResources

It contains information about the engine nodes, such as the name of the engine, percentage of CPU used, percentage of physical memory used by the node. The following section lists the metrics and dimensions of Component engineSystemResources.

Metrics

The following metrics are available for the component.

Engine System Resources Cpu Percentage Used

• Used CPU of the Host System in percentage.
• The type is double.
• The unit is percent.

Engine System Resources Physical Memory Percentage Used

• Used Physical memory of the Host System in percentage.
• The type is double.
• The unit is percentage.

EngineSystemResources Timestamp

• This is the local time when the data was collected. This is available for thresholds, but not in the UI.
• The type is timestamp.
• The unit is timestamp.

Dimensions

The following dimensions are available for the component.

Engine System Resources HostId

• The ID of the host system on which project is created for this job.
• The type is string.

Engine System Resources Hostname

• The host name of the system where DataStage engine tier is installed.
• The type is string.

Engine System Resources Node

• Node.
• The type is string.

Resource dataStageServiceTier

Server which binds multiple DataStage Engines. The following section lists the dimensions and components of Resource dataStageServiceTier.

Dimensions

The following dimensions are available for the resource.

Agent Hostname

• The hostname of the system where agent is installed.
• The type is string.

Agent Instance Name

• It shows the name of agent instance.
• The type is string.

Database Hostname

• The hostname of the system where metadata repository tier is installed.
• The type is string.

Database Type

• The database server which is used as Metadata repository.
• The type is string.

DB IP

• The IP address of the system where metadata repository tier is installed. This is available for thresholds, but not in the UI.
• The type is string.

dbPort

• The port on the system where metadata repository database service is available. This is available for thresholds, but not in the UI.
• The type is string.

FQDN

• The fully qualified name of DataStage machine.
• The type is string.

Node

• agent nodename.
• The type is string.

Rest Hostname

• The hostname of system where service tier is installed.
• The type is string.

restIp

• The IP address of system where service tier is installed. This is available for thresholds, but not in the UI.
• The type is string.

restPort

• The port on the system where service tier service is available. This is available for thresholds, but not in the UI.
• The type is string.

timestamp

• This is the local time when the data was collected. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

Component: jobDetailsForJobRunSummary

It contains configuration information about the job such as project name that the job was designed in, type of job(server, parallel, sequence)type, whether the job is multi instance or not. The following section lists the metrics and dimensions of Component jobDetailsForJobRunSummary.

Metrics

The following metrics are available for the component.

Job Details For Job Run Summary Compilation Timestamp

• The compilation time for the version of the job executable that is running.
• The type is string.
• The unit is timestamp.

Dimensions

The following dimensions are available for the component.

Job Details For Job Run Summary Is Multiinstance

• It is set to Yes if the job is a multi-instance job. It is set to No if the job is not a multi-instance job.
• The type is string.

Job Details For Job Run Summary Job Type

• The type of the job.
• The type is string.

Job Details For Job Run Summary JobId

• The ID used while creating the job.
• The type is string.

Job Details For Job Run Summary Node

• jobDetailsForJobRunSummary_node.
• The type is string.

Job Details For Job Run Summary Project Name

• The project that the executable was run from.
• The type is string.

Job Details ForJ ob Run Summary Job Name

• The name of the job, which is unique within the project.
• The type is string.

jobDetailsForJobRunSummary_hostId

• The ID of the host system on which project is created for this job. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is int.

jobDetailsForJobRunSummary_hostName

• The host name of the system where DataStage engine tier is installed. This is available for thresholds, but not in the UI.
• The type is string.

jobDetailsForJobRunSummary_timestamp

• This is the local time when the data was collected. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is timestamp.

Component: jobRunLogs

It contains an information about all the events that are logged for a job run such as type of the log, full text of the logged message, and timestamp. The following section lists the metrics and dimensions of Component jobRunLogs.

Metrics

The following metrics are available for the component.

JobRunLogs EventId

• A number that differs according to the log message which can be used to sort log events for a job run. This is available for thresholds, but not in the UI.
• The type is int.
• The unit is Id.

Dimensions

The following dimensions are available for the component.

Job Run Logs Message Text

• The full text of the logged message.
• The type is string.

Job Run Logs node

• agent nodename.
• The type is string.

Job Run Logs RunId

• The run ID of the job that is running or that completed execution with some return code.
• The type is string.

Job Run Logs Timestamp

• The time, to the nearest second, when the log event was generated as output by the run.
• The type is string.

jobRunLogs timestamp

• This is the local time when the data was collected. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is timestamp.

Component: jobRunParameters

It contains an information about the parameters used for configuring a job or parameters with which a job was configured such as param name and value. The following section lists the dimensions of Component jobRunParameters.

Dimensions

The following dimensions are available for the component.

Job Run Parameters Node

• Not Available.
• The type is string.

Job Run Parameters Param Name

• The name of the parameter.
• The type is string.

Job Run Parameters Param Value

• The value of the parameter.
• The type is string.

Job Run Parameters RunId

• The run ID of the job that is running or that completed execution with some return code.
• The type is string.

jobRunParameters timestamp

• This is the local time when the data was collected. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is timestamp.

Component: runDetails

It provides a table that contains run specific information such as run id, start time of the run, end time of the run,total physical nodes, total logical nodes and the queue name to which the job is assigned. The following section lists the metrics and dimensions of Component runDetails.

Metrics

The following metrics are available for the component.

Configuration ID

• It is used in the case of parallel job runs to get information about parallel configuration. This is available for thresholds, but not in the UI.
• The type is int.
• The unit is id.

Controlling Run ID

• The job ID of the parent job that started the execution of the current job. This is available for thresholds, but not in the UI.
• The type is int.
• The unit is id.

Number Messages Fatal

• The number of fatal messages that are logged by the job run. This is available for thresholds, but not in the UI.
• The type is int.
• The unit is count.

Number Messages Total

• The number of messages that are logged by the job run. This is available for thresholds, but not in the UI.
• The type is int.
• The unit is count.

Number Messages Warning

• The number of warnings that are logged by the job run. This is available for thresholds, but not in the UI.
• The type is int.
• The unit is count.

Run Details Elapsed Run Secs

• The length of time in seconds that the job ran for.
• The type is int.
• The unit is secs.

Run Details Minor Status Name

• The detailed status of the job run. The column is updated when the first warning or fatal message appears.
• The type is string.
• The unit is status.

Run Details Run End Timestamp

• The time when the job execution was completed. If job is running or in waiting queue, the value is null.
• The type is string.
• The unit is timestamp.

Run Details Run Start Timestamp

• The time that the job execution started.If the job is in waiting queue, the value is null.
• The type is string.
• The unit is timestamp.

Run Details Total Cpu

• The total usage of CPU, in milliseconds, for all stages and control processes.
• The type is int.
• The unit is milliseconds.

Run Details Total Logical Nodes

• For parallel jobs, the number of different logical node names found in the configuration file.
• The type is int.
• The unit is count.

Run Details Total Physical Nodes

• For parallel jobs, the number of different physical node names (fastnames) found in the configuration file.
• The type is int.
• The unit is count.

Total Rows Consumed

• The total row counts of data from the source stage links that needs to be processed.
• The type is int.
• The unit is count.

Total Rows Produced

• The total row counts of data from the target stage links that was processed successfully.
• The type is int.
• The unit is count.

Dimensions

The following dimensions are available for the component.

Creation Timestamp

• The time that is set when the run event for the job is created by the engine. This is available for thresholds, but not in the UI.
• The type is timestamp.

DS User Name

• The credential-mapped InfoSphere DataStage user. This is available for thresholds, but not in the UI.
• The type is string.

Invocation ID

• For a multi-instanced job, it is a string that identifies the instance. For a job that is not multi-instanced, it is N/A. This is available for thresholds, but not in the UI.
• The type is string.

IS User Name

• The Information Server user that started the job run. This is available for thresholds, but not in the UI.
• The type is string.

Last Update Timestamp

• The time that this JobRun details were captured by Datastage. This is available for thresholds, but not in the UI.
• The type is string.

Major Status

• The status of job run : Submitted, Started or Finished. This is available for thresholds, but not in the UI.
• The type is string.

Run Details Host Name

• The host name of the system where DataStage engine tier is installed.
• The type is string.

Run Details Job ID

• The job ID that was used while creating the job.
• The type is string.

Run Details Job Name

• The name of the job, which is unique within the project.
• The type is string.

Run Details Node

• Not Available.
• The type is string.

Run Details Queue Name

• The name of the workload management queue that the job run was submitted to.
• The type is string.

Run Details RunId

• The run ID of the job that is running or that completed execution with some return code.
• The type is string.

Run Details Timestamp

• This is the local time when the data was collected.
• The type is timestamp.

Run Type

• The type of job run: Run,Validate or Reset. This is available for thresholds, but not in the UI.
• The type is string.

Component: runStageDetails

It provides a table that contains run specific information such as name of the stage, type of the stage, CPU usage(milliseconds), run duration, number of instances. The following section lists the metrics and dimensions of Component runStageDetails.

Metrics

The following metrics are available for the component.

Run Stage Details Elapsed Run Secs

• The length of time in seconds that the stage ran for. The length of time is calculated from the StageStartTimeStamp and the StageEndTimeStamp columns.
• The type is int.
• The unit is seconds.

Run Stage Details Number Of Instances

• For parallel jobs, the number of instances of the stage that were used. For server jobs, it is always 1.
• The type is int.
• The unit is count.

Run Stage Details RunId

• The run ID of the job that is running or that completed execution with some return code.
• The type is string.
• The unit is id.

Run Stage Details Total Cpu

• The total CPU time in milliseconds consumed by the job stage.
• The type is int.
• The unit is milliseconds.

RunStage Details Stage Status

• The status of the stage:Ready,Running,Finished.
• The type is string.
• The unit is status.

Dimensions

The following dimensions are available for the component.

Job Name(Job Stage)

• The name of the job, which is unique within the project.
• The type is string.

Run Stage Details JobId

• The Job ID used while creating the job. This is not available in the UI by default. It can be made available through dynamic configuration.
• The type is string.

Run Stage Details node

• Not Available.
• The type is string.

Run Stage Details Stage Name

• The name of the stage.
• The type is string.

Run Stage Details Stage Type

• The stage type of the stage, as shown in the Designer client.
• The type is string.

Run Stage Details StageId

• A unique key that represents stage of job creation.
• The type is string.

RunStageDetails Timestamp

• This is the local time when the data was collected. This is available for thresholds, but not in the UI.
• The type is timestamp.

Stage Description

• The stage description. This is available for thresholds, but not in the UI.
• The type is string.

Stage End TimeStamp

• The time when the stage is finished. If the stage is still running, this column is set to null. This is available for thresholds, but not in the UI.
• The type is string.

Stage Start TimeStamp

• The time when the stage is started. This is available for thresholds, but not in the UI.
• The type is string.

Component: serviceTierJobActivitySummary

it contains information about the historical trend of number of jobs that were in running state at a particular time across all engines. The following section lists the metrics and dimensions of Component serviceTierJobActivitySummary.

Metrics

The following metrics are available for the component.

Service Tier Job Activity Summary Job Count

• Count of jobs.
• The type is int.
• The unit is count.

Dimensions

The following dimensions are available for the component.

Service Tier Job Activity Summary Node

• agent nodename.
• The type is string.

serviceTierJobActivitySummary timestamp

• This is the local time when the data was collected. This is available for thresholds, but not in the UI.
• The type is timestamp.