Monitoring Agent for Hadoop metrics

The metrics for Hadoop resource types collect data for monitoring with IBM Cloud Pak for Multicloud Management. Every Hadoop resource type defines a set of dimensions and metrics. The descriptions provide such information as data type, dimension key, and metric unit.

Resource hadoopCluster

A set of nodes that orchestrate and execute various jobs across the Hadoop distributed file system. The following section lists the metrics, dimensions and components of Resource hadoopCluster.

Metrics

The following metrics are available for the resource.

Average MemoryHeapUsed Percent

Capacity Total GB

Capacity Used GB

Capacity Used Percent

Count of Active Node Managers

Count of Dead Datanodes

Count of Decommissioned Node Managers

Count of Live Datanodes

Count of Lost Node Managers

Count of Rebooted Node Managers

Count of Unhealthy Node Managers

Process CPULoad Percentage

Dimensions

The following dimensions are available for the resource.

Applications Failed

Cluster ID

Containers Failed

Count of Nodes

Custom Cluster Name

Ha State

Missing Blocks

Namenode uptime

OriginNode

Timestamp

Total Disk failures

Component: hadoopClusterJobsDetails

Hadoop Cluster Jobs Details. The following section lists the metrics and dimensions of Component hadoopClusterJobsDetails.

Metrics

The following metrics are available for the component.

Failed Map Tasks

Failed Map Tasks Percent

Failed Reduce Tasks

Failed Reduce Tasks Percent

Finish Time

Killed Map Tasks

Killed Map Tasks Percent

Killed Reduce Tasks

Killed Reduce Tasks Percent

Map Output Record Spills

Maps Total

Reduces Total

Start Time

Submit Time

Succeeded Map Tasks

Succeeded Map Tasks Percent

Succeeded Reduce Tasks

Succeeded Reduce Tasks Percent

Dimensions

The following dimensions are available for the component.

Job ID

Job Name

Jobs Details Timestamp

Node

Queue

State

Subnode Host

User

Component: hadoopServices

Hadoop Services. The following section lists the metrics and dimensions of Component hadoopServices.

Metrics

The following metrics are available for the component.

Desired State

Hadoop Service Component State

Init Count in Cluster

Init Count on Host

Install failed Count in Cluster

Install failed Count on Host

Installed Count in Cluster

Installed Count on Host

Other Count in Cluster

Other Count on Host

Service Status

Started Count in Cluster

Started Count on Host

Total Count in Cluster

Total Count on Host

Unknown Count in Cluster

Unknown Count on Host

Dimensions

The following dimensions are available for the component.

Cluster Name

Component Name

Custom Cluster Name

Custom State

Desired Stack Id

HDP Version

Hostname

Maintenance State

Node

Service Name

Stack Id

Stale Configs

Timestamp

Upgrade State

Resource hadoopHost

Provides information about Hadoop Host. The following section lists the dimensions and components of Resource hadoopHost.

Dimensions

The following dimensions are available for the resource.

Cluster ID

Daemon

DataNode Status

File System Status

Hostname

IP Address

Java Virtual Machine Status

Node FQDN

OriginNode

Port

Queue Status

Remote Procedure Call Status

Roles Hostname

Roles OriginNode

Roles Timestamp

Subnode Host

Timestamp

Component: hadoopHostDataNodeMetrics

Component monitors the DataNode metrics of the Hadoop cluster. The following section lists the metrics and dimensions of Component hadoopHostDataNodeMetrics.

Metrics

The following metrics are available for the component.

DataNode Metrics Block Checksum Average Time ms

DataNode Metrics Block Checksum Operations

DataNode Metrics Block Reports Average Time ms

DataNode Metrics Block Reports Operations

DataNode Metrics Block Verification Failures

DataNode Metrics Blocks Cached

DataNode Metrics Blocks Local Path Information

DataNode Metrics Blocks Read

DataNode Metrics Blocks Removed

DataNode Metrics Blocks Replicated

DataNode Metrics Blocks Uncached

DataNode Metrics Blocks Verified

DataNode Metrics Blocks Written

DataNode Metrics Bytes Read (Rate)

DataNode Metrics Bytes ReadMB

DataNode Metrics Bytes Written (Rate)

DataNode Metrics Bytes WrittenMB

DataNode Metrics Cache Report

DataNode Metrics Cache Report Average Time ms

DataNode Metrics Copy Block Operation Average Time ms

DataNode Metrics Copy Block Operations

DataNode Metrics File Synchronization Count

DataNode Metrics File Synchronization Nanos Average Time

DataNode Metrics File Synchronization Nanos Operations

DataNode Metrics Flush Nanos Average Time

DataNode Metrics Flush Nanos Operations

DataNode Metrics Heartbeats Average Time ms (Rate)

DataNode Metrics Heartbeats Operations

DataNode Metrics Packet Round Trip Nanos Operations

DataNode Metrics Packet Round Trip Time Nanos Average Time

DataNode Metrics Read Block Operation

DataNode Metrics Read Block Operation Average Time ms

DataNode Metrics Reads From Local Client (Rate)

DataNode Metrics Reads From Remote Client (Rate)

DataNode Metrics Replace Block Average Time ms

DataNode Metrics Replace Block Operations

DataNode Metrics Send DataPacket Blocked On Network Nanos Average Time

DataNode Metrics Send DataPacket Blocked On Network Nanos Operations

DataNode Metrics Send DataPacket Transfer Nanos Average Time

DataNode Metrics Send DataPacket Transfer Nanos Operations

DataNode Metrics Volume Failures

DataNode Metrics Write Block Average Time ms

DataNode Metrics Write Block Operations

DataNode Metrics Writes From Local Client (Rate)

DataNode Metrics Writes From Remote Client (Rate)

Dimensions

The following dimensions are available for the component.

DataNode Metrics Collect Timestamp

DataNode Metrics Context

DataNode Metrics Hostname

DataNode Metrics OriginNode

DataNode Metrics Session Id

DataNode Metrics Timestamp

Component: hadoopHostFSNamesystemMetrics

Component monitors the Hadoop Distributed File System metrics of the Hadoop Cluster. The following section lists the metrics and dimensions of Component hadoopHostFSNamesystemMetrics.

Metrics

The following metrics are available for the component.

FSNamesystem Metrics Block Capacity

FSNamesystem Metrics Block Capacity MB

FSNamesystem Metrics Blocks Total

FSNamesystem Metrics Capacity Remaining Bytes

FSNamesystem Metrics Capacity Remaining in GB

FSNamesystem Metrics Capacity Total Bytes

FSNamesystem Metrics Capacity Total GB

FSNamesystem Metrics Capacity Used Bytes

FSNamesystem Metrics Capacity Used GB

FSNamesystem Metrics Capacity Used NonDFS Bytes

FSNamesystem Metrics Capacity Used NonDFS GB

FSNamesystem Metrics Capacity Used Percent

FSNamesystem Metrics Corrupt Blocks

FSNamesystem Metrics Corrupt Blocks Percent

FSNamesystem Metrics Excess Blocks

FSNamesystem Metrics Expired Heartbeats

FSNamesystem Metrics Files Total

FSNamesystem Metrics Last Checkpoint Time ms

FSNamesystem Metrics Last Loaded Edits

FSNamesystem Metrics Missing Blocks

FSNamesystem Metrics Missing Blocks Percent

FSNamesystem Metrics Pending DataNode Message Count

FSNamesystem Metrics Pending Deletion Blocks

FSNamesystem Metrics Pending Replication Blocks

FSNamesystem Metrics Postponed Misreplicated Blocks

FSNamesystem Metrics Scheduled Replication Blocks

FSNamesystem Metrics Snapshots

FSNamesystem Metrics Snapshottable Directories

FSNamesystem Metrics Stale DataNodes

FSNamesystem Metrics Total Files

FSNamesystem Metrics Total Load

FSNamesystem Metrics Transactions Since Last Checkpoint

FSNamesystem Metrics Transactions Since Last Rollback

FSNamesystem Metrics UnderReplicated Blocks

Dimensions

The following dimensions are available for the component.

FSNamesystem Metrics Collect Timestamp

FSNamesystem Metrics Context

FSNamesystem Metrics Has State

FSNamesystem Metrics Hostname

FSNamesystem Metrics OriginNode

FSNamesystem Metrics Timestamp

Last Written Transaction Id

Component: hadoopHostJobsDetails

Hadoop Host Jobs Details. The following section lists the metrics and dimensions of Component hadoopHostJobsDetails.

Metrics

The following metrics are available for the component.

Failed Map Tasks

Failed Map Tasks Percent

Failed Reduce Tasks

Failed Reduce Tasks Percent

Finish Time

Killed Map Tasks

Killed Map Tasks Percent

Killed Reduce Tasks

Killed Reduce Tasks Percent

Map Output Record Spills

Maps Total

Reduces Total

Start Time

Submit Time

Succeeded Map Tasks

Succeeded Map Tasks Percent

Succeeded Reduce Tasks

Succeeded Reduce Tasks Percent

Dimensions

The following dimensions are available for the component.

Job ID

Job Name

Jobs Details OriginNode

Jobs Details Subnode Host

Jobs Details Timestamp

Queue

State

User

Component: hadoopHostJVMMetrics

Component monitors the Java Virtual Machine (JVM) metrics of the Hadoop cluster. The following section lists the metrics and dimensions of Component hadoopHostJVMMetrics.

Metrics

The following metrics are available for the component.

Garbage Collection Rate

JVM Metrics Garbage Collection Count

JVM Metrics Garbage Collection Count Mark Sweep

JVM Metrics Garbage Collection Count Scavenge

JVM Metrics Garbage Collection Information Threshold

JVM Metrics Garbage Collection Sleep Time ms

JVM Metrics Garbage Collection Time Mark Sweep

JVM Metrics Garbage Collection Time ms

JVM Metrics Garbage Collection Time Scavenge ms

JVM Metrics Garbage Collection Warning Threshold

JVM Metrics Logs Error

JVM Metrics Logs Fatal

JVM Metrics Logs Information

JVM Metrics Logs Warn

JVM Metrics Memory HeapCommitted MB

JVM Metrics Memory HeapMaximum MB

JVM Metrics Memory HeapUsed MB

JVM Metrics Memory Maximum MB

JVM Metrics Memory NonHeap Committed MB

JVM Metrics Memory NonHeap Free MB

JVM Metrics Memory NonHeap Maximum MB

JVM Metrics Memory NonHeap Used MB

JVM Metrics Threads Blocked

JVM Metrics Threads New

JVM Metrics Threads Runnable

JVM Metrics Threads Terminated

JVM Metrics Threads Timed Waiting

JVM Metrics Threads Waiting

Memory Heap Used Percent

Dimensions

The following dimensions are available for the component.

JVM Metrics Collect Timestamp

JVM Metrics Context

JVM Metrics Hostname

JVM Metrics OriginNode

JVM Metrics Process Name

JVM Metrics Session Id

JVM Metrics Timestamp

Component: hadoopHostOperatingSystem

Compoenent monitors the Operating System information of Hadoop Hosts. The following section lists the metrics and dimensions of Component hadoopHostOperatingSystem.

Metrics

The following metrics are available for the component.

Operating System AvailableProcessors

Operating System ProcessCPULoad

Operating System SystemCpuLoad

Dimensions

The following dimensions are available for the component.

Operating System Hostname

Operating System Oracting System Collect Timestamp

Operating System OriginNode

Operating System Timestamp

Component: hadoopHostQueueMetrics

Component monitors the Queue metrics of the Hadoop cluster. The following section lists the metrics and dimensions of Component hadoopHostQueueMetrics.

Metrics

The following metrics are available for the component.

Queue Metrics Active Applications

Queue Metrics Active Users

Queue Metrics Aggregate Containers Allocated

Queue Metrics Aggregate Containers Released

Queue Metrics Allocated Containers

Queue Metrics Allocated Space MB

Queue Metrics Allocated Virtual Cores

Queue Metrics Applications Completed

Queue Metrics Applications Failed

Queue Metrics Applications Killed

Queue Metrics Applications Pending

Queue Metrics Applications Running

Queue Metrics Applications Submitted

Queue Metrics Available Space MB

Queue Metrics Available Virtual Cores

Queue Metrics Failed Applications Percent

Queue Metrics Fair Share MB

Queue Metrics Fair Share Virtual Cores

Queue Metrics Killed Applications Percent

Queue Metrics Maximum Share MB

Queue Metrics Maximum Share Virtual Cores

Queue Metrics Minimum Share MB

Queue Metrics Minimum Share Virtual Cores

Queue Metrics Pending Containers

Queue Metrics Pending Space MB

Queue Metrics Pending Virtual Cores

Queue Metrics Reserved Containers

Queue Metrics Reserved Space MB

Queue Metrics Reserved Virtual Cores

Queue Metrics Running 0

Queue Metrics Running 1440

Queue Metrics Running 300

Queue Metrics Running 60

Dimensions

The following dimensions are available for the component.

Queue Metrics Allocated Memory Percent

Queue Metrics Collect Timestamp

Queue Metrics Context

Queue Metrics Hostname

Queue Metrics OriginNode

Queue Metrics Queue Name

Queue Metrics Short Queue Name

Queue Metrics Timestamp

Queue Metrics User

Component: hadoopHostRPCMetrics

Component monitors the RPC metrics of the Hadoop cluster. The following section lists the metrics and dimensions of Component hadoopHostRPCMetrics.

Metrics

The following metrics are available for the component.

RPC Metrics Authentication Failures

RPC Metrics Authentication Successes

RPC Metrics Authorization Failures

RPC Metrics Authorization Successes

RPC Metrics Call Queue Length

RPC Metrics Open Connections

RPC Metrics Processed Requests Count

RPC Metrics Processing Average Time in minutes

RPC Metrics Processing Average Time ms

RPC Metrics Queue Average Time in minutes

RPC Metrics Queue Average Time ms

RPC Metrics Queued Requests Count

RPC Metrics Received Bytes

RPC Metrics Received Data MB

RPC Metrics Sent Bytes

RPC Metrics Sent Data MB

Dimensions

The following dimensions are available for the component.

RPC Metrics Collect Timestamp

RPC Metrics Context

RPC Metrics Hostname

RPC Metrics IP Address

RPC Metrics OriginNode

RPC Metrics Port

RPC Metrics Timestamp

Resource hadoopService

Provides the details of Amabari Supported Services. The following section lists the metrics, dimensions and components of Resource hadoopService.

Metrics

The following metrics are available for the resource.

Desired State

Init Count in Cluster

Init Count on Host

Install failed Count in Cluster

Install failed Count on Host

Installed Count in Cluster

Installed Count on Host

Other Count in Cluster

Other Count on Host

Started Count in Cluster

Started Count on Host

State

Total Count in Cluster

Total Count on Host

Unknown Count in Cluster

Unknown Count on Host

Dimensions

The following dimensions are available for the resource.

Cluster Name

Component Name

Custom Cluster Name

Custom State

Desired Stack Id

HDP Version

Hostname

Maintenance State

Node

Service Name

Service Status

Stack Id

Stale Configs

Timestamp

Upgrade State

Component: bigSQL

BigSQL. The following section lists the metrics and dimensions of Component bigSQL.

Metrics

The following metrics are available for the component.

BigSQL Init Count in Cluster

BigSQL Install failed Count in Cluster

BigSQL Installed Count in Cluster

BigSQL Other Count in Cluster

BigSQL Started Count in Cluster

BigSQL Total Count in Cluster

BigSQL Unknown Count in Cluster

Dimensions

The following dimensions are available for the component.

BigSQL Cluster Name

BigSQL Component Name

BigSQL Custom Cluster Name

BigSQL Desired Stack Id

BigSQL Desired State

BigSQL HDP Version

BigSQL Hostname

BigSQL Maintenance State

BigSQL Node

BigSQL Service Name

BigSQL Stack Id

BigSQL Stale Configs

BigSQL State

BigSQL Timestamp

BigSQL Upgrade State

Component: nifi

NiFi. The following section lists the metrics and dimensions of Component nifi.

Metrics

The following metrics are available for the component.

NiFi Init Count in Cluster

NiFi Install failed Count in Cluster

NiFi Installed Count in Cluster

NiFi Other Count in Cluster

NiFi Started Count in Cluster

NiFi Total Count in Cluster

NiFi Unknown Count in Cluster

Dimensions

The following dimensions are available for the component.

NiFi Cluster Name

NiFi Component Name

NiFi Custom Cluster Name

NiFi Desired Stack Id

NiFi Desired State

NiFi HDP Version

NiFi Hostname

NiFi Maintenance State

NiFi Node

NiFi Service Name

NiFi Stack Id

NiFi Stale Configs

NiFi State

NiFi Timestamp

NiFi Upgrade State

Component: nifiRegistry

NiFI Regitry. The following section lists the metrics and dimensions of Component nifiRegistry.

Metrics

The following metrics are available for the component.

NiFi Registry Init Count in Cluster

NiFi Registry Install failed Count in Cluster

NiFi Registry Installed Count in Cluster

NiFi Registry Other Count in Cluster

NiFi Registry Started Count in Cluster

NiFi Registry Total Count in Cluster

NiFi Registry Unknown Count in Cluster

Dimensions

The following dimensions are available for the component.

NiFi Registry Cluster Name

NiFi Registry Component Name

NiFi Registry Custom Cluster Name

NiFi Registry Desired Stack Id

NiFi Registry Desired State

NiFi Registry HDP Version

NiFi Registry Hostname

NiFi Registry Maintenance State

NiFi Registry Node

NiFi Registry Service Name

NiFi Registry Stack Id

NiFi Registry Stale Configs

NiFi Registry State

NiFi Registry Timestamp

NiFi Registry Upgrade State

Component: schemaRegistry

Schema Registry. The following section lists the metrics and dimensions of Component schemaRegistry.

Metrics

The following metrics are available for the component.

Schema Registry Init Count in Cluster

Schema Registry Install failed Count in Cluster

Schema Registry Installed Count in Cluster

Schema Registry Other Count in Cluster

Schema Registry Started Count in Cluster

Schema Registry Total Count in Cluster

Schema Registry Unknown Count in Cluster

Dimensions

The following dimensions are available for the component.

Schema Registry Cluster Name

Schema Registry Component Name

Schema Registry Custom Cluster Name

Schema Registry Desired Stack Id

Schema Registry Desired State

Schema Registry HDP Version

Schema Registry Hostname

Schema Registry Maintenance State

Schema Registry Node

Schema Registry Service Name

Schema Registry Stack Id

Schema Registry Stale Configs

Schema Registry State

Schema Registry Timestamp

Schema Registry Upgrade State

Component: streamingAnalyticsManager

Streaming Analytics Manager. The following section lists the metrics and dimensions of Component streamingAnalyticsManager.

Metrics

The following metrics are available for the component.

Streaming Analytics Manager Init Count in Cluster

Streaming Analytics Manager Install failed Count in Cluster

Streaming Analytics Manager Installed Count in Cluster

Streaming Analytics Manager Other Count in Cluster

Streaming Analytics Manager Started Count in Cluster

Streaming Analytics Manager Total Count in Cluster

Streaming Analytics Manager Unknown Count in Cluster

Dimensions

The following dimensions are available for the component.

Streaming Analytics Manager Cluster Name

Streaming Analytics Manager Component Name

Streaming Analytics Manager Custom Cluster Name

Streaming Analytics Manager Desired Stack Id

Streaming Analytics Manager Desired State

Streaming Analytics Manager HDP Version

Streaming Analytics Manager Hostname

Streaming Analytics Manager Maintenance State

Streaming Analytics Manager Node

Streaming Analytics Manager Service Name

Streaming Analytics Manager Stack Id

Streaming Analytics Manager Stale Configs

Streaming Analytics Manager State

Streaming Analytics Manager Timestamp

Streaming Analytics Manager Upgrade State

Component: superset

Superset. The following section lists the metrics and dimensions of Component superset.

Metrics

The following metrics are available for the component.

Superset Init Count in Cluster

Superset Install failed Count in Cluster

Superset Installed Count in Cluster

Superset Other Count in Cluster

Superset Started Count in Cluster

Superset Total Count in Cluster

Superset Unknown Count in Cluster

Dimensions

The following dimensions are available for the component.

Superset Cluster Name

Superset Component Name

Superset Custom Cluster Name

Superset Desired Stack Id

Superset Desired State

Superset HDP Version

Superset Hostname

Superset Maintenance State

Superset Node

Superset Service Name

Superset Stack Id

Superset Stale Configs

Superset State

Superset Timestamp

Superset Upgrade State

Component: unifiedConsole

Unified Console. The following section lists the metrics and dimensions of Component unifiedConsole.

Metrics

The following metrics are available for the component.

Unified Console Init Count in Cluster

Unified Console Install failed Count in Cluster

Unified Console Installed Count in Cluster

Unified Console Other Count in Cluster

Unified Console Started Count in Cluster

Unified Console Total Count in Cluster

Unified Console Unknown Count in Cluster

Dimensions

The following dimensions are available for the component.

Unified Console Cluster Name

Unified Console Component Name

Unified Console Custom Cluster Name

Unified Console Desired Stack Id

Unified Console Desired State

Unified Console HDP Version

Unified Console Hostname

Unified Console Maintenance State

Unified Console Node

Unified Console Service Name

Unified Console Stack Id

Unified Console Stale Configs

Unified Console State

Unified Console Timestamp

Unified Console Upgrade State

Resource hadoopServiceComponent

Provides the data for Hadoop Service Components. The following section lists the metrics, dimensions and components of Resource hadoopServiceComponent.

Metrics

The following metrics are available for the resource.

Desired State

Init Count in Cluster

Init Count on Host

Install failed Count in Cluster

Install failed Count on Host

Installed Count in Cluster

Installed Count on Host

Other Count in Cluster

Other Count on Host

Service Status

Started Count in Cluster

Started Count on Host

State

Total Count in Cluster

Total Count on Host

Unknown Count in Cluster

Unknown Count on Host

Dimensions

The following dimensions are available for the resource.

Cluster Name

Component Name

Custom Cluster Name

Custom State

Desired Stack Id

HDP Version

Hostname

Maintenance State

Node

Service Name

Stack Id

Stale Configs

Timestamp

Upgrade State