mmperfmon command

Configures the Performance Monitoring tool and lists the performance metrics.

Synopsis

  mmperfmon config generate --collectors CollectorNode[,CollectorNode...] 
                          [--config-file InputFile]

  mmperfmon config add --sensors SensorFile

  mmperfmon config add --apikey key_name[--filter FilterSpecFile]

  mmperfmon config update { [--collectors CollectorNode[,CollectorNode...] ]
                            [--config-file InputFile] [ Attribute=value ... ] }

  mmperfmon config update --apikey key_name[--filter FilterSpecFile]

  mmperfmon config delete {--all |--sensors  Sensor[,Sensor...] }

  mmperfmon config delete --apikey key_name

  mmperfmon config show [--config-file OutputFile]

  mmperfmon config show --apikey  [key_name | all]

mmperfmon query --list keystat

mmperfmon query netstat --sensor -b 30

  mmperfmon query  Metric[,Metric...] | Measurement | NamedQuery 
                  [StartTime EndTime ] [Options]

  mmperfmon query compareNodes ComparisonMetric [StartTime EndTime] [Options]

  mmperfmon report top [StartTime EndTime ] [Options]

  mmperfmon delete {--expiredKeys | --key Key[,Key...]} [--wait seconds] [--force]

Availability

Available on all IBM Storage Scale editions.

The protocol functions provided in this command, or any similar command, are generally referred to as CES (Cluster Export Services). For example, protocol node and CES node are functionally equivalent terms.

Description

The mmperfmon config command modifies the performance monitoring tool by updating the configuration that is stored in IBM Storage Scale. It can be used to generate an initial configuration to update reporting periods of different sensors, or to restrict sensors to a defined set of nodes.

The mmperfmon query command is used to query metrics in a cluster from the performance metrics collector. Output can be delivered either in a raw format, formatted table layout or as a CSV export.

In addition to metrics known by the performance collector, the mmperfmon query command can also run predefined named queries or use predefined computed metrics. You can specify a bucket size for each record to return in number of seconds and the number of buckets to retrieve. You can also specify the duration or a time range for which the query can run.

When the performance monitoring tool is being used in a container environment, the transient network interfaces and file system mount points can increase the amount of metadata or keys within the tool and slowdown the metric queries. To prevent this issue, use the filter attribute of the Network and DiskFree sensors. Usage of the mmperfmon command to set the filter conditions is shown in Example 5.

Parameters

config

generate

Generates the configuration of the performance monitoring tool.

Updating the performance monitoring tool packages do not change the configuration, the configuration remains on the release when it was created. /opt/IBM/zimon/defaults/ZIMonSensors.cfg contains the latest package configuration defaults and should be used as reference when updating the performance monitoring tool configuration.

Note: After the configuration is generated, do not forget to turn on monitoring through the mmchnode command.

Important: In a mixed cluster configuration wherein nodes have different IBM Storage Scale versions that might happen during an upgrade, issuing the mmperfmon config generate command can cause issues. The proposed method of configuration is to place all nodes of the cluster with the same IBM Storage Scale version before the initial configuration of the performance monitoring tool is generated. However, if the proposed configuration method cannot be implemented, then you must run the mmperfmon config generate command on the node with the oldest IBM Storage Scale version.

--collectors CollectorNode[,CollectorNode...] specifies the set of collectors to which the sensors report their performance measurements. The number of collectors that each sensor reports to can be specified through colRedundancy parameter in the template sensor configuration file (see --config-file). Federated collectors are automatically configured between these collectors. For more information, see Configuring multiple collectors

--config-file InputFile specifies the template sensor configuration file to use. If this option is not provided, the /opt/IBM/zimon/defaults/ZIMonSensors.cfg file is used.

add

Adds a new sensor to the performance monitoring tool.

--sensors SensorFile adds the sensors that are specified in SensorFile to the sensor configuration. Multiple sensors in the configuration file need to be separated by a comma. Following is a sample SensorFile:

sensors = {
name = "MySensor"
# sensor disabled by default
period = 0
type = "Generic"
}

The generic sensor and a sensor-specific configuration file need to be installed on all the nodes where the generic sensor is to be activated.

--apikey key_name creates an API key entry for a specific key_name, and stores the information in CCR, so that all the cluster nodes are aware of it. If the key_name already exists, an error message is displayed.

--filter FilterSpecFile associates an allow-list filter with this API key. The file is a plain text specification that restricts which performance data is visible to clients that authenticate with this API key. Keys are filter names (for example, node, gpfs_fs_name, mountPoint), and values can be literal strings or regular expressions.

Note: The --apikey flag is mutually exclusive with other flags. The default API key scale_default is system-defined, and is generated automatically.

update

Updates the existing configuration.

--collectors CollectorNode[,CollectorNode...] updates the collectors to be used by the sensors and for federation (see config generate for details).

--config-file InputFile specifies a template sensor configuration file to use. This file overwrites the currently used configuration with the configuration that is specified in InputFile.

Attribute=value ... specifies a list of attribute value assignments. This sets the value of attribute Attribute to value. Attribute is a combination of a sensor name and one of its parameters that are separated by a dot as shown: <sensor>.<parameter>. For example, CPU.period.

The following attributes are supported:

<sensor>.period: Specifies the seconds between each invocation of the sensor. This parameter is supported for all sensors.
<sensor>.restrict: Specifies a node or node class. The invocation of the sensor is limited to the nodes specified. This parameter is supported for all sensors.
<sensor>.filter: Specifies a sensor-specific element to be ignored when the sensor retrieves data. This parameter is supported for sensors Network and DiskFree.

--apikey key_name updates an existing key entry for a given key_name, and stores the information in CCR. A new key value is generated and replaces the previous one. If the key_name does not exist, then an error message is displayed.

--filter FilterSpecFile sets or replaces the allow-list filter for this API key. The file uses a simple key-value format, where keys represent filter names and values are either literal strings or regular expressions. Omitting --filter leaves any existing filter unchanged.

Note: The --apikey flag is mutually exclusive with other flags.

delete

Removes configuration of the performance monitoring tool or the specified sensors.

--sensors Sensor[,Sensor...] removes the sensors with the specified names from the performance monitoring configuration.

--all removes the entire performance monitoring configuration from IBM Storage Scale.

--apikey key_name deletes an existing key entry for a given key_name, and stores the information in CCR. If the key_name does not exist, an error message is displayed.

Note: The --apikey flag is mutually exclusive with other flags. The default API key scale_default is system-defined, and cannot be deleted.

show

Displays the currently active performance monitoring configuration. Specifies the following options:

--config-file OutputFile specifies that the output is saved to the OutputFile.

--apikey key_name shows existing key entries. If key_name is specified, then only the corresponding data is shown. If the key_name does not exist, then an error message is shown. If no argument or all is given, then all entries are displayed.

Note: The --apikey flag is mutually exclusive with other flags.

query

Metric[,Metric...] specifies a comma-separated list of metrics for displaying in the output.

Measurement specifies a computed metric.

NamedQuery specifies the name of a predefined query.

compareNodes compares the specified metrics for all nodes in the system. The query creates one column per existing node and only one metric can be compared.

ComparisonMetric specifies the name of the metric to be compared when using the compareNodes query.

StartTime specifies the start timestamp for query in the YYYY-MM-DD-hh:mm:ss format.

EndTime specifies the end timestamp for query in the YYYY-MM-DD-hh:mm:ss format. If it is not specified, the query returns results until the present time.

You can specify the --hour, --day, --week, or --month to get data for a specified time.

Options specifies the following options:

-h or --help shows the help message and exits.
--interactive or -i displays results in an interactive table, which supports scrolling for large result sets.
--statistics displays statistics for data for each column.
--downsample DOWNSAMPLE reduces the sampling rate by aggregating data into the specified time interval, in seconds. The downsampling interval must be larger than the bucket size.
--downsamplefn {min,max,sum,avg} selects the aggregation function used for downsampling.
--top [TOP] limits the output to the top X elements with the highest metric values at each point in time. Results are sorted by the first two metric values to highlight the most significant contributors.
-N or --Node NODENAME specifies the node for which the metrics are retrieved.
For general information on how to specify node names, see Specifying nodes as input to GPFS commands.
--bucket-size BUCKET_SIZE specifies the bucket size (number of seconds), default is 1. For BUCKET_SIZE, it is recommended to use the same value as the sensor period. For example, if the sensor DiskFree has period 600 (default), then metrics of the sensor DiskFree must be queried with BUCKET_SIZE 600 (--bucket-size 600).
--number-buckets NUMBER_BUCKETS specifies the number of buckets (records) to display, default is 10.
--hour retrieves data for the last hour. The default bucket size is 60 seconds.
--day retrieves data for the last 24 hours. The default bucket size is 30 minutes.
--week retrieves data for the last 7 days. The default bucket size is 4 hours.
--month retrieves data for the last 30 days. The default bucket size is 1 day.
--filter FILTER specifies the filter criteria for the query to run. To see the list of filters in the node use the mmperfmon query --list filters command.
--group-by makes a group of columns to use if aggregate functions such as sum() are used in the query.
--sort sorts by a node name and an identifier instead of a metric name.
--json formats the output in JSON.
--csv provides the output in the CSV format.
--legend includes a separate legend in the output.
--raw does not format byte values.
--legend prints the keys for the returned data in a separate Legend block. Useful when key strings are longer than the space available for each column.
--nice displays the column headers in the output in a bold and underlined typeface.
--sensor retrieves all metrics of the specified sensor name.
--resolve displays the resolved computed metrics and metrics that are used.
--list {computed | metrics | measurements | keys | filters | queries | expiredKeys | keystat} lists the following information:
- computed displays the computed metrics.
- measurements displays the metrics.
- filters lists the filters.
- queries lists the available predefined queries.
- expiredKeys lists the group keys for the entities that do not return any metrics values within the default retention period of 14 days.
- keystat displays the number of connected entities for each sensor. The metric name and keys are not included.
--suppressnull removes all rows that contain null values from the output.

Note: The --suppressnull option is applied after the query is executed. When this option is used together with row-limiting options such as -n, the resulting output might contain fewer rows than requested because rows with null values are removed.

report

Returns a report.

top

Returns a report of the top processes by the CPU.

StartTime specifies the start timestamp for query in the YYYY-MM-DD-hh:mm:ss format.

EndTime specifies the end timestamp for a query in the YYYY-MM-DD-hh:mm:ss format. If it is not specified, the query returns results until the present time.

Options specifies the following options:

-N or --Node NODENAME specifies the node from where the metrics are retrieved.
For general information on how to specify node names, see Specifying nodes as input to GPFS commands.
--bucket-size BUCKET_SIZE specifies the bucket size in number of seconds. Default is 1.
--number-buckets NUMBER_BUCKETS or -n NUM_BUCKETS specifies the number of buckets to display. Default is 10.
--json provides the output in a JSON format.
--raw provides the output in a RAW format.
Note: The --json and --raw options display similar outputs.
--filter FILTER specifies the filter criteria for the query to run. To see the list of filters in the node use the mmperfmon query --list filters command. Currently, the only supported filter is node.

delete

Removes expired keys from the performance monitoring tool database.

--key Key[,Key...] specifies the key or list of keys that must be removed from the performance monitoring tool database, if they are expired keys. The keys are displayed as a comma-separated string.

--expiredKey specifies all the expired keys must be removed from the performance monitoring tool database.

Note: A group key is the part of the metric key string that represents a base entity. For example, for the keys:


gpfsgui-cluster-1.novalocal|GPFSInodeCap|nfs_shareFS|gpfs_fs_inode_alloc
gpfsgui-cluster-1.novalocal|GPFSInodeCap|nfs_shareFS|gpfs_fs_inode_free
gpfsgui-cluster-1.novalocal|GPFSInodeCap|nfs_shareFS|gpfs_fs_inode_max
gpfsgui-cluster-1.novalocal|GPFSInodeCap|nfs_shareFS|gpfs_fs_inode_used

the group key would be gpfsgui-cluster-1.novalocal|GPFSInodeCap|nfs_shareFS.
Expired keys are group keys, which are detected by IBM Storage Scale monitoring tool. However, these group keys are not found in the current cluster configuration and do not return metrics for the default retention period of at least 14 days.

--wait seconds specifies the maximum processing timeout in seconds to search and delete expired keys from the performance monitoring tool database.

Note: By default, the value is 300 seconds. However, a user can overrule this default value.

--force removes expired keys from the performance monitoring database irrespective of the default retention period. When this option is used, the expired keys are removed without further verification.

Exit status

0: Successful completion.
1: Invalid arguments given
2: Invalid option
3: No node found with a running performance collector
4: Performance collector backend signaled bad query, for example, no data for this query.

Security

You must have root authority to run the mmperfmon command.

The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages. The performance monitoring tool uses the GPFS™ cluster daemon node names and network to communicate between nodes. For more information, see Requirements for administering a GPFS file system.

Examples

To generate configuration for the c89f8v03 collector node, issue the following command:

# mmperfmon config generate --collectors c89f8v03

A sample output is as follows:

mmperfmon: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
Tue Oct 27 20:40:07 EDT 2015: mmcommon pushSdr_async: mmsdrfs propagation started

To add /tmp/SensorFile sensor to the performance monitoring tool, issue the following command:

# mmperfmon config add --sensors /tmp/SensorFile

A sample output is as follows:

mmperfmon: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
Tue Oct 27 20:44:33 EDT 2015: mmcommon pushSdr_async: mmsdrfs propagation started
# mmperfmon config show | tail -12
{
        name = "NFSIO"
        period = 0
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "TestAdd"
        period = 4
}
smbstat = ""

To create a new API key for the following cases:

If no keys are defined yet, then issue the following command:

# mmperfmon config add --apikey scale_default

A sample output is as follows:

File _perfmon.keys does not exist in CCR
Create a new empty API key Json file to be uploaded to CCR
API key file _perfmon.keys successfully uploaded to CCR
[root@gpfs-21 ~]#


[root@gpfs-21 ~]# mmperfmon config show --apikey scale_default
{
    "key": "aedc2798-e2b0-4cda-8f12-da1e778c1504",
    "comment": "user comment"
}
[root@gpfs-21 ~]#

If keys are already defined, then issue the following command:

# mmperfmon config add --apikey scale_user1

A sample output is as follows:

API key file _perfmon.keys successfully uploaded to CCR
[root@node-11 ~]#

To update the NFSIO.period value to 5, issue the following command:

# mmperfmon config update NFSIO.period=5

A sample output is as follows:

mmperfmon: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
Tue Oct 27 20:47:53 EDT 2015: mmcommon pushSdr_async: mmsdrfs propagation started

# mmperfmon config show | tail -9
},
{
        name = "NFSIO"
        period = 5
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

To update the Network.filter value to ignore multiple network interfaces, issue the following command:

# mmperfmon config update Network.filter="netdev_name=veth.*|docker.*|flannel.*|cali.*|cbr.*"

A sample output is as follows:

mmperfmon: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
Tue Jun 23 17:25:45 CEST 2020: mmcommon pushSdr_async: mmsdrfs propagation started
Tue Jun 23 17:25:51 CEST 2020: mmcommon pushSdr_async: mmsdrfs propagation completed; mmdsh rc=0

To update an API key value, issue the following command:

# mmperfmon config update --apikey animal

A sample output is as follows:

API key file _perfmon.keys successfully uploaded to CCR
[root@gpfs-21 ~]#

To remove the TestAdd sensor, issue the following command:

# mmperfmon config delete --sensors TestAdd

A sample output is as follows:

mmperfmon: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
Tue Oct 27 20:46:23 EDT 2015: mmcommon pushSdr_async: mmsdrfs propagation started
Tue Oct 27 20:46:28 EDT 2015: mmcommon pushSdr_async: mmsdrfs propagation completed; mmdsh rc=0

# mmperfmon config show | tail -12
{
        name = "GPFSDiskCap"
        period = 0
},
{
        name = "NFSIO"
        period = 0
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

To display the currently active performance monitoring configuration, issue the following command:

# mmperfmon config show

A sample output is as follows:

cephMon = "/opt/IBM/zimon/CephMonProxy"
cephRados = "/opt/IBM/zimon/CephRadosProxy"
colCandidates = "c89f8v03"
colRedundancy = 1
collectors = {
        host = ""
        port = "4739"
}
config = "/opt/IBM/zimon/ZIMonSensors.cfg"
ctdbstat = ""
daemonize = T
delay_initial_poll = T
hostname = ""
ipfixinterface = "0.0.0.0"
logfile = "/var/log/zimon/ZIMonSensors.log"
loglevel = "info"
mmcmd = "/opt/IBM/zimon/MMCmdProxy"
mmdfcmd = "/opt/IBM/zimon/MMDFProxy"
mmpmon = "/opt/IBM/zimon/MmpmonSockProxy"
piddir = "/run"
release = "4.2.0-0"
sensors = {
        name = "CPU"
        period = 1
},
{
        name = "Load"
        period = 1
},
{
        name = "Memory"
        period = 1
},
{
        name = "Network"
        period = 1
},
{
        name = "Netstat"
        period = 0
},
{
        name = "Diskstat"
        period = 0
},
{
        name = "DiskFree"
        period = 600
},
{
        name = "GPFSDisk"
        period = 0
},
{
        name = "GPFSFilesystem"
        period = 1
},
{
        name = "GPFSNSDDisk"
        period = 1
        restrict = "nsdNodes"
},
{
        name = "GPFSPoolIO"
        period = 0
},
{
        name = "GPFSVFS"
        period = 1
},
{
        name = "GPFSIOC"
        period = 0
},
{
        name = "GPFSVIO"
        period = 0
},
{
        name = "GPFSPDDisk"
        period = 1
        restrict = "nsdNodes"
},
{
        name = "GPFSvFLUSH"
        period = 0
},
{
        name = "GPFSNode"
        period = 1
},
{
        name = "GPFSNodeAPI"
        period = 1
},
{
        name = "GPFSFilesystemAPI"
        period = 1
},
{
        name = "GPFSLROC"
        period = 0
},
{
        name = "GPFSCHMS"
        period = 0
},
{
        name = "GPFSAFM"
        period = 0
},
{
        name = "GPFSAFMFS"
        period = 0
},
{
        name = "GPFSAFMFSET"
        period = 0
},
{
        name = "GPFSRPCS"
        period = 0
},
{
        name = "GPFSFilesetQuota"
        period = 3600
},
{
        name = "GPFSDiskCap"
        period = 0
},
{
        name = "NFSIO"
        period = 0
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SwiftAccount"
        period = 1
        restrict = "cesNodes"
        type = "generic"
},
{
        name = "SwiftContainer"
        period = 1
        restrict = "cesNodes"
        type = "generic"
},
{
        name = "SwiftObject"
        period = 1
        restrict = "cesNodes"
        type = "generic"
},
{
        name = "SwiftProxy"
        period = 1
        restrict = "cesNodes"
        type = "generic"
}
smbstat = ""

To show all the API key values, issue the following command:

# mmperfmon config show --apikey all

A sample output is as follows:

{
    "scale_default": {
        "key": "aedc2798-e2b0-4cda-8f12-da1e778c1504",
        "comment": "user comment"
    },
    "forrest": {
        "key": "c07abeb9-b137-4074-95c7-e49eaf92dfce",
        "comment": "user comment"
    },
    "buttermilk": {
        "key": "f2f08be9-c743-4626-bde2-99e05502732c",
        "comment": "user comment"
    },
    "honey": {
        "key": "7d54d052-2600-405c-b316-8b36a88b95ce",
        "comment": "user comment"
    },
    "animal": {
        "key": "ab2e3a71-d869-4e4e-a690-68de58ecbc6e",
        "comment": "user comment"
    }
}
[root@gpfs-21 ~]#

To list the two metrics nfs_read_lat and nfs_write_lat for a specific time range, filtered by an export and NFS version with 60 seconds buckets (one record represents 60 seconds), issue the following command:

# mmperfmon query nfs_read_lat,nfs_write_lat 2026-01-21_23:08:00  2026-01-21_23:12:00 --filter export=/mnt/gpfs0/nfsV3export,nfs_ver=NFSv3 -b 60

A sample output is as follows:

                                          gpfs-23                      gpfs-23
                     /mnt/gpfs0/nfsV3export|NFSv3 /mnt/gpfs0/nfsV3export|NFSv3
 Time                                nfs_read_lat                nfs_write_lat
 2026-01-21_23:09:00                            -                            -
 2026-01-21_23:10:00                            0                            0
 2026-01-21_23:11:00                      4502573                      2453623
 2026-01-21_23:12:00                      3856796                      8793246

To list all available filters, issue the following command:

# mmperfmon query --list filters

A sample output is as follows:

Available Filters:

node
        gpfs-21.localnet.com
        gpfs-22.localnet.com
protocol
        smb2
db_name
        account_policy
        autorid
        brlock
        ctdb
        dbwrap_watchers
        g_lock
        group_mapping
        leases
        locking
        netlogon_creds_cli
        notify_index
        passdb
        registry
        secrets
        serverid
        share_info
        smbXsrv_open_global
        smbXsrv_session_global
        smbXsrv_tcon_global
        smbXsrv_version_global
gpfs_fs_name
        fs0
        gpfs0
gpfs_cluster_name
        gpfs-cluster-2.localnet.com
mountPoint
        /
        /boot
        /dev
        /dev/shm
        /gpfs/fs0
        /mnt/gpfs0
        /run
        /sys/fs/cgroup
operation
        break
        cancel
        close
        create
        find
        flush
        getinfo
        ioctl
        keepalive
        lock
        logoff
        negprot
        notify
        read
        sesssetup
        setinfo
        tcon
        tdis
        write
sensor
        CPU
        CTDBDBStats
        CTDBStats
        DiskFree
        GPFSFilesystemAPI
        GPFSVFS
        Load
        Memory
        Network
        SMBGlobalStats
        SMBStats
netdev_name
        eth0
        lo

To list the number of collected entities for a sensor, issue the following command:

# mmperfmon query --list keystat

A sample output is as follows:

Sensor            Entities
GPFSmmhealth      109
DiskFree          41
GPFSNSDDisk       40
SMBStats          38
GPFSWaiters       35
CTDBDBStats       30
GPFSNSDPool       30
Network           20
GPFSPoolCap       16
GPFSNSDFS         15
GPFSFilesystemAPI 10
GPFSFilesystem    10
CPU               5
GPFSVFSX          5
Load              5
Memory            5
GPFSRPCS          5
GPFSNode          5
Netstat           5
GPFSNodeAPI       5
GPFSBufMgr        5
TopProc           5
GPFSDiskCap       4
GPFSPool          4
SMBGlobalStats    2
CTDBStats         2
GPFSInodeCap      2
IPFIX             2
GPFSFileset       2

To query the statistics of a sensor, issue the following command:

# mmperfmon query netstat --sensor -b 30

where, netstat is a name of a sensor.

A sample output is as follows:


                         nnode-12       nnode-12  nnode-12         nnode-12         nnode-12     nnode-12          nnode-12          nnode-12      nnode-12    nnode-12
Time                ns_closewait ns_established ns_listen ns_local_bytes_r ns_local_bytes_s ns_localconn ns_remote_bytes_r ns_remote_bytes_s ns_remoteconn ns_timewait
2025-03-04_08:17:00            0             24        10            0   B            0   B            0             0   B             0   B            24         127
2025-03-04_08:17:30            0             24        10            0   B            0   B            0             0   B            16   B            24         124
2025-03-04_08:18:00            0             24        10            0   B            0   B            0           336   B           273   B            24         116
2025-03-04_08:18:30            0             24        10            0   B            0   B            0             0   B             0   B            24         123
2025-03-04_08:19:00            0             24        10            0   B            0   B            0             0   B           273   B            24         122
2025-03-04_08:19:30            0             24        10            0   B            0   B            0            62   B            62   B            24         120
2025-03-04_08:20:00            0             24        10            0   B            0   B            0           273   B             0   B            24         120
2025-03-04_08:20:30            0             24        10            0   B            0   B            0             0   B            62   B            24         121
2025-03-04_08:21:00            0             24        10            0   B            0   B            0             0   B             0   B            24         130
2025-03-04_08:21:30            0             24        10            0   B            0   B            0             0   B           737   B            24         123

To find the values for the threshold feature , issue the following command:

# mmperfmon query cpu -a --statistics

As sample output is as follows:

     Item ID       Metric  count       mean        std        min        25%        50%        75%        max
nnode-11      cpu_system     10       0.40       0.40       0.00       0.00       0.50       0.63       1.01
nnode-12      cpu_system     10       5.75       6.21       0.50       0.89       3.00      10.10      16.67
nnode-13      cpu_system     10       3.53       4.10       0.50       0.51       1.25       6.63      12.18
nnode-14      cpu_system     10       0.38       0.50       0.00       0.19       0.25       0.31       1.75
nnode-15      cpu_system     10       0.85       1.03       0.00       0.00       0.50       1.64       3.02
nnode-11        cpu_user     10       0.45       0.29       0.00       0.38       0.50       0.50       1.01
nnode-12        cpu_user     10      23.12      25.76       1.00       1.39       9.28      50.63      62.12
nnode-13        cpu_user     10      15.40      22.77       0.51       0.88       1.25      37.08      56.35
nnode-14        cpu_user     10       0.60       1.22       0.00       0.00       0.25       0.38       4.01
nnode-15        cpu_user     10       2.31       4.39       0.00       0.00       0.50       2.77      14.00
nnode-11    cpu_contexts     10     594.20     237.16     462.00     469.50     489.00     625.50    1225.00
nnode-12    cpu_contexts     10    2302.10    1367.77    1345.00    1354.50    2009.00    2540.50    5834.00
nnode-13    cpu_contexts     10    1949.50     914.49    1127.00    1188.50    1659.00    2601.00    3875.00
nnode-14    cpu_contexts     10     471.40     369.53     228.00     242.75     306.50     597.25    1423.00
nnode-15    cpu_contexts     10     842.10     754.75     334.00     383.50     499.50    1062.00    2782.00

You can add, --hour, --day, --week, --month to get data for a specified time.

To sort by the same measurement for CLI and thresholds, issue the following command:

# mmperfmon query cpu -a --sort

A sample output is as follows:

                             cesSharedRoot/root        cesSharedRoot/root       cesSharedRoot/root cesSharedRoot/root                 gpfs0/root                gpfs0/root               gpfs0/root    gpfs0/root
Time                sum(gpfs_fset_allocInodes) sum(gpfs_fset_freeInodes) sum(gpfs_fset_maxInodes)      Fileset_inode sum(gpfs_fset_allocInodes) sum(gpfs_fset_freeInodes) sum(gpfs_fset_maxInodes) Fileset_inode
2025-03-04_08:25:00                      65792                     61712                    65792               6.20                     133376                    129338                   133376          3.03
2025-03-04_08:30:00                      65792                     61712                    65792               6.20                     133376                    129338                   133376          3.03
2025-03-04_08:35:00                      65792                     61712                    65792               6.20                     133376                    129338                   133376          3.03

To make the column headers in the output in a bold and underlined typeface, issue the following command:
```
# mmperfmon query cpu --nice
```
A sample output is as follows:

To run a named query for export /ibm/gpfs/nfsexport and nfs_ver NFSv3, using the default bucket size as defined by the sensor period for the given time range, issue the following command:

# mmperfmon query nfsIORate   2026-01-21_23:08:10  2026-01-21_23:08:40 --filter export=/mnt/gpfs0/nfsV3export,nfs_ver=NFSv3

A sample output is as follows:

                                          gpfs-23                      gpfs-23
                     /mnt/gpfs0/nfsV3export|NFSv3 /mnt/gpfs0/nfsV3export|NFSv3
 Time                                nfs_read_ops                nfs_write_ops
 2026-01-21_23:08:20                            80                          0
 2026-01-21_23:08:30                            83                         90
 2026-01-21_23:08:40                            78                         99

To run a compareNodes query for the cpu_user metric, issue the following command:

# mmperfmon query compareNodes cpu_user

A sample output is as follows:

                     nnode-11 nnode-12 nnode-13 nnode-14 nnode-15
 Time                cpu_user cpu_user cpu_user cpu_user cpu_user
 2026-01-29_09:40:46     0.00    48.26    38.07     1.76     0.50
 2026-01-29_09:40:47     0.50    77.89     2.99     0.00     0.00
 2026-01-29_09:40:48     0.50    72.73     0.51     0.25     0.00
 2026-01-29_09:40:49     1.49    50.77     1.49     0.75     0.00
 2026-01-29_09:40:50     0.50    17.77     2.50     8.75     0.00
 2026-01-29_09:40:51     9.18    27.64    15.66     5.00     0.50
 2026-01-29_09:40:52     0.50    45.73    44.95     0.25     0.00
 2026-01-29_09:40:53     0.00    50.25    26.13     0.25     0.50
 2026-01-29_09:40:54     0.50    49.49     0.50     0.50     0.00
 2026-01-29_09:40:55     0.00    39.90     4.52     9.02     0.00

To view expired keys, issue the following command:

# mmperfmon query –list=expiredKeys

A sample output is as follows:


Found expired keys:
test_nodename|GPFSFilesystem|gpfsgui-cluster-2.novalocal|fs1
test_nodename|GPFSFilesystem|gpfsgui-cluster-2.novalocal|fs2
test_nodename|GPFSFilesystemAPI|gpfsgui-cluster-2.novalocal|fs2
test_nodename|GPFSFilesystemAPI|gpfsgui-cluster-2.novalocal|fs1
test_nodename|GPFSFilesystem|gpfsgui-cluster-2.novalocal|gpfs0
test_nodename|DiskFree|/mnt/gpfs0
test_nodename|Netstat
test_nodename|GPFSFilesystem|gpfsgui-cluster-2.novalocal|objfs
test_nodename|GPFSVFS
test_nodename|GPFSNode
test_nodename|GPFSFilesystemAPI|gpfsgui-cluster-2.novalocal|gpfs0
test_nodename|GPFSFilesystemAPI|gpfsgui-cluster-2.novalocal|objfs
test_nodename|DiskFree|/gpfs/fs2
test_nodename|DiskFree|/gpfs/fs1
test_nodename|GPFSRPCS
test_nodename|CPU
test_nodename|GPFSNodeAPI
test_nodename|Load
test_nodename|DiskFree|/mnt/objfs
test_nodename|Memory

To suppress rows with null data in query results, use the --suppressnull option. This option removes all rows containing null values from the output.

For example:

# mmperfmon query gpfs_disk_free_fullkb

A sample output is as follows:


                     cesSharedRoot|data|disk02 cesSharedRoot|system|disk01
Time                    gpfs_disk_free_fullkb       gpfs_disk_free_fullkb
2026-03-29_01:00:00                     1.9GB                     689.0MB
2026-03-29_08:00:00                         -                           -
2026-03-29_14:00:00                     1.9GB                     689.0MB
2026-03-29_20:00:00                         -                           -
2026-03-30_02:00:00                     1.9GB                     689.0MB
2026-03-30_08:00:00                         -                           -
2026-03-30_14:00:00                     1.9GB                     689.0MB
2026-03-30_20:00:00                         -                           -
2026-03-31_02:00:00                     1.9GB                     689.0MB
2026-03-31_08:00:00                         -                           -

To remove rows with null data, issue the following command:

# mmperfmon query gpfs_disk_free_fullkb --suppressnull

A sample output is as follows:


                     cesSharedRoot|data|disk02 cesSharedRoot|system|disk01
Time                    gpfs_disk_free_fullkb       gpfs_disk_free_fullkb
2026-03-29_01:00:00                     1.9GB                     689.0MB
2026-03-29_14:00:00                     1.9GB                     689.0MB
2026-03-30_02:00:00                     1.9GB                     689.0MB
2026-03-30_14:00:00                     1.9GB                     689.0MB
2026-03-31_02:00:00                     1.9GB                     689.0MB

When you specify -n 10, the command first retrieves the last 10 records and then removes rows that contain only null values. As a result, the final output might contain fewer than 10 rows if some of the retrieved records are purged because of null data.

To delete expired key, issue the following command:

# mmperfmon delete --key ‘test_nodename|DiskFree|/mnt/gpfs0‘

A sample output is as follows:

Check expired keys completed. Successfully 3 keys deleted.

To set a processing timeout (for example, 3600 seconds) for deleting an expired key, issue the following command:

# mmperfmon delete --expiredkeys --wait 3600

A sample output is as follows:

[root@node-11 ~]# mmperfmon delete --expiredkeys --wait 3600
Check expired keys completed.
Nothing to delete.
[root@node-11 ~]#

To delete a key irrespective of the default retention period, issue the following command:

# mmperfmon delete --key  " test_nodename|DiskFree|/mnt/gpfs0 " --force

A sample output is as follows:

[root@node-11 ~]# mmperfmon delete --key  " test_nodename|DiskFree|/mnt/gpfs0 " --force
Check expired keys completed. Nothing to delete.

To list a CPU report, issue the following command:

# mmperfmon report top --bucket-size 60 --number-buckets 1

A sample output is as follows:

[root@node-11 ~]# mmperfmon report top --bucket-size 60 --number-buckets 1
Top values in format:
{process name | PID} {cpu per mil} {memory per mil}

Time                 node-11.localnet.com
2020-04-03-17:11:00  11684 33 12
                     9205 23 302
                     26331 9 0
                     27671 9 1
                     1 3 2
                     9 3 0
                     386 1 0
                     707 1 0
                     1012 1 5
                     1016 1 2
[root@node-11 ~]#

To disable the delay_initial_poll feature, follow the given steps:
1. Export the current performance monitoring sensor configuration to a local file. To do this, create a working copy, myFile, of the current sensor configuration data by using the following command:
```
# mmperfmon config show --config-file myFile
```
  A sample output is as follows:
```
[root@scale-11 ~]# mmperfmon config show --config-file myFile
[root@scale-11 ~]#
```
2. Edit this file in an editor, such as vi or vim editor. Set the value for delay_initial_poll to F by using the following command:
```
[root@scale-11 ~]# grep delay_initial_poll myFile
delay_initial_poll = F
[root@scale-11 ~]#
```
3. Import the working copy file, myFile, that replaces the current sensor configuration cluster-wide. The changes are validated and a new performance monitoring sensor configuration file is created and published on all relevant cluster nodes.
```
# mmperfmon config update --config-file myFile
```
  A sample output is as follows:
```
[root@scale-11 ~]# mmperfmon config update --config-file myFile
mmperfmon: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@scale-11 ~]# Thu Aug  5 09:54:46 EDT 2021: mmcommon pushSdr_async: mmsdrfs propagation started
Thu Aug  5 09:54:52 EDT 2021: mmcommon pushSdr_async: mmsdrfs propagation completed; mmdsh rc=0

[root@scale-11 ~]#
```
4. Optionally, verify that the configuration for delay_initial_poll is now changed or disabled by using the following command:
```
# mmperfmon config show | grep delay_initial_poll
```
  A sample output is as follows:
```
[root@scale-11 ~]# mmperfmon config show | grep delay_initial_poll
delay_initial_poll = F
root@scale-11 ~]#
```

To display CPU metrics with keys listed in a separate legend, issue the following command:

# mmperfmon query cpu -a --legend -n 2

A sample output is as follows:

Legend:
001 nnode-11|CPU|cpu_system
002 nnode-12|CPU|cpu_system
003 nnode-13|CPU|cpu_system
004 nnode-14|CPU|cpu_system
005 nnode-15|CPU|cpu_system
006 nnode-11|CPU|cpu_user
007 nnode-12|CPU|cpu_user
008 nnode-13|CPU|cpu_user
009 nnode-14|CPU|cpu_user
010 nnode-15|CPU|cpu_user
011 nnode-11|CPU|cpu_contexts
012 nnode-12|CPU|cpu_contexts
013 nnode-13|CPU|cpu_contexts
014 nnode-14|CPU|cpu_contexts
015 nnode-15|CPU|cpu_contexts
Time 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015
2026-01-02_10:18:09 4.52 12.76 12.06 8.93 5.56 12.56 57.14 56.28 24.74 6.06 1609 5938 5394 7158 967
2026-01-02_10:18:10 2.00 9.41 8.42 5.05 9.09 2.00 37.13 54.46 8.84 14.14 1146 4435 4201 4969 1717

Location

/usr/lpp/mmfs/bin

mmperfmon command

Synopsis

Availability

Description

Parameters

Exit status

Security

Examples

See also

Location