mmwatch command

Administers clustered watch folder watches and lists all of the active watch folder API watches.

Synopsis


mmwatch Device list [--events] [ --watch-id WatchID ] [-Y]

or

mmwatch Device list --watch-id WatchID --config [-Y]

or

mmwatch all list [--events] [-Y]

or

mmwatch {Device | all} cleanup --auto-disabled

or

mmwatch all status

or

mmwatch Device status [ --watch-id WatchID [-v] ]

or

mmwatch Device enable { -F ConfigFilePath |
          [ --fileset Start of changeFSetNameEnd of change ] [ --watch-id WatchID ]
          [ --events {Event[,Event...] | ALL} ]
          --event-handler handlertype
          --sink-brokers BrokerIP:Port[,BrokerIP:Port...]
          --sink-topic Topic
          [ --sink-auth-config Path ]
          Start of change[ --secondary-sink [ --sec-sink-fileset Start of changeFSetNameEnd of change ]]End of change
          [ --degraded ] }

or

mmwatch Device disable --watch-id WatchID

or

mmwatch Device suspend --watch-id WatchID

or

mmwatch Device resume --watch-id WatchID

Availability

Available with IBM Spectrum Scale Advanced Edition, IBM Spectrum Scale Data Management Edition, Start of changeIBM Spectrum Scale Developer Edition, End of changeor IBM Spectrum Scale Erasure Code Edition.

Description

The mmwatch command is used to enable, disable, list, and generally administer persistent and fault-tolerant clustered watches. The purpose of clustered watch folder is to monitor file system event activity and produce events to an external event handler, which is referenced as a sink. The mmwatch command is also used to list active watch folder API watches. Errors are logged to /var/adm/ras/mmwatch.log, /var/adm/ras/mmwfclient.log, and /var/log/messages.

Remember: To use this command to list watch folder API watches, your cluster code level must be at IBM Spectrum® Scale 5.0.3. To use this command to enable, disable, and administer clustered watch folder, your cluster code level must be at IBM Spectrum Scale 5.0.3, and the file system on which the clustered watch folder or watch folder API watch is set must be at IBM Spectrum Scale 5.0.2.

Parameters

Device
Specifies the device name of the file system upon which the clustered watch folder configuration change or listing is to occur.
Note: You must specify the Device or use the all keyword for mmwatch operations.
all
Lists the active clustered watches for all file systems.
Note: You must specify the Device or use the all keyword for mmwatch operations.
list [--events] [-Y]
Displays the details of active clustered watches for the specified device. The optional --events option lists the monitored clustered watch folder events for each specific watch for the specified device. The -Y parameter provides output in machine-readable (colon-delimited) format.
Note: Fields that have a colon (:) are encoded to prevent confusion. For the set of characters that might be encoded, see the command documentation of mmclidecode. Use the mmclidecode command to decode the field.
cleanup --auto-disabled
If more than 50% of the conduits for a clustered watch cannot send messages to the sink Kafka, the watch will be auto-disabled. Although the watch is essentially stopped, the information in the message queue services monitor database remains for later debugging. You can use the cleanup option to clear out the auto-disabled clustered watch folder information.
status
Provides the status of the consumer conduit processes that are running on the nodes within the kafkaClWatchConsumerServers node class, and is subject to the following:
  • If all is used in place of a device name, the status is provided for all clustered watches that are enabled within the cluster. If no clustered watches are found, a statement stating this is returned but this scenario is not considered an error.
  • If a Device is specified without a watch ID, then the status is provided for all clustered watches that are enabled within the cluster associated with that device. If no clustered watches exist for the device, a statement stating this is returned and this is considered an error.
  • If Device and --watch-id WatchID are both specified, then the status is provided for the single clustered watch that is associated with the given device and watch ID. If the clustered watch cannot be found, a statement stating this is returned and this is considered an error.
    Note: The -v flag can only be used when a watch ID is given. It provides up to the last 10 entries for the given watch ID for each component.
enable
Enables a clustered watch for the given device. The scope of the watch is the whole file system unless the --fileset option is provided. If --fileset is provided, then the watch type will be fileset watch for dependent filesets and inode space watch for independent filesets. Enablement entails setting up and validating the configuration, creating a Kafka topic on the local Kafka cluster, starting the watch consumer processes, and applying the respective policy partition rules based on the watched events.
-F ConfigFilePath
Specifies the path to the configuration file.
The configuration file is populated with key:value pairs, which include all of the required parameters for enable and any optional fields desired in the following format:
  • WATCH_ID:<Watch ID>
  • FILESET:<fileset name>
  • EVENTS:<Event1>,<Event2>
  • EVENT_HANDLER:kafkasink
  • SINK_BROKERS:<sink broker:port>
  • SINK_TOPIC:<sink topic>
  • SINK_AUTH_CONFIG:</path/to/auth/config>
  • DEGRADED:<true|false>
Note: If this option is given, then the other command line options that are used to enable a watch are invalid and produce a syntax error. Using the --fileset, --watch-type, --events, --event-handler, --sink-brokers, --sink-topic, or --sink-auth-config options in combination with the -F option produces a syntax error.
--fileset Start of changeFSetNameEnd of change
Specifies the name of the fileset within the given file system to watch. If the fileset is dependent, then the watch type is fileset. No nested filesets are watched within a dependent fileset. If the fileset is independent, then the watch type is inodespace. Only nested dependent filesets are included in the watch of an independent fileset. The --fileset option is optional.
--watch-id WatchID
The WatchID is the unique identifier for a clustered watch. The WatchID is optional for the enable operation; if it is not specified, a calculated name is given with the following value:
CLW<#seconds since epoch>
The WatchID of a current clustered watch is required for the disable operation.
Note: The WatchID can only be alphanumeric. The WatchID cannot contain the keywords all, list, enable, disable, or suspend.
--config
Displays the configuration details for the specified active watch WatchID.
--events Event[,Event...] | ALL}
Defines a comma-separated string of events to monitor. The --events option is optional. The following events can be watched:Start of change
  • IN_ACCESS
  • IN_ATTRIB
  • IN_CLOSE_NOWRITE
  • IN_CLOSE_WRITE
  • IN_CREATE
  • IN_DELETE
  • IN_DELETE_SELF
  • IN_MODIFY
  • IN_MOVED_FROM
  • IN_MOVED_TO
  • IN_MOVE_SELF
  • IN_OPEN
End of change If the --events option is not included, all events are watched.
--event-handler handlertype
The only type of event handler that is currently supported is kafkasink. The --event-handler option is required.
--sink-brokers BrokerIP:Port[,BrokerIP:Port...]
Includes a comma-separated list of broker:port pairs for the sink Kafka cluster (the external Kafka cluster where the events are sent). The --sink-brokers option is required.
--sink-topic Topic
The topic that producers write to in the sink Kafka cluster. The --sink-topic option is required.
--sink-auth-config Path
The full path to the file that contains the authentication details to the sink Kafka cluster. The --sink-auth-config option is optional.
Start of change--secondary-sink [ --sec-sink-fileset Start of changeFSetNameEnd of change ]End of change
Start of changeThe --secondary-sink option means that when the suspend operation is run for an active clustered watch, the events received while the clustered watch is suspended are written to files within a fileset. When the corresponding clustered watch is resumed (the resume operation is run), events from the file system activity as well as those that were written previously to the secondary sink fileset will be sent to the external Kafka sink.
If a secondary sink fileset name is not provided, the default name will be .secondary_sink. It will be created as an independent fileset that is mounted under the corresponding clustered watch's associated device mount point. If a fileset name is provided and the fileset that the customer defines does not exist, the fileset will be created as an independent fileset on the file system that is associated with the clustered watch.
Note: The secondary sink fileset must always be located within the file system that is associated with the clustered watch. In addition, the secondary sink fileset cannot be named .msgq.
End of change
--degraded
Uses half of the standard number of partitions (the number of brokers instead of 2X the number of brokers) to save local disk space. The --degraded option is optional.
disable
Disables a clustered watch for the given device. Disablement stops the watch consumer processes, removes the local Kafka topic, and deletes the policy partition rules for the watch.
suspend
Suspends an active clustered watch, which means that the consumer conduits are stopped and will not send events to the external Kafka queue. Events will accumulate within the message queue.
Note: If the clustered watch is suspended for a lengthy period of time or if a lot of file system activity occurs while the clustered watch is suspended, events might be missed due to the limited amount of local disk space on the broker nodes of the message queue.
resume
Resumes a previously suspended clustered watch. The consumer conduits are restarted. They will read events that built up while the consumer conduits were suspended and send them to the external Kafka queue.

Exit status

0
Successful completion.
nonzero
A failure occurs.

Security

You must have root authority to run the mmwatch command.

Examples

  1. To list all current clustered watches that are active, issue this command:
    mmwatch demo list
    The system displays information similar to the following output:
    Filesystem demo has 2 watcher(s):
      Watcher                                PID     Type       Start Time                Path
      -----------------------------------------------------------------------------------------------------
      node1.ibm.com                          7047    directory  Mon Aug 20 20:39:20 2018  /demo/somedir
      node2.ibm.com                          6225    directory  Tue Aug 21 10:45:05 2018  /demo
  2. To list all current clustered watches with associated events, issue this command:
    mmwatch demo list --events
    The system displays information similar to the following output:
    Filesystem demo has 2 watcher(s):
      Watcher                                PID     Monitored Events
      ---------------------------------------------------------------------------------------
      node3.ibm.com                          7047    IN_CLOSE_WRITE,IN_DELETE_SELF
      node4.ibm.com                          6225    IN_ACCESS,IN_ATTRIB,IN_CLOSE_NOWRITE,
                                                     IN_CLOSE_WRITE,IN_CREATE,IN_DELETE,
                                                     IN_DELETE_SELF,IN_MODIFY,IN_MOVED_FROM,
                                                     IN_MOVE_SELF
     
  3. To list the current configuration for a clustered watch, issue this command:
    mmwatch Device list --watch-id WatchID --config
    The system displays information similar to the following output:
    node1:~ # mmwatch fs2 list --watch-id CLW1552340432 --config
    DEVICE:fs2
    WATCH_ID:CLW1552340432
    EVENTS:all
    EVENT_HANDLER:kafkasink
    SINK_BROKERS:ibmserver30:9092
    SINK_TOPIC:ibm30topic
    DEGRADED:TRUE
    SINK_AUTH_TYPE:none
    node1:~ #
  4. You can see if clustered watches are auto-disabled in the status:
    # mmwatch all status
    Device                      Watch ID                 Watch State  
    fs2                         CLW1556232964            Auto Disabled
    Node Name                   Status    
    machine1.company.com        DOWN      
    machine2.company.com        DOWN      
    machine3.company.com        DOWN      
    machine4.company.com        DOWN      
    
    fs1                         CLW1556232741            Auto Disabled
    Node Name                   Status    
    machine1.company.com        DOWN      
    machine2.company.com        DOWN      
    machine3.company.com        DOWN      
    machine4.company.com        DOWN          
    
    To remove the corresponding database entries, use the cleanup option:
    # mmwatch all cleanup --auto-disabled
    [I] Successfully cleaned up 2 clustered watch(es).
    
    Then, if you view the status again, you will not see the auto-disabled clustered watches:
    # mmwatch all status
    [I] No clustered watch topics found.  Status cannot be obtained.
    
  5. To check the status, issue this command:
    mmwatch all status
    The system displays information similar to the following output:
    node1:~ # mmwatch fs2 status --watch-id CLW1552340432
    Device                      Watch ID
    fs2                         CLW1552340432
    Node Name                   Status
    node1.gpfs.ss               HEALTHY
    node3.gpfs.ss               HEALTHY
    node4.gpfs.ss               HEALTHY
    node1:~ #
  6. To enable a clustered watch using an input configuration file, issue this command:
    mmwatch Device enable -F ConfigFilePath
    The system displays information similar to the following output:
    node1:~ # mmwatch fs2 enable -F watch.config
    [I] Beginning enablement of Clustered Watch with newly created watch ID: CLW1552340432
    [I] Verifying MsgQueue nodes meet minimum local space requirements 
        for Clustered Watch to be enabled for watch: CLW1552340432.
        Depending on cluster size, this may take some time.
    [I] Successfully verified all configured MsgQueue nodes meet minimum local space requirements 
        for Clustered Watch to be enabled for watch: CLW1552340432
    [I] Verified the watch type is FSYS for filesystem fs2
    [I] Successfully created Clustered Watch topic on the MsgQueue for watch: CLW1552340432
    [I] Successfully added Clustered Watch configuration file into CCR for watch: CLW1552340432
    [I] Successfully enabled Clustered Watch consumers for watch: CLW1552340432
    [I] Successfully added Clustered Watch policy rules for watch: CLW1552340432
    [I] Successfully enabled Clustered Watch: CLW1552340432
    node1:~ #
     
  7. Enabling an independent fileset displays information similar to the following output:
    node1.gpfs.ss:~ # mmwatch fs2 enable --fileset fileset10 --event-handler kafkasink 
        --sink-brokers "sink1.server.net:9092,sink2.server.net:9092" 
        --sink-topic sink_topic --sink-auth-config "/root/sink.auth" --degraded
    [I] Beginning enablement of Clustered Watch with newly created watch ID: CLW1549311488
    [I] Verifying MsgQueue nodes meet minimum local space requirements for 
        Clustered Watch to be enabled for watch: CLW1549311488.
        Depending on cluster size, this may take some time.
    [W] Broker node node1.gpfs.ss meets the minimum amount of local disk space required 10485760KB 
        to run in degraded mode
        but not the recommended amount of local disk space 20971520KB
    [I] Successfully verified all configured MsgQueue nodes meet minimum local space requirements for 
        Clustered Watch to be enabled for watch: CLW1549311488
    [I] Successfully created Clustered Watch topic on the MsgQueue for watch: CLW1549311488
    [I] Successfully added Clustered Watch configuration file into CCR for watch: CLW1549311488
    [I] Successfully enabled Clustered Watch consumers for watch: CLW1549311488
    [I] Successfully added Clustered Watch policy rules for watch: CLW1549311488
    [I] Successfully enabled Clustered Watch: CLW1549311488
    node1.gpfs.ss:~ #
  8. To disable a clustered watch, issue this command:
    mmwatch Device disable --watch-id WatchID
    The system displays information similar to the following output:
    node1:~ # mmwatch fs2 disable --watch-id CLW1552340432
    [I] Successfully removed Clustered Watch policy rules for watch: CLW1552340432
    [I] Successfully disabled Clustered Watch consumers for watch: CLW1552340432
    [I] Successfully deleted Clustered Watch topic from the MsgQueue for watch: CLW1552340432
    [I] Successfully removed Clustered Watch configuration file from the CCR for watch: CLW1552340432
    [I] Successfully disabled Clustered Watch: CLW1552340432
    node1:~ #
  9. Before suspending a clustered watch, you can check the status with the mmwatch all status command:
    # mmwatch all status
    Device                      Watch ID                 Watch State  
    fs1                         CLW1556906797            Active      
    Node Name                   Status    
    computer1.company.com       HEALTHY    
    computer2.company.com       HEALTHY    
    computer3.company.com       HEALTHY    
    computer4.company.com       HEALTHY    
    
    To suspend a clustered watch, issue this command:
    mmwatch Device suspend --watch-id WatchID
    The system displays information similar to the following output:
    # mmwatch fs1 suspend --watch-id CLW1556906797
    [I] The clustered watch associated with device: fs1 and watch ID: CLW1556906797 is not currently suspended.  
        Will attempt to suspend.
    [I] Successfully suspended Clustered Watch associated with device: fs1 and watch ID: CLW1556906797
    # mmwatch all status
    Device                      Watch ID                 Watch State  
    fs1                         CLW1556906797            Suspended    
    Node Name                   Status    
    computer1.company.com       SUSPENDED  
    computer2.company.com       SUSPENDED  
    computer3.company.com       SUSPENDED  
    computer4.company.com       SUSPENDED  
  10. To resume a suspended clustered watch, issue this command:
    mmwatch Device resume --watch-id WatchID
    The system displays information similar to the following output:
    # mmwatch fs1 resume --watch-id CLW1556906797
    [I] The clustered watch associated with device: fs1 and watch ID: CLW1556906797 is currently suspended.  
        Will attempt to resume.
    [I] Successfully resumed Clustered Watch associated with device: fs1 and watch ID: CLW1556906797
    # mmwatch all status
    Device                      Watch ID                 Watch State  
    fs1                         CLW1556906797            Active      
    Node Name                   Status    
    computer1.company.com       HEALTHY    
    computer2.company.com       HEALTHY    
    computer3.company.com       HEALTHY    
    computer4.company.com       HEALTHY    

See also

Location

/usr/lpp/mmfs/bin