Using the nodetool drain command

Problem

Common Usage:

This command is usually executed during a maintenance activity and before the Cassandra process is shut down.
 

Action taken:

• Stops listening on the native transport and storage(gossip) port, thereby stop accepting client connections
• Flushes all memtables from this node to SSTables on disk.

Note: After executing nodetool drain command on a node, the Cassandra process requires a restart to start accepting connections on that node.

Why to execute this command:

NOTE: Stopping the Cassandra process is still required after executing the nodetool drain command for a clean shutdown.

Usage:

$ nodetool <connection_options> drain

Help:

$ nodetool help drain

SYNOPSIS
        nodetool [(-h <host> | --host <host>)] [(-p <port> | --port <port>)]
                [(-pw <password> | --password <password>)]
                [(-pwf <passwordFilePath> | --password-file <passwordFilePath>)]
                [(-u <username> | --username <username>)] drain

OPTIONS
        -h <host>, --host <host>
            Node hostname or ip address

        -p <port>, --port <port>
            Remote jmx agent port number

        -pw <password>, --password <password>
            Remote jmx agent password

        -pwf <passwordFilePath>, --password-file <passwordFilePath>
            Path to the JMX password file

        -u <username>, --username <username>
            Remote jmx agent username

Example:

• Say we want to drain the node - 10.10.xx.yy in a Cassandra cluster, execute:

$ nodetool -h 10.10.xx.yy drain

• To view all stages of the drain process, review Cassandra's debug.log (Default location: /var/log/cassandra)
• A typical debug.log snippet during the drain process:

INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:47,621  StorageService.java:1862 - DRAINING: starting drain process
INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:47,659  NodeSyncService.java:485 - Disabled NodeSync service
INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:47,660  HintsService.java:222 - Paused hints dispatch
INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:47,662  Gossiper.java:1975 - Announcing shutdown
DEBUG [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:47,662  StorageService.java:2941 - Node /10.10.xx.yy state shutdown, token [-9223372036854775808]
INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:47,663  StorageService.java:2944 - Node /10.10.xx.yy state jump to shutdown
:
INFO  [ACCEPT-/10.10.xx.yy] 2021-06-02 13:56:53,670  MessagingService.java:1081 - MessagingService has terminated the accept() thread
DEBUG [MemtablePostFlush:1201] 2021-06-02 13:56:54,792  ColumnFamilyStore.java:1050 - forceFlush requested but everything is clean in local
:
:
<truncated>
:
:
DEBUG [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:54,800  StorageService.java:1864 - DRAINING: stopping mutations
DEBUG [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:54,802  StorageService.java:1864 - DRAINING: clearing background IO stage
INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:54,802  HintsService.java:222 - Paused hints dispatch
INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:54,827  StorageService.java:1862 - DRAINED
INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:54,831  PluginManager.java:195 - Plugins are stopped.
INFO  [RMI TCP Connection(57270)-10.10.xx.yy] 2021-06-02 13:56:54,831  Daemon.java:954 - shutdown complete.

Note:

• Expected behavior: nodetool status will show the node as 'DS' (Down Stopped) after the drain completes on that node:

$ nodetool status | grep 10.10.xx.yy 
DS 10.10.xx.yy 1.48 GiB 1 ? c7d3d6dd-2a71-4b5f-b84f-8ae2587a7a6b rack1

• Restart Cassandra after drain
• Re-check the nodetool status to validate its status as 'UN' (UP Normal)