Problem

Purging gossip state on a node

The DataStax Enterprise Help Center also provides troubleshooting information.

Gossip information is persisted locally by each node to use immediately on node restart without having to wait for gossip communications.

Procedure

1. Use the nodetool assassinate to shut down the problem node.

   This takes approximately 35 seconds to complete, so wait for confirmation that the node is deleted.

2. If this method doesn't solve the problem, stop your client application from sending writes to the cluster.
3. Take the entire cluster offline:
   1. Drain each node.

      nodetool options drain

   2. Stop each node.
4. Clear the data from the peers directory, remove all directories in the peers-UUID directory, where UUID is the particular directory that corresponds to the appropriate node:

   sudo rm -r /var/lib/cassandra/data/system/peers-UUID/*

   CAUTION:
   Use caution when performing this step. The action clears internal system data from the database and may cause application outage without careful execution and validation of the results. To validate the results, run the following query individually on each node to confirm that all of the nodes are able to see all other nodes.

   select * from system.peers;

5. Clear the gossip state when the node starts:
   • For tarball or Installer No-Services installations, use a command line option or edit the cassandra-env.sh. To use the command line:

     installation_location/bin/cassandra -Dcassandra.load_ring_state=false

   • For package or Installer Services installations or if you are not using the command line option above, add the following line to the cassandra-env.sh file:

     JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"

6. Bring the cluster online one node at a time, starting with the seed nodes.
   See Starting and stopping DataStax Enterprise.

What's next

Remove the line you added in the cassandra-env.sh file.