Handling schema disagreements and "Schema version mismatch detected" on node restart

Problem

Summary

A warning 'schema version mismatch detected' occurs on restarting some DataStax Enterprise (DSE) nodes.

Applies to

• DSE 5.1
• DSE 6.7
• DSE 6.8

Symptoms

If some nodes are unavailable while restarting another node, the following warning occurs:

Warning: schema version mismatch detected; check the schema versions of your nodes in system.local and system.peers.

Cause

This error can be a false positive if some nodes are unavailable. This confusing error message when nodes are down is known behavior as tracked in CASSANDRA-11142.

Determine cause

Run the nodetool describecluster command:

nodetool describecluster

If the nodes are unreachable, the command output looks like this:

Cluster Information:
  Name: Cluster12345abcde
  Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
  Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
  Schema versions:
  UNREACHABLE: 1176b7ac-8993-395d-85fd-41b89ef49fbb: [10.202.205.203]
               9b861925-1a19-057c-ff70-779273e95aa6: [10.80.207.102]
               8613985e-c49e-b8f7-57ae-6439e879bb2a: [10.116.138.23]

Solution

1. Restart unreachable nodes.
2. Repeat step 1 until the output of the nodetool describecluster command shows that all nodes have the same schema version number and only one schema version appears in the output.

Related DataStax documentation

• Fixing a table schema collision
• nodetool describecluster