Problem

Summary

The error message in the title is thrown when the node is started without the metadata directory correctly configured.
This article refers to a particular scenario: when upgrading DSE to version 6.8 from previous versions (5.1, 6.0 or 6.7) and reusing the previous cassandra.yaml. However, everytime the metadata directory is not found, will cause the same error.
Infact, when using a cassandra.yaml from versions 5.1, 6.0 or 6.7, the property "metadata_directory" is not explicitly defined in the cassandra.yaml, therefore apache cassandra will use the default location and will create a metadata directory in: $CASSANDRA_HOME/data/metadata, however in some cases this location may not be reachable by the upgraded nodes and the error is thrown preventing the node to startup.

Applies to

DSE 6.8.x

Symptoms

This error message is thrown in the system.log when a node is started and the metadata directory is not available:

ERROR [DSE main thread] 2021-12-20 06:42:50,217 CassandraDaemon.java:932 - Exception encountered during startup
java.lang.RuntimeException: A node with address /10.160.103.11 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:790)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:1161)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:922)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:856)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:419)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:541)
at org.apache.cassandra.service.CassandraDaemon.activate0(CassandraDaemon.java:754)
at org.apache.cassandra.service.CassandraDaemon.access$100(CassandraDaemon.java:88)
at org.apache.cassandra.service.CassandraDaemon$3.run(CassandraDaemon.java:715)

which means that the node is trying to JOIN a cluster as a new node, rather than just starting up after a stop.
 

Cause

With DSE 6.8 a new metadata directory is added to the list of data directories required by apache cassandra.
When restarting a node after an upgrade to version DSE6.8, the node needs to read from a file called "local" in the metadata directory. The file "local" contains the node_ID and other information about the peer nodes.
Failing to read the "local" file, will cause the node to startup as a new node with a new node_ID and token range. However because a node with the same IP address exists in the peers table, the startup is aborted and the error message is generated.

Solution

To resolve this scenario, the cassandra.yaml must be correctly updated with the "metadata_directory" property and the Apache Cassandra user must be able to access the metadata directory and read the files within.

See also

https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/config…

Last Reviewed Date: 2023/12/20