DSE5.1.0 node will not start if vnodes split across disks

Summary

Post Upgrade to DSE5.1.0 some nodes have were observed to fail to start due to a problem spanning vnodes over more than one disk partition

Applies to

DSE5.1.0, Apache Cassandra 3.0.0 - 3.11.x

Symptoms

The node will fail to start with an error similar to the following in the /var/log/cassandra/system.log

ERROR main 2017-05-03 09:18:42,719 CassandraDaemon.java:705 - Exception encountered during startup
java.lang.IndexOutOfBoundsException: Index: 33, Size: 33
at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~na:1.8.0_102
at java.util.ArrayList.get(ArrayList.java:429) ~na:1.8.0_102
at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5255) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5229) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.compaction.CompactionStrategyManager.getCompactionStrategyIndex(CompactionStrategyManager.java:239) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.compaction.CompactionStrategyManager.getCompactionStrategyFor(CompactionStrategyManager.java:205) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.compaction.CompactionStrategyManager.startup(CompactionStrategyManager.java:181) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.compaction.CompactionStrategyManager.reload(CompactionStrategyManager.java:312) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.compaction.CompactionStrategyManager.<init>(CompactionStrategyManager.java:91) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:433) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:618) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:592) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:583) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:420) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:333) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:104) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:471) ~dse-core-5.1.0.jar:5.1.0
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:599) ~cassandra-all-3.10.0.1652.jar:3.10.0.1652
at com.datastax.bdp.DseModule.main(DseModule.java:93) dse-core-5.1.0.jar:5.1.0

Cause

The problem arises when splitting vnode ranges over more than one disk, which is mapped to the data_file_directories parameter in the cassandra.yaml

The following Apache Cassandra jira captures this issue:

CASSANDRA-13229

Workaround

Moving all the sstables to one disk, as long as there is room, will allow the node to start successfully.

Solution

Upgrade to DSE5.1.1, which includes the fix for DB-654

Last Reviewed Date: 2023/11/07