FAQ - OS Upgrade for DSE/Cassandra environments

Problem

Summary

With CentOS 7 EOL, a number of question arise on the topic of OS upgrade with DataStax Enterprise DSE) and Apache Cassandra ™️ (C*) running on it. This article will provide a checklist and some recommended practices on the topic.

Applies to

• DataStax Enterprise (all versions)
• Cassandra (all versions)

Pre-requisites

For DSE, check first if the ongoing version is compatible with the new OS. Refer to the Supported Platforms documentation

For example, it is important to note that DSE 6.8 and 5.1 have a minimum required patch level for upgrade to RHEL 9, Oracle Linux 9, or Ubuntu 22.

Avoid upgrading OS and DSE at the same time

DataStax Support would recommend to decouple DSE and the OS upgrade. The reasoning behind this is that schema clashes could occur and that bootstrapping and updating DSE/Cassandra nodes should be 2 separate operations. This article is written by support that does not get called when things go smoothly, and in case of an issue during the upgrade, finding if DSE or the OS is causing a problem may be challenging

It may be possible to patch DSE at the same time, but that requires pre-validation to confirm if the 2 versions used (existing on current OS and new DSE on new OS) are compatible. Schema clashes could occur and this require review in nodetool describecluster to make sure there's no schema disagreement - something common in upgrade and the reason why DDL and other pre-requisites exist in upgrade process.

Another potential issue is the streaming process when leveraging Zero Copy Streaming in DSE, which would copy some files bypassing the streaming process. This should be therefore avoided for versions with different sstables versions. Refer to the Product Compatibility documentation

General recommendations

While it may be possible to run an in-place OS upgrade, it is generally advisable to avoid major OS upgrade (the author spent a chunk of his holiday reinstalling his Ubuntu desktop following an OS upgrade failure), which means a new machine with new OS should be provisioned. Most organizations would also prefer this approach. In this scenario, a new set of machines would be made available.

It is also a good time to give the DB a refresh. Originally, when vnodes came out, the recommendation was to set it to 256 tokens and the topology cannot be changed after a DC is built. However, the current recommendation is to set it to 8 or 16 tokens as experience showed some administration tasks such as bootstrapping and repair are drastically slowed down by large amount of tokens.

Best approach: Building a new DC

Considering the pre-requisites are met (DSE and OS are compatible), building a new DC with new OS and same version of DSE as the existing DC, taking care of reviewing our documentations:

Vnodes documentation

Recommended Production Settings documentation

Adding a DataCenter to a Cluster

This approach has many advantages, such as these:

• the application can remain on the existing environment and be tied to the existing DataCenter, provided the LoadBalancingPolicy are correctly defined and consistency level (CL) is set to LOCAL_, LOCAL_QUORUM being the most popular, in the application
• it allows to follow more recent topology practices and make sure the system uses the better snitches (GossipingPropertyFileSnitch as an example) and replication factor (NetworkTopologyStrategy), or to update them accordingly.
• the new OS can be built and integrated in the cluster without disruption
• some testing can be done on the new DC and the environment can be validated for read performance independently. Again, the premise being that the reads and writes are using a DC based CL
• minimal downtime
• most risk averse
• allows a streaming to a new DC which would potentially erase some old/shadow tombstones that may not have had a chance to compact in the past.

When the system is considered ready, the application can be switched to the new DC and the old DC can be decomissioned.

If the pre-requisites aren't met, 2 solutions to consider:

1- Leverage another OS which is not EOL, but compatible with the version of DSE/Cassandra in use

2- Upgrade DSE/Cassandra prior to introducing the new DC

Second best approach Lift and Shift nodes one at a time

It is recognized that allocating a full new set of machines for a new DC may not be possible, particularly on larger clusters. It is possible to do a "lift and shift" approach and rotate the nodes. Note that the most common misconception is to use replace_address to restream the data to a new node. However, this is an approach to use when a node has become unavailable. The "lift and shift" approach mimics what kubernetes deployments of Cassandra and DSE do, which is to attach the volume to a new machine.

• Preconfigure DSE/Cassandra on the new node with cassandra.yaml settings of the old node, with new IP configuration. If other configuration files have been tweaked, also consider reconfiguring them. See the Configuration documentation for a non-exhaustive list
• Make sure the directories are correctly configured and the mount points for the disks are correctly set in the new machine
• Stop DSE/Cassandra on the old node
• Copy the data folder (and the solr data directory) to the new machine
• Note for DSE 6.8: Make sure to copy the metadata folder which contains topology info
• Start DSE/Cassandra on the new node

This allows for minimal downtime, but depending on the data density - and by extension the time it takes to copy, may require a repair.

In this scenario however

• the topology cannot be updated (use the same num_tokens)
• best practices/recommended settings are still good to review

Other approaches

Using replace_address flag

It is possible to re-stream the data to a node, but avoid re-using the existing IPs. This stems from this JIRA limitation that both Cassandra and DSE suffer from. If re-streaming to an existing IP, make sure to repair post upgrade https://issues.apache.org/jira/browse/CASSANDRA-12344

OS in-place upgrade

In case of an OS in-place upgrade, this is beyond DataStax responsibility and the question should be refered to the Linux System Administrator and tested thoroughly prior to roll out in production - note this last statement applies to all the scenarios though