Data Resurrection in DSE and Cassandra

Problem

Summary

This article explains the cause, mitigation and prevention of data resurrection in DSE and Cassandra.

Applies to

• DataStax Enterprise 6.8

• DataStax Enterprise 6.7

• DataStax Enterprise 5.1

• Cassandra 3.x

• Cassandra 4.x

Symptoms

Data resurrection or zombie data refers to the unintended reappearance of deleted data. Due to the distributed nature of Cassandra and its eventual consistency model, this phenomenon can occur if expired tombstones are not managed properly. There are two stages of data resurrection:

• Stage 1 : Expired tombstones not yet purged

A tombstone is kept for a certain period defined by the gc_grace_seconds property, which allows sufficient time for all nodes to receive the deletion marker. Once gc_grace_seconds expires, the tombstone is subject to be purged by compaction process. In this stage, a tombstone is not propagated to all replicas due to temporary network issues or node failures and the expired tombstone has not yet been purged from the system.

• Stage 2: Expired tombstones purged

In this stage, the expired tombstone has been purged yet the original data still persists. Please refer to
Deletion in a distributed system for a diagram about how this stage is developed.
 

Cause

Expired tombstones are handled specially in DSE/Cassandra in that they are not propagated in read path nor considered as valid data when returning the replica to a coordinator. See also Read Repair.
Let's create a scenario of stage 1 - with the following steps in a 3 node test cluster.
1. Create a table and insert some data

cassandra@cqlsh> CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1':'3'};
cassandra@cqlsh> create table test.t1 (   pk1 ascii,
       ck1 ascii,
       c3 ascii,
       PRIMARY KEY (pk1, ck1)
   )WITH gc_grace_seconds = 5 and nodesync = {'enabled': 'false'};
cassandra@cqlsh> insert into test.t1(pk1,ck1,c3) values ('1','k1','v1');

Run "nodetool flush test t1" on all 3 nodes.

2. Create an expired tombstone and leave one node missing with the tombsone
- Stop one node, then issue the following 

cassandra@cqlsh> consistency LOCAL_QUORUM;
cassandra@cqlsh> delete from test.t1 where pk1='1' and ck1='k1';

Issue "nodetool flush test t1" and "nodetool disablehandoff" on the two available nodes.
After 5 seconds (as configured by gc_grace_seconds property), bring back the stopped node. We now have two nodes having the expired tombstone and one node missing the tombstone.

3. See zombie data in action
When querying using consistency LOCAL_QUORUM, we can see the deleted data is back:

The query result may be inconsistent depending on which replicas are retrieved. When the two nodes with expired tombstones are queried, the above record will not be returned.

Workaround

If expired tombstones are not yet purged(stage 1), you can perform either of steps below to mitigate the situation:

• Run nodetool repair to sync expired tombstones in the cluster. Note nodesync would not be able to propagate expired tombstones.
• Extend gc_grace_seconds so that tombstones are not expired

If expired tombstones are already purged, you need to reissue deletion to remove the old data.

Solution

To prevent old data from being reintroduced to the nodes accidentally, perform regular node repairs to ensure all replicas are consistent within gc_grace_seconds and to reduce the likelihood of data resurrection due to missing tombstones.
When possible, you may also consider expiring data using TTL instead of deleting data manually. In this way, there is no separation of tombstone and data to delete, hence prevent data resurrection. See Expiring data with TTL for more details.