Problem

Summary

Solr indexes can become out of sync with the underlying data and may need to be rebuilt. To avoid system overloading issues, we recommend to rebuild the indexes one by one and node by node, using the steps described below.
 

Applies to

The following versions:

• DataStax Enterprise 6.8
• DataStax Enterprise 6.9
• DataStax Enterprise 5.1

Steps to Rebuild SOLR index(es)

 Prepare DSE 6.8 and DSE 6.9. Skip for DSE 5.1.

1.  Change the Cover Finder from STATIC to DYNAMIC on all Indexes. The settings are documented here.

dsetool set_core_property demo.health_data shard.set.cover.finder=DYNAMIC

This is a required step; otherwise, blacklisting the node will not work.

Blacklist the nodes

1. Mark the node as unavailable for Solr queries:

nodetool sjk mx -b com.datastax.bdp:type=core,name=EndpointStateTracker -f Blacklisted --set -v true

2. Rebuild index:

dsetool reload_core <keyspace>.<index> reindex=true deleteAll=true distributed=false;

3. Check the status of the index rebuild:

nodetool sjk mxdump -q "com.datastax.bdp:index=<keyspace>.<table>,metricType=ReindexMetrics,name=Progress Percentage,scope=search,type=metrics"

          When the index rebuild is complete, rebuild the next index.

4. After all the indexes required to be rebuilt have completed, mark the node as available for SolR queries:

nodetool sjk mx -b com.datastax.bdp:type=core,name=EndpointStateTracker -f Blacklisted --set -v false

          Repeat for each node one at a time


Once the index has been rebuilt across the cluster, you may check the status of this on each node:

nodetool sjk mxdump -q "com.datastax.bdp:index=<keyspace>.<table>,metricType=ReindexMetrics,name=Progress Percentage,scope=search,type=metrics" 

Or for the whole cluster:

dsetool core_indexing_status <keyspace>.<table>;