Troubleshooting: cluster_block_exception, index blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark error

Reason

The Elasticsearch exception occurs due to insufficient disk space when the Elasticsearch disk space is not planned properly.

Solution

The error indicates that a data node is critically low on disk space and reached the threshold. To prevent the disk from getting full, when a node reaches the threshold, Elasticsearch blocks the writes to any index with a shard on that node. Elasticsearch removes the blocking when the affected node's disk usage goes below the threshold. To achieve this result, Elasticsearch automatically moves some of the affected node's shards to other nodes in the same data cluster. To restore the write operations immediately, Elasticsearch provides APIs to temporarily increase the threshold, and remove the write block.

However, the long-term solution is to either add more nodes to the data cluster or increase the disk space for the existing nodes. After this is done, the temporary increase of threshold is removed. It is recommended that you have a data cluster with two nodes, two shards, and one replica. To reduce the occurrences of this error, increase the nodes (and increase the disk space for every node) to three nodes, two shards, and two replicas.

Note: Assess your current and future order volume and allocate sufficient disk space for the Elasticsearch data instead of encountering and resolving this error.

For more information, see Elasticsearch disk plan.