IBM Support

Elasticsearch/Opensearch indices are locked after a shortage of disk space

How To


Summary

When disk space reaches 95% used, Elasticsearch/OpenSearch has a protective function that locks indices and prevents new data from being written. This prevents the search service from consuming additional disk space and exhausting the filesystem.

This document explains how to unlock indices after the disk space issue has been resolved.

Objective

While indices are locked, new data is not indexed and therefore is not searchable. Users might see inconsistent search results where new incidents or updates are not returned.

Common causes of disk space issues include:

  • Backups running
  • Log file rotation
  • Increase in database or attachment size
  • Logs accumulating in directories where rotation is not configured

Review the appropriate search service log file:

Version before 51.0.9

/var/log/elasticsearch/elasticsearch.log

Version 51.0.9 and later

/var/log/opensearch/opensearch.log

You may see messages similar to:

[2019-08-06T11:22:54,950][WARN ][o.e.c.r.a.DiskThresholdMonitor] [VQNpg25] flood stage disk watermark [95%] exceeded on [VQNpg25iQlK6wVi_5uMvYA][VQNpg25][/var/lib/elasticsearch/nodes/0] free: 2.8gb[4.8%], all indices on this node will marked read-only

In /usr/share/co3/logs/client.log you see this message when data is added to the database but it cannot be indexed

11:24:50.831 [http-nio-443-exec-1] ERROR com.co3.search.ElasticSearchManager - Failed to update object 36 of index artifact  com.co3.search.exception.ElasticSearchQueryException: ElasticSearch operation PUT on https://localhost:9200/artifact/_doc/36 failed with response 403: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];.   
at com.co3.search.ElasticSearchClient$ClientHelper.execute(ElasticSearchClient.java:246)   
at com.co3.search.ElasticSearchClient.put(ElasticSearchClient.java:333)   
at com.co3.search.ElasticSearchManager.updateData(ElasticSearchManager.java:306)   
at com.co3.search.ElasticSearchManager.sendToElasticUpdateData(ElasticSearchManager.java:881)   
at com.co3.search.ElasticSearchUpdater.onObjectsCommitted(ElasticSearchUpdater.java:98)

Steps

Version 51.0.11.0 and later

Stop the search provider service:

sudo systemctl stop restrict-search-provider

Get the state of the indices:

sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"

Look for:

"read_only_allow_delete" : "true"

Example:

{
  "incident" : {
    "settings" : {
      "index" : {
        "blocks" : {
          "read_only_allow_delete" : "true"
        }
      }
    }
  }
}

Remove the restriction for all indices:

sudo curl -k -X PUT -H "Content-Type: application/json" \
http://localhost:9200/_all/_settings \
-d ' {"index.blocks.read_only_allow_delete": null} '

Verify the lock has been removed:

sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"

The following should no longer be present:

"read_only_allow_delete" : "true"

Restart the services:

sudo systemctl restart resilient
sudo systemctl start restrict-search-provider

Versions before 51.0.11.0

Stop the Elasticsearch service:

sudo systemctl stop restrict-elastic

Get the state of the indices:

sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"

Look for:

"read_only_allow_delete" : "true"

Example:

{
  "incident" : {
    "settings" : {
      "index" : {
        "blocks" : {
          "read_only_allow_delete" : "true"
        }
      }
    }
  }
}

Remove the restriction for all indices:

sudo curl -k -X PUT -H "Content-Type: application/json" \
http://localhost:9200/_all/_settings \
-d ' {"index.blocks.read_only_allow_delete": null} '

Verify the lock has been removed:

sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"

The following should no longer be present:

"read_only_allow_delete" : "true"

Restart the services:

sudo systemctl restart resilient
sudo systemctl start restrict-elastic

Rebuild Search Index

After unlocking the indices, new data will be indexed immediately. However, data created while the indices were locked will not be indexed.

A full reindex is required.

Run:

sudo resutil configset -key elastic_server.init_schema -bvalue true

Restart IBM SOAR:

sudo systemctl restart resilient

Search functionality resumes immediately after the commands are executed, but results may remain incomplete until the rebuild finishes.

The rebuild time depends on the amount of data stored.

Data is indexed from newest to oldest, so newer incidents become searchable before older incidents.

Additional Information

This could happen more than once. Asses your disk requirements, which depend on the usage of IBM Resilient and plan to extend or add extra disk.
 
Consider removing unnecessary files
 
  • /usr/share/co3/logs/client_access_log<YYYY-MM-DD>.log
  • /var/log/elasticsearch/
  • /tmp/
  • /home/resadmin/
  • /crypt/backups/
 

Document Location

Worldwide

[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSIP9Q","label":"IBM Security SOAR"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":"","Line of Business":{"code":"LOB77","label":"Automation Platform"}}]

Document Information

Modified date:
30 June 2026

UID

ibm11160026