How To
Summary
When disk space reaches 95% used, Elasticsearch/OpenSearch has a protective function that locks indices and prevents new data from being written. This prevents the search service from consuming additional disk space and exhausting the filesystem.
This document explains how to unlock indices after the disk space issue has been resolved.
Objective
While indices are locked, new data is not indexed and therefore is not searchable. Users might see inconsistent search results where new incidents or updates are not returned.
Common causes of disk space issues include:
- Backups running
- Log file rotation
- Increase in database or attachment size
- Logs accumulating in directories where rotation is not configured
Review the appropriate search service log file:
Version before 51.0.9
/var/log/elasticsearch/elasticsearch.logVersion 51.0.9 and later
/var/log/opensearch/opensearch.logYou may see messages similar to:
[2019-08-06T11:22:54,950][WARN ][o.e.c.r.a.DiskThresholdMonitor] [VQNpg25] flood stage disk watermark [95%] exceeded on [VQNpg25iQlK6wVi_5uMvYA][VQNpg25][/var/lib/elasticsearch/nodes/0] free: 2.8gb[4.8%], all indices on this node will marked read-onlyIn /usr/share/co3/logs/client.log you see this message when data is added to the database but it cannot be indexed
11:24:50.831 [http-nio-443-exec-1] ERROR com.co3.search.ElasticSearchManager - Failed to update object 36 of index artifact com.co3.search.exception.ElasticSearchQueryException: ElasticSearch operation PUT on https://localhost:9200/artifact/_doc/36 failed with response 403: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];.
at com.co3.search.ElasticSearchClient$ClientHelper.execute(ElasticSearchClient.java:246)
at com.co3.search.ElasticSearchClient.put(ElasticSearchClient.java:333)
at com.co3.search.ElasticSearchManager.updateData(ElasticSearchManager.java:306)
at com.co3.search.ElasticSearchManager.sendToElasticUpdateData(ElasticSearchManager.java:881)
at com.co3.search.ElasticSearchUpdater.onObjectsCommitted(ElasticSearchUpdater.java:98)Steps
Version 51.0.11.0 and later
Stop the search provider service:
sudo systemctl stop restrict-search-providerGet the state of the indices:
sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"Look for:
"read_only_allow_delete" : "true"Example:
{
"incident" : {
"settings" : {
"index" : {
"blocks" : {
"read_only_allow_delete" : "true"
}
}
}
}
}Remove the restriction for all indices:
sudo curl -k -X PUT -H "Content-Type: application/json" \
http://localhost:9200/_all/_settings \
-d ' {"index.blocks.read_only_allow_delete": null} 'Verify the lock has been removed:
sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"The following should no longer be present:
"read_only_allow_delete" : "true"Restart the services:
sudo systemctl restart resilient
sudo systemctl start restrict-search-providerVersions before 51.0.11.0
Stop the Elasticsearch service:
sudo systemctl stop restrict-elasticGet the state of the indices:
sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"Look for:
"read_only_allow_delete" : "true"Example:
{
"incident" : {
"settings" : {
"index" : {
"blocks" : {
"read_only_allow_delete" : "true"
}
}
}
}
}Remove the restriction for all indices:
sudo curl -k -X PUT -H "Content-Type: application/json" \
http://localhost:9200/_all/_settings \
-d ' {"index.blocks.read_only_allow_delete": null} 'Verify the lock has been removed:
sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"The following should no longer be present:
"read_only_allow_delete" : "true"Restart the services:
sudo systemctl restart resilient
sudo systemctl start restrict-elasticRebuild Search Index
After unlocking the indices, new data will be indexed immediately. However, data created while the indices were locked will not be indexed.
A full reindex is required.
Run:
sudo resutil configset -key elastic_server.init_schema -bvalue trueRestart IBM SOAR:
sudo systemctl restart resilientSearch functionality resumes immediately after the commands are executed, but results may remain incomplete until the rebuild finishes.
The rebuild time depends on the amount of data stored.
Data is indexed from newest to oldest, so newer incidents become searchable before older incidents.
Additional Information
- /usr/share/co3/logs/client_access_log<YYYY-MM-DD>.log
- /var/log/elasticsearch/
- /tmp/
- /home/resadmin/
- /crypt/backups/
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
30 June 2026
UID
ibm11160026