Magento 2.4 reindex catalogsearch_fulltext issue with elasticsearch

Question

I have Magento2.4 with an elastic search version (7.9.2), Indexing of the catalogsearch_fulltext shows error

{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [magen to2_product_1_v272] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-
stage watermark, index has read-only-allow-delete block];"}],"type":"cluster_blo
ck_exception","reason":"index [magento2_product_1_v272] blocked by: [TOO_MANY_RE QUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-d
elete block];"},"status":429}

I tried running the following commands to free up.

- curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
- curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
- sudo systemctl restart elasticsearch

and now if I run the index for the first time it works the second time again it shows the same error

sudo php bin/magento indexer:reindex catalogsearch_fulltext ;

I tried multiple times with free command facing the same issue.

Also executed.

sudo php bin/magento setup:upgrade;
sudo php bin/magento setup:di:compile;
sudo php bin/magento setup:static-content:deploy -f;
sudo php bin/magento indexer:reindex;
sudo php bin/magento cache:clean;
sudo php bin/magento cache:flush;
sudo chmod -R 0777 pub/;
sudo chmod -R 0777 var/;
sudo chmod -R 0777 generated/

Note: I have a stock update cron that runs each night, Around 200+updates will happen and I have scheduled the index of all products from save to schedule (Default 1 min cron).

This only happens at night, The clear command for elastic and restart I executed during the morning noticing the issue.

Since its showing TOO_MANY_REQUESTS how that's being triggered. Is it from Magento or from Elasticsearch?

How to fix it? Please help.

Answer 1

My disk was full, Try curl -XGET 'http://localhost:9200/_cat/allocation?v' to see the free space available at Elasticsearch.

I cleared a few spaces and it's working now, Have to allocate space.

We have three states on Elasticsearch Disk Watermark, Based on your config file this will show, the Flood state is the one where read_only_allow_delete will be enabled by Elasticsearch

• Low disk watermark
• High disk watermark
• Flood stage disk watermark

PUT _cluster/settings
{
  "transient": {
   
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
    "cluster.info.update.interval": "1m"
  }
}

For more details, you can refer to the link.

https://opster.com/guides/elasticsearch/capacity-planning/elasticsearch-disk-watermark/
https://opster.com/guides/elasticsearch/capacity-planning/elasticsearch-high-disk-watermark/