I'm using Magento 2.3 with cloud ElasticSearch and sometimes reindex cron gives the following error
[2019-09-18 19:24:13] main.ERROR: Cron Job indexer_reindex_all_invalid has an error: {"error":{"root_cause":[{"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot."}],"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot."},"status":400}. Statistics: {"sum":0,"count":1,"realmem":0,"emalloc":0,"realmem_start":171966464,"emalloc_start":157486136} [] []
[2019-09-18 19:24:13] main.CRITICAL: {"error":{"root_cause":[{"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot."}],"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot."},"status":400} {"exception":"[object] (Elasticsearch\\Common\\Exceptions\\BadRequest400Exception(code: 400): {\"error\":{\"root_cause\":[{\"type\":\"snapshot_in_progress_exception\",\"reason\":\"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot.\"}],\"type\":\"snapshot_in_progress_exception\",\"reason\":\"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot.\"},\"status\":400} at /vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:620)"} []
The error occurs when magento try to reindex during the snapshot of elasticsearch platform (once a day).
After this error some indexes are in suspended status and product categories on frontend became empty.
To unlock them I usually do a manual reset and reindex.
Is there a way to check with an ssh script if one or more indexes are suspended? If it is possible, I can create a script that automatically reset and reindex them when necessary.
Other ideas?