Magento 2 Reindex error: snapshot in progress

Question

I'm using Magento 2.3 with cloud ElasticSearch and sometimes reindex cron gives the following error

[2019-09-18 19:24:13] main.ERROR: Cron Job indexer_reindex_all_invalid has an error: {"error":{"root_cause":[{"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot."}],"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot."},"status":400}. Statistics: {"sum":0,"count":1,"realmem":0,"emalloc":0,"realmem_start":171966464,"emalloc_start":157486136} [] []

[2019-09-18 19:24:13] main.CRITICAL: {"error":{"root_cause":[{"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot."}],"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot."},"status":400} {"exception":"[object] (Elasticsearch\\Common\\Exceptions\\BadRequest400Exception(code: 400): {\"error\":{\"root_cause\":[{\"type\":\"snapshot_in_progress_exception\",\"reason\":\"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot.\"}],\"type\":\"snapshot_in_progress_exception\",\"reason\":\"Cannot delete indices that are being snapshotted: [[product_1_v1284]]. Try again after snapshot finishes or cancel the currently running snapshot.\"},\"status\":400} at /vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:620)"} []

The error occurs when magento try to reindex during the snapshot of elasticsearch platform (once a day).

After this error some indexes are in suspended status and product categories on frontend became empty.

To unlock them I usually do a manual reset and reindex.

Is there a way to check with an ssh script if one or more indexes are suspended? If it is possible, I can create a script that automatically reset and reindex them when necessary.

Other ideas?

Answer 1

Magento having an inbuilt processor for the reindexing if found invalid.

vendor/magento/module-indexer/Cron/ReindexAllInvalid.php

the execute function called reindexAllInvalid of the vendor/magento/module-indexer/Model/Processor.php

Using shell script, triggering custom console command will be helpful in such cases.

Shell Script

#!/usr/bin/env bash
echo "Checking for any suspended indexers"
php bin/magento custom:findandfixindex
echo "Done!!";

app/code/Custom/Indexer/etc/di.xml

<type name="Magento\Framework\Console\CommandListInterface">
    <arguments>
        <argument name="commands" xsi:type="array">
            <item name="customIndexerFixer" xsi:type="object">Custom\Indexer\Console\Command\FixIndexerCommand</item>
        </argument>
    </arguments>
</type>

app/code/Custom/Indexer/Console/Command/FixIndexerCommand.php

<?php
namespace Custom\Indexer\Console\Command;

use Magento\Framework\Exception\LocalizedException;
use Magento\Framework\Indexer\StateInterface;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class FixIndexerCommand extends \Magento\Indexer\Console\Command\AbstractIndexerManageCommand
{

    protected function configure()
    {
        $this->setName('custom:findandfixindex')
            ->setDescription('Resets indexer if found in processing or suspended')
            ->setDefinition($this->getInputList());

        parent::configure();
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $indexers = $this->getIndexers($input);
        $output->writeln('Looking for invalid indexers....');
        foreach ($indexers as $indexer) {
            try {
                if ($indexer->getStatus() == StateInterface::STATUS_WORKING) {

                    //Remove/add a comment to reset the only working indexers
                    $indexer->getState()
                        ->setStatus(StateInterface::STATUS_INVALID)
                        ->save();

                    $output->writeln($indexer->getTitle() . ' indexer found in working status.');
                }
            } catch (LocalizedException $e) {
                $output->writeln($e->getMessage());
            } catch (\Exception $e) {
                $output->writeln($indexer->getTitle() . ' indexer unknown error while fixing:');
                $output->writeln($e->getMessage());
            }
        }
        $output->writeln('Finished .....!!!');
    }
}

Non-Magento Way

To wait until a snapshot is completed on aws only

The following wait snapshot-completed example pauses and resumes running only after it confirms that the specified snapshot is completed.

aws ec2 wait snapshot-completed \
    --snapshot-ids snap-1234567890abcdef0