Catalog Search Indexer runs indefinitely with alot of backlog

Question

I've run into this issue with my indexers lately where my Catalog Search Indexer will run indefinitely with a lot of backlogged tasks. And if, and when, I edit any product or even add any new attribute without even assigning it to an attribute set this backlog will increase exponentially and will cause certain, if not most, products to disappear COMPLETELY from the frontend, but NOT from the backend. I had this issue the day before where I found my Catalog Search indexer giving this when I run the indexer:status command

Catalog Search | Reindex required | Schedule | working (952 in backlog) | 2022-03-08 15:33:26|

My indexers will be delayed for hours before being completed, which is probably why the products disappear.

These are my current cron jobs I thought they might be relevant information to the issue I posed.

Can someone provide some insight as to why that may be happening?

Answer 1

This can happen due to poor indexer optimization. If you have set it up in the schedule mode, it may be that the new process is "adding" itself to the waiting queue because the previous ones are still running - if your Catalog Search indexer has status "working" in the indexer_state table, it means that you can't run a new indexer job.

If you have made any changes to the native Magento indexer, try to optimize them.