Just upgraded my DEV server to Magento 2.4.2-p1.
I am very confused about why its using HUGE amounts of RAM when reindexing Catalog Search.
You can see on this screenshot it's filled 115GB of RAM.
The catalog is big (300k +). This issue is not happening on current PRODUCTION site which is running on 2.3.3.
when I was upgrading Magento. So it cant really be an issue with duplicate keys.
My env.php settings for cache look like this (both sites use the same settings):
'cache' => [ 'frontend' => [ 'default' => [ 'backend' => 'Cm_Cache_Backend_Redis', 'backend_options' => [ 'server' => '127.0.0.1', 'database' => '0', 'port' => '6379' ], 'id_prefix' => '123_' ], 'page_cache' => [ 'backend' => 'Cm_Cache_Backend_Redis', 'backend_options' => [ 'server' => '127.0.0.1', 'port' => '6379', 'database' => '1', 'compress_data' => '0' ], 'id_prefix' => '123_' ] ] ],
We're running Varnish as Full Page Cache, but its not connected yet for testing purposes.
I mentioned this cache overload in my other thread: https://stackoverflow.com/questions/68278489/docker-compose-mnt-sdb-is-always-full/68287144#68287144
^^^ here because cache was set to file storage it would fill up the hard-drive within 15 minutes. It would create 100GB just like that.
I was thinking maybe it's using DDL cache is doing this?
After about 2 hours, Redis finally blew up and crashed:
It looks like maybe the plugin that is managing the ElasticSearch + Catalog Search indexes is not actually saving the data into the DB? I contacted devs to see what they say.
UPDATE: Looks like I'm not alone in this. There is a GitHub issue with other people reporting the same issue: https://github.com/magento/magento2/issues/32118