After upgrading from Magento 2.4.1 to 2.4.2 on Feb 25th our install has been experiencing very poor performance. The 2.4.1 install ran flawlessly with an uptime of in excess of 80 days, now symptoms include;
- Random (every hour or so) high server loads and very high frontend response times, these sometimes settle back down
- Random server hangs (no response and 500s every 24-48hrs) requiring an external reboot to get it back online
- Magento double processing orders (ie customers are charged twice and two orders appear backend)
- Magento takes customer payment (via stripe or Paypal) but does not complete order and no order appears in the backend (this usually coincides with a high server load event) - a while later a failed payment email will come through saying products are out of stock that should have a saleable quantity
- when server hangs and recovers sometimes Elasticsearch will not recover in this instance the site and server will run fine but no products will be available on the site obviously!
- During these high load events if i execute top -c the CPUs seem to be tied up with a high %wa
I'm guessing that there are some changes between 2.4.1 and 2.4.2 that are causing elasticsearch to overload now possibly with reindexing or a cron job change? Sooo...
Environment (off the top of my head)
- Centos 7.9 VPS
- WHM/cPanel
- 2CPUs
- 2gb Ram
- 3gb Swap
- 60gb disk sapce (69% used)
- Litespeed websserver 5.4.12 and Litemage cache plugin
- PHP 7.4.16
- MySql 8.0.23
Things I have tried so far...
- switched from PHP7.3 to PHP7.4.16
- Upgaraded the MFTF from 2.5.4 to 3.4
- Upgraded from MySQL 7 to MySQL 8
- Set elasticsearch Xms1g Xmx1g in jvm.options.d - this was done before upgrade and worked fine with 2.4.1
- added LimitMEMLOCK=infinity in etc/systemd/system/elasticsearch.service.d/override.conf - this was done before upgrade and worked fine with 2.4.1
- added Restart=always in etc/systemd/system/elasticsearch.service.d/override.conf - this hasnt stopped the overloads though.
- Installed sodium PHP module - Had to be done otherwise users could not log in after upgrade
Anyone else having similar issues after upgrading from 2.4.1 to 2.4.2? any thoughts? I' sure this is impacting on my sales.
Update 27/3/21
After spending time observing the process manager during these events it seems to be the cron jobs causing the high server load. When it happens i am seeing multiple instances of:
opt/cpanel/ea-php74/root/usr/bin/php /home/andy/public_html/bin/magento cron:run
and
jailshell (andy) [init] ilshell -c /opt/cpanel/ea-php74/root/usr/bin/php /home/andy/public_html/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /home/andy/public_html/var/log/magento.cron.log
When the load is really high (40+) ther are upwards of 10 of each of these processes. If i kill these processes he load returns to normal.
I tried deleting the crontab temporarily and truncating the cron_schedule table and the site ran fine with minimal load for about 12hrs, but the high load issues returned as soon as i reinstalled the crontab.
I have compared cron.php in 2.4.1 and 2.4.2 and there are no changes.The release note for 2.4.2 don't point to any obvious changes to the crons either. I didn't change anything or add any modules when i upgraded from 2.4.1 to 2.4.2 either.
Any ideas or pointers on how i can work out why the crons are now causing such high load?
