Magento 2.3 performance drop after running php bin/magento setup:di:compile

Question

While working on a development instance I installed the stripe payment module and then ran

php bin/magento setup:di:compile

I hadn't previously done this on this install and it's been running for over a month totally stable and with acceptable performance for a small aws instance. However since this was run the performance of the system has dropped to unacceptable levels and pins php7.2-fpm to 100%.

It's so slow even the cron jobs start backing up and consuming more and more cpu time.

Any ideas what might be going on and what I can do to debug it?

EDIT I've added new relic onto the system and can see the following

Category    Slowest components  Count   Duration    %
Custom  Magento\Framework\Config\Dom::_mergeNode    731 38,700 ms   53%
Custom  Magento\Framework\Config\Reader\Filesystem::_readFiles  18  7,950 ms    11%
Custom  Magento\Framework\Config\Dom::_getMatchedNode   113 5,020 ms    7%
Custom  Magento\Framework\Config\Dom::validateDomDocument   8   2,790 ms    4%
Custom  Magento\Framework\ObjectManager\Config\Mapper\Dom::convert  7   2,740 ms    4%
Custom  Magento\WebapiAsync\Code\Generator\Config\RemoteServiceReader\Communication::read   1   2,570 ms    4%
Remainder   Remainder   1   12,700 ms   18%
Total time          72,500 ms   100%

Answer 1

Edited - This didn't work

Finally found the solution to this, it appears the for some reason when doing the di compile the system had turned the cache off or somehow otherwise invalidated it but the status still showed as on when doing

php /bin/magento cache:status

I fixed it by running

php bin/magento cache:disable
php bin/magento cache:enable
php bin/magento setup:upgrade

This seems to have reset everything back to a working state.

update So this seemed to work for a short period, performance then reverted to very poor levels. I've disabled all 3rd party and custom modules, cleared the cache, deleted the generated/ folder, tried default mode, developer mode and production to no effect.

Answer 2

I ran into the same issue and it turned out it was caused by running composer update as described in this bug report: https://github.com/magento/magento2/issues/17634#issuecomment-492500726

You can sometimes fix it by running bin/magento cache:enable, but sometimes you'll need to go into env.php and manually edit the cache_types section to look like this:

    'cache_types' => [
        'config' => 1,
        'layout' => 1,
        'block_html' => 1,
        'collections' => 1,
        'reflection' => 1,
        'db_ddl' => 1,
        'eav' => 1,
        'customer_notification' => 1,
        'config_integration' => 1,
        'config_integration_api' => 1,
        'target_rule' => 1,
        'full_page' => 1,
        'config_webservice' => 1,
        'translate' => 1,
        'compiled_config' => 1
    ],