Magento 2.2.x cache disabled automatically

Question

First of all, I couldn't find any info about this kind of issue anywhere on the web.

We have a production environment with git integration. We pull our changes to it only via git (git pull).

The problem is that , somehow in one of the steps Magento caches disables automatically (all zeros when checking cache:status). That causes a problem if this is missed via programmer further causing server overload due to high traffic 'bashing' to the Magento without cache.

Maybe some of you have seen this issue before? We do not know when or how it's exactly happens.
And it kinda appears randomly.

Usual steps we do:

• enabling maintenance
• git pull
• composer install (if needed)
• module enable Vendor_ModuleName (if needed)
• setup:upgrade (if needed)
• clearing static stuff
• deployment command
• clearing caches
• clearing opcache
• disabling maintenance

I would appreciate any valuable suggestions that could help to solve this kind of issue.

Answer 1

This looks like a known issue:
This does happen from time to time on the project I'm working on, but I wasn't able to find the steps to reproduce. All I can say is that it happens during a deployment process.
All I could find is that under certain circumstances a file .regenerate is written in the var folder (either at setup upgrade or composer install / upgrade) and if that file is present when running setup:di:compile the cache is disabled and re-enabled when the compilation process is finished.
For some reason, sometimes the cache is not re-enabled.
We took the quick and dirty approach and made the last step of the deployment process php bin/magento cache:enable to be sure. SO basically we hid the dirt under the rug.

You can find the code that disables the cache in here
It is wrapped in a TODO: remove statement.

Answer 2

For anyone interested, I think I can bring some light about this issue. It seems to be a concurrency (race condition) problem in \Magento\Framework\Code\GeneratedFiles::cleanGeneratedFiles, when the var/.regenerate flag is set, and more than one process/request tries to clean generated files.

It is more likely to happen when you have cron enabled, and many cron groups using use_separate_process config. When more than one process are trying to clean the same, FileIterator fails with different messages similar to: FilesystemIterator::__construct(/Users/adrianmartinez/Sites/r2-project-develop-b2b/environments/2-2-develop-b2b/magento/generated/code): failed to open dir: No such file or directory.

Moving up call to $this->write->delete(self::REGENERATE_FLAG);, just after flag existence check resolves the problem, as the first arriving process marks itself as responsible of cleaning files.

I leave here a demo video about how to replicate the problem: https://youtu.be/9-X1cIIY7y8

And the script used for it:

#!/bin/bash

# \Magento\Framework\Code\GeneratedFiles has a concurrency problem
# Create regenerate flag and launch parallel commands that try to regenerate at the same time
# This is a real case, cron:run launches stand alone processes in parallel

# Created by magento composer installer upon code install or module enable/disable
touch var/.regenerate

# Launch parallel commands
# Error differs each execution, sometimes it even works
bin/magento cron:run --group=ddg_automation --bootstrap=standaloneProcessStarted=1 2>&1 &
bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1 2>&1 &

wait
echo "All done"