Magento 2 update.log increases significant size every minute

Question

UPDATE:

I ended up using logrotate utility to handle the ever-growing log files. I've also set up few cron jobs that will check for unneeded temporary files from time to time.

---

I have the standard Magento 2 cron configured and running. After two days, I encountered some problems while running some actions in the Admin panel. When I checked the server logs, I found out that it was a disk space issue. After further investigation, I found out that this update.log.txt is the culprit. The file size was increasing by >1MB every minute. There was a time it reached more than 20GB.

Is it normal for a standard Magento 2 setup? Or can & how I can disable it or at least minimize it?

It happens to both developer and production mode.

Cron setup was based on this guide.

Cron tab:

* * * * * <path to php binary> <magento install dir>/bin/magento cron:run | grep -v "Ran jobs by schedule" >> <magento install dir>/var/log/magento.cron.log
* * * * * <path to php binary> <magento install dir>/update/cron.php >> <magento install dir>/var/log/update.cron.log
* * * * * <path to php binary> <magento install dir>/bin/magento setup:cron:run >> <magento install dir>/var/log/setup.cron.log

SAMPLE CONTENTS OF THE LOG

/var/www/html/setup/src/Magento/Setup/Console/Command/I18nCollectPhrasesCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/MaintenanceStatusCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/BackupCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/DependenciesShowFrameworkCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/UninstallCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/InfoTimezoneListCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/MaintenanceAllowIpsCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/ModuleStatusCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/InstallStoreConfigurationCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/InfoBackupsListCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/MaintenanceDisableCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/DiCompileCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/InstallCommand.php /var/www/html/setup/src/Magento/Setup/Console/Command/UpgradeCommand.php /var/www/html/setup/src/Magento/Setup/Fixtures/IndexersStatesApplyFixture.php /var/www/html/setup/src/Magento/Setup/Fixtures/StoresFixture.php /var/www/html/setup/src/Magento/Setup/Fixtures/CategoriesFixture.php /var/www/html/setup/src/Magento/Setup/Fixtures/SimpleProductsFixture.php /var/www/html/setup/src/Magento/Setup/Fixtures/tax_rates.csv /var/www/html/setup/src/Magento/Setup/Validator /var/www/html/setup/src/Magento/Setup/Validator/DbValidator.php /var/www/html/setup/src/Magento/Setup/Validator/IpValidator.php /var/www/html/setup/src/Magento/Setup/Validator/AdminCredentialsValidator.php /var/www/html/setup/index.php /var/www/html/setup/performance-toolkit /var/www/html/setup/performance-toolkit/.htaccess /var/www/html/setup/performance-toolkit/profiles /var/www/html/setup/performance-toolkit/profiles/ce /var/www/html/setup/performance-toolkit/profiles/ce/large.xml /var/www/html/setup/performance-toolkit/profiles/ce/small.xml /var/www/html/setup/performance-toolkit/profiles/ce/medium.xml /var/www/html/setup/performance-toolkit/profiles/ce/extra_large.xml /var/www/html/setup/performance-toolkit/README.txt /var/www/html/setup/performance-toolkit/benchmark.jmx /var/www/html/vendor/.htaccess [] []

I found few places where this log might be generated / updated :

<INSTALL_DIR>setup\src\Magento\Setup\Console\Command\CronRunCommand.php:
       75      protected function execute(InputInterface $input, OutputInterface $output)
       76      {
       77:         $notification = 'setup-cron: Please check var/log/update.log for execution summary.';
       78  
       79          if (!$this->deploymentConfig->isAvailable()) {

    <INSTALL_DIR>setup\src\Magento\Setup\Model\Cron\ReadinessCheck.php:
      164          $write->writeFile(self::SETUP_CRON_JOB_STATUS_FILE, $resultJson);
      165  
      166:         // write to permanent log file, var/log/update.log
      167          foreach ($errorLogMessages as $errorLog) {
      168              $this->status->add($errorLog, \Psr\Log\LogLevel::ERROR, false);

    <INSTALL_DIR>setup\src\Magento\Setup\Model\Cron\SetupLoggerFactory.php:
       22      {
       23          $logger = new \Monolog\Logger($channelName);
       24:         $path = BP . '/var/log/update.log';
       25          $logger->pushHandler(new \Monolog\Handler\StreamHandler($path));
       26          return $logger;

    <INSTALL_DIR>setup\src\Magento\Setup\Model\Cron\SetupStreamHandler.php:
       17       * @var string
       18       */
       19:     protected $fileName = '/var/log/update.log';
       20  
       21      /**

    <INSTALL_DIR>setup\src\Magento\Setup\Model\Cron\Status.php:
       81          $this->varReaderWriter = $filesystem->getDirectoryWrite(DirectoryList::VAR_DIR);
       82          $this->statusFilePath = $statusFilePath ? $statusFilePath : '.update_status.txt';
       83:         $this->logFilePath = $logFilePath ? $logFilePath : DirectoryList::LOG . '/update.log';
       84          $this->updateInProgressFlagFilePath = $updateInProgressFlagFilePath
       85              ? $updateInProgressFlagFilePath

    <INSTALL_DIR>setup\src\Magento\Setup\Test\Unit\Model\Cron\StatusTest.php:
       71          $this->varReaderWriter->expects($this->any())
       72              ->method('getAbsolutePath')
       73:             ->with('log/update.log')
       74:             ->willReturn('DIR/var/log/update.log');
       75:         $this->assertEquals('DIR/var/log/update.log', $this->status->getLogFilePath());
       76      }
       77  

    <INSTALL_DIR>update\app\code\Magento\Update\CronReadinessCheck.php:
       16      const CRON_JOB_STATUS_FILE = '.update_cronjob_status';
       17  
       18:     const UPDATE_CRON_LOG_FILE = 'var/log/update.log';
       19  
       20      /**#@+
       ..
       91          file_put_contents(MAGENTO_BP . '/var/' . self::CRON_JOB_STATUS_FILE, $resultJson);
       92  
       93:         // If non-accessible paths are found, log an 'error' entry for the same in update.log
       94          if ( !$success && !empty($outputString) ) {
       95              $updateLoggerFactory = new UpdateLoggerFactory();

    <INSTALL_DIR>update\app\code\Magento\Update\Status.php:
       67      ) {
       68          $this->statusFilePath = $statusFilePath ? $statusFilePath : MAGENTO_BP . '/var/.update_status.txt';
       69:         $this->logFilePath = $logFilePath ? $logFilePath : MAGENTO_BP . '/var/log/update.log';
       70          $this->updateInProgressFlagFilePath = $updateInProgressFlagFilePath
       71              ? $updateInProgressFlagFilePath

    <INSTALL_DIR>update\app\code\Magento\Update\UpdateLoggerFactory.php:
       29       */
       30      public function __construct(
       31:         $logFile = '/var/log/update.log',
       32          $channelName = 'update-cron'
       33      ) {

    <INSTALL_DIR>update\cron.php:
       15  $status = new \Magento\Update\Status();
       16  $cronReadinessChecker = new \Magento\Update\CronReadinessCheck();
       17: $notification = 'update-cron: Please check var/log/update.log for execution summary.' . PHP_EOL;
       18  
       19  if (!$cronReadinessChecker->runReadinessCheck()) {

    <INSTALL_DIR>update\dev\tests\integration\testsuite\Magento\Update\StatusTest.php:
       43          $this->statusFilePath = __DIR__ . '/_files/update_status.txt';
       44          $this->tmpStatusFilePath = TESTS_TEMP_DIR . '/update_status.txt';
       45:         $this->tmpStatusLogFilePath = TESTS_TEMP_DIR . '/update.log';
       46          $this->updateInProgressFlagFilePath = TESTS_TEMP_DIR . '/update_in_progress.flag';
       47          $this->updateErrorFlagFilePath = TESTS_TEMP_DIR . '/update_error.flag';

---

UPDATE:

I ended up using logrotate utility to handle the ever-growing log files. I've also set up few cron jobs that will check for unneeded temporary files from time to time.

Answer 1

You probably set wrong permissions on some folders. /update/cron lists all folders with wrong permissions

Easy way is just comment out in crontab update and setup

#* * * * * <path to php binary> <magento install dir>/update/cron.php >> <magento install dir>/var/log/update.cron.log
#* * * * * <path to php binary> <magento install dir>/bin/magento setup:cron:run >> <magento install dir>/var/log/setup.cron.log

You don't need them if you don't need to update and install from admin panel.

It's dangerous idea for production server. You can manually update via composer on test/local server; test it; then copy updates to production server.

The one you really need is

* * * * * <path to php binary> <magento install dir>/bin/magento cron:run | grep -v "Ran jobs by schedule" >> <magento install dir>/var/log/magento.cron.log

It manages "indexers, sends automated e-mails, generates the sitemap, and so on" - as described in devdocs

Answer 2

As far as I know, the var/log/update.log file is used with the cron indeed.

It is declared in \setup\src\Magento\Setup\Model\Cron\Status.php

$this->logFilePath = $logFilePath ? $logFilePath : DirectoryList::LOG . '/update.log';

And then retrieved via the getLogFilePath() method.

If you check where this method is being called, it's under the \setup\src\Magento\Setup\Model\Cron\JobFactory.php :

$logStream = fopen($cronStatus->getLogFilePath(), 'a+');
$streamOutput = new MultipleStreamOutput([$statusStream, $logStream]);

Then the $streamOutput variable is passed as a parameter to the job class, for example:

return new JobUpgrade(
    $this->serviceLocator->get('Magento\Setup\Console\Command\UpgradeCommand'),
    $objectManagerProvider,
    $streamOutput,
    $this->serviceLocator->get('Magento\Setup\Model\Cron\Queue'),
    $cronStatus,
    $name,
    $params
);

If I keep going with this example, the $this->output is used in the execute() method:

$this->command->run(new ArrayInput($this->params), $this->output);

The run() method is declared in vendor\symfony\console\Symfony\Component\Console\Command\Command.php

The $output variable is then used in several places such as :

    $this->initialize($input, $output);

As well as :

    if ($input->isInteractive()) {
        $this->interact($input, $output);
    }

    $input->validate();

    if ($this->code) {
        $statusCode = call_user_func($this->code, $input, $output);
    } else {
        $statusCode = $this->execute($input, $output);
    }

Another interesting one:

        } elseif (OutputInterface::VERBOSITY_VERY_VERBOSE === $output->getVerbosity()) {
            $output->writeln('<comment>Install the proctitle PECL to be able to change the process title.</comment>');
        }

According to this last code, some stuff can be written in case of very verbose parameter. Maybe you setup your cron jobs with -vv and that's why stuff is getting logged.

This is not a definite answer, but you should follow that stack trace I just gave you to investigate the issue.

Answer 3

/var/log/update.log is cron log file and this issue especially comes when you have files permission issue. "This is what i got in my case when i open that file."

and this could be the fix: https://community.bitnami.com/t/update-log-file-grows-rapidly-in-magento-2-1-1/49253/6

You need to set correct permissions and ownership to your Magento directories and files.

Answer 4

As per I know update.log is your custom log file because Magento 2 generate log this kind of files

• var/log/debug.log – all information that is saved by Psr\Log\LoggerInterface::debug() is stored there.
• var/log/exception.log – all exceptions information is stored there.
• var/log/system.log – information from other logging levels is saved there.

I am not sure why It generate so much log data in your file might be some syntax error or something like that.

But I would suggest just search with the update.log with in your Magento 2 code directory and remove that log from your code.