Magento 2 - setup:di:compile throws No tokens were provided error

Question

In magento 2 when I run setup:di:compile I get the following error:

[Zend\Code\Exception\RuntimeException]
No tokens were provided

How to solve this problem?

I disabled all additional modules, ran setup:upgrade, but still I get the tokens error.

Answer 1

I experienced this problem and put breakpoints on each instance of 'No tokens were provided' I could find using:

grep -lr 'No tokens were provided' vendor/

which returned:

vendor/zendframework/zend-code/src/Scanner/ConstantScanner.php
vendor/zendframework/zend-code/src/Scanner/TokenArrayScanner.php
vendor/zendframework/zend-code/src/Scanner/MethodScanner.php
vendor/zendframework/zend-code/src/Scanner/PropertyScanner.php
vendor/zendframework/zend-code/src/Scanner/ClassScanner.php

My exception was being thrown in the TokenArrayScanner.php file. The "tokens" instance value is set in the constructor, so I put a break point there and re-launched the process. Unfortunately, this constructor is called many times on numerous files, and data for the file being tokenized is not included in the construct call. I was able to determine that the "tokens" are just lines of the file being inspected. Based on the post by CarComp, I used this to find the offending file:

find . -type f -name '*.php' -empty

This turned up a number of files that lived under paths that contained "dev/tests", which are excluded from scanning, and a single file from a third party developer. removing this file solved the issue and the compile command completed with only complaints and not exceptions.

Answer 2

Here, if you want to know HOW to find those empty files, just run this command (assuming you have linux, and access to the shell) in your magento root directory, or whatever directory you are working in that might have empty files...

find -type f -empty

Answer 3

Seems you have an empty file which is parsed by the \Magento\Setup\Module\Di\Code\Reader\ClassesScanner, which uses the PHP token_get_all method. The exception is thrown, if the method does not return any value, which is the case on an empty file.

Answer 4

Empty files also provokes the No tokens were provided issue, so there should be a check on ClassesScanner to evaluate if used directory entry or any class file is a valid file and if its content is not empty. You need to add proper content within created files OR it shouldn't be defined/created.

Answer 5

You can put this inside if condition:

    if (!$this->tokens) {
        \Zend_Debug::dump(get_object_vars($this); exit;
        throw new \Zend\Code\Exception\RuntimeException('No tokens were provided');
    }

Then check system.log to see which file produced this error.