7

I'm trying to add PHP Static test at Frontools to simplify and improve testing process and if it's possible increase performance, b/c it takes so long to get any results.

GH Issue - https://github.com/SnowdogApps/magento2-frontools/issues/45

I'm not familiar with PHP testing libs / tools, so have few questions for you:

  1. Is there any reason why M2 use testing libs (PHPUnit and PHP_CS) from 2014 instead of fresh one?
  2. Is it normal that output of this test looks like a mess and it's hard to understand what and where something wrong happened? I compare it with output of code quality tests for CSS/JS and it's a nightmare. Is there any better reporter available or some other way to get a meaningful report, instead of something looking like a PHP backtrace?
  3. Is there any reason why it's so slow? It take ~7-8 minutes to analyse template files. Most of front-end tests in worst case take a few seconds, so there is no way to get live feedback about issues.
  4. How to run this type of tests when we have single module (i.e. theme), not whole Magento 2 instance (CI tests)?
  5. It looks like PHP_CS has already a simple wrapper for Gulp, but I'm not sure where configuration is stored. Is it in /.php_cs file?
Improve this question

2 Answers 2

Reset to default
2

I'm just going to give you my thoughs about that, I may be wrong for some points but maybe it'll clarify some things:

Is there any reason why M2 use testing libs (PHPUnit and PHP_CS) from 2014 instead of fresh one?

My guess is that, as Magento 2 development started a few years ago, the team used the library that were available at that time. As they have written a lot of tests in Magento 2, they probably stick with the version they used at the time they wrote the first tests in order not to break the tests. Pretty sure they could upgrade that at some point in a major version. You can now make Feature Requests on the forum about that: https://community.magento.com/t5/Magento-2-Feature-Requests-and/idb-p/feature-requests

Is normal that output of this test looks like a mess and it's hard to understand what and where something wrong happens? I compare it with output of code quality tests for CSS/JS and it's nightmare. Is there any better reporter available or other way to get meaningful report, instead of something looking like a PHP backtrace?

Well, yeah the default PHPUnit output is not super nice. Most of the IDE supports PHPUnit and give a better nicer output. For example here's the official documentation to run the tests in PHPStorm: http://devdocs.magento.com/guides/v2.0/test/unit/unit_test_execution_phpstorm.html There's also tools like VisualPHPUnit that provides a GUI for the unit tests: https://github.com/VisualPHPUnit/VisualPHPUnit

Is there any reason why it's so slow? It take ~7-8 minutes to analyse template files. Most of front-end tests in worst case take a few seconds, so there is no way to get live feedback about issues.

First, Magento 2 comes with a lot of tests which definitely has an impact on performance (but hey that's what it takes to have your code cover right ;) ). I'm pretty sure Vinai Kopp mentionned some performance improvements you can do to make the tests run faster, I hope he'll give us some insights when he's back from holidays.

How to run this type of tests when we have single module (i.e. theme), not whole Magento 2 instance (CI tests)?

Do you mean run the tests for one module ? Yes you can definitely do that, I suggest you check that answer from KAndy (he's part of the Magento 2 team) : Run tests for a specific module in Magento2

It looks like PHP_CS has already a simple wrapper for Gulp, but I'm not sure where configuration is stored. It's in /.php_cs file?

I'm not sure where is that simple wrapper. I don't think it is the .php_cs file. From what I know this file is only used for the GitHub pre-commit static reviews

Improve this answer
1
  • Thanks for answer! I comment most of things below @fschmengler answer, but looks like point 5. is unclear. I'm not asking about things related with Gulp implementation, b/c I'm familiar with JS tools :) I just need to know how PHP_CS configuration is stored on Magento level.
    – igloczek
    Commented Sep 2, 2016 at 10:28
2
+200

These are quite a lot questions at once, but I can answer at least some:

  1. Is there any reason why M2 use testing libs (PHPUnit and PHP_CS) from 2014 instead of fresh one?

Major development in Magento 2 started around 2014, so they used the tools available at that time. When PHPUnit 5 came out, there was already a huge number of tests that were not compatible with the new version (see this forum thread for an example), so it's understandable that they postponed an update.

I assume, the reasons to stick with an old PHP_CS version are similar, though I don't have a concrete example here.

  1. Is normal that output of this test looks like a mess and it's hard to understand what and where something wrong happens? I compare it with output of code quality tests for CSS/JS and it's nightmare. Is there any better reporter available or other way to get meaningful report, instead of something looking like a PHP backtrace?

IDEs like PHPStorm have good integration with these tools, where you can see code sniffer results directly in the source files and get a nice GUI around PHPUnit tests as well.

Besides that, PHPUnit has various output options. For example with the --testdox argument, you'll get a human readable checklist of passed and failed tests. It provides less infos but a readable overview. You can also get it in HTML format with --testdox-html=OUTPUTFILE. Similarily you can get the code coverage report in HTML with --coverage-html OUTPUTDIR.

But the more useful output formats are XML and JSON formats that can be read by other applications like VisualPHPUnit or CI servers.

PHPUnit parameters for report generation:

Code Coverage Options:

  --coverage-clover <file>  Generate code coverage report in Clover XML format.
  --coverage-crap4j <file>  Generate code coverage report in Crap4J XML format.
  --coverage-html <dir>     Generate code coverage report in HTML format.
  --coverage-php <file>     Export PHP_CodeCoverage object to file.
  --coverage-text=<file>    Generate code coverage report in text format.
                            Default: Standard output.
  --coverage-xml <dir>      Generate code coverage report in PHPUnit XML format.

Logging Options:

  --log-junit <file>        Log test execution in JUnit XML format to file.
  --log-tap <file>          Log test execution in TAP format to file.
  --log-json <file>         Log test execution in JSON format.
  --testdox-html <file>     Write agile documentation in HTML format to file.
  --testdox-text <file>     Write agile documentation in Text format to file.

More info: https://phpunit.de/manual/current/en/textui.html

PHP_CS parameters for report generation

PHP_CS also has different report formats:

--report=xml         PHP_CS XML format
--report=checkstyle  Checkstyle XML format
--report=csv         CSV

(other formats: emacs, svnblame, gitblame)

More info: https://github.com/squizlabs/PHP_CodeSniffer/wiki/Reporting

  1. Is there any reason why it's so slow? It take ~7-8 minutes to analyse template files. Most of front-end tests in worst case take a few seconds, so there is no way to get live feedback about issues.

I can't tell what the reasons are for PHP_CS to take 8 minutes just for template files, but it should be possible for your watcher to only check changed files. The PHPStorm integration does this quite well.

  1. How to run this type of tests when we have single module (i.e. theme), not whole Magento 2 instance (CI tests)?

Simply run phpcs /path/to/theme to only check files in this directory.

  1. It looks like PHP_CS has already a simple wrapper for Gulp, but I'm not sure where configuration is stored. It's in /.php_cs file?

It doesn't look like this wrapper includes a file watcher, so I don't see the benefit.

The .php_cs file defines which files to check and which coding standards to use. This is a PHP_CS configuration file and independent from the gulp wrapper.

Improve this answer
4
  • Thanks for answer! 1. I'd move this topic to improvement proposal thread, b/c looks like no one really know what is stoping i.e. to update PHPUnit to latest 4.x. 2. I'm a Atom user, b/c I don't need this IDE feature at all (it's mostly useless on front-end), so I'm looking only on improvements in console (CI test output is important too). I'd test this parameters, but looks like building a simple custom printer module, will be better option. 3. Looks like a found a cause - static tests run some part of integrity tests too and it's take about 4-5 minutes to finish. 1/2
    – igloczek
    Commented Sep 2, 2016 at 10:12
  • 4. It's not only about running PHP_CS. Static test contains also XXS vulnerability checks and custom functions build on top of PHPUnit, so probably it's necessary to have Magento core package to run anything :< 5. It's a Gulp plugin, so watching files is handled on Gulp level, but it's not a case. I'd have to know where it Magento is stored configuration of this plugin (if exist, but I can't imagine a linter without configuration file :O) 2/2
    – igloczek
    Commented Sep 2, 2016 at 10:18
  • 3.-4.) Indeed, there's the PHPUnit static test suite which takes some time, checking dependencies and whatnot. I found out that you can put a list of files in dev/tests/static/testsuite/Magento/Test/_files/changed_files* (can be generated with get_github_changes.php) and some of the tests will only process these files. But for the rest I don't see a possibility to restrict them per module/theme. What you can do is run a single testsuite, e.g. with phpunit --testsuite "Less Static Code Analysis" (see phpunit.xml.dist for the list of test suites) 5.) yes that would be .php_cs
    – Fabian Schmengler
    Commented Sep 2, 2016 at 12:41
  • I'm thinking about extracting tests to separate composer package and then adding it as a dev dependency to our packages, to run tests on CI env separately. Hope it's going to work somehow :D
    – igloczek
    Commented Sep 2, 2016 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.