As an integrator, should I expect the provided Travis CI configuration to work properly?

Question

As an integrator, after creating a Magento 2.2 project, should I expect the provided Travis CI configuration to work properly?

The Magento Testing Guide explains how to run individual test suites, but does not provide any guidance for the provided .travis.yml.

Why I want this

To meet the requirements of <project>, I'm going to be adding and customizing Magento 2 modules of all kinds: console commands, payment gateways, themes, shipping providers, API endpoints, etc. During this process, I think it would be really nice if an automated build system could tell me, "hey, you goofed and broke this important core thing".

It would also be really nice if there were a standard "Magento 2" way of making this happen, instead of everybody having to reinvent their own square wheel of testing. After creating a project using the given template, one would think that the provided configuration would either (a) work properly, or (b) be documented in such a way that an integrator could get up and running quickly.

...right?

Project setup steps

I've created a brand new, fresh Magento 2.2 project by following the "integrator" path, creating a project via composer by fetching the Magento CE metapackage. More specifically, I ran the following command:

$ composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition testrepo

The resulting project directory comes with (among other things) a .travis.yml file. I've configured the connection between this project and Travis CI, including valid credentials for the custom Magento composer repository. Unfortunately, the Travis CI build doesn't work out of the box.

Issues

Some issues I've run into using the provided configuration (and the workarounds I've tried):

1. The dev/travis/before_*.sh scripts are not executable. This causes the build to error out.

   Workaround: Set executable bit on scripts.

2. I created the project on a machine with PHP 7.1.9, so the composer.lock file ended up with dependency versions which are incompatible with PHP 7.0. One of the specified PHP versions in the Travis matrix is 7.0, which won't work at all.

   Workaround: Remove PHP 7.0 from the build matrix.

3. After composer install runs, it overwrites the dev/ directory. This undoes the changes to my travis scripts, making them non-executable again!

   Workaround: Set executable bit on scripts in the Travis config, directly before running them.

After these changes, it works! Sort of...

What works

• JS specs
  • TEST_SUITE=js GRUNT_COMMAND=spec
• Some integration tests
  • TEST_SUITE=integration INTEGRATION_INDEX=1
  • TEST_SUITE=integration INTEGRATION_INDEX=2
• Functional tests (wow, these take a long time)
  • TEST_SUITE=functional ACCEPTANCE_INDEX=1
  • TEST_SUITE=functional ACCEPTANCE_INDEX=2

What doesn't

• PHP Unit tests
  • TEST_SUITE=unit (very few failures)
• Static analysis / lint
  • TEST_SUITE=static (fails with lint errors on core stuff that I can't change)
  • TEST_SUITE=js GRUNT_COMMAND=static (holy cow, all the errors!)
• Some integration tests
  • TEST_SUITE=integration INTEGRATION_INDEX=3 (fails on reading an XML file from the vendor directory)

Back to the question (and follow-up questions)

Is the .travis.yml intended to be used by integrators?

If so,

1. Why did I trip over so many hurdles while attempting to set it up?
2. What did I do wrong, and/or how do I fix it so that it works as intended?

If not,

1. Why is the .travis.yml file included in the generated project directory?
2. Why is my custom .travis.yml file overwritten with the default one each time I run composer install?

Answer 1

The problem with overriding changes in default files can be easily solved by preparing your composer.json with additional data. All those files are copied from magento modules into the project root by function called after composer install completes. Check MagentoHackathon\Composer\Magento\Command\DeployCommand::execute() for that. Basically magento parses all modules composer.json files and reads extra/map field. All entries in that array is then used to copy file from module directory to project root directory. To prevent that from happening and make sure your changes to those files are intact after deploy/upgrade in your root composer.json add an entry like this:

{
  ...
  "extra": {
    "magento-deploy-ignore": {
      "magento/magento2-base": [
        "/.travis.yml"
      ]
    }
  }
  ...
}

This config will make .travis.yml to be ignored by next composer install or composer upgrade command.

The structure is rather self-explanatory. magento-deploy-ignore is a hash where key is the module name and value an array of elements to ignore. One note is that magento/magento2-base module in its composer.json defines files to copy like this:

"extra": {
  "map": [
    [
      ".travis.yml",
      ".travis.yml"
    ],
    ...

And in ignore you need to start with / that denotes the project root folder.

Answer 2

I have been struggling with this myself (see https://github.com/magento/magento2/issues/12334) and at the moment, I decided that the best way to move forward with Magento2 project development and Travis is to do as follows:

For static tests: Edit the provided black and white lists to only check your own maintained code

For Unit Tests: Edit phpunit.xml.dist to only test your own code

For integration and functional test: Filter out any tests that are failing out of the box