Run Integration/Functional Tests on an Existing Database for Customer Specific Projects

Question

We develop customer specific projects in Magento 2.3.x, with different customer specific extensions.

Development is based on a copy of the production database (master.sql), so we do not usually use install scripts to setup the full customer data and products, but do this in the admin panel, if possible.

The master.sql contains the product catalog, but not the customer data.

Thus said, we did not do much automatic testing yet - for reasons.

We want to introduce automated testing now, on "all levels" Magento 2 provides. But it would not make sense to reengineer the full project fixture wise. This is why we would like to base test tests for our custom modules on the master.sql database.

For ApiFunctional tests for example, we found out already, that we can set in phpunit_rest.xml

    <const name="TESTS_CLEANUP" value="disabled"/>
    <const name="TESTS_MAGENTO_INSTALLATION" value="disabled"/>

and in install-config-mysql.php we can use the MySQL credentials for the just imported master.sql and set 'cleanup-database' => false,

Next we found out, that when using fixtures from the sales module vendor/magento/magento2-base/dev/tests/integration/testsuite/Magento/Sales/_files/default_rollback.php cleans up all orders, products and stocks. That is why we would patch that file.

And here it begins to feel messy, that is why I asking here

1. Is anybody else using this approach?
2. Is it a very bad idea or could it work?
3. What are you "best practices" and tricks to make this work?

Answer 1

I totally understand your motivation to simply run the tests against the copy of the production database. I've done that in the past, too, but it also came back to bite me like it is happening to you.

In the following response I'll focus on integration testing, since I think that is where the most value lies. For MFTF or legacy functional tests I can't give any recommendations.

I think it would make sense to encapsulate the state in master.sql into a set of course grained fixtures. Do not use the fixture script files like the Magento core team does, they are hard to manage, inflexible and full of hardcoded values.

Instead, I suggest writing a set of classes that set up parts of master.sql state. One for the websites and stores, one for the customers, one for the catalog, one for the orders and so on.

In each test suite you can then add @before, @after, @beforeClass and @afterClass annotated methods to trigger the fixtures needed for a specific test, and clean them up afterwards.

Because the fixtures will be rather course grained, it would be very slow to create them with Magento's ORM models. Also, converting the existing DB dump into pure PHP takes a lot of work.

When faced with that issue I decided to split the full DB dump into smaller dumps and imported those by shelling out to MySQL CLI commands to load the files. It worked well enough for me, much better than always working with a fixed DB.

It's probably still a fair bit of work to convert the master.sql dump to individual SQL files, but in my experience having proper test isolation is very much worth it.