I am trying to only run the test StorefrontVerifySecureURLRedirectCheckout. To run it I am using the following command:
vendor/bin/mftf run:test StorefrontVerifySecureURLRedirectCheckout -k
(I have to use
-k to skip generating due to an error see the update)
When executing the above command it executes much more tests than just the desired one:
Magento\FunctionalTestingFramework.functional Tests (935) ---------------------- Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, Asserts -------------------------------------------------------------------------------- AdminAddConfigurableProductWithDropDownAttributeAndCustomSourceTestCest: Signature: Magento\AcceptanceTest\_MSI_Multi_Mode\Backend\AdminAddConfigurableProductWithDropDownAttributeAndCustomSourceTestCest:AdminAddConfigurableProductWithDropDownAttributeAndCustomSourceTest Test: tests/functional/Magento/FunctionalTest/_generated/MSI_Multi_Mode/AdminAddConfigurableProductWithDropDownAttributeAndCustomSourceTestCest.php:AdminAddConfigurableProductWithDropDownAttributeAndCustomSourceTest Scenario -- /******** Beginning execution of MSI_Multi_Mode suite before block ********/ [...] /******** Execution of MSI_Multi_Mode suite before block complete ********/ [...] AdminAddConfigurableProductWithDropDownAttributeWithNewSourceTestCest: Admin add configurable product with drop down attribute with new source test Signature: Magento\AcceptanceTest\_MSI_Multi_Mode\Backend\AdminAddConfigurableProductWithDropDownAttributeWithNewSourceTestCest:AdminAddConfigurableProductWithDropDownAttributeWithNewSourceTest Test: tests/functional/Magento/FunctionalTest/_generated/MSI_Multi_Mode/AdminAddConfigurableProductWithDropDownAttributeWithNewSourceTestCest.php:AdminAddConfigurableProductWithDropDownAttributeWithNewSourceTest Scenario -- [..]
It seems like the whole MSI_Multi_Mode suite is being executed.
I've compared both tests but I didn't find any difference that could lead to such a behavior.
I found the following in the official docs:
<test>is included in
<suite>, it cannot be generated in isolation to the rest of the contents of the suite (see suites for details).
If I understand it correctly is that if a test is included it cannot be run separately.
The test StorefrontVerifySecureURLRedirectCheckout actually is defined in the suite SecureStorefrontURLSuite due to SecureStorefrontURLSuite.xml which includes the group secure_storefront_url.
vendor/bin/mftf run:test StorefrontVerifySecureURLRedirectCheckout without the
-k argument as written above it will result in the following error:
In TestGenerator.php line 147: Cannot reference test configuration for generation without accompanying suite.
How is it possible to run a single test that is included in a suite?
A possible solution might be to create a new suite that only includes this one test but it doesn't seems more like a hack to me and it's quite some effort to always create or change suites just to run a single test. I wonder if there is any other solution for this problem.
I just verified, it is possible to wrap the single test in a suite and then execute the suite with
vendor/bin/mftf run:group SingleTestSuiteName --remove but this IS NOT really an solution/answer I am looking for due to the inconveniences it brings with having to keep changing the suite all the time.