Magento Test Framework(MTF) vs Selenium with Java - Magento 2.0.1

Question

I am a newbie in magento testing framework, I had good knowledge about selenium webdriver with Java. I put some little research about MTF sample test module(CreateWidgetEntity), When I run this its follow some flow to create widget in admin. This same thing is possible in selenium with java also(eclipse IDE). so I am not sure MTF provide what best features in testing. I could found one its generate screenshot,logs as per individual test.

My question is

What are the best features MTF provide over selenium with Java(eclipse)?

Answer 1

The MTF uses Selenium under the hood so it's not really an either/or question.

The biggest benefit of using the MTF is that you can use the tests that someone else has written already vs. if you used just selenium you would need to write all the tests yourself.

According to Magento there are about 300 test scenarios that the MTF for Magento 2 provides.

Answer 2

Kristof at Fooman, is right, for you as a benefit could be that there are quite a big amount of existing tests that simplify creating new test for your module/functionality.

Of course you can create/rewrite any test using another framework or don't use it at all, like it with Java + Selenium in Eclipse.

One of the difference that the MTF uses variations instead of data provider in PHPUnit. It allows to use one test flow but with different data and specific assertions for this data. The data is passed to test and constraints for verification. Based on the information in variation specific fixture class can be created that represents the entity under test (for example, widget), that can be persisted to the Magento if needed or serves as data storage for the entity data.

Using only variation allows you easily extend basic test from another modules with the data the is specific to a custom modules, this allows to follow modularity in tests without duplicating test flow. The same approach is used with repositories (data storage), fixtures (entity containers), pages and blocks on it, that increase it's re-usability.

Also MTF allows to create custom test suites based on tagging, where you can specify rules according where you can determine what tests, variations, contraints must be run. So you can create flexible rule for each testing activity and then run all tests in parallel on different Magento instances.

You can familiar with some MTF structure from the documentation http://devdocs.magento.com/guides/v2.0/mtf/mtf_introduction.html

BTW, Selenium it's just one of the possible drivers that can be used for running test and creating screenshot is default function.