Can Magento 2 Functional Test Framework run inside a Vagrant box?

Question

I'm trying to run the Magento 2 Functional Testing Framework (FTF) inside a Vagrant box, but without success.

My setup is a host machine, which runs Selenium and a Vagrant box which runs Magento 2. What I'm currently doing:

• I'm running Selenium in hub mode on my host machine:
  java -jar selenium-server-standalone-3.0.0-beta3.jar -role hub -port 5555
• In my guest machine, I'm also running selenium, but as a node (which connects to the hub on my host):
  java -jar /vagrant/selenium-server-standalone-3.0.0-beta3.jar -role node -hub http://192.168.56.1:5555/grid/register -port 4444

The connection is established, and I see activity in both console windows that they're connected.

But... as soon as I try the out-of-the box functional tests that come bundled with Magento 2, I get the following error in as well PHP as Selenium (node):

Cannot find firefox binary in PATH. Make sure firefox is installed. OS appears to be: LINUX

Needless to say, I don't have Firefox installed in my Vagrant box. It is however installed on my host machine (together with some other selenium driven browsers, like Chrome and Safari).

But the fact that I get this error makes me wonder: is the FTF capable of running it's test in a setup like this? Or is it really intended to run native on a system? Or am I perhaps overseeing a simple configuration here?

EDIT:

Here are some more details about my setup:

• I'm using Vagrant 1.9.1, Virtualbox 5.1.14 and Selenium standalone server 3.0.0-beta3.
• The forwarded ports are 8080 -> 80, 8443 -> 443, 8025 -> 8025. Note that I don't forward port 4444 nor 5555 (If I'm not mistaken this is not necessary due to the direction of the traffic).
• The IP address of my host machine (eth0) is 10.31.40.176
• My vagrant is setup as a private network on IP 192.168.56.101
• The IP address of my guest machine (eth0) is 10.0.2.15
• On my host machine, I run selenium as a hub on port 5555. The hub shows Nodes should register to http://10.31.40.176:5555/grid/register/.
• On my guest machine, I run selenium as a node on port 4444. I register it to the hub on http://10.31.40.176:5555/grid/register. Both the node as the hub state that they're connected:

node:

07:57:19.092 INFO - Registering the node to the hub: http://10.31.40.176:5555/grid/register
07:57:19.110 INFO - The node is registered to the hub and ready to use

hub:

08:59:29.590 INFO - Registered a node http://192.168.56.101:4444
08:59:29.590 WARN - Cleaning up stale test sessions on the unregistered node http://192.168.56.101:4444

• I've edited my etc/config.xml-file and added the following parameters after </install>:

XML:

<server>
    <item name="selenium"
          type="default"
          browser="Mozilla Firefox"
          browserName="firefox"
          host="10.0.2.15"
          port="4444"
          seleniumServerRequestsTimeout="90"
          sessionStrategy="shared">
        <desiredCapabilities>
            <platform>ANY</platform>
        </desiredCapabilities>
    </item>
</server>

Now when I run the built-in PHPUnit with FTF, I still get the error message:

PHP Fatal error:  Uncaught PHPUnit_Extensions_Selenium2TestCase_WebDriverException: Cannot find firefox binary in PATH. Make sure firefox is installed. OS appears to be: LINUX
Build info: version: '3.0.0-beta3', revision: 'c7b525d', time: '2016-09-01 14:57:03 -0700'
System info: host: 'magento2', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-45-generic', java.version: '1.8.0_121'
Driver info: driver.version: FirefoxDriver in /var/www/public/dev/tests/functional/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase/Driver.php:165

I know that the configuration is read, because when I change browser and browserName to Google Chrome and chrome I get the following error:

PHP Fatal error:  Uncaught PHPUnit_Extensions_Selenium2TestCase_WebDriverException: The path to the driver executable must be set by the webdriver.chrome.driver system property; for more information, see https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver. The latest version can be downloaded from http://chromedriver.storage.googleapis.com/index.html in /var/www/public/dev/tests/functional/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase/Driver.php:165

Any help is greatly appreciated.

Answer 1

Yes, you can run functional tests inside Vagrant box. You need to follow this guide http://devdocs.magento.com/guides/v2.1/mtf/mtf_quickstart.html for configuring environment.

Note, that selenium with specific browser driver must to be run on host machine, but on guest machine you need to set host machine ip (10.0.0.2 for example) as host in config.xml file.

<server>
<item name="selenium"
      type="default"
      browser="Google Chrome"
      browserName="chrome"
      host="{{HOST_MACHINE_IP}}"
      port="4444"
      seleniumServerRequestsTimeout="90"
      sessionStrategy="shared">
    <desiredCapabilities>
        <platform>ANY</platform>
    </desiredCapabilities>
</item>
</server>

Additionally, you can use headless running of functiona tests on guest machine, using xvfb (on CI server for example).