Robots.txt for Magento 2.1+ - where did it go?

Question

Where is the actual robots.txt file like Magento 1.9.x if it exists?

My site URL /robots.txt does not work.

The official dev docs for this http://docs.magento.com/m2/ce/user_guide/marketing/search-engine-robots.html have an image with a note below it stating

This will be included in before head closing tag in page HTML.

When I inspect the homepage in the tag of the HTML I do not see a reference to the robots.txt or any directives like "INDEX, NOINDEX, FOLLOW, NOFOLLOW". I was searching for more information on this, and think others here probably have as well.

Can someone explain how robots.txt functions in Magento2?

Answer 1

You can find your robots.txt in the folder pub/robots.txt, I recommend you remove it and generate using the admin panel, via the route below you can choose the second option listed and change the SEO robots.txt.

Content > Design > Configuration

Answer 2

This excellent guide explains this in detail. In M2, a separate module called module-robots is responsible for routing and rendering the robots file.