Where is the actual robots.txt file like Magento 1.9.x if it exists?

My site URL /robots.txt does not work.

The official dev docs for this http://docs.magento.com/m2/ce/user_guide/marketing/search-engine-robots.html have an image with a note below it stating

This will be included in before head closing tag in page HTML.

When I inspect the homepage in the tag of the HTML I do not see a reference to the robots.txt or any directives like "INDEX, NOINDEX, FOLLOW, NOFOLLOW". I was searching for more information on this, and think others here probably have as well.

Can someone explain how robots.txt functions in Magento2?

  • robots.txt is just a file.configure it. click save. and forget.
    – MagenX
    Commented Mar 5, 2018 at 7:23

2 Answers 2


You can find your robots.txt in the folder pub/robots.txt, I recommend you remove it and generate using the admin panel, via the route below you can choose the second option listed and change the SEO robots.txt.

Content > Design > Configuration


This excellent guide explains this in detail. In M2, a separate module called module-robots is responsible for routing and rendering the robots file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.