How can I have different robots.txt files for each store in a Magento installation with multiple stores?

Question

I have a Magento installation with multiple stores, and I need to have different robots.txt files for each store. Currently, I have a single robots.txt file in the root directory of my Magento installation, but I want to customize it separately for each store.

Is there a specific configuration or code modification that needs to be done to have different robots.txt files for each store? I want to ensure that each store has its own set of rules and directives within the robots.txt file.

Any guidance or code examples would be greatly appreciated. Thank you!

Answer 1

By default, you shouldn't create robots.txt manually in the webroot. Just update the Store configuration in: Content > Design > Configuration > Edit Custom instruction of robots.txt for each stores.

Answer 2

At some point Magento added a module called Magento_Robots that will dynamically render robots.txt content for the website being viewed.

Everything is already wired up, so all you would need to do is:

1. Set the robots.txt for each website in admin from: Content > Design > Configuration > Search Engine Robots > Edit custom instruction of robots.txt file
2. Make sure whether your various stores are loading from separate domains or loading from subfolders....that the correct store scope / store code is used to run the app

Note: Looking at the robots model, it looks like it only fetches robots.txt configuration data from as low as the website scope. I think that makes sense sense store views are typically used for languages

Robots Router

The \Magento\Robots\Controller\Router::match() router matches requests to /robots.txt and dispatches a robots controller action

Robots Action

The \Magento\Robots\Controller\Index\Index::execute() action adds robots_index_index layout handle and sets the result pages content type to plain text.

Robots Layout + Block + Model

The \Magento\Robots\Block\Data::_toHtml() outputs robots.txt content from the \Magento\Robots\Model\Robots::getData() method. The data for robots.txt is fetched from the design/search_engine_robots/custom_instructions system configuration.