0

I have a Magento installation with multiple stores, and I need to have different robots.txt files for each store. Currently, I have a single robots.txt file in the root directory of my Magento installation, but I want to customize it separately for each store.

Is there a specific configuration or code modification that needs to be done to have different robots.txt files for each store? I want to ensure that each store has its own set of rules and directives within the robots.txt file.

Any guidance or code examples would be greatly appreciated. Thank you!

2 Answers 2

1

By default, you shouldn't create robots.txt manually in the webroot. Just update the Store configuration in: Content > Design > Configuration > Edit Custom instruction of robots.txt for each stores.

0

At some point Magento added a module called Magento_Robots that will dynamically render robots.txt content for the website being viewed.

Everything is already wired up, so all you would need to do is:

  1. Set the robots.txt for each website in admin from: Content > Design > Configuration > Search Engine Robots > Edit custom instruction of robots.txt file
  2. Make sure whether your various stores are loading from separate domains or loading from subfolders....that the correct store scope / store code is used to run the app

Note: Looking at the robots model, it looks like it only fetches robots.txt configuration data from as low as the website scope. I think that makes sense sense store views are typically used for languages

Robots Router

The \Magento\Robots\Controller\Router::match() router matches requests to /robots.txt and dispatches a robots controller action

Robots Router

Robots Action

The \Magento\Robots\Controller\Index\Index::execute() action adds robots_index_index layout handle and sets the result pages content type to plain text.

Robots Action

Robots Layout + Block + Model

The \Magento\Robots\Block\Data::_toHtml() outputs robots.txt content from the \Magento\Robots\Model\Robots::getData() method. The data for robots.txt is fetched from the design/search_engine_robots/custom_instructions system configuration.

Robots layout

Robots block

Robots model

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.