0

Situation is: we have no robots.txt file in this project. It is not in root folder. It is not in /pub folder. But... If we browse ourdomain/robots.txt it is loaded & printed (containing just the sitemap url)

I am not asking here about that concrete content (for this project, it is ok as it is), but the responsible of generating that content

So, question is... If the file does not exist in filesystem, should we asume its content is generated on the fly? Does Magento do this natively? Or should we asume this is caused by some 3rd party module?

1

According to the documentation, Magento generates this page according to settings in the backend.

  1. On the Admin sidebar, go to Content > Design > Configuration.

  2. Find the Global configuration in the first row of the grid and click Edit.

  3. Scroll down and expand the Search Engine Robots section and do the following:

  4. Set Default Robots to the preferred setting.

  5. If needed, enter custom instructions into the Edit Custom instruction of robots.txt file box. For example, while a site is in development, you might want to disallow access to all folders.

To restore the default instructions, click Reset to Default.

When complete, click Save Configuration.

https://docs.magento.com/m2/ce/user_guide/marketing/search-engine-robots.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.