Magento 2.3 robots.txt file

Question

Situation is: we have no robots.txt file in this project. It is not in root folder. It is not in /pub folder. But... If we browse ourdomain/robots.txt it is loaded & printed (containing just the sitemap url)

I am not asking here about that concrete content (for this project, it is ok as it is), but the responsible of generating that content

So, question is... If the file does not exist in filesystem, should we asume its content is generated on the fly? Does Magento do this natively? Or should we asume this is caused by some 3rd party module?

Answer 1

According to the documentation, Magento generates this page according to settings in the backend.

1. On the Admin sidebar, go to Content > Design > Configuration.

2. Find the Global configuration in the first row of the grid and click Edit.

3. Scroll down and expand the Search Engine Robots section and do the following:

4. Set Default Robots to the preferred setting.

5. If needed, enter custom instructions into the Edit Custom instruction of robots.txt file box. For example, while a site is in development, you might want to disallow access to all folders.

To restore the default instructions, click Reset to Default.

When complete, click Save Configuration.

https://docs.magento.com/m2/ce/user_guide/marketing/search-engine-robots.html