At some point Magento added a module called Magento_Robots that will dynamically render robots.txt content for the website being viewed.
Everything is already wired up, so all you would need to do is:
- Set the robots.txt for each website in admin from: Content > Design > Configuration > Search Engine Robots > Edit custom instruction of robots.txt file
- Make sure whether your various stores are loading from separate domains or loading from subfolders....that the correct store scope / store code is used to run the app
Note: Looking at the robots model, it looks like it only fetches robots.txt configuration data from as low as the website scope. I think that makes sense sense store views are typically used for languages
Robots Router
The \Magento\Robots\Controller\Router::match()
router matches requests to /robots.txt
and dispatches a robots controller action

Robots Action
The \Magento\Robots\Controller\Index\Index::execute()
action adds robots_index_index layout handle and sets the result pages content type to plain text.

Robots Layout + Block + Model
The \Magento\Robots\Block\Data::_toHtml()
outputs robots.txt content from the \Magento\Robots\Model\Robots::getData()
method. The data for robots.txt is fetched from the design/search_engine_robots/custom_instructions
system configuration.


