At some point Magento added a module called Magento_Robots that will dynamically render robots.txt content for the website being viewed.
Everything is already wired up, so all you would need to do is:
- Set the robots.txt for each website in admin from: Content > Design > Configuration > Search Engine Robots > Edit custom instruction of robots.txt file
- Make sure whether your various stores are loading from separate domains or loading from subfolders....that the correct store scope / store code is used to run the app
Note: Looking at the robots model, it looks like it only fetches robots.txt configuration data from as low as the website scope. I think that makes sense sense store views are typically used for languages
\Magento\Robots\Controller\Router::match() router matches requests to
/robots.txt and dispatches a robots controller action
\Magento\Robots\Controller\Index\Index::execute() action adds robots_index_index layout handle and sets the result pages content type to plain text.
Robots Layout + Block + Model
\Magento\Robots\Block\Data::_toHtml() outputs robots.txt content from the
\Magento\Robots\Model\Robots::getData() method. The data for robots.txt is fetched from the
design/search_engine_robots/custom_instructions system configuration.