Where is the actual robots.txt file like Magento 1.9.x if it exists?
My site URL /robots.txt does not work.
The official dev docs for this http://docs.magento.com/m2/ce/user_guide/marketing/search-engine-robots.html have an image with a note below it stating
This will be included in before head closing tag in page HTML.
When I inspect the homepage in the tag of the HTML I do not see a reference to the robots.txt or any directives like "INDEX, NOINDEX, FOLLOW, NOFOLLOW". I was searching for more information on this, and think others here probably have as well.
Can someone explain how robots.txt functions in Magento2?