Webmaster tool robots.txt fetch failed, fresh magento 2 web site

Question

Problem is google webmaster tools cant fetch my magento2 web site,

My backend settings,

I want to use secure base url so that reason I chanced base url as a https, but I think it doesn't matter because when I tried to fetch as google "http://" look like succeed but render says "This website temporarily unavailable."

I verified all domains at google webmaster, when I tired fetch as google for "https://" error is "Temporarily unreachable"

Any Ideas ?

Answer 1

The official docs are your friend.

1. On the admin sidebar, tap Stores.
2. Then under Settings, choose Configuration.
3. In the panel on the left under General, choose Design.
4. Expand the Search Engine Robots section.
5. Enter your values in the text box or click Reset to Default to auto-populate Magento's recommendations.
6. Flush your caches.

Answer 2

I found that the official docs let me down and the best thing to do was to create a robots.txt in the pub folder instead.