Problem is google webmaster tools cant fetch my magento2 web site,

robot txt fetch failed

My backend settings,

enter image description here

I want to use secure base url so that reason I chanced base url as a https, but I think it doesn't matter because when I tried to fetch as google "http://" look like succeed but render says "This website temporarily unavailable."

I verified all domains at google webmaster, when I tired fetch as google for "https://" error is "Temporarily unreachable"

Any Ideas ?

  • I might be your Firewall please check... it's not blocking it. – Raheel Aqeel Dec 13 '19 at 13:02

The official docs are your friend.

  1. On the admin sidebar, tap Stores.
  2. Then under Settings, choose Configuration.
  3. In the panel on the left under General, choose Design.
  4. Expand the Search Engine Robots section.
  5. Enter your values in the text box or click Reset to Default to auto-populate Magento's recommendations.
  6. Flush your caches.

enter image description here

| improve this answer | |
  • Thanks for your time, I forget to mention about that, I already did that but still same. – Whatatimetobealive Jun 17 '16 at 13:53
  • Do you have an SSL certificate installed/configured on the server @YasinUzun – Mat Hellums Jun 17 '16 at 13:54
  • yes i have it @MatHellums – Whatatimetobealive Jun 17 '16 at 13:56

I found that the official docs let me down and the best thing to do was to create a robots.txt in the pub folder instead.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.