For some reason our site is not being indexed. I am trying to step through all the things that might be causing this. Any help or pointers would be greatly appreciated.
I created 4 properties in Google Search Console for the 4 ways of accessing the site (https://www , http://www, https://, http:// ). I also created a set holding these 4.
The sitemap that I generated has the URLs as https://. From this property I test one of the site maps and get 5599 warnings. Issue “ URL blocked by robots.txt” - Sitemap contains urls which are blocked by robots.txt.
I checked the robots.txt and I don’t see anything that should restrict it. If I test these URLs in the “Robots.txt Tester” they are allowed.
I go into the “Fetch as Google” and enter one of the examples. I get “Complete” so it seems that it is OK. A couple of times I did get “Temporarily unreachable” .
If I enter some of the products in the site map I always get “Temporarily unreachable”. Again, in the “Robots.txt Tester” they are allowed. I checked the URLS in a browser and they are OK.
This is a Magento 2.1.4 site. We will be updating Magento soon but I don’t think that this is a Magento bug, but rather something that I am doing wrong. Any help is greatly appreciated.