0

For some reason our site is not being indexed. I am trying to step through all the things that might be causing this. Any help or pointers would be greatly appreciated.

I created 4 properties in Google Search Console for the 4 ways of accessing the site (https://www , http://www, https://, http:// ). I also created a set holding these 4.

The sitemap that I generated has the URLs as https://. From this property I test one of the site maps and get 5599 warnings. Issue “ URL blocked by robots.txt” - Sitemap contains urls which are blocked by robots.txt.
I checked the robots.txt and I don’t see anything that should restrict it. If I test these URLs in the “Robots.txt Tester” they are allowed. I go into the “Fetch as Google” and enter one of the examples. I get “Complete” so it seems that it is OK. A couple of times I did get “Temporarily unreachable” .
If I enter some of the products in the site map I always get “Temporarily unreachable”. Again, in the “Robots.txt Tester” they are allowed. I checked the URLS in a browser and they are OK.

This is a Magento 2.1.4 site. We will be updating Magento soon but I don’t think that this is a Magento bug, but rather something that I am doing wrong. Any help is greatly appreciated.

Improve this question

1 Answer 1

Reset to default
0

First of all, you can configure if the site is allowed to be indexed.

Stores -> Configuration -> General (Left side) -> Design (section) -> Search Engine Robots -> Default Robots (Select)-> Select
INDEX, FOLLOW
NOINDEX, FOLLOW
INDEX, NOFOLLOW
NOINDEX, NOFOLLOW

For product sites this should be INDEX, FOLLOW. Then its also possible that you server (apache or nginx) block some google traffic. or that your site is irregular unavailable like you said before.

Another good tip is try to download the tool Screaming Frog, its free. you can crawl to the site to check for errors.

Also when you talk about http vs https and www vs non-www. Be sure your browser redirects properly to only one channel. And your sitemap also reflects this setting.

These would be the first steps to follow personally. Let me know if you've done this and if you find anything.

Improve this answer
6
  • Thanks Kay. The site is set to INDEX, FOLLOW. I found that out a couple of days ago. I ran Screaming Frog. The first 13 URL's work fine. Then it starts failing with an error of Connection Time. Does that mean that the site is not responding quickly enough ? It is a dedicated server with reasonable horsepower.
    – shimon
    Mar 1, 2018 at 19:27
  • Ok, then it seems to be either 2 isuses. 1. The site is not deployed / cached properly so a lot of calls from a bot gives timeouts... or else 2. there is a ratelimiting on your server
    – Kay Int Veen
    Mar 1, 2018 at 19:38
  • Now I am a bit confused. It tried it again (early Sunday morning) Screaming Frog crawled all 500 URL's (free version is limited) with no problems. 1 or 2 URLs were Robots restricted (like checkout) but essentially it crawled fine. I submitted the sitemaps under the https:// site.com property, which is the one that everything redirects to. It is still pending. Any ideas ?
    – shimon
    Mar 4, 2018 at 15:51
  • The sitemap crawled. All the pages came back with blocked by robots. The meta tag at the top of the page says <meta name="robots" content="INDEX,FOLLOW"/> which it gets from Stores -> Configuration as you instructed above. When I search for NOINDEX I can find it in an included file that has to do with the "Add to Wish List" that I have on the page. I may have to disable this feature. I am still clueless in this matter.
    – shimon
    Mar 5, 2018 at 13:16
  • Maybe you can provide me a url so i can take a look ;-)
    – Kay Int Veen
    Mar 5, 2018 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.