0

I am working on this site and some how (I don't know why) Google stopped crawling the home page URL (http://www.indusuno.com/)

When I search for the site in Google, it shows "A description for this result is not available because of this site's robots.txt Learn more".

I'm attaching the screenshot.

And also index.php is coming in search result URLs. Earlier we have blocked index.php from robots.txt, but now allow this.

Current contents of indusunocom/robots.txt:

User-agent: *
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Disallow: /catalogsearch/

Sitemap: https://www.indusuno.com/sitemap.xml

enter image description here

4
  • Original post: stackoverflow.com/questions/41732697/… Please do not cross post on different StackExchange sites or at least provide a link for context. Jan 22 '17 at 15:23
  • @FabianSchmengler actually this is the best place for this post and he was sent here from there.
    – SR_Magento
    Jan 22 '17 at 15:31
  • @SR_Magento makes sense, the original post is the one that should be deleted Jan 22 '17 at 16:08
  • I added the current content of robots.txt to the question, so that it stays relevant after the issue is fixed. The spirit of Stack Exchange is that future visitors will also benefit from questions and answers. Jan 22 '17 at 16:14
0

Your robots.txt is not correctly setup:

Current:

User-agent: *
...

What it should be:

User-agent: Googlebot
Disallow: 

User-agent: Googlebot-image
Disallow:

User-agent: *
...

Make the change and then give it some time before your site gets crawled properly. Also, check your webmaster tools to make sure that there are no additional errors.

0

Works for me:

enter image description here

Earlier we have blocked index.php

This probably was the issue, and after you removed that rule, it took a while until Google crawled the site again.

1
  • Thanks for your precious comment, yes now it is working, may be this issues comes as we have blocked index.php.
    – Indus Uno
    Jan 29 '17 at 7:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.