Skip to main content
added 334 characters in body; edited tags
Source Link
Fabian Schmengler
  • 66.1k
  • 25
  • 188
  • 422

I am working on this site and some how (I don't know why) Google stopstopped crawling Homethe home page URL (http://www.indusuno.com/http://www.indusuno.com/), while checking this

When I search for the site in Google It, it shows (A"A description for this result is not available because of this site's robots.txt Learn more)more". 

I'm attaching the snapshotscreenshot.

And also index.php is coming in url which any one search on websiteresult URLs. Earlier we have blocked index.php from robots.txt, but now allow this.

Current contents of (indusunocom/robots.txt):

Kindly help me to resolve this issue.

User-agent: *
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Disallow: /catalogsearch/

Sitemap: https://www.indusuno.com/sitemap.xml

enter image description here

I am working on this site and some how (I don't know why) Google stop crawling Home page URL (http://www.indusuno.com/), while checking this site in Google It shows (A description for this result is not available because of this site's robots.txt Learn more). I'm attaching the snapshot.

And also index.php is coming in url which any one search on website. Earlier we have blocked index.php from robots.txt, but now allow this. (indusunocom/robots.txt)

Kindly help me to resolve this issue.

enter image description here

I am working on this site and some how (I don't know why) Google stopped crawling the home page URL (http://www.indusuno.com/)

When I search for the site in Google, it shows "A description for this result is not available because of this site's robots.txt Learn more". 

I'm attaching the screenshot.

And also index.php is coming in search result URLs. Earlier we have blocked index.php from robots.txt, but now allow this.

Current contents of indusunocom/robots.txt:

User-agent: *
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Disallow: /catalogsearch/

Sitemap: https://www.indusuno.com/sitemap.xml

enter image description here

minor formatting, removed unecessary comment
Source Link
SR_Magento
  • 5.2k
  • 13
  • 62
  • 104

0 down vote favorite

I am working on this site and some how (I don't know why) Google stop crawling Home page URL (http://www.indusuno.com/), while checking this site in Google It shows (A description for this result is not available because of this site's robots.txt Learn more). I'm attaching the snapshot.

And also index.phpindex.php is coming in url which any one search on website. Earlier we have blocked index.phpindex.php from robots.txtrobots.txt, but now allow this. (indusunocom/robots.txtindusunocom/robots.txt)

Kindly help me to resolve this issue.

enter image description here

0 down vote favorite

I am working on this site and some how (I don't know why) Google stop crawling Home page URL (http://www.indusuno.com/), while checking this site in Google It shows (A description for this result is not available because of this site's robots.txt Learn more). I'm attaching the snapshot.

And also index.php is coming in url which any one search on website. Earlier we have blocked index.php from robots.txt, but now allow this. (indusunocom/robots.txt)

Kindly help me to resolve this issue.

enter image description here

I am working on this site and some how (I don't know why) Google stop crawling Home page URL (http://www.indusuno.com/), while checking this site in Google It shows (A description for this result is not available because of this site's robots.txt Learn more). I'm attaching the snapshot.

And also index.php is coming in url which any one search on website. Earlier we have blocked index.php from robots.txt, but now allow this. (indusunocom/robots.txt)

Kindly help me to resolve this issue.

enter image description here

Source Link

Magento Website Crawl Issue - A description for this result is not available because of this site's robots.txt

0 down vote favorite

I am working on this site and some how (I don't know why) Google stop crawling Home page URL (http://www.indusuno.com/), while checking this site in Google It shows (A description for this result is not available because of this site's robots.txt Learn more). I'm attaching the snapshot.

And also index.php is coming in url which any one search on website. Earlier we have blocked index.php from robots.txt, but now allow this. (indusunocom/robots.txt)

Kindly help me to resolve this issue.

enter image description here