0

I've noticed my indexed pages are dropping from Google like a rock. I'm reviewing everything that was changed in the last month. I noticed in my Google Webmaster Tools that there were some inaccessible pages, to compensate I blocked them with my robots.txt file, see bellow:

User-agent: Baiduspider
Disallow: /
User-agent: * 
Disallow: /index.php/ 
Disallow: /*? Disallow: /*.js$ 
Disallow: /*.css$ 
Disallow: /checkout/ 
Disallow: /tag/ 
Disallow: /catalogsearch/ 
Disallow: /review/ 
Disallow: /app/ 
Disallow: /downloader/ 
Disallow: /js/ 
Disallow: /lib/ 
Disallow: /media/ 
Disallow: /*.php$ 
Disallow: /pkginfo/ 
Disallow: /report/ 
Disallow: /skin/ 
Disallow: /var/ 
Disallow: /customer/ 
Disallow: /productdata/
Disallow: /productscripts/
Disallow: /includes/
Disallow: /wishlist/
Disallow: /shop/
Disallow: /supplier/
Disallow: /eng/catalog/gallery/
Disallow: /fr/catalog/gallery
Disallow: /eng/catalog/product/gallery/
Disallow: /fr/catalog/product/gallery
Disallow: /eng/cms/index/noCookies
Disallow: /fr/cms/index/noCookies
Disallow: /eng/catalog/view/_ignore_category/
Disallow: /fr/catalog/view/_ignore_category/
Disallow: /eng/customer/
Disallow: /fr/customer/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/


Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*`

Is my robots.txt file too restrictive?

Improve this question
2

2 Answers 2

Reset to default
1

According to Google the best, most complete answer to your question is available in Google Webmaster Tools under Crawl > Blocked URLs. Depending on the number of results you have on category pages and the pagination configuration, the

Disallow: /*?

line will block Googlebot from crawling anything but the first page of a category list.

Also, I would not block access to /media/, or at least not to /media/catalog/ or /media/wysiwyg/. Images have value too!

Improve this answer
1
  • Good catch! This would definitely explain why my pages are falling and less and less bots are crawling my site. I have no idea where that came from but this is the most probably cause. I hope it can bounce back soon... Thanks benmarks!
    – SR_Magento
    Commented Apr 21, 2014 at 12:35
2

Some technical notes.

Empty lines

According to the original robots.txt specification,

  • records must be separated by (at least) one empty line, and
  • records must not contain empty lines.

So this means that you should add a new line after Baiduspider’s Disallow: / (the first record ends there):

User-agent: Baiduspider
Disallow: /

User-agent: * 
Disallow: /index.php/
# …

And remove the empty lines in the second record:

# …
Disallow: /catalog/product/view/
Disallow: /*?dir*
# …

Several Disallow in the same line

Each line must also contain only one Disallow, so you should change

Disallow: /*? Disallow: /*.js$

to

Disallow: /*? 
Disallow: /*.js$

Pattern matching

Also note that characters like * and $ don’t have a special meaning in the original robots.txt specification.

Parsers that follow the spec would block the following URLs when parsing Disallow: /*.php$:

  • http://example.com/*.php$
  • http://example.com/*.php$$$$
  • http://example.com/*.php$/foobar

But they would still crawl, for example:

  • http://example.com/*.php
  • http://example.com/foo.php
  • http://example.com/foo.php/bar/

(Note that, for example, Google does not follow the original specification; i.e., they interpret characters like * etc. with a special meaning.)

Improve this answer
5
  • That's great spec information, though OP was asking specifically about Google: "my indexed pages are dropping from Google like a rock". Google's docs state that Googlebot uses * as a wildcard and $ as EOURL. I wonder how many spiders follow Google once Google goes a certain way?
    – benmarks
    Commented Apr 14, 2014 at 12:21
  • 1
    @benmarks: True. But as the OP used User-agent: * (instead of targeting Google’s bots only) I assumed that the rules should work for all bots.
    – unor
    Commented Apr 14, 2014 at 12:32
  • Damn, that's a great point. I suppose Bing is somewhat important...
    – benmarks
    Commented Apr 14, 2014 at 13:06
  • Sorry unor, had to take away the accepted answer. Your answer was more complete, however, after revisiting this trying to find answers, benmarks seems to have hit the nail on the head. The disallow: / would block bots from crawling my site.
    – SR_Magento
    Commented Apr 21, 2014 at 12:37
  • I've thrown an upvote your way - great content for future users!
    – benmarks
    Commented Apr 22, 2014 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.