1

I am having trouble with googlebot crawling through our layered navigation. I get numerous logs like:

GET /shop/dresses?clothing_size=250&color=247&dir=desc&limit=15&mode=list&order=position 
GET /shop/dresses?clothing_size=268&color=248&dir=asc&order=name 
GET /shop/sale?clothing_size=252&dir=asc&limit=5&mode=grid&order=name&price=50-100 

Here is a copy of my robots.txt

User-agent: Googlebot-Image
Disallow: /
Allow: /shop/media/catalog/product/
Allow: /shop/media/wysiwyg/

# Crawlers Setup
User-agent: *

# Directories
Disallow: /shop/404/
Disallow: /shop/app/
Disallow: /shop/cgi-bin/
Disallow: /shop/downloader/
Disallow: /shop/errors/
Disallow: /shop/includes/
#Disallow: /shop/js/
#Disallow: /shop/lib/
Disallow: /shop/magento/
#Disallow: /shop/media/
Disallow: /shop/pkginfo/
Disallow: /shop/report/
Disallow: /shop/scripts/
Disallow: /shop/shell/
Disallow: /shop/skin/
Disallow: /shop/stats/
Disallow: /shop/var/
Disallow: /offline/

# Paths (clean URLs)
Disallow: /shop/index.php/
Disallow: /shop/catalog/product_compare/
Disallow: /shop/catalog/category/view/
Disallow: /shop/catalog/product/view/
Disallow: /shop/catalogsearch/
#Disallow: /shop/checkout/
Disallow: /shop/control/
Disallow: /shop/contacts/
Disallow: /shop/customer/
Disallow: /shop/customize/
Disallow: /shop/newsletter/
Disallow: /shop/poll/
Disallow: /shop/review/
Disallow: /shop/sendfriend/
Disallow: /shop/tag/
Disallow: /shop/wishlist/
Disallow: /shop/catalog/product/gallery/

# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /.DS_Store
Disallow: /api.php
Disallow: /get.php
Disallow: /mage



# Paths (no clean URLs)
#Disallow: /*.js$
#Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=

How can I rewrite my robots.txt to block these urls?

3 Answers 3

3

robots.txt dont work as expected anymore. use something simple like in server config:

APACHE:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot|Yahoo) [NC]
RewriteCond %{QUERY_STRING} ^clothing_size=([0-9]*) [NC]
RewriteRule .* - [G]

NGINX:

if ($http_user_agent ~* "Baiduspider|Googlebot|bingbot|Yahoo|YandexBot") { set $layered A; }
if ($args ~ ^(clothing_size|color|price)=.+) { set "${layered}B"; }
if ($layered = AB) { return 410; }

this will tell bots this page was gone, remove from index also. you can extend regex to catch more args and bots.

5
  • I'm assuming this goes in .htaccess? Can go anywhere in the file?
    – Pragman
    Commented Mar 17, 2017 at 9:12
  • htaccess or httpd config
    – MagenX
    Commented Mar 17, 2017 at 9:56
  • I tried implementing your solution in htaccess but afterwards it returned 404 errors for robots.txt and also sitemap.xml???
    – Pragman
    Commented Mar 21, 2017 at 17:15
  • there is nothing in these snippets that affects robots.txt or sitemap.xml you probably has some incorrect changes in htaccess
    – MagenX
    Commented Mar 21, 2017 at 18:06
  • @MagenX how about if have another bot craw? because we don't know how many bot index to website
    – xanka
    Commented Oct 22, 2021 at 1:34
3

See solution from here as copy below to fix your issue and prevent bots crawling through your layered navigation

Denying layered navigation for crawlers and fix SEO issues caused by the huge number of layered navigation URLs can be done by using PRG Pattern.

This works like a charm, i. e. not changing the UX regarding Layered Navigation and 100% reliable in terms of preventing crawlers from wasting crawl budget on useless duplicate content URLs.

Simply said, it's about replacing the GET request to a layered navigation/filter URL with a POST request (which search engine crawlers do not follow) before redirecting the user to the original layered navigation/filter URL.

For further details and reading, please see

  1. Detailed explanation incl. sample request flow
  2. Why robots.txt, rel=nofollow etc. are no satisfying solutions here
  3. PRG Pattern Magento 2 Extension
  4. PRG Pattern Demo
0

I actually ran across this as well and believe I have found a simple solution that works. Whenever you set up your robots.txt you can simply disallow the attribute attribute. For example I don't want Google to crawl or index all the potential options for color.

Disallow: /*color=*

Just use the attribute code followed by an = and preceded and followed by *.

I spent the last few hours testing in console and it appears to be working smoothly. I had found a few other potential solutions online but they all ended up having problems that would have caused Google to miss pages that I did want indexed and followed.

Hope this helps someone else out there, it took me way to long to find such a simple solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.