googlebot and layered navigation

Question

I am having trouble with googlebot crawling through our layered navigation. I get numerous logs like:

GET /shop/dresses?clothing_size=250&color=247&dir=desc&limit=15&mode=list&order=position 
GET /shop/dresses?clothing_size=268&color=248&dir=asc&order=name 
GET /shop/sale?clothing_size=252&dir=asc&limit=5&mode=grid&order=name&price=50-100 

Here is a copy of my robots.txt

User-agent: Googlebot-Image
Disallow: /
Allow: /shop/media/catalog/product/
Allow: /shop/media/wysiwyg/

# Crawlers Setup
User-agent: *

# Directories
Disallow: /shop/404/
Disallow: /shop/app/
Disallow: /shop/cgi-bin/
Disallow: /shop/downloader/
Disallow: /shop/errors/
Disallow: /shop/includes/
#Disallow: /shop/js/
#Disallow: /shop/lib/
Disallow: /shop/magento/
#Disallow: /shop/media/
Disallow: /shop/pkginfo/
Disallow: /shop/report/
Disallow: /shop/scripts/
Disallow: /shop/shell/
Disallow: /shop/skin/
Disallow: /shop/stats/
Disallow: /shop/var/
Disallow: /offline/

# Paths (clean URLs)
Disallow: /shop/index.php/
Disallow: /shop/catalog/product_compare/
Disallow: /shop/catalog/category/view/
Disallow: /shop/catalog/product/view/
Disallow: /shop/catalogsearch/
#Disallow: /shop/checkout/
Disallow: /shop/control/
Disallow: /shop/contacts/
Disallow: /shop/customer/
Disallow: /shop/customize/
Disallow: /shop/newsletter/
Disallow: /shop/poll/
Disallow: /shop/review/
Disallow: /shop/sendfriend/
Disallow: /shop/tag/
Disallow: /shop/wishlist/
Disallow: /shop/catalog/product/gallery/

# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /.DS_Store
Disallow: /api.php
Disallow: /get.php
Disallow: /mage



# Paths (no clean URLs)
#Disallow: /*.js$
#Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=

How can I rewrite my robots.txt to block these urls?

Answer 1

robots.txt dont work as expected anymore. use something simple like in server config:

APACHE:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot|Yahoo) [NC]
RewriteCond %{QUERY_STRING} ^clothing_size=([0-9]*) [NC]
RewriteRule .* - [G]

NGINX:

if ($http_user_agent ~* "Baiduspider|Googlebot|bingbot|Yahoo|YandexBot") { set $layered A; }
if ($args ~ ^(clothing_size|color|price)=.+) { set "${layered}B"; }
if ($layered = AB) { return 410; }

this will tell bots this page was gone, remove from index also. you can extend regex to catch more args and bots.

Answer 2

See solution from here as copy below to fix your issue and prevent bots crawling through your layered navigation

Denying layered navigation for crawlers and fix SEO issues caused by the huge number of layered navigation URLs can be done by using PRG Pattern.

This works like a charm, i. e. not changing the UX regarding Layered Navigation and 100% reliable in terms of preventing crawlers from wasting crawl budget on useless duplicate content URLs.

Simply said, it's about replacing the GET request to a layered navigation/filter URL with a POST request (which search engine crawlers do not follow) before redirecting the user to the original layered navigation/filter URL.

For further details and reading, please see

1. Detailed explanation incl. sample request flow
2. Why robots.txt, rel=nofollow etc. are no satisfying solutions here
3. PRG Pattern Magento 2 Extension
4. PRG Pattern Demo

Answer 3

I actually ran across this as well and believe I have found a simple solution that works. Whenever you set up your robots.txt you can simply disallow the attribute attribute. For example I don't want Google to crawl or index all the potential options for color.

Disallow: /*color=*

Just use the attribute code followed by an = and preceded and followed by *.

I spent the last few hours testing in console and it appears to be working smoothly. I had found a few other potential solutions online but they all ended up having problems that would have caused Google to miss pages that I did want indexed and followed.

Hope this helps someone else out there, it took me way to long to find such a simple solution.