I have the following in my robots.txt for months now:
Disallow: /catalogsearch/
Disallow: /catalogsearch/*
Disallow: /webshop/catalogsearch/*
Disallow: /webshop/catalogsearch/
My webshop url is like www.webshopdomain.com/webshop
The problem is: Google still shows and crawling all search results (39.000 ..)
What am I doing wrong?
The code you have in your robots.txt file is spot on. Submitting the urls in your Google Webmaster Tools account for removal from search index will be the best way to prevent them showing.
There are a number of other urls and prefixes you need to think about too.
The below is more or less optimum Magento robots.txt file default entries:
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by search engines.
# Google Image Crawler Setup
User-agent: Googlebot-Image
Disallow:
# Crawlers Setup
User-agent: *
# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/
# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?SID=
while robots.txt is good, but i guess search index will be maintained as it is. so you have to login to google webmaster tools and remove those urls, just if you need them gone.
also do not count on robots.txt and robot's good behavior, add something like this to your .htaccess:
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot|Yahoo|Yandex|rogerbot|Baiduspider) [NC]
RewriteCond %{REQUEST_URI} /(catalogsearch|checkout|wishlist|customer|catalog/product_compare)/
RewriteRule .* - [G]
first this will tell search bot that those pages are really gone, no need to index them, and no need to maintain them in search index.
