Prevent Search Engine Index

Question

We have number of categories/sub-catgories and products assigned to them in our Magento store.

Is it feasible to make sure that specific category landing pages / product detail pages cannot be indexed by search engines ?

For ex:

http://mysite.com/category_1/sub-category_1.html

is indexed by search engines, while

http://mysite.com/category_2/sub-category_2.html

cannot be indexed by search engines.

The same goes for product details page.

We do not want specific list pages or product detail pages to show up in Google, Bing, etc search results.

How can I achieve this ?

Thanks in advance.

Answer 1

You can use robots.txt to do so. To stop certain category to be indexed by search engine you can use below code in your robots.txt file

User-agent: *
Disallow: http://mysite.com/category_2/sub-category_2.html

You can add the links of whichever category you want in similar fashion. Hope you are clear on this.

To learn more about robots.txt you can refer some links robots.txt, robots.txt tutorial, robots.txt in Magento

To add robots.txt you just need to create a file named robots.txt and upload it on the root of the folder so it is accessible like www.mysite.com/robots.txt

After that you can write the above code in that file and upload it in your root folder

Answer 2

Yes you will need to use Robots:

A sample code that you can use is :

## robots.txt for Magento Community and Enterprise

## GENERAL SETTINGS

## Enable robots.txt rules for all crawlers
User-agent: *

## Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
## Set a custom crawl rate if you're experiencing traffic problems with your server.
# Crawl-delay: 30

## Magento sitemap: uncomment and replace the URL to your Magento sitemap file
# Sitemap: http://www.example.com/sitemap/sitemap.xml

## DEVELOPMENT RELATED SETTINGS

## Do not crawl development files and folders: CVS, svn directories and dump files
Disallow: /CVS
Disallow: /*.svn$
Disallow: /*.idea$
Disallow: /*.sql$
Disallow: /*.tgz$

## GENERAL MAGENTO SETTINGS

## Do not crawl Magento admin page
Disallow: /admin/

## Do not crawl common Magento technical folders
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/

## Do not crawl common Magento files
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt

## MAGENTO SEO IMPROVEMENTS

## Do not crawl sub category pages that are sorted or filtered.
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*

## Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
## Disallow: /index.php/

## Do not crawl links with session IDs
Disallow: /*?SID=

## Do not crawl checkout and user account pages
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

## Do not crawl seach pages and not-SEO optimized catalog links
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/

## SERVER SETTINGS

## Do not crawl common server technical folders and files
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php

## IMAGE CRAWLERS SETTINGS

## Extra: Uncomment if you do not wish Google and Bing to index your images
# User-agent: Googlebot-Image
# Disallow: /
# User-agent: msnbot-media
# Disallow: /

Although Turnkeyeye has explained it very well here

Answer 3

Dexter's answer is the easiest way of doing what you want on a category by category basis however you could also look into some third party "SEO" extensions that might allow you to accomplish the same from the Magento admin area e.g. http://www.magentocommerce.com/magento-connect/creare-seo.html (FREE) which allows you to edit your robots.txt file as well as a do lot of other useful features you might not realise you need yet.

I can't remember using a Magento SEO type extension (paid or free) that allows you to NOT allow certain categories or individual products to be indexed from the category area of Magento admin however extensions like the above might be able to help you also add canonical tags to the sub categories to also give search engine bots hints that you want their parent categories indexed instead.