If Google does not crawl your site, nobody find you :) It is vise to restrict some areas like

1. layered navigation
2. search and advanced search
3. paginated content

Also you can disallow some user agents (bots and crawlers) in the .htaccess file.

If Google does not crawl your site, nobody find you :) It is vise to restrict some areas like

1. layered navigation
2. search and advanced search
3. paginated content

Also you can disallow some user agents (bots and crawlers) in the .htaccess file.

Taken from stackoverflow:

You need to have mod_rewrite enabled. Placed it in .htaccess in that folder. If placed elsewhere (e.g. parent folder) then RewriteRule pattern need to be slightly modified to include that folder name).

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]

1. I have entered only few bots -- you add any other yourself (letter case does not matter).
2. This rule will respond with "403 Access Forbidden" result code for such requests. You can change to another response HTTP code if you really want (403 is most appropriate here considering your requirements).