where to add meta tag to stop robots/spiders in magento site

Question

In magento site, 32 gb is wasted by "Not viewed traffic includes traffic generated by robots, worms, or replies with special HTTP status codes."

Googlebot alone is using 29.41 gb. I wanted to avoid all these "Robots/spiders" consuming bandwidth.

SO I added robots.txt file :

But I need more security.

Where to add in magento

to stop all robots/spiders/googlebot to stop indexing or crawling magento site.

Let me know if you need any clarifications.

Thanks in advance.

Answer 1

If Google does not crawl your site, nobody find you :) It is vise to restrict some areas like
1. layered navigation
2. search and advanced search
3. paginated content

Also you can disallow some user agents (bots and crawlers) in the .htaccess file.

Taken from stackoverflow:

You need to have mod_rewrite enabled. Placed it in .htaccess in that folder. If placed elsewhere (e.g. parent folder) then RewriteRule pattern need to be slightly modified to include that folder name).

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]

1. I have entered only few bots -- you add any other yourself (letter case does not matter).
2. This rule will respond with "403 Access Forbidden" result code for such requests. You can change to another response HTTP code if you really want (403 is most appropriate here considering your requirements).