In magento site, 32 gb is wasted by "Not viewed traffic includes traffic generated by robots, worms, or replies with special HTTP status codes."

Googlebot alone is using 29.41 gb. I wanted to avoid all these "Robots/spiders" consuming bandwidth.

SO I added robots.txt file : Demo => "http://albedoeventz.com/robots.txt"

But I need more security.

Where to add in magento

to stop all robots/spiders/googlebot to stop indexing or crawling magento site.

Let me know if you need any clarifications.

Thanks in advance.

In magento site, 32 gb is wasted by "Not viewed traffic includes traffic generated by robots, worms, or replies with special HTTP status codes."

Googlebot alone is using 29.41 gb. I wanted to avoid all these "Robots/spiders" consuming bandwidth.

SO I added robots.txt file :

But I need more security.

Where to add in magento

to stop all robots/spiders/googlebot to stop indexing or crawling magento site.

Let me know if you need any clarifications.

Thanks in advance.