Can you block Google from crawling filters?

Question

Is there a way to block Google and other engines from seeing your filtered pages? We get emails talking about how Google is seeing "an extremely high number of URLs" and when I look, it's all the color options and other kinds of sidebar options we have - but it's useless content.

I tried robots.txt and nothing really is being blocked from the testing tools. I added shopby to be be disallowed and it seems to block individual pages but nothing after .com/category/shopby/ (anything else I wanted blocked)

Ideas?

Answer 1

Magento has the setRobots method to insert the values in the robots tag within the head block. So you can create a extension to change the head block via observer. In your config.xml add a event to change layout before Magento get layout xml done.

<frontend>
    <events>
        <controller_action_layout_generate_xml_before>
            <observers>
                <change_noindex>
                    <type>singleton</type>
                    <class>namespace_extension/observer</class>
                    <method>changeRobots</method>
                </change_noindex>
            </observers>
        </controller_action_layout_generate_xml_before>
    </events>
<frontend>

There are other events that you can use to do this, maybe even more elegant than that. But now you can handle the layout settings dynamically:

public function changeRobots($observer) {
    if ($observer->getEvent()->getAction()->getFullActionName() === 'catalog_category_view') {
        $uri = $observer->getEvent()->getAction()->getRequest()->getRequestUri();
        if (stristr($uri, "?")) {
            $layout = $observer->getEvent()->getLayout();
            $product_info = $layout->getBlock('head');
            $layout->getUpdate()->addUpdate('<reference name="head"><action method="setRobots"><value>NOINDEX,FOLLOW</value></action></reference>');
            $layout->generateXml();
        }
    }
}