3

I've been periodically checking Google Webmaster Tools and recently noticed a large number of ERROR 500 messages for links to pages that do not exist.

Example:

http://www.mysite.com/eng/catalog/product/view/id/21826/s/URL_KEY/ RESPONSE CODE 500

The ID and URL KEY is valid on their own and correctly identify the product. The product does exist and if I change the URL to:

http://www.mysite.com/eng/URL_KEY.html

the page opens. Using the ID in the backend also shows the correct product.

All links to product pages in the frontend are in this format: http://www.mysite.com/eng/URL_KEY.html there is no reference to any URL that is using /catalog/product/view/id/21826/s/URL_KEY/.

How are Google bots finding these links if there is no reference in the frontend?

Can I safely Disallow: /catalog/ in my robots.txt to avoid error message in Google Webmaster Tools without negatively impacting my site?

Thanks!

2 Answers 2

4

The quick easy solution is to setup your robots.txt to ignore the parent path:

## Do not crawl seach pages and not-SEO optimized catalog links
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/

Snippet from:

To answer your question as how, dissecting the controller referenced will help:

File: app/code/core/Mage/Catalog/controllers/ProductController.php

 /**
 * Product view action
 */
public function viewAction()
{
    // Get initial data from request
    $categoryId = (int) $this->getRequest()->getParam('category', false);
    $productId  = (int) $this->getRequest()->getParam('id');
    $specifyOptions = $this->getRequest()->getParam('options');

    // Prepare helper and params
    $viewHelper = Mage::helper('catalog/product_view');

    $params = new Varien_Object();
    $params->setCategoryId($categoryId);
    $params->setSpecifyOptions($specifyOptions);

    // Render page
    try {
        $viewHelper->prepareAndRender($productId, $this, $params);
    } catch (Exception $e) {
        if ($e->getCode() == $viewHelper->ERR_NO_PRODUCT_LOADED) {
            if (isset($_GET['store'])  && !$this->getResponse()->isRedirect()) {
                $this->_redirect('');
            } elseif (!$this->getResponse()->isRedirect()) {
                $this->_forward('noRoute');
            }
        } else {
            Mage::logException($e);
            $this->_forward('noRoute');
        }
    }
}

$specifyOptions would be the value to follow during a request.

Most likely the URL is being found via a sitemap.xml link generated, the SEO Sitemap catalog Magento Core module itself has /catalog/seo_sitemap/category/, or such.

The /s/ in your URL could mean a possible alternative store view could be the culprit as well.

Not sure what version of Magento you are on. If you are running the latest version of Magento CE or EE I believe the latest release causes the Google sitemap sitemap.xml generation return the true controller path like yours and not the url_key.

2
3

It's better to use a canonical tag for this, so whenever Google indexes http://www.mysite.com/eng/catalog/product/view/id/21826/s/URL_KEY/

it should have a canonical tag in the head like

<link rel="canonical" href="http://www.mysite.com/eng/URL_KEY.html"/>

You can turn canonical tags on under System > Configuration > Catalog > Use Canonical Link Meta Tag For Products

1
  • Hi Sander, this is already enabled. That is why I'm trying to find out how Goolge is finding these links and how to prevent them from being accessed. Thanks! SR
    – SR_Magento
    Feb 24, 2014 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.