2

From December 12th, crawl errors have increased on our website: www.digital-pianism.com

We have received today a warning email from Google Webmaster Tool saying that there was a significant increased in crawling errors so we connected to GWT and every page is flagged as unreachable (500 error).

Website is up, I can crawl it and we haven't done any modification on December 12th.

Worse, when I try to crawl as Google, state is "unreachable".

Has something changed in Google crawler recently ? What can I do to solve this ?

Only error in the server log is this one:

FastCGI: comm with server "/path/to/index.php" aborted: error parsing headers: duplicate header 'Content-Type'

My .htaccess has not changed for a while and there is definitely no blocking IP in here.

Could it be related to my automatic language detection in my index.php ?

Here's the code:

function getStoreForLanguage()
{
    if (isset($_SERVER['HTTP_ACCEPT_LANGUAGE']))
    {
        foreach (explode(",", strtolower($_SERVER['HTTP_ACCEPT_LANGUAGE'])) as $accept)
        {
            if (preg_match("!([a-z-]+)(;q=([0-9.]+))?!", trim($accept), $found))
            {
                $langs[] = $found[1];
                $quality[] = (isset($found[3]) ? (float) $found[3] : 1.0);
            }
        }
        // Order the codes by quality
        array_multisort($quality, SORT_NUMERIC, SORT_DESC, $langs);
        // get list of stores and use the store code for the key
        $stores = Mage::app()->getStores(false, true);
        // iterate through languages found in the accept-language header
        foreach ($langs as $lang)
        {
            $lang = substr($lang,0,2);
            if (isset($stores[$lang]) && $stores[$lang]->getIsActive()) return $stores[$lang];
        }
        return Mage::app()->getStore();
    }
    return Mage::app()->getStore();
}

/* Auto redirect to language store view if request is for root */
if ($_SERVER['REQUEST_URI'] === '/') {
    header('Location: '.getStoreForLanguage()->getBaseUrl(), false);
    exit;
}

EDIT: problem is definitely caused by a 500 error but site is still displayed. See my solution for details.

EDIT 2: switching to the original Magento index.php keeps the 500 error -_-

  • Does this really have something to do with Magento ? Any changes in GWT should not be relevant here. – Prateek Dec 16 '15 at 13:40
  • I asked the Google forum and they told me it has nothing to do with GWT, ask Magento forum ^^ – Raphael at Digital Pianism Dec 16 '15 at 13:44
  • Can I assume you have checked with the error logs at the time when Google said 500 errors on your site ? Did you find anything there ? Any special crawl rule in your robots ? – Prateek Dec 16 '15 at 13:47
  • Are you sure there is not some directive in the .htaccess file that is blocking bots – ProxiBlue Dec 16 '15 at 13:51
  • I have updated my post – Raphael at Digital Pianism Dec 16 '15 at 13:57
1

Ok guys, we have found the issue and how weird is the fix, we've been working with Magento for over 5 years and never faced that before. Maybe one of you can enlight us ?

The problem was definitely caused by a 500 error. Nothing was displayed in the error log (different from the error we pasted in the OP) and the pages were still getting displayed. Enabling firebug would show there was a 500 error though.

After several tests (theme switch, modules disabling, original Magento htaccess and index.php backup), we have found the lines causing the issues:

#Varien_Profiler::enable();

if (isset($_SERVER['MAGE_IS_DEVELOPER_MODE'])) {
    Mage::setIsDeveloperMode(true);
}

#ini_set('display_errors', 1);

Uncommenting the two commented lines got rid of the 500 error!! If we comment it back (even using //), the error pops back!

If one of you knows why those lines are triggering the error, we'd be glad to hear the reason.

  • And you really needed to uncomment BOTH lines? – Fabian Schmengler Dec 17 '15 at 21:42
  • Eh, turning on the profiler and turning on all PHP error messages is needed to make a 500 error go away?? Something's not right here. I'm with @fschmengler on this one. – Fiasco Labs Dec 18 '15 at 3:38
  • After some tests, it seems like I only need to uncomment the ini_set('display_errors', 1); . If I delete the line, the error is gone, if I comment it, it's back. I have no clue what's going on here, I have tried to use the original default Magento index.php, same error! – Raphael at Digital Pianism Dec 18 '15 at 9:17
  • 1
    Open you site's page source when you disable the 'display_errors'. There is a fatal error at the end of page. Or better, try to register on the site, you'll see what's wrong. – Prateek Dec 18 '15 at 9:32
  • You're the man, there was indeed a fatal error in the source!!! Thanks for that I really appreciate your help. – Raphael at Digital Pianism Dec 18 '15 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.