Increased in crawl errors and unable to crawl as Google

Question

From December 12th, crawl errors have increased on our website: www.digital-pianism.com

We have received today a warning email from Google Webmaster Tool saying that there was a significant increased in crawling errors so we connected to GWT and every page is flagged as unreachable (500 error).

Website is up, I can crawl it and we haven't done any modification on December 12th.

Worse, when I try to crawl as Google, state is "unreachable".

Has something changed in Google crawler recently ? What can I do to solve this ?

Only error in the server log is this one:

FastCGI: comm with server "/path/to/index.php" aborted: error parsing headers: duplicate header 'Content-Type'

My .htaccess has not changed for a while and there is definitely no blocking IP in here.

Could it be related to my automatic language detection in my index.php ?

Here's the code:

function getStoreForLanguage()
{
    if (isset($_SERVER['HTTP_ACCEPT_LANGUAGE']))
    {
        foreach (explode(",", strtolower($_SERVER['HTTP_ACCEPT_LANGUAGE'])) as $accept)
        {
            if (preg_match("!([a-z-]+)(;q=([0-9.]+))?!", trim($accept), $found))
            {
                $langs[] = $found[1];
                $quality[] = (isset($found[3]) ? (float) $found[3] : 1.0);
            }
        }
        // Order the codes by quality
        array_multisort($quality, SORT_NUMERIC, SORT_DESC, $langs);
        // get list of stores and use the store code for the key
        $stores = Mage::app()->getStores(false, true);
        // iterate through languages found in the accept-language header
        foreach ($langs as $lang)
        {
            $lang = substr($lang,0,2);
            if (isset($stores[$lang]) && $stores[$lang]->getIsActive()) return $stores[$lang];
        }
        return Mage::app()->getStore();
    }
    return Mage::app()->getStore();
}

/* Auto redirect to language store view if request is for root */
if ($_SERVER['REQUEST_URI'] === '/') {
    header('Location: '.getStoreForLanguage()->getBaseUrl(), false);
    exit;
}

EDIT: problem is definitely caused by a 500 error but site is still displayed. See my solution for details.

EDIT 2: switching to the original Magento index.php keeps the 500 error -_-

Answer 1

Ok guys, we have found the issue and how weird is the fix, we've been working with Magento for over 5 years and never faced that before. Maybe one of you can enlight us ?

The problem was definitely caused by a 500 error. Nothing was displayed in the error log (different from the error we pasted in the OP) and the pages were still getting displayed. Enabling firebug would show there was a 500 error though.

After several tests (theme switch, modules disabling, original Magento htaccess and index.php backup), we have found the lines causing the issues:

#Varien_Profiler::enable();

if (isset($_SERVER['MAGE_IS_DEVELOPER_MODE'])) {
    Mage::setIsDeveloperMode(true);
}

#ini_set('display_errors', 1);

Uncommenting the two commented lines got rid of the 500 error!! If we comment it back (even using //), the error pops back!

If one of you knows why those lines are triggering the error, we'd be glad to hear the reason.