2

This question is specific to Magento, but as an FYI, here is a link to Google definition of cloaking: https://support.google.com/webmasters/answer/66355?hl=en

One of our Magento stores has been banned by Google for "cloaking". We have done a full investigation of our stack and can't seem to find anything specifically that we have done to trigger this ban. We have absolutely nothing checking specifically for the google bot user agent. Our site treats Google bot like any other user.

Unfortunately Google is unwilling to tell us exactly we we did to get banned. Google has told us that a test is to request a page with a normal user agent (chrome for example) and then request the page with a google bot user agent. Take both of the HTML results and hash them (we are using md5) and compare the strings. If the strings don't match then the page is "cloaking" (at least that's what we have been told).

Our script identified the following diffs causing our hashes to not match:

  1. Magento FPC related comment tags like this:

    <!--{TOPMENU_7642615c1b9a72d412676bd2c56a7e9a}-->

  2. Magento form key tags like this:

    <input name="form_key" type="hidden" value="H8ojGUSABI6wsLVP" />

Has anyone else had this issue?

Is there any good way to resolve this?

Thanks!

  • Sorry, it's EE 1.13.1.0. Both of the code snippets I mentioned are NOT from 3rd party modules. – jmspldnl Sep 16 '14 at 23:27
  • Yes, in our staging system turning off FPC does eliminate the 1st tag, since that is added by FPC. We refuse to turn off FPC on our LIVE system though, as it would drastically hurt customer experience. – jmspldnl Sep 16 '14 at 23:48
  • Maybe "drastically" was a bad choice of words on my part... Our page loads times aren't terrible with FPC disabled, but they aren't as fast we get with FPC. We expect the best for our customers, so the slightest decrease in page load time isn't really acceptable. – jmspldnl Sep 17 '14 at 2:03
1

There's a problem with your page but we're not telling you what...very helpful of Google.

The form_key is always going to differ whether you are using FPC or not because the value string is specific to the user session so every new session will get a different string. The frontend form_key was introduced a year ago now so I suspect if Google was going to start banning sites for this it would have happened already. This kind of approach also isn't specific to Magento, plenty of other sites use the same kind of approach for form validation.

For the FPC comment tags, I don't believe this should differ - if you are seeing a difference it's worth investigating why. Basically using FPC does not mean Google is going to accuse you of cloaking as the entire point of FPC is to serve the same HTML (but faster) for each request whether that request comes from Google bot or a standard browser.

  • Yea, Google isn't being very helpful on the issue at all. I agree with your statements about the form_key. As for the FPC comment tags the variation seems to largely be around whether they exist or not. It seems that FPC will replace the comment tags once the dynamic block is in cache. Since we are doing back to back requests, the cache might not be fully primed on the first request, and then second request it will be and the tags will be gone, causing the hash to fail. I think these minor differences are acceptable tho, and we must be missing something else... – jmspldnl Sep 17 '14 at 14:16
  • In case the site got hacked, diff the source against a known good backup. A good computer crook can make sure the output changes are hidden to a select few if they've done their homework to extend the time they're serving out their package. The FPC comment tags are kind of stupid on Magento's part, there really is no need for that kind of code bloat. If they're placeholders, they should be processed or removed before allowing anyone outside to see the page. – Fiasco Labs Sep 17 '14 at 14:37
  • @FiascoLabs, I agree with you about the FPC comments tags. I've actually been considering stripping them out of the response... – jmspldnl Sep 18 '14 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.