3

I've got to deal with a Magento 1.x site that has been run on unsecure http for over a decade. Just reading that, you're immediately thinking it's time to move it to https, right?

Well the trouble started when we did switch it to use https. We got the certificates etc, set up apache redirects to https and applied it in both web/unsecure/base_url and web/secure/base_url in admin config.

Google tanked us. Completely. It was a disaster.

We're a small regional store and we never experienced huge traffic but we did get quite a bit of traffic from Google and frequently showed up on the first page of search results. That died overnight on Feb 25th when we did the switch.

I'm forced to switch it back now to http and hope it somehow recovers.

I'm pulling my hair here trying to think of workarounds, but Magento isn't cooperating.

Also, an obvious workaround should/would be to "use secure on frontend" setting and let search engines gradually update their links. But this clearly is not doing what I would expect it to do. (ie. offer https links on link rels, hrefs and img srcs) instead it only does it for the account login link and a couple of others.

What's the point of having the option if it isn't obeyed? Is there a module that fixes that behaviour? If I manually visit the https version of the site the page will be unsecure because all the images inside it are on http.

  1. Why did Google behave so strongly?

  2. What can I do to survive this mess?

  3. Is there a module that makes Magento frontend obey "use secure on frontend"?

Improve this question

2 Answers 2

Reset to default
2

From the sound of your question, you simply switched to https and did not think about some of the problems that might arise:

  • did you change all your internal links and image-sources to https?
  • did you update your sitemaps?
  • do you redirect all your unsecure traffic to secure pages using 301?
  • did the performance stay the same?

Also, that you see the visitor numbers go down does not necessarily mean that less visitors get directed to your site - maybe some browsers, especially firefox, reports your site as insecure now because an image, a font or a 3rd party script does not use https, which will result in many users going back to the place where they came from, and they will never trigger your google analytics code (or whatever you use for visitor tracking).

As for your sub-questions:

1. Why did Google behave so strongly?

You did something wrong, but I can only guess what it was. Check your search console, try to look out for changes in the crawling budget, 404, etc. sections.

2. What can I do to survive this mess?

Find out what it was and correct it.

3. Is there a module that makes Magento frontend obey "use secure on frontend"?

The setting is intended to work as you describe. If you want to be fully on https, you already did the right thing and set the web/unsecure/base_url value.

Improve this answer
8
  • yes/yes/yes*1/yes*2 on all four of your questions. *1 we noticed it was broken out of the box shortly after the switch and we fixed it with an .htaccess redirect to send all http hits to https, *2 the perfomance has been improving because of upgrades/cleanup (unrelated to https)
    – DM8
    Commented Mar 8, 2018 at 15:35
  • Ok, then sorry - it's pretty impossible imho to diagnose this special problem from away without access to the system, search console, GA etc... so I can't help you here. Maybe head over to webmasters.stackexchange.com, that's not so much a Magento topic
    – simonthesorcerer
    Commented Mar 8, 2018 at 15:43
  • I've been trying to figure out the url factory logic... In my view it's not working as intended :/ ... There's even a facility to check Mage_core_model_store::isCurrentlySecure() and it doesn't seem to be fully used when generating urls everywhere, some elements are https but some others aren't, ie. the category listing image srcs are https, but the hrefs on them are http, the main menu is http, also some widgets ignore it too... I am hoping to make it prefer to use https over http but not force 301 redirects so that Google doesnt kill us again.
    – DM8
    Commented Mar 8, 2018 at 16:17
  • 301 redirects don't kill you (they are actually best practise), and a complete move to https is not your problem if, as you say, everything is done correctly; we've done that a few times and never lost rankings. your problem must be somewhere else. in contrast, if you allow http and https, you may have a duplicate content issue if your canonicals are not correct
    – simonthesorcerer
    Commented Mar 8, 2018 at 16:28
  • I've almost done it, still need some more testing to iron out a bug that emerges in one page checkout/login. Working on it, local/Mage/Core/Model/store.php function getBaseUrl() is the main change, plus a change in the config to not 'Auto-redirect to Base URL' ... but this might break ajax hrmff... I'll have to change the pages that have ajax so they do not rely on getBaseUrl for http/https so it passes CORS.
    – DM8
    Commented Mar 8, 2018 at 18:02
0

TL-DR: My problems all emerged from the "Auto-redirect to Base Url" config messing up visits to http. I've fixed it and now I'm praying to the google gods they re-crawl the site ASAP to get everything out of the 'really really bad links' bin...

OK, so after digging a bit I believe I found the cause of the mess

1. Why did Google behave so strongly?

There is a config setting in Magento 1x

 System > Configuration > 'Web' (left side tab) > 'Url Options' (right side tab).
Auto-redirect to Base Url > Yes/No

This makes all urls that do not exactly match (including the protocol) what Magento expects a url to look like, to redirect to the home page (base_url).

Unfortunately this setting is protocol agnostic illierate. Meaning, while this setting was enabled, right after I switched everything to https:// all the old http:// urls being visited by webcrawlers redirected them to the home page.

If you think this is a bug, good. So do I.

This is (most likely) the cause that made the switch backfire so heavily by Google dumping on us...

Unfortunately this broke site crawling on the first day. It's likely Googlebot saw this and threw all the links it crawled on that day in a bin of 'bad links'

I have since changed this to No.

2. What can I do to survive this mess?

sigh, fix it... and hope it will somehow recover. But the old http urls had been in good standing for so long; and high up enough in the results; that it will take time to get that back methinks.

I don't know how long Google takes to re-crawl links that it previously thought had 'bad content' ... if someone can point me to an action that I need to do to tell GoogleBot that "Hey I've fixed this, go crawl them again ASAP", please let me know. (yes, I've updated the sitemaps etc to be correct, read on)

3. Is there a module that makes Magento frontend obey "use secure on frontend"?

I didn't find one, so I had to dig through the code and figure it out. The Classes of interest are:

Mage_Core_Model_Store (this is the url factory)

Thankfully all of these classes use getBaseUrl() in Mage_Core_Model_Store so, no need to change these:

Mage_Core_Model_Url
Mage_Catalog_Block_Category_View (makes link rel canonical for category)
Mage_Catalog_Block_Product_View (makes link rel canonical for product)

So I only needed to modify App/code/core/Mage/Core/Model/Store.php (I made a module out of it, but for testing you can just copy it over to /App/code/local/Mage/Core/Model/Store.php and apply the changes there. Do not overwrite core files in-place)

(This is the diff that Netbeans makes, sorry...)

--- a/<html>Store.php (<b>Today 1:06:57 μμ</b>)</html>
+++ b/<html><b>Current File</b></html>
@@ -585,6 +585,9 @@
      */
     public function getBaseUrl($type = self::URL_TYPE_LINK, $secure = null)
     {
+        if ($this->getConfig('web/secure/use_in_frontend')) {
+              $secure = true;
+        }
     $cacheKey = $type . '/' . (is_null($secure) ? 'null' : ($secure ? 'true' : 'false'));
     if (!isset($this->_baseUrlCache[$cacheKey])) {
         switch ($type) {

Expected effects: Since the 'Auto-redirect 'option is now disabled, http links will be valid and will not be redirected. https links will also be valid. The visitor will be seeing https in all img src, a href, link rel, etc. and navigate to https on their next click inside the site. (and stay at https)

I have made certain that none of the content has hardcoded img src with http:// on any products/categories/blocks. They all use either {{media ...}} or the fallback <img src=//domain.tld/foo"> thank you MySQL REPLACE()

This also generates https link rel canonical, so there's no duplicate content issues. and the sitemaps are generated with https as well. I cannot think of anything else I might need to fix to have the effect there but since Mage::getBaseUrl redirects to Mage::app()->getStore()->getBaseUrl(), and that's the class I modified, I think I'm covered (?).

My Intent is to let it run like this for a couple of months so whatever engine visits the http site will still be able to do so and switch their canonical links to https. And then switch it all over to https

Issues: visitors of http:// pages cannot use Ajax to a https:// target, this makes the search box that has ajax auto-complete not work for the visitor's first page load. (I'm hoping Googlebot does not care about this since it's js... am I wrong? please let me know)

Since all links have https, the visitor's next page load will have ajax working fine. I also had to change the main navigation menu from ajax to static, but I can live with that.

I initially thought I'd have trouble with onepagecheckout because of this, but because the user will have to switch to https long before they reach that page, it is not an issue.

I do not think there's any other ajax elements in the page anywere else.. this site uses a pretty simplistic theme, so no auto-scroll categories or other autoloaders like that.

I'm posting this in the hopes that it might help anyone that made similar mistakes I did when switching from http to https.

This should not be needed for any other scenario.

If you're building a new site on https from the get-go, there will not be any old crawler history on old http links to worry about.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.