I have been setting up a new magento site for a client and while I was still learning my way around Magento (only new to it) Google managed to index the entire site and has indexed +7000 entries for a site with like 46 pages in its sitemap.

Looking further at it, the majority has been the individual combinations of parameters available for each of the pages.

And I also tidied up the urls for SEO by removing the .html at the end (but the .html versions got indexed too and now have 404's)

I have the correct robots.txt in place to block such a thing happening.

The problem now is because google bet me to it, I have thousands of indexed pages with 404 errors.

Some reading up suggests that using the robots.txt to block them won't remove those that exist on there already, so I was wondering if anyone knew a magento specific way I could update those urls to NOINDEX rather than having to delete them from the index one at a time.

I was thinking maybe allow those pages through the robots.txt but use .htaccess to update all the links with certain parameters to be NOINDEX so google removes them from the index then once removed, block them again in the robots.txt, but am not too sure on what solution would be best...

Any ideas?

  • Upon doing some further digging around, I found a site that suggests I can unblock the parameters in the robots.txt and add some code to my htaccess file that will set the X-Robots-Tag to noindex (thus telling google to unindex it) once they are all removed then I can block in robots.txt again remove the htaccess modification. The code looked something like the following.. Can anyone confirm for me that this would work? RewriteCond %{QUERY_STRING} ^\? RewriteRule ^(.*)$ - [E=NOINDEX:1,L] Header set X-Robots-Tag "noindex" env=NOINDEX? – KadeOK Mar 11 '15 at 16:34
  • The theory with the rewrite being, that any string with '?' in it (so pretty much all parameter strings) will get a header rewrite to noindex would this work on a Magento site? – KadeOK Mar 11 '15 at 16:41
  • Adrian, <meta name="robots" content="noindex"> - Where do we place this code for the search results, could you tell us please? Kade, Did the FME extension do the trick? – user31283 Sep 18 '15 at 8:40

There are three steps to this

  1. add no-index tag

  2. add canonical tags

  3. Add them to robots.txt with disallow mark

Wait for few days for changes to implement Ignore / mark the 404 errors as fixed

For change from .html to non html - You need to apply 301 redirection. here is code

Rewrite valid requests on .html files

RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^ %{REQUEST_URI}.html?rw=1 [L,QSA]

  • awesome, thanks for the advice. However how exactly do I add a no-index tag to pages with parameters? or will the 'no-index' apply to the page and all parameters for that page as well? – KadeOK Mar 11 '15 at 13:02
  • Here is how to add no-index tag - support.google.com/webmasters/answer/93710?hl=en – Adrian Lynch Mar 11 '15 at 13:18
  • Let me ask you does the parameter URLs gives 404? – Adrian Lynch Mar 11 '15 at 13:21
  • the actual tag code to enter I was aware of, I just was not sure where to enter it in magento (if it has to be manually coded into the phtml/xml/php pages) or if it was done from one of the tabs/sections in the admin backend. Turns out the extension you suggested aside from canonical tags applies the no-index to Search Results, Pop-up Gallery and filtered category pages... So that should cover most of the ones that would be parameters. – KadeOK Mar 11 '15 at 14:54
  • Majority of the indexed links for the parameters do give 404 errors at present because they are all indexed as 'site.com/categorya.html?parameter', 'categoryb.html?parameter' etc and I turned off the product/category url suffix to make it look a bit neater. I did just notice some of the newer indexed page links without the .html suffice do actually work still but. – KadeOK Mar 11 '15 at 14:55

If the core issue is having urls indexed that you don't then the core fix is redirected if they no longer exist. If you use features like robots.txt you slow down your crawling rates for pages you've configured when actually you want them to speed up so they are re-crawled and fixed in Google's index quicker.

For the HTML pages then just add a redirect rule that redirects all HTML pages to non-HTML

RewriteCond %{THE_REQUEST} \.html
RewriteRule ^(.*)\.html$ /$1 [R=301,L]

If you've change things like category names you can do something similar and so using rules rather than unique redirect for the 7000 will be much quicker. I'd put them all in a spreadsheet and then sort and work out what you have to do.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.