1

Is there a way to block access to the system URL's so a customer or bot would get a 404 or 403? I've done research into the rewrite rules and can't seem to find a way to mass rewrite the products' system URLs because of the complexity of those links. It would be great to just block them all from being accessed so that I don't have to worry about this moving forward.

  • Are you trying to block urls such as mystore.com/catalog/product/view/id/10 ? Why do you want to block these? – pspahn Apr 17 '15 at 17:31
  • Yes those, here's an example: example.com/catalog/product/view/id/25117/s/… I'm trying to block them because they've been indexed by the SERPs. I've since created rules to hide them in WMTs and the robots file. For now they are only causing major issues. – Marshall Bishop Apr 17 '15 at 18:26
2

If you block these URLs, users won't be able to access your products when your URL-index is not up to date. This is something you don't want, I guess. To prevent them from appearing in the SERPs, use a robots.txt file. It will only block robots from indexing these URLs and will still let your users access them.

Be careful, though! If these URLs are the only URLs indexed by Google so far, and you block them now using a robots.txt, you can lose lots of traffic!

Here are some examples: http://inchoo.net/ecommerce/ultimate-magento-robots-txt-file-examples/

...and because a link is a poor answer, here is the first example in full:

# Google Image Crawler Setup
User-agent: Googlebot-Image
Disallow:

# Crawlers Setup
User-agent: *

# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
#Disallow: /js/
#Disallow: /lib/
Disallow: /magento/
#Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/

# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt

# Paths (no clean URLs)
#Disallow: /*.js$
#Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?SID=
0

it can be done using mod_rewrite, however you must go through each module and check which administrative urls it uses and then block all of them

example:

RewriteRule "^/index\.php/admin" "-" [F]

http://httpd.apache.org/docs/trunk/rewrite/flags.html#flag_f

  • I looked at using the [F] flag but can't figure out how to make it work based on URL parameters. If I want to block all URLs containing 'catalog' can I use the forbidden rule like such: <br/> 'RewriteRule "^/catalog\ /admin" "-" [F]' – Marshall Bishop Apr 17 '15 at 18:29
  • also add L-flag for so it stops to process other rules [F,L] – greenone83 Apr 20 '15 at 11:04
  • parameters are in QUERY_STRING, they are not part of the url rewrite like the path, so you need to use a query_cond with it – greenone83 Apr 20 '15 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.