Setting up Magento Staging Environment with Restricted Access

Question

I'm trying to figure out the best way to setup a staging environment with some access restrictions.

The simple solution would be to throw up Basic Authentication, but then I won't be able to point Google Page Speed Insights at it while testing performance optimizations, as well as other similar external services that I want to access it.

Could make it completely public with robots.txt in order to prevent it from showing up in search engines. But my concern there is that the risk of any mistake in the robots.txt is fairly high, and I'd rather not have to worry about that.

If you don't block search engines (or if some ignore it), then you'll be getting live customers placing orders to your staging site, which won't make them happy.

Or even worse, if you accidentally deploy the robots.txt to production, you'll lose all your Google juice and a good chunk of sales.

So the option that I'm liking is a simple IP address restriction. But I'd love to be able to add/remove restrictions without having to restart Nginx, just to again minimize risk while making changes.

So I'm beginning to lean towards a quick module that, when enabled, will look at developer IP addresses and only allow access to the site (front and backend) if the user's IP address (or X_FORWARDED_FOR) match it.

Wondering if this sounds like a reasonable solution or if there's something simpler that I'm missing.

UPDATE: Given that the robots.txt can be controlled via a native backend switch and the demo store notice will prevent any legitimate customer orders, and since I'm really not concerned about public access to the staging site, I like Phil's solution.

But for anyone that does want to restrict access to their staging site, I think Kris's solution is the way to go.

UPDATE 2: Not 100% sure what the robots.txt options are supposed to do in System Config > Design > HTML Head, but in my case - and from a brief search this appears to be common - I just have a flat robots.txt text file in place that's being used, so that config option isn't being respected.

So I'm going with the maintenance module for now: https://github.com/aleron75/Webgriffe_Maintenance

Answer 1

A few suggestions - some are built-in!

- Developer IP restriction is built-in in System Config > Developer:

This doesn't restrict IP access. Move along.

• IP restriction is tough and I prefer to handle this at the firewall, personally. IP tables is also a candidate, as is htaccess restriction or via $_SERVER['REMOTE_ADDR'] in index.php.

• Update the default per-page robots meta in the CMS to NOINDEX/NOFOLLOW while in staging in System Config > Design > HTML Head:

• In the same config area, is the ability to display a demo store notice:

Answer 2

Our primary means of locking down (most) staging environments is BASIC authentication. But we also have preventative measures in place to prevent them from being discovered by engines, barring a link showing up on a public website (this never happens), and also to prevent them from being indexed by Google.

I've setup a rule in /etc/httpd/conf.d/robots.conf with the following alias:

Alias /robots.txt /<path_to_public_html>/robots.txt
<Location /robots.txt>
  Satisfy any
</Location>

The alias routes all requests to the robots.txt location to a locked down file. This means it doesn't matter what is in the robots.txt file in the Magento staging root, the server will always serve up the following rules:

User-agent: *
Disallow: /

The location and satisfy any allows the robots.txt file to be served up to anyone regardless of the authentication since we do not have global disallow from any rules.

For the password authentication, I've got the rules setup so that I can open the authentication on a single site temporarily by adding Satisfy any to the .htaccess file. This is because we run multiple stage sites on the same dedicated internal staging server. It will also allow setting allow from rules along with the Satisfy any to remove password authentication to specific IP addresses while maintaining it for everyone else (if I really need to).

The reason I do not like IP based whitelisting across the board (i.e. with no password based authentication) is because client's IP addresses are not always static. Which means we then would have to update their IPs to get them access on a (potentially) daily or weekly basis depending on how long their ISPs DHCP holds the lease.

For DNS, we use wildcard DNS so that DNS crawlers will not pickup on all the stage site hostnames which need to have public DNS. Google will actually pick up a site from DNS records. This prevents that, meaning the only way for them to find it is if someone leaves a link laying somewhere. But with forcing the robots file to serve a disallow rule will stop them indexing it if they do find a link.

Were I in the place of a merchant running a stage site for the company website, I would do things a bit differently and would just straight up block all traffic coming to the stage box unless it came known IP addresses. Anyone working on the site remotely (in-house) would be required to connect to a company VPN to access if they did not have a static IP which I could whitelist.

Having public DNS is a must if you need to test things like payment processor integrations, which, unlike most gateways, must make callbacks to the server to go through the payment process.

Answer 3

I have developed a module for enabling a maintenance mode which can be used with the same effect of blocking users accessing the fronted (not the admin which can be limited with Magento's native IP blocking feature).

You can infact allow some IPs to access the frontend even with maintenance mode enabled.

Maybe you can give it a try, hoping it could help. It's free and open source: https://github.com/aleron75/Webgriffe_Maintenance

Answer 4

Theres several different ways of doing this.

One way would be to make your developers edit their /hosts file with the correct IP address.

There is a extension out there that claims to do this in a more elegant way: http://www.magentocommerce.com/magento-connect/et-ip-security.html

Answer 5

Since you asked about Varnish in the comments, I'm going to share my configuration with HTTP Basic Authentication using Varnish, including exceptions. You have to set it up in the VCL, otherwise cached pages would be always accessible.

Varnish VCL configuration

I want to allow certain IP addresses and ranges for callbacks of payment providers and such, which I define as ACL at the top of the VCL file:

acl payment {
  "1.2.3.4"/28;
  "1.3.3.7";
}

Then add the following at the end of vcl_recv, just before return (lookup):

if (! req.http.Authorization ~ "Basic XXXXXXXXX"
&& ! client.ip ~ payment
&& ! req.url ~ "^/index.php/ADMIN/.*/upload") {
    error 401 "Restricted";
}

payment is the ACL defined above. I also allow access to the upload route because the Flash uploader does not send authentication headers and thus fails behind HTTP Basic Auth. Replace ADMIN with your actual admin URL. You can add any other exceptions this way.

XXXXXXXXX is the base64 encoded username and password. Run the following on the shell to generate this string:

$ echo -n "username:password" | base64

Then add this at the end of the VCL to define the 401 error response:

sub vcl_error {
if (obj.status == 401) {
  set obj.http.Content-Type = "text/html; charset=utf-8";
  set obj.http.WWW-Authenticate = "Basic realm=Secured";
  synthetic {" 

 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 
 "http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">

 <HTML>
 <HEAD>
 <TITLE>Error</TITLE>
 <META HTTP-EQUIV='Content-Type' CONTENT='text/html;'>
 </HEAD>
 <BODY><H1>401 Unauthorized (varnish)</H1></BODY>
 </HTML>
 "};
  return (deliver);
}
}