Magento - Production and Test environment - Best practices [closed]

Question

I know there are some posts related to this topic available, but I do think it covers my questions:

I have now set up two environment: Staging and Production, as the shop is not live yet it is quite easy to maintain, as the both sites are identical. However when orders, customers etc starts to flow in the production environment and we start creating test orders in the test environment, that is when it becomes messy.

Scenario 1: Take db backup daily and take backup of files if changes are done. This can be done through ssh on the server. Not ideal, but it works I guess.

Scenario 2: Lets say I want to install a new module, that will impact both the db and the files. Then I create it in the test environment, see that it works, and then I install it directly in the production environment. "Crossing my fingers that it will work".

So currently what I have done is that I have connected my test and prod env to mageflow, so when I add products, changes to cms etc, I can push it to the prod env. In this way I can work in test and push changes when im ready. This works fine.

However my biggest question is how should I handle magento upgrades, installing of custom modules etc. I tried creating a bitbucket account, but I was not able to put the magento files into the .git. After spending some time on that it struck me that I didnt have a versioning on the database.

Can somebody explain to me step by step how I should set up my shop, so that I can get it correctly from day 1. So that it is easy to test out new modules, upgrade magento etc without having to much downtime on the prod environment.

Looking forward to hear your reply on this topic.

Thanks

Answer 1

After much trial and error I've developed a workflow that allows me to quickly and efficiently setup sandbox environments as needed in a matter of minutes. A 'sandbox' and can be used for 'staging' of Magento development (dev -> test -> production) and involves a combination of database backups, scm (e.g. git), shell scripts, subdomains and localhosting.

In addition to using these techniques for a staging workflow for development, these techniques can be extended to provide a way of working with 3rd parties (contractors or extension publishers) in a secure / scrubbed environment that firewalls them from production environment and data.

There is a lot of detail here, but once this is setup it is a very productive workflow for developing and testing your Magento install in isolated sandbox(es).

Setup New Sandbox

To setup a new 'sandbox' for my Magento installation from an existing product site, take the following steps.

It isn't uncommon to have multiple sandboxes, one might be a subdomain dev.mydomain.com (on same host) and another might be a localhost domain localhost.mydomain.com (using hosts file to map the domain)

1. Setup a new hosting environment for sandbox
   • can be a subdomain, or localhost of my magento site
   • including new hosted domain, perhaps use a subdomain of my site on the same hosted server, or something local
   • create a new database (with different name and credentials to avoid unfortunate accidents)
2. 'clone' the site repository into the sandbox environment
   • create (or use existing) branch for the sandbox 'role' (e.g. test or dev)
   • perhaps separate branch for production and use 'master' as an integration branch
3. edit the localhost.xml, not in scm to keep it environment specific
   • using the new database and different credentials to be safe!

Once you have an established sandbox environment you can repeat the following steps each time you want to refresh the sandbox with the latest production version.

Refresh Sandbox

The following steps can be done manually, or automated with shell scripts.

4. Refresh the source from the scm

   • Something like 'git pull' and 'git checkout devbranch' for your preferred scm and branch

5. Load the latest production database backup into the sandbox database server

   • Typically using 'mysql' command line

6. 'Localize' the database to the sandbox environment very important

   • Without localizing your database to your sandbox environment many unexpected behaviors can occur. For example, existing cookies in your browser could affect the behavior of your sandbox environment if you don't have some uniqueness to the cookies for your sandbox.

The following is an excerpt from a shell script that localizes the database to the 'sandbox' domain. These steps can be normalized or you can define the shell variables in your own shell script and for the command-line database tool appropriate for your environment.

    echo -e "* Step 1: change site URLs from production domain to staging domain"
    $DBCALL -e "UPDATE core_config_data SET value = REPLACE(value, 'http://$OLDDOMAIN', 'http://$NEWDOMAIN') WHERE value like  'http://$OLDDOMAIN%'"
    $DBCALL -e "UPDATE core_config_data SET value = REPLACE(value,'https://$OLDDOMAIN', 'http://$NEWDOMAIN') WHERE value like https://$OLDDOMAIN'" 

    echo -e "* Step 2: Replace all production notification email addresses with a test email address"
    $DBCALL -e "UPDATE core_config_data SET value = 'test@$ROOTDOMAIN' WHERE value like '%@$ROOTDOMAIN'"

    echo -e "* Step 3: VERY IMPORTANT!  Update cookie domain to avoid cookie collision with live site"
    $DBCALL -e "UPDATE core_config_data SET value='$NEWDOMAIN' WHERE path='web/cookie/cookie_domain'"

7. Anonymize the database

   • This is a critical step if you want to allow 3rd parties access to a dev / testing environment.
   • This shell script can be customized to your own needs

8. Delete the contents of the var/cache directory

Ready for Development! You are now ready for development in your sandbox. This might involve multiple commits to your scm branch, staging from dev to test (following similar techniques) and

9. Complete development and testing

   • optional testing in a separate staging environment.
   • one approach is localhost for development, and a subdomain on the production hosting environment before bring your changes into to the live enviornment.

10. Commit and push your changes to your

Staging your changes to production After you have completed development and testing, time to move your changes to production.

11. Merge your changes into your production branch

    • Use your preferred workflow, one approach is to merge all dev changes to a master branch and from their to live or production

12. Pull/Checkout your changes into your live site

    • Bring your changes (now in your live branch) from your scm
    • clear your cache

Done!

With this workflow, and some shell automation, it is possible to refresh a sandbox environment from production environment and data to a sandbox, develop/test changes, and deploy safely back to production in a matter of minutes.

Answer 2

The way I approached these very scenarios will be much different than what people are going to suggest.

Virtualization: By virtualizing my Magento installation, I have taken out all the stress associated with testing OS patches, Magento upgrades and Custom Module implementations.

I am personally using VMWARE which offers a FREE platform that will get you started. I clone my production server with little to no noticeable performance hit on the front end and from that clone I can test everything exactly as it would be on the production server.

Snapshots the beauty about virtualization is that you can easily take a snap shot of your working state and then with a click go back to an earlier time before the changes were made. If something goes wrong in either your production or test environment you are right back to before the problem occurred.

Regular backups there are plenty of free utilities that allow you to backup your virtual machine(s), I have this automated daily and archived.

No doubt someone will chime in with backing up directories, databases and while I agree that it isn't a bad idea to have a good understanding about how to perform these tasks manually (and requires much less storage space), I would trade the speed in which you can get your systems up and running again (seconds) over lengthy restores and file transfers.