1

I am considering a migration over to AWS for my Magento instance since I have been told that it is the best solution for handling high traffic volume and huge catalog 100, 000+ products.

Could anyone recommend how to go about the migration over to AWS please?

I have found this link to their page but wanted to make sure that I am signing up for the correct service with the right configuration including Varnish cache beforehand...

I would really like a step by step guide to the process if its not too much trouble. Is it as simple as a regular migration from a shared host or dedicated server?

Any advice will be much appreciated!

  • 4
    Oh dear, millions of products are a huge site but that is another story. AWS is an enterprise class environment, you need enterprise class knowledge to use it effectively. If you are asking for step-by-step guides you are better with a managed host such as Nexcess, Vanquish, Crucial, AWS will not help you in this context. – user2935 May 7 '14 at 23:16
  • 1
    Use HHVM on dual web/db clusters - you will outperform almost all Magento hosting out there and it reduces your server specs by 10fold - it's the retailer win:win no service provider wants in the public domain. You will see (3) posts if you search for HHVM here - so where did all the other posts go - it's insane given Facebook developed and run on it. – user15949 Nov 4 '14 at 13:35
  • 1
    Seriously - they deleted our user again today - someone doesn't want HHVM to become known even though for a retailer you don't need FPC, Google loves them especially on due node clusters, and you get 5-10x the performance for the same hardware. – user15963 Nov 4 '14 at 14:47
12

You've been advised poorly.

Remember that AWS doesn't do infinite scaling miracles, its just a virtual server provider

AWS is nothing more than a set of virtual servers, running in a contended, shared environment. Coupled with the fact it runs older generation hardware, that has its resources crippled by the AWS hypervisor - its performance is less than stellar if speed is important.

Its multi-tenant, so your performance isn't guaranteed

What you need to understand is that AWS is an entirely multi-tenant environment - from the physical hardware, to the internal networking, to the external networking. The performance you see will be heavily influenced by activity from other users of AWS. So you could have moments of "normal" performance, followed by catastrophically slow performance. You could opt for enhanced/clustered networking - but it is still effectively a public network performing a balancing act with QoS/CoS to try and improve latency/bandwidth to some users whilst limiting it for others.

Underlying hardware is a gamble of old technology

Also consider that you really have no idea what hardware your different instances are sat on. Some may benefit from newer generation CPU's, others significantly older - it is effectively pot luck as to what you get - and computationally how one instance may perform compared to another.

Scaling traffic isn't that easy, you'll hit a vertical ceiling almost instantly

In terms of handling high traffic, you are talking scalability, and AWS is constrained there too. You can only effectively scale horizontally. The capacity to scale vertically is limited to 32 cores - which is going to cripple any efforts to scale vertically.

But you can almost infinitely scale horizontally

So at this point, horizontal scaling is your only mechanism. Once you start scaling horizontally, you've got to add in considerations for

  1. Load balancing
  2. Shared/common storage
  3. Availability

... and worrying about the fact that with each additional instance you run, the risk of failure is amplifying by as much. So high availability must then be built into your service.

When scaling horizontally, you are also back to the common bottleneck of contented networking. So whilst you can spin up infinite instances for load balancing, web servers, db servers and cache servers - your baseline performance will be dominated by the low communication speed between said instances.

AWS isn't that cheap

Once you dig into the costs of "the cloud", you'll suddenly realise that you're actually paying close to 2x or 3x as much you actually would for a dedicated server. Now this is for a reason, you can create/destroy instances on a whim - so you are paying a premium for the flexibility of reducing cost.

But your, and many others', main reason for looking to use cloud hosting is to have infrastructure that will support your growing business, this means that your business is growing (or plans to),

Ie. the plans are to outgrow the original VPS and you want infrastructure to support it

So now, the original premise of cost savings (but running only what you need) is effectively gone - and once your business has grown, and is using the extra resources that you wished for - you'll suddenly realise that you are actually paying about 3x as much as you would have been if you just opted for a dedicated box to begin with - not to mention, have significantly worse performance.

The burden of management, deployment and configuration is entirely on you

The very fact you are asking about setting up AWS is evidence enough that you shouldn't attempt this endeavour at all. At best on AWS, you'll get an average performance Magento store - at worst, you'll have a terrifically slow, unreliable store.

Don't expect any support from Amazon - not even email support. You are limited to community forums for help (which, if the worst is happening and you are losing sales (£££) by the minute, is costing your business significantly). You can opt for paid support, if you can afford the costs (read: not cheap) - but don't expect the person on the other end to be able to offer any application specific help for Magento.


My advice

There are plenty of capable Magento hosts out there that run specialised infrastructure designed to scale - and plenty of which who have experience with both high volume stores and large catalogue stores.

You would be mad to go it alone, especially if you think that your magic bullet resides in AWS's promise of infinite scalability.

  • Fantastic answer. I wish I could up vote more than once – Joe Constant May 7 '14 at 23:54
  • 1
    Unforutnately the above answer is both correct and incorrect. As for average performance, AWS is more than that and we have used 1-2s page loads with first page Google next to Amazon & eBay on Magento, it was a special architecture sitting on AWS but that is something else. What the answer above is trying to say is that for 95% AWS is a neutral to bad choice, what they have missed out is that for the top 1-5% it is a good choice. 50% of all online revenue is generated by the top 1% of companies, if you want to be in that top few % you need hosting like AWS - along with a high end architecture – user2935 May 8 '14 at 10:10
  • 1
    1-2 second page load times? Anything more than 0.7s is slow, not exactly a convincing argument for AWS. What all Magento store owners need is a competant, capable host that understands Magento inside out. How to architect the underlying equipment to be high performance, to be able to scale and be reliable at the same time, whilst being able to provide store level support in times of need - but they are few and far between and certainly not what you will get DIYing a solution via AWS. – Ben Lessani - Sonassi May 8 '14 at 19:05
  • What marketing nonsense, Amazon, Nordstrom all have 1-2s page loads (Radware State of the Union), technical selling again. Put a company on some new third party hosting which produces a few 100ms page loads without FPC (fully dynamic content) - tier1 as well but sme prices - nice, but then they are architects for multi-nationals. A bit like saying a small retailer knows how to rank higher than Amazon for their products, oh, hold on, they ranked a new 30,000 product site first page next to Amazon & eBay for the same products in a few weeks - with 1.5s non-fpc page loads on AWS - oxymoron. – user2935 May 22 '14 at 15:55
1

AWS best solution for handling high traffic volume and huge catalog 100, 000+ products.

wrong and wrong. but anyway you just looking in front of you. the problem is not "how to transfer magento to AWS", but "what im goin to do when i get there...".

From my experience: take a piece of paper and start drawing your infrastructure. Make sure you know how Nginx proxy/load balancing works. Prepare all configuration templates.

  1. start from sizing your database and comparing it to AWS RDS pries/instance size. go to AWS calculator, select your RDS instance with enough RAM, and add some storage(more storage = faster DB), you need to start with about ~1000 IOPS and ~500Gb(as example), you will see how monthly price will jump up. think twice.
  2. handling high traffic - i was so naive thinking that AWS auto-scaling and load balancing for magento is simple as a snap your fingers. in a real life it is a complete nightmare!!
  3. use just a simple raw operating system image, do not use anything else, all pre-configured AMI and images on marketplace built specifically for magento is just a CRAP. Install your software and create your server template, etc.(if you go for CentOS6.5 use Bashton (PV) images)
  4. when you create your instance you will have EBS root drive as default, BUT! do not forget to add remaining instance "store volumes". because when you boot your instance you wont be able to add store volumes. think about space before you click launch, for admin server you will probably connect few EBS drives, like root|magento|media. Do not install magento on admin server to your root drive.
  5. prepare for synchronization: github|rsync|csync2 , etc.
  6. be ready for opcode cache cleanup. this is a real pain.
  7. configure your logs LEMP|magento|apps -> to go to one centralized log processing location. if not do so immediately, then later you will tear your hair on the ass in search of a problem.
  8. learn AWS api.

configure admin server, make it ready. then go create one frontend server. connect with admin, synchronize things, etc. then create varnish cache server. when you happy, roll out more frontend servers and go live.

but it all depends on your experience, and your shop configuration. no one will tell what to do exactly step by step.

1

I'm not going to argue for or against the other answers here. I guess it has its pros and cons to host at Amazon. We use AWS for all our websites, but no high traffic websites. From your question I can't tell how complicated your site is, how many servers are needed, load balancing, etc.

We have about seven servers, all medium instances, two database servers, one Postgres, one Mysql. For me these are normal servers, same as if it was hosted here in a datacenter. If you have an AWS server, you do not share this with other users! You have all the power, and if you do not use it, maybe Amazon can divert it to other users, but if you need it you should have all the power.

One big pro: snapshots

You can set daily snapshots, and you can restore them in a few minutes. You have to get into this, but if you know what you do, the server can crash, and you are back online within ten minutes. Maybe other providers offer this as well, but the way it's setup is magnificent I think.

Hire a good system admin to set it up properly! If you're going to try this yourself, then first set it up and leave the production site as it is.

Support

Amazon does offer support, and at a reasonable price. We have used it several times. But once again - for a big high traffic website, have local support available. Amazon won't set up your server, but will help you with problems with snapshots, etc, and stuff specific to all their services.

  • Thanks for the answers guys, I've spoken to some other Magento devs that I have worked with in the past and can vouch for their experience. The consensus is AWS isn't the way to go for some of the same reasons posted here. I was advised to consider using nexcess.co.uk/magento/hosting so I will make further enquiries and post back my honest opinion for others to gain from... – user1704524 May 8 '14 at 20:46
  • Check out MageStack - its effectively everything you've described. – choco-loo May 8 '14 at 20:53
  • Take into consideration the location of the servers, which country! – SPRBRN May 8 '14 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.