What is the Best Magento 1 Server Setup?

Question

We are currently working with a requirement that first response from the web server must come in under 200ms in the UK. Currently under 2 dedicated web servers under load balancer and 1 db server, we are coming in at 800ms.

The site at the moment has less than 5 customers, 2 products, 4 categories, there is no frontend to the site at the moment, it is style free and image free.

It is also being run on nginx with Varnish.

Can anyone give me any advice on web server setups? Why is ours coming in slowly? What can you recommend to optimize this? Need to get 400% quicker!

Answer 1

I'll bite.

That first response from the web server must come in under 200ms in the UK
There is no frontend to the site at the moment, it is style free and image free.

You won't achieve those figures without the aid of either Varnish or FPC (or both). I would certainly hope that figure doesn't also have to include static content (whenever you decide to add it) - as its near impossible to achieve (short of having little to no images/js/css).

We are coming in at 800ms
It is also being run on Nginx with Varnish

You've got Varnish configured wrong.

A properly configured installation of Varnish will deliver <100ms page load times (we see closer to <10ms).

In fact, for Magento, you should expect to see something like this,

When a customer is not logged in ...
Ie. Not having created a unique session (add-to-cart/wishlist, logging in etc.)

--1.2s--------0.8s-----------------0.6s----------------0.1s--------------0.08s----
  Uncached    Mage default cache   Partial FPC cache   Total FPC cache   Varnish

When a customer is logged in ...
Ie. Having created a unique session (logged in, items in cart etc.). At this point Varnish will likely be off. And if you've chose to use ESI's - depending on the reverse calls, it can either maintain a similar page load time as the FPC cache (due the bootstrap overheads) - or actually increase page load times beyond being uncached.

--1.4s--------0.8s-----------------0.6s--------------
  Uncached    Mage default cache   Total FPC cache   

Its not a case of tuning Varnish - its a case of - "you are not actually caching anything at all".

---

The ideal Magento server configuration files

There isn't one, well, not quite.

We operate over 400 servers, all purely Magento stores - of varying sizes and capacity. And it is rare that the configuration files we have on one - will match that of another. That is because not all businesses are alike.

Bottlenecks can form due many different reasons:

1. High numbers of visitor concurrency, with active sessions
2. Victims of 'bad' crawl bots, generating necessary, unvaluable load
3. High proportion of layered navigation hits
4. High numbers of search queries
5. High volume of transactions per hour
6. Poorly built template
7. Too many/slow/bulky 3rd party extensions
8. Outdated inbound links leading to high proportion of 404 hits
9. Network interface capacity at limit
10. Large/complex catalogue (lots of products/categories/attributes)

So with stores all across this spectrum, each have a different approach to more optimum performance.

To solve the issues outlined above; we'll deliberately avoid just stating "more/better hardware"

1. Use a FPC beyond that of Varnish
2. Filter out/block bad bots at the network edge - or redirect all requests to Varnish regardless of cookie state/URL
3. Change stock layered navigation to SOLR, make layered navigation filters dependant
4. Change stock search to SOLR
5. Distribute MySQL load across Master/Slave configuration - only do this when you've guaranteed browsing load is absorbed by Varnish/FPC
6. Re-build the template
7. Strip them out
8. Monitor access logs continuously and rewrite URLs at Nginx/Varnish prior to delivery. If doing it at Nginx level - ensure Varnish is caching 301/302 redirects.
9. Split off static content to a CDN, or improve connectivity
10. Add more hardware (well, we had to say it at some point)

So with this in mind - you'll see there probably isn't going to be an Nginx config file, PHP fcgi pool config file, MySQL config file or Varnish config file that are going to be the same. Couple that with hardware changes itself; available memory, I/O performance (HDD and Network) and CPU - and you'll find there is subtle variations that lead to the 400% performance gain you desire - but no one quick answer you'll readily find on-line.

You could copy+paste the Peer 1 sponsored Magento white paper on peformance (we wouldn't recommend it); hope that the settings don't exceed your available memory, thread limits, TCP/IP states, I/O capacity and lead to lesser performance than a vanilla Apache/mod_php configuration.

So lets continue on.

The ideal Magento server stack

This is more likely to get you closer to reality. A good example to demonstrate this is to show how a dedicated Magento OS is configured, MageStack

Take the separate sub-components and you've got a list of the most optimum/critical software, when configured properly, to run a Magento store. Notably:

Firewall, DOS filter, Load Balancer, Varnish, Nginx, PHP, Redis, Memcached, MySQL

So when you ask:

What is the Best Magento Server Setup?

What is your goal exactly?

1. High availability
2. Reliability
3. Simplicity of administration
4. Performance
5. Scalability

Enough lecturing, how would we do it

To partially mirror the answer given on server fault down a similar vein. You've got 3 servers at your disposal - so first orient them as optimally as possible. We'll avoid a highly-available solution as that is far beyond the scope of this answer.

The sub-components required for a multi-server configuration are:

• Firewall
• Load Balancer
• Web Server
• MySQL Server
• Common Storage

So we'll multi-purpose some of the systems. PCI-DSS compliance dictates a role, for each server. So with 5 roles and 3 servers - you'll be in breach immediately. MageStack gets round this by using virtualisation - you could do the same.

Server 1: Load Balancer + Web server
Server 2: Web server
Server 3: Database server

Without low-latency and significant network bandwidth (>1Gbps, <125µs), rather than having common storage - its better for you to merely store the store root directory on each machine and replicate the data, either in real-time using ionotify or lapsed using a cron job. Again, we'll avoid network file systems like NFS, or replicated block devices like Gluster or DRBD - as vast tuning and decent network bandwidth is required.

Varnish needs to sit as close to the front as possible. But Varnish cannot decrypt SSL. So combine it with an SSL terminator; Nginx, Pound, Stunnel, Stud etc. The built in load balancer in Varnish isn't great - but would be adequate for a 2 server set up.

Nginx + PHP-FPM is fine, but don't drink too much of the Nginx kool-aid. It will perform almost identically to a traditional Apache/mod_php configuration, here's some good reading on why not to use Nginx. Nginx is good, very good infact, but its certainly not a bottleneck of a Magento store - and given its complexity and lack of native Magento support. Most novice system administrators would benefit from using Apache/mod_php over anything else. This may seem like an archaic recommendation over using PHP-FPM - but our performance tests have shown performance is only ~7% faster with the Nginx solution - when properly configured. The tuning and experience required for a high-performance, reliable Nginx/PHP-FPM set-up is fairly vast to get it to outperform Apache/mod_php. Whichever you choose to use, is your call.

The database server is simple, MySQL. But as mentioned earlier, if you have a high converting site, a Master/Slave configuration is advised. Whether you should follow this approach can be determined by reading this article.

Then your peripheral back-end caches, Memcached and Redis. On smaller stores, storing sessions in one Memcache instance and the fast backend cache in another will yield good performance benefits. We don't advocate storing the cache tags in a slow backend - as it causes more problems than it gives. So with a Memcached set-up, you'll have to forfeit cache tagging. Instead, we use a configuration like this.

Redis isn't native to Magento, but with the extension from Colin Mollenhour - its a better solution than Memcache, supports cache tags, session storage and even persistent cache storage - its not quite as volatile as Memcache. But it does have its drawbacks. We've found on large scale production stores (>500 orders/hour, >30k uniques/hour) that the cache (and tags) can fill up very quickly and once the point of saturation has been hit, the LRU mechanism fails somewhat (despite different settings) and causes a massive immediate bottleneck. So it is prudent to regularly prune old records manually.

So what hardware should be used for what?

Web servers: Fastest CPU, most CPU cores, ratio of 2GB RAM/ Core
DB server: Fast CPU, fastest disk I/O, most RAM

So when multi-purposing your 3 machines, the best layout would be:

Server 1: SSL Terminator -> Varnish -> Nginx/Apache > PHP
Server 2: Nginx/Apache > PHP, Redis, (MySQL Slave)
Server 3: MySQL

As to the specific configuration of each application. Well, that's down to your hardware specifications, your store complexity, your type and nature of visitor and the sheer volume of visitors.

Answer 2

You’re on a great path with that cluster configuration. I recommend adding a dedicated cache host for Redis; select one with high CPU power and a lot of RAM (~64 GB).

Here’s the full list of configurations I’ve used for a highly available, fault tolerant, distributed, and load balanced LEMP cluster. It includes app/etc/local.xml, the core_config_data table, and configurations for MySQL, php-fpm, nginx, and Redis. All run Ubuntu 12.04 LTS 64-bit. The configurations include a lot of optimizations without drawback.

Highlights

• Admin users: 46
• Categories: 2,450 (largest one has 2,400 products)
• Product entities: 101,000
• Combo products: 484
• Product relations: 54,000
• In stock and enabled configurable products: 10,100
• CMS blocks: 3,100
• CMS pages: 1,400

August 2013 traffic:

• 40 million monthly pageviews
• 2.3 million unique visitors
• 46,000 monthly checkouts
• 89% of visitors from the USA

---

Web hosts

There are 10 hosts behind redundant, highly available hardware firewalls and hardware load balancers. Most static assets are offloaded to a CDN.

• site-wide average response time: 282 ms
• Average FPC response: 48 ms
• load average: 0.6 to 1.0 (in tests, performance degrades by 35% when load averages hit ~5.0)
• Dual Intel Xeon CPU E3-1230 V2 @ 3.30GHz (4 cores each)
• 32 GB DDR3 1333 MHz RAM

Modules

• https://github.com/colinmollenhour/Cm_RedisSession
• https://github.com/colinmollenhour/Cm_Cache_Backend_Redis
• https://github.com/aligent/CacheObserver.git --branch=models
• https://github.com/aligent/Lokey_ShippingAdjustments
• https://github.com/aligent/VF_CustomMenu.git --branch=production
• https://github.com/Vinai/customer-activation
• https://github.com/gknoppe-guidance/magento-cachebuster

---

Cache hosts

There are two hosts running Redis in a master-slave configuration with automated failover. Three Redis instances (sessions, backend, and FPC) are used to increase throughput and provide fine-tuning of persistence behaviors.

• 3,000 commands per second
• 0.7 ms average response time
• load average of 1.0 to 1.5
• Quad Intel Xeon CPU E5-2620 0 @ 2.00GHz (6 cores each)
• 128 GB buffered DDR3 1333 MHz RAM
• Mechanical disks, RAID 1, hardware controller

---

Database hosts

There are two hosts running MySQL 5.6.11 in a master-slave configuration with warm failover.

• 1,500 commands per second
• 1.1 ms average response time
• load average of 0.1 (master) and 0.4 (slave)
• Quad Intel Xeon CPU E7- 2860 @ 2.27GHz (10 cores each)
• 128 GB buffered DDR3 1333 MHz RAM
• SSD, RAID 1+0, hardware controller
• MySQL 5.6.11 with tcmalloc

Answer 3

Another must have server tip for Magento is installing PHP 5.4 or 5.5 with OPcache. PHP 5.4 is much more faster than 5.3 (http://www.eschrade.com/page/magento-performance-on-php-5-3-5-4-and-5-5rc3/).

Answer 4

I want to add another important tip that improved magento page speed when not cached by varnish and is not enabled by default (cart page load time changed from 6sc to 1.5sc).

Activate mysql query cache in /etc/mysql/my.conf

query_cache_size = 268435456
query_cache_type= 1
query_cache_limit=1048576

the cache_type enable it, cache size is the value used by cache in memory and cache limit is the maximum size of query result to cache

Answer 5

With our current configuration we are receiving initial response in 400 ms and document complete in 2 seconds (using a standard connection of 5mbps). Our homepage size is 1mb.

Our setup is based on AWS as follows: We have an ec2 instance with a load balancer connected to an RDS database (with failover). We have also implemented full page cache with a redis cache backend for both cache storage and session storage.

On average we have 300 - 400 visitors a day but with redis caching enabled we have had minimal ec2 resource usage while maintaining speed and bringing down costs.

The reason we have a load balancer is that the ec2 is setup to automatically boot up a new instance if on the rare chance we have traffic spikes that the current setup cannot handle.