Full Page Cache on CE 1.8 - An FPC Magento Module? Varnish? Both?

Question

So I am a bit confused as I go about researching Full Page Caching for Community Edition 1.8. I have already implemented a Two Level Redis Cache, CDN, tuned MySQL's my.cnf for max performance (w/ the DB being on a separate server of course), and I have 2 servers hosting our store behind a load balancer. I say that to point out that I am not immediately jumping for the FPC before doing the initial performance tweaks.

I've never used Varnish before on any kind of site, let alone Magento, and I have never set up an FPC in Magento either. I understand Varnish to be a proxy that acts as a cross between a CDN and a page cache on it's own, sending data to the browser before the request even gets to the web server. And to my understanding, an FPC Module creates a cache locally that the webserver itself dishes out. I know that for both setups, you need to do some "Hole Punching" to get the dynamic content through to the browser (though the techniques are different, between using a module or using Varnish). Please correct me if I am misunderstanding anything here.

Up until just now, I thought of them as two separate entities that you could implement it help out your site, but now somethings I have read seem to imply the opposite. My original plan was to buy the "Warp Advanced Full Page Cache" module for Magento (Formerly the "Tiny Brick Lightspeed FPC", I believe) as it seems to be the most popular, if a touch on the pricier side (but, frankly, $350 isn't much for our company, especially for what it can do). Myself and 2 of my fellow developers were planning on learning out to implement it properly and fully within our own custom, homemade theme to maximize what we can get out of it. After that was done, at some point down the road, I figured I would look into implementing Varnish as well - but, as I said earlier, I had understood them to be separate.

Now, however, I am starting to come across extensions like this PageCache Powered by Varnish that is free, or this Vortex Cache Powered by Varnish Cache which is nearly $800 USD, that are Magento Full Page Cache modules that work directly with Varnish.

My question to you, stack exchange, is how should I be seeing a FPC and Varnish? As separate entities? If so, are they mutually exclusive? Are they two sides of the same coin that I should implement together? Or are they similar but neither exclusive nor inclusive of one another?

Can I use the Warp Advanced FPC I mentioned above with Varnish? Should I use it with Varnish? Or would it be better to use a different FPC if I plan on using Varnish? OR even further, is there an FPC so good that I don't need Varnish? Or vice-versa, should I just use Varnish and ditch the FPC idea?

Sorry for the wall of text, but I have been looking at a lot of articles, blogs, and forum posts, and I haven't been able to discern a definitive answer to those questions. I truly appreciate your help and input in this matter =)

Oh and lastly, a quick question about Varnish and webservers. Currently I am using the normal Apache LAMP stack setup, but for a while now I've been seeing people rave about using Nginx with Magento. I've done some tests myself, stress and load tests, and it seems that it can definitely work a bit better in the right conditions. As such, I was considering switching at some point in the near future. Would this in anyway affect my desire and decision to use an FPC and/or Varnish?

Thank you!!!

EDIT: Oh! And one more quick question - Since I have two servers hosting my site behind a load balancer (which is also a setup that can be increased horizontally should the need arise), I make full use of Redis and Memcached hosted on a separate server from the Web and DB ones for my sessions and each level of Magento's (well, Zend's) Two Level Cache. I assume the FPC would store it's data in one of those to systems? Would I need to have a specific extension to store it there or do they all do it? And while I assume not, would this affect Varnish in anyway? Thanks again!!

Answer 1

There are two difficult things in computer science:

1. Naming things
2. Cache invalidation.

Hole punching falls into category #2 :)

General

The best approach is to start at the lower points of the stack and optimize up to the frontend of Magento.

---

Database and Filesystem

Should always be the first areas to focus on. Because. I/O.

MyTop is a handy Linux based perl script that will mimic the Linux 'top' command and give you insight on the state of your MySQL instance(s).

• https://github.com/jzawodn/mytop

Htop is a more robust top, The strace feature can help determine ins/outs of a process to find potential bottlenecks.

• https://github.com/hishamhm/htop

Iotop is another tool to consider for monitoring I/O.

• http://guichaz.free.fr/iotop/

Other handy utility scripts like mysqltuner.pl and mysql tunning primer can offer insight into your MySQL runtime variables and offer advice to help. Keep in mind these are meant to be guides as the best approach is always an evaluation of requirements and tuning based on known data gathered. Blindly doing so can cause more damage at times than good. And prematurely running these without at least 24 hours of mysql runtime variables may offer bad advice.

• http://mysqltuner.pl
• https://launchpad.net/mysql-tuning-primer

Keep in mind Percona, MariaDB and standard MySQL should work with all of the above. Favoring Percona as a MySQL fork, since Magento is so heavy on InnoDB and XtraDB offers many tools and enhancements to the db engine.

• http://www.percona.com/software
• https://mariadb.org/en/about/

---

Apache or Nginx

Still using Apache as it has served many others well, myself included. I have used and configured Nginx as well. While it does offer some advantages there is a learning curve. While the two are both popular options, it does offer some advantages over Apache, one would be a smaller memory footprint. However a slim downed Apache running PHP-FPM will have a similar memory footprint.

• http://arstechnica.com/business/2011/11/a-faster-web-server-ripping-out-apache-for-nginx

Case in point:

Since this article was about performance, I should point out that one of the easiest ways to help apache get out of its own way is to not use .htaccess files. Put what you'd put there in your Directory stanzas, set AllowOverride to "None" and you end up not asking apache to traverse the whole document path to figure out if it needs to pay attention to .htaccess or not. This is a basic, simple tuning hint that many people seem to miss.

To help facilitate this check out:

• https://github.com/preinheimer/htaccessConverter

Utilizing a CDN to help take the ease off of either will help obviously but will have added benefit on frontend optimization since most end users browsers will be able to connect to both servers with the same number of connection limits. This also frees up Apache from not having to jump through checks and such just to serve up a simple static image. Lighthttpd is an option if you want to run a static web server just for content besides a CDN.

PHP

PHP-FPM and APC. Use them, strip out any unneeded or unrequired PHP modules not needed for Magento.

---

Magento codebase

AOE_TemplateHints is great to determine if your blocks are caching properly:

• https://github.com/fbrnc/Aoe_TemplateHints
• http://www.fabrizio-branca.de/magento-advanced-template-hints-20.html

AOE_Profiler is good for profiling, be sure and enable its DB layer profiling (in a local/dev environment obviously). This in conjunction with the mytop tool mentioned previously makes finding bad behaving SQL an easier task.

• https://github.com/fbrnc/Aoe_Profiler

3rd Party modules & Custom code

Some very good best practices for optimization from Magento themselves is a good read, and to keep in mind when reviewing 3rd party modules before using them. (there are lots of bad behaving ones IMO).

• http://info.magento.com/rs/magentocommerce/images/Conquer_the_5_Most_Common_Magento_Coding_Issues_to_Optimize_Your_Site_for_Performance.pdf

A tool Magniffer from Magento ECG will help easily identify bad behaving code based on the PDF provided above. It is symfony/php-parser based however but installable via composer.

• https://github.com/magento-ecg/magniffer

---

Varnish

As an advocate of Varnish being the author was a FreeBSD kernel dev, it offers some crazy sub second load times. However if you even have some of the slightest differences in your templates that isn't out of box, you will spend time configuring varnish / magento to holepunch the content you need. Most I've seen will simply AJAX'ify the needed items uncached from Varnish.

There are a number of Magento modules to help facilitate this hole punching and caching:

• AOE_Static: https://github.com/AOEmedia/Aoe_Static
• Nexcess Turpentine: https://github.com/nexcess/magento-turpentine

Ultimately this should be at the last end of your optimization journey, and MAY require some customization to get things right.

---

Magento CE FPC

So far the best CE FPC I have found is: Lesti::FPC

it is a very well put together (all observer based) open-source and free FPC for Community.

• http://gordonlesti.com/lestifpc/
• https://github.com/GordonLesti/Lesti_Fpc (github in case of dead future link)

---

At the end of the day use your own testing and judgement.

Some further reading:

• http://www.kingletas.com/2012/08/caching-performance-optimization.html
• https://www.sonassi.com/knowledge-base/magento-with-varnish/
• http://info.magento.com/rs/magentocommerce/images/HostandBoastBestPracticesForMagentoHosting.pdf

Answer 2

A bit late to this thread I know but if you are still looking for a solution then you may want to consider, Evolved Caching. It's the same price as Warp, but it:

• Is very quick and easy to install and configure - all hole punching and configuration is done from within admin
• Integrates directly with Varnish and lets you clear and warm your Varnish cache from within Magento
• Works with the frontend form_key introduced in 1.8 CE in both Varnish and it's own cache.
• Is very actively developed with responsive support. Regular new versions with the aim to release bug fixes within a few days of reporting
• Has extensive documentation which is updated with every release

Setting up with Varnish is straight forward, you just need to enable an admin setting and use the .vcl found here. You are also not restricted to Varnish only serving cache when no cookies are present as per normal - you get a very high cache hit rate.

Answer 3

There is no correct answer. A store should have sub 3s dynamic page loads and ideally 1-2s dynamic page loads, sub-second is not necessary and is primarily a marketing driven statistic. Apache is easy to learn and hard to make perform, Nginx is hard to learn and easy to make perform, many sites are moving over to Nginx however to have a high quality architecture based on Nginx and Magento is not simple.

Multi-server Magento clusters are already complex to implement, and even harder to maintain if not on the correct architecture, we normally work with larger clusters which makes everything run more smoothly including ranking. We do this with standard install config with minor changes for medium to long term stability targeting the 1-2s dynamic page loads, it makes everything much simpler for maintenance.

Varnish can be an FPC, CDN among others, however in your case it is best to think of it as an FPC. An FPC allows more visitors on the server and provides faster static delivery of which Varnish is one such tool, however there are various issues with it including dynamic content, stock control, pricing. The answer comes down to how your business is structured, how your data is loaded, how frequently, your hosting type and more, simply is your business affected by providing static content to visitors. You can technically mitigate much of this with FPC config however it complicates the business environment, from a business owner perspective it may not produce a balanced return on investment.

The FPC is the last part if you have sub 3s or better dynamic loading, your architecture can handle beadth in visitor requests as this affects ranking, absorb marketing and holiday spikes, and have the budget to add complexity in server architecture - hosting should be 0.5-1% of revenue for smaller businesses, most run substantially under this causing many indirect business issues.

The reason you have not found a definitive answer is due to the fact that those questions take months to answer as they are qualitative (business based) which require information a company would not want to post publicly, the page load speeds are quantitative (technical based) which can be posted publicy, it is how your combine the two that makes the solution.

Answer 4

We have written a FPC that is compatible with Magento 1.8 new form key. Brim's Full Page Cache: http://ecommerce.brimllc.com/full-page-cache-magento.html

BOOMER makes a great point about starting out low on the stack and working your way up. An FPC or Varnish should be about the last you do. We do performance audits and commonly find issues with MySQL and APC configurations that are just really off. Like Innodb buffer sizes set to the default and the database has grown way way past it.

We recommend against using any FPC with Varnish, unless specifically designed to work together. Generally, we don't recommend Varnish unless you have a handful of beefy servers which have all been tuned along with your code base and are still struggling to keep with the traffic. Updating dynamic content can be tricky with Varnish specifically when trying to limit your requests to the Magento backend and in turn reducing load. If you have one or two web heads, the gains may not be worth the time and complication.

In most situations a good FPC will get you the performance your need, of course after your server and codebase have been tuned. With our FPC you can get sub 15ms generation times on the level 1 cache and sub 100ms on the standard cache. Our level 1 cache is used for cases where the user is not logged in and has nothing in their cart as it doesn't do hole punching. When either of those conditions is false the standard cache is used with full hole punching support.

Our FPC has easy hole punching built in and works out of the box with all standard Magento blocks as well as any custom blocks you may have. It's all configurable via the admin panel.

I'd recommend sticking with Redis unless you are having scaling issues with it. It has tag support and is much faster than memcached with file or database as the slow backend. If you want consistent tags and cleaning you have to use memcached with database when you have multiple web heads. With Redis's tag support built in, you don't have to worry about it. You can also use Redis for your sessions.

I can speak for all FPC's, but with ours, you can configure via the admin where to store it. You can choose to use the default Magento cache backend or specify custom settings to use either File, Database, APC, Redis, Memcache, and an Optimized File backend.

Answer 5

You can use this Magento page cache which will fit your needs and is similar to varnish. It is used by many of the largest Magento stores. Some features:

1. Like Varnish, it does not make use of a database connection for 90% of requests. As a result, it is extremely fast
2. It has capability to auto-flush pages when things like product inventory changes and it is very good at that
3. It is a multi-layered cache so it also supports hole punching when users login (these requests require database use)

As a multi-level cache it is scalable for even the highest traffic stores and has been used on many extremely high traffic sites that receive peak traffic such as stores being featured on SharkTank (tv show)