3

in a setup with multiple magento frontends i discovered that using a shared memcached as a cache backend over the network slows down the frontends (probably because of network latency) in comparison to a memcached which runs on each frontend and is accessed via socket.

my idea was thus to write a cache backend which stores and reads cache entries locally on each frontend. so far this is simple, only invalidation is the problem, if an action occurs where one frontend invalidates a cache entry the other frontends would not notice that.

so to invalidate the other frontends i send all remove and flush commands to all other nodes using curl and a php-script to issue the same command locally.

an example:

node1: get X : false
node1: store X

node2: get X : false
node2: store X

node1: remove X : true

now node2 remains to have the old version of X

now with the "distributed command":

node1: get X : false
node1: store X

node2: get X : false
node2: store X

node1: remove X : true
node1: --> node2: remove X : true

now both nodes would store a fresh version of X next time it is needed

My Question:

is this insane or does it make sense? is there something out there which does that already? am i missing a point about memcached using multiple servers? should the network latency not be a bottleneck? to clarify: by reading locally from memcached i was able to speed up a category view by 100ms (currently it is at around 300) so imho this is worth the trouble.

  • Surely this is just an obvious bottleneck with both your network and TCP/IP settings. Even on a (slow) 100Mb connection, the packet size would have to be about 500Kb to incur 100ms of latency (with 3-way handshake etc.). Using any "local" cache backend prohibits horizontal scalability, equally, having exceptionally poor network connectivity does the same. NB. I wouldn't touch Memcache with a barge pole - just use Redis. – Ben Lessani - Sonassi Mar 4 '15 at 21:16
  • what exactly would redis do differently? the network is 10GE, what i am observing is fluctuations in access time when fetching cache items over the network from memcached (for one cache entry between 0,5ms and 1,5ms, this happens 100 of times so it adds up, using a local memcached instance via unix socket brought this value done and made it stable). – greenone83 Mar 4 '15 at 21:34
  • You are using 10GbE and are seeing latency like that? Is it a wholly owned network or shared? Can you account for other traffic over the switches? Are the packet buffers deep enough? Are you using visualisation? If so, what driver (emulated or pass through)? If so, what switching method (bridging or ovs)? Have you tuned the sysctl settings on your machine for a 10GbE network? There's a lot of factors at play here, outside of just using Memcached over a network. – Ben Lessani - Sonassi Mar 5 '15 at 9:05
  • We've got thousands of deployments, all using network based Redis instances, that add near-zero overhead compared to a local socket. We don't use Memcached (and I've posted A LOT of times about the two) because it is volatile - it offers no persistence and doesn't support tagging. Redis is the more appropriate product by quite a measure. Your issue screams network contention or misconfiguration - not a limitation of Magento's interface with Memcached. – Ben Lessani - Sonassi Mar 5 '15 at 9:08
  • so what you're telling me is that 2-3ms is too long to fetch an entry from memcached over the network and can not be considered normal? – greenone83 Mar 6 '15 at 11:42
1

Memcached over TCP/IP is always going to be slower, especially over a network, but it's a tradeoff between performance and scalability (Alan Kent made a recent quick blog post about this actually).

I found an article comparing the speeds of various caches, although somewhat old, it's quite an interesting read - essentially the findings relevant to you is that a file cache could do 27000 cache gets a second, vs 12200 for memcached, so a local FS cache is more than 2 times faster, bear in mind though I'm not sure what the round trip delay was to the memcached server being tested, so it's probably best to do your own benchmarks to be certain, Colin Mollenhour has a tool to do this called Magento Cache Benchmark.

With your approach I'd be concerned that the hosts could easily have different cache entries because they read whatever ended up in cache (from DB or wherever) at different times, so you could end up with a situation where host A have Z in the cache, and host B has Y in the cache.

I'd probably advise going down the route of having a shared memcached or redis and a HTTP cache such as varnish in front of the hosts in order to bring that page load speed down, it's a more complex architecture but it ensures that all hosts have a single source of truth which, when debugging production issues, will be a godsend!

  • i am not so concerned about the possibility of different cache entries, which ever host invalidates a cache entry will inform the other hosts about it, thus they will recreate their local entry, the only problem occurs if a host does not receive this signal, but for that to happen the network would basically have to fail and that is unlikely. varnish will not serve us because of the high fluctuation we have (new products every few seconds, lots of customizations) – greenone83 Mar 4 '15 at 11:13
  • Are you using EE? If so, will you be using this caching mechanism for the full page cache? If so, the underlying data might change between the time host A and host B cache the entry, resulting in differences when customers refresh the page. If you're not using EE, or another full page cache, then the problem isn't so bad - Magento caches configuration, layouts, block HTML output (which isn't used extensively by core), translations, collection data, EAV types, and web services config - none of these are likely to change outside of Magento, so presuming your invalidation works you should be ok. – Mike Whitby Mar 4 '15 at 13:14
  • I'd still consider varnish though, as you'll reduce TTFB on a category page from something like 500ms or so to 30ms or so, and you can invalidate URLs specifically if your product range changes frequently - this might mitigate the need for a very fast cache. – Mike Whitby Mar 4 '15 at 13:16
  • imagine that we let another product into the shop every few seconds/minutes, this would in fact invalidate thousands of pages each time... and we would have to change all session-specific parts to ajax... not really a great idea for us... also the different options to filter and sort would create another gazillion versions of pages which would also be invalid every few moments... i really think our site is not suited for a FPC – greenone83 Mar 11 '15 at 9:58
1

i know now that it won't work as described.

due to the way magento uses the cache it must be one coherent/consistent cache.

why:

  1. magento uses the cache for keeping states (like config_global_lock)
  2. magento calls remove on the cache after it detects a miss, right before it stores, this could lead to ping-pong of invalidations between nodes

i am now looking into:

  1. repcached with multi-master
  2. redis cluster (although i don't know if it can provide a master/master-setup)
  3. NFS with cachefilesd

[update]

as requested some details (this is magento on 1.5):

load(h) means hit and load(m) means miss

global lock:

request /index.php/admin/cache/massRefresh/key/fb71891674cbe8f67f268863d93cf5ae//POST
1425474402: load(h): 51a_CORE_CACHE_OPTIONS
1425474402: load(m): 51a_CONFIG_GLOBAL_LOCK
1425474402: load(h): 51a_CONFIG_GLOBAL
1425474402: load(h): 51a_APP_4E4ABDD8DC00C3DACB3C1597944A3B6C
1425474402: load(h): 51a_STORE_ADMIN_CONFIG_CACHE
1425474402: load(h): 51a_APP_B1FB6E8F13287C01E5C05063633DDA4C
1425474402: load(h): 51a_APP_E4D52B98688947405EDE639E947EE03D
1425474402: load(h): 51a_STORE_DEFAULT_CONFIG_CACHE
1425474402: load(h): 51a_STORE_DUTCH_CONFIG_CACHE
1425474402: load(h): 51a_STORE_FRENCH_CONFIG_CACHE
1425474402: load(h): 51a_STORE_SPANISH_CONFIG_CACHE
1425474402: load(h): 51a_CONFIG_GLOBAL_STORES_DEFAULT
1425474402: load(h): 51a_CONFIG_GLOBAL_ADMIN
1425474402: load(h): 51a_CONFIG_GLOBAL_STORES_ADMIN
1425474402: load(h): 51a_CONFIG_GLOBAL_ADMINHTML
1425474402: load(h): 51a_TRANSLATE_DE_DE_ADMINHTML_0_DEFAULT
1425474402: load(h): 51a_ADMIN_NOTIFICATIONS_LASTCHECK
1425474402: load(h): 51a_AMBASE_NOTIFICATIONS_LASTCHECK
1425474402: removeMany: 
    51a_CONFIG_GLOBAL
    51a_CONFIG_GLOBAL_ADMIN
    51a_CONFIG_GLOBAL_ADMINHTML
    51a_CONFIG_GLOBAL_CRONTAB
    51a_CONFIG_GLOBAL_INSTALL
    51a_CONFIG_GLOBAL_STORES
    51a_CONFIG_GLOBAL_STORES_ADMIN
    51a_CONFIG_GLOBAL_STORES_DEFAULT
    51a_CONFIG_GLOBAL_STORES_DUTCH
    51a_CONFIG_GLOBAL_STORES_FRENCH
    51a_CONFIG_GLOBAL_STORES_SPANISH
    51a_CONFIG_GLOBAL_WEBSITES
    51a_STORE_ADMIN_CONFIG_CACHE
    51a_STORE_DEFAULT_CONFIG_CACHE
    51a_STORE_DUTCH_CONFIG_CACHE
    51a_STORE_FRENCH_CONFIG_CACHE
    51a_STORE_SPANISH_CONFIG_CACHE
1425474402: load(h): 51a_CORE_CACHE_INVALIDATE
1425474402: save: 51a_CORE_CACHE_INVALIDATE
1425474402: load(h): 51a_CORE_CACHE_INVALIDATE

remove after miss and before save

1425475004: load(m): 51a_TRANSLATE_DE_DE_FRONTEND_1_DEFAULT
1425475004: remove: 51a_TRANSLATE_DE_DE_FRONTEND_1_DEFAULT
1425475004: save: 51a_TRANSLATE_DE_DE_FRONTEND_1_DEFAULT
1425475004: load(h): 51a_TRANSLATE_DE_DE_FRONTEND_1_DEFAULT
  • Interesting, I've not seen config_global_lock before, would you be able to elaborate on it to help myself and others? Also, did you manage to trace down where the code responsible for removing a cache entry before saving it? I had a quick check but could not see such code – Mike Whitby Mar 4 '15 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.