5

For all the Magento people running multiple web servers, how do you handle media folder?

  • NFS
  • each server has its own media folder and keep it in sync (lsyncd?)
  • other "smarter" fs (glusterfs)
  • other
7

NFS and nothing else for all our servers. Writes and reads are so very, very limited that there is no reason to use anything else. Varnish caches almost everything so activity over NFS is negligible.

Performance is quick, reliable and secure. No reason at all to use anything else.

In the past we've used deviations based around rsync - but only in odd circumstances where we felt NFS wasn't appropriate (geographically diverse servers).

There is absolutely no use case for Gluster or any other complex distributed file system. It shouldn't even be considered.

  • 100k products, you import images for 5k (distributed across various categories)... varnish cache needs to be cleaned... how will NFS handle that? or your customer asks for a bigger thumbnail... – FlorinelChis Feb 24 '13 at 16:49
  • 1
    It will handle it absolutely fine, without a hiccup. NFS is extremely low latency and can handle reads well enough to saturate the disks/network. We can (and do) saturate 1Gb/s links with conventional SATA disks. And on our 10Gb/s NICs, can saturate the link with a 4 disk RAID10 array of SSDs. NFS is more than adequate and perfectly suited to this role. – Ben Lessani - Sonassi Feb 24 '13 at 17:12
  • This mirrors my experience. NFS works very well unless you have a very write heavy application. – Aaron Bonner Feb 24 '13 at 17:36
0

I used NFS on two occasions and it failed for me during high IO load, similary Glusterfs is a big fat guy, not recommended, it is slower then NFS, IMHO I would recommend using Amazon S3 with combination of CloudFront or some other cloud based file storage, NFS and Gluster FS would give you tough time during high IO.

If you have small number of files, in thousands may be rsync would be a better option for you

Edited Version

Since I could not comment on the original post, I am editing my current answer.

There is already an extension available https://github.com/magefm/cdn I tried that myself and most of the parts just works fine, I had to enhance add a few features according to my requirements, it can be slow but once the cached files has been created, it is even more efficient, Its not an overhead but rather convenient IMHO. If you can then write a bash script to generate thumbnails and small images overnight or once a week, that too will make things efficient for you.

Note** If you have millions of files, I strongly recommend using NOSQL instead of MYSQL to store cached image information, and then query that for fileExiss

to summerize if you have millions of files that you would want to sync, NFS, rsync, GlusterFS all these local distributed storage systems cannot be an option, Object Storage can be the only better solution.

  • Im curious to know if you have actually got S3 working well with Magento? From my experience it is a lot more difficult than it should be due to how Magento pre-generates its thumbnails on page load. Tools like s3fs weren't really an option as they slow down the page load too much. – Andrew Kett Feb 12 '15 at 7:36
  • making s3 work as a storage for Magento is a pain. Quite complicated, there is quite an overhead if you want to send images to s3 while the page is rendered (when resize happens and so on)... s3fs is really slow... – FlorinelChis Feb 12 '15 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.