21

I was asked how to save all the magento media assets in Amazon S3. I thought about using a PHP Stream Wrapper like this one: https://github.com/punkave/aS3StreamWrapper and just set the media folder to s3:\mybucket\mediaFolder

Does anyone has experience with this?

Or better use something like this: https://github.com/sstoiana/magento-s3 But this extension doesn't look good. Any other recommendations?

  • 2
    Just as an aside. What is your motivation exactly? Unless you are catering to a global audience, or pushing more traffic than your server's interface (>100Mbit/1Gbit), or have a server provider with poor transit (which is an issue in itself) - CDN's usually are slower than serving the content direct from the same machine. Congested networks, distant break out points and poor routes usually lead to slower content delivery than simply hosting it yourself. – Ben Lessani - Sonassi Feb 1 '13 at 20:53
  • 1
    The motivation is: Someone asked me for my experience and opinion and I thought about it and prefer the stream wrapper solution, but we have this cool site and maybe someone has his two cents for me :-) – Fabian Blechschmidt Feb 1 '13 at 22:12
  • @sonassi your experience with CDNs seems to be somewhat dated. All the mature CDN solutions distribute content geographically at a much higher scale than you will be able to afford for your web servers and will serve content up from the closest geographical location to the requesting client. It's also important to optimize # of HTTP requests on your site by using an asset pipeline for js/css concatenation and minification, but that's much more complicated especially for Magento sites. – Ralph Tice May 24 '13 at 17:24
  • @RalphTice Not that outdated considering we run our own BGP anycast network across 3 continents. Most 'affordable' CDNs have single PoPs per country (some just per continent), that isn't necessarily geographically any closer than your web server itself. Eg. GB's nearest PoP for Amazon is Ireland - which has far, far greater latency than a England based machine. So I reiterate, that unless you are targeting a global audience, its unlikely your domestic customers would see any benefit at all from a CDN. In some cases it is slower, not to mention more expensive and complicated. – Ben Lessani - Sonassi May 24 '13 at 22:04
  • guys, i have used thaiphan/magento-s3 extension for s3 bucket, its working with upload product images on s3 and connect with product images , but it will also generate product images in my system server also. i just need to use s3 bucket for images so i also tried to delete my system's pub/media/catalog but after load site page product images is recreated in my system server also. is there need to stop this image created in local server ? – Camit1dk Apr 19 '18 at 12:09
7

it seems these guys find a solution: ( http://thinkglobal.co/resources/moving-the-magento-media-directory-to-s3/ )

There are 2 possible approaches, one being to have Magento push all media assets to S3 as they are created and the other being to simply mount the media directory to an S3 bucket and let Magento think that it’s writing to the local disk. We chose the latter.

We ended up using a tool called s3fs, which uses a broader technology called fuse. With some fairly straightforward configuration (described below), you can get s3fs up and running. Be careful not to hardcode S3 bucket names or API access credentials if you are using AWS or other PAAS provider.

Make sure you replace the following variables with values for your specific configuration:

  • {{S3_BUCKET}} (name of your S3 bucket)
  • {{API_PUBLIC_ACCESS_KEY}}(provided by AWS)
  • {{API_SECRET_ACCESS_KEY}} (provided by AWS)
  • {{UID}}(user id for nginx/apache user)
  • {{GID}} (group id for nginx/apache user)
  • {{MOUNTED_DIRECTORY}} (path to your magento media directory)

Here is the setup:

yum install -y gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap

cd /usr/local/src
wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.9.3/fuse-2.9.3.tar.gz
tar -xzf fuse-2.9.3.tar.gz
rm -f fuse-2.9.3.tar.gz
mv fuse-2.9.3 fuse
cd fuse/
./configure –prefix=/usr
make
make install
export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/
ldconfig
modprobe fuse
pkg-config –modversion fuse

wget http://s3fs.googlecode.com/files/s3fs-1.74.tar.gz
tar -xzvf s3fs-1.74.tar.gz
rm -f s3fs-1.74.tar.gz
mv s3fs-1.74 s3fs
cd s3fs
./configure –prefix=/usr
make
make install

echo “{{S3_BUCKET}}:{{API_PUBLIC_ACCESS_KEY}}:{{API_SECRET_ACCESS_KEY}}” > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

echo “user_allow_other” > /etc/fuse.conf

echo “s3fs#{{S3_BUCKET}} {{MOUNTED_DIRECTORY}} fuse use_cache=/tmp,allow_other,uid={{UID}},gid={{GID}} 0 0″ >> /etc/fstab

s3fs -o allow_other -o uid={{UID}} -o gid={{GID}} -ouse_cache=/tmp {{S3_BUCKET}} {{MOUNTED_DIRECTORY}}
| improve this answer | |
  • 2
    This also seems to be the recommended way by AWS s3.amazonaws.com/quickstart-reference/magento/latest/doc/… page 11 (at least until AWS EFS gets available) – pHiL Jun 2 '16 at 17:49
  • 1
    While this solution is good, I'm finding that it runs very slow. How did you deal with this? It seems like when Magento generates URL's it's checking the filesystem to make sure the file exists. – greatwitenorth Jan 10 '19 at 21:41
2

OnePica's CDN extension is very handy out of the box for this functionality and supports more than just S3. Quote: "Currently, the extension integrates with Amazon S3, Coral CDN, Mosso/Rackspace Cloud Files, and any CDN, server or service that supports FTP, FTPS, or SFTP."

| improve this answer | |
  • +1 - I can confirm that OnePica works quite nicely and supports both S3 and Rackspace Cloud. – philwinkle Feb 1 '13 at 20:59
  • 4
    OnePica is only CDN manager, means the files are still local – Fabian Blechschmidt Feb 4 '13 at 20:01
  • @FabianBlechschmidt You'll have to be more specific with your comment. OnePica's extension most certainly does not leave the files locally when properly installed/configured and I'm using it in production myself w/ Amazon S3. – Ralph Tice Feb 6 '13 at 15:51
  • Maybe I missed something. You mean the files are uploaded to S3 and no more in media/catalog/... ? – Fabian Blechschmidt Feb 6 '13 at 19:15
  • 1
    FYI, this should also work with DreamObjects drop-in S3 replacement solution. It has worked great with every other S3 bucket so far. – ylluminate Apr 24 '13 at 16:53
1

OnePica's CDN extension is not the solution it only works for product images not to mention you need to set the correct TTL for its own cache life time which will timeout much earlier than default Magento. All you cms wysiwyg image will be in media folder but not using S3, you will end up uploading all your media image to S3.

Use cloudfront will be good but you will still need a NFS to share your media folder across all your frontend nodes and backend nodes. Or you need to build your own interface to save file on S3, and you also need to handle errors (think about someone upload extreme large file whether your code can handle that before S3 API timeout)

Other way just save media storage in your extra RDS (database) and individual resize cache image will still save in your app server local media folder, but it is ok as cloudfront will cache them already.

| improve this answer | |
1

I prefer having a separate "admin" node(admin.example.com) and keeping media folders there. This way you would save yourself with NFS cost as well as issue with syncing etc. any CDN (I use cloudfront) could then be connected with admin node media folders, and be configured for all stores.

Other benefits of using admin node is, you could move all crons(reindexing and other crons) , scheduled tasks(product imports, inventory & order sync etc) at admin node and save frontend nodes cpu processing.

Edit: As last answer didnt actually addressed OP's question. Magento Media Assets in Amazon S3? Well I believe Magento already has got so much processing overhead, that I think it'd be good idea to keep it from s3 syncing/streaming, and use local media folder instead.

| improve this answer | |
  • Thanks! Actually I checked out a cloud host where no upload was possible. Dropped it later :-) – Fabian Blechschmidt Oct 28 '14 at 8:26
1

Well, now that a couple years have gone by I think it's important to note that there are multiple plugins available that will move and serve assets from S3 / CDN.

Google searches turn up plenty of options.

Here's one on GitHub:

https://github.com/magefm/cdn

Here's one on Magento's Connect marketplace:

http://www.magentocommerce.com/magento-connect/aws-s3-cdn-for-product-images.html

I have not tried these particular extensions yet but will be for an upcoming project. There are others out there, too.

I'd have to disagree with @Sonassi, using separate storage to allow the cheapest server configuration still with good performance is really considered the best practice today. For example, simply use S3 to move all static content off your web server helps the server use far less bandwidth and allow more concurrent users and less overall server power needed. And, if you're in a single datacenter like most people these days, and say it's in Dallas, then there's no doubt that people in Western Europe will be served much faster with a Global CDN like Cloudfront. So, using storage separate from your web server to lower bandwidth usage allows you to use only 1 server when you would need more sooner without this approach, and using CDN will greatly increase global load times.

| improve this answer | |
0

Found another extension built by arkadedigital https://github.com/arkadedigital/magento-s3

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.