Skip to main content
Magento

Return to Answer

added external reference as further reading / evidence.
Source Link
Alistair
Alistair
  • 401
  • 4
  • 9

After some part time messing around with a copy of the site on a dev server, i finally got to the bottom of the lag issue. What I found totally infuriating is the smaller and far less powerful dev server would return pages faster! Which made no sense at all, as it was a mirror image of the site and db with the same OS and even the same versions of nginx php etc..

But then it hit me, what i didn't rsync over was the cache and sessions folders.. As I had an SSD webserver, i had allowed magento to use "file storage" for the cache and sessions. This had managed to accumulate hundreds of thousands of tiny files in mageDIR/var/session and /cache folders which i thought was no problem.. turns out it's a very big problem if left unchecked for a couple weeks.

Conclusion (for me, on my system) even with dedicated server with SSDs in raid, file storage is never the way to go. It would seem the speed of IO on the storage in this case wasn't an issue, i guess the way in which it handles huge amounts of tiny files is?

In any case Switching to a localhost redis-server for cache and sessions has reduced all lag and has been running very well for a month on the live machine. Page "wait" as reported by the speed test sites was in the 240ms area

Further savings were made when I switched ngx_pagespeed to memcached storage instead of file storage.

Now initial wait is down to 140ms odd plus 20ms odd per browser connection, but more importantly to me google no longer suggests I need to "reduce your server response time" which is great news as that warning from them is what sent me down this rabbit hole.

-more reading:

...the drawback of putting potentially thousands of files in a single directory. If you've been around servers long enough, you know that directories become slower and slower as more files are added...

Redis: Better Session Storage for Magento October 28, 2015 by Scott Hebert

After some part time messing around with a copy of the site on a dev server, i finally got to the bottom of the lag issue. What I found totally infuriating is the smaller and far less powerful dev server would return pages faster! Which made no sense at all, as it was a mirror image of the site and db with the same OS and even the same versions of nginx php etc..

But then it hit me, what i didn't rsync over was the cache and sessions folders.. As I had an SSD webserver, i had allowed magento to use "file storage" for the cache and sessions. This had managed to accumulate hundreds of thousands of tiny files in mageDIR/var/session and /cache folders which i thought was no problem.. turns out it's a very big problem if left unchecked for a couple weeks.

Conclusion (for me, on my system) even with dedicated server with SSDs in raid, file storage is never the way to go. It would seem the speed of IO on the storage in this case wasn't an issue, i guess the way in which it handles huge amounts of tiny files is?

In any case Switching to a localhost redis-server for cache and sessions has reduced all lag and has been running very well for a month on the live machine. Page "wait" as reported by the speed test sites was in the 240ms area

Further savings were made when I switched ngx_pagespeed to memcached storage instead of file storage.

Now initial wait is down to 140ms odd plus 20ms odd per browser connection, but more importantly to me google no longer suggests I need to "reduce your server response time" which is great news as that warning from them is what sent me down this rabbit hole.

After some part time messing around with a copy of the site on a dev server, i finally got to the bottom of the lag issue. What I found totally infuriating is the smaller and far less powerful dev server would return pages faster! Which made no sense at all, as it was a mirror image of the site and db with the same OS and even the same versions of nginx php etc..

But then it hit me, what i didn't rsync over was the cache and sessions folders.. As I had an SSD webserver, i had allowed magento to use "file storage" for the cache and sessions. This had managed to accumulate hundreds of thousands of tiny files in mageDIR/var/session and /cache folders which i thought was no problem.. turns out it's a very big problem if left unchecked for a couple weeks.

Conclusion (for me, on my system) even with dedicated server with SSDs in raid, file storage is never the way to go. It would seem the speed of IO on the storage in this case wasn't an issue, i guess the way in which it handles huge amounts of tiny files is?

In any case Switching to a localhost redis-server for cache and sessions has reduced all lag and has been running very well for a month on the live machine. Page "wait" as reported by the speed test sites was in the 240ms area

Further savings were made when I switched ngx_pagespeed to memcached storage instead of file storage.

Now initial wait is down to 140ms odd plus 20ms odd per browser connection, but more importantly to me google no longer suggests I need to "reduce your server response time" which is great news as that warning from them is what sent me down this rabbit hole.

-more reading:

...the drawback of putting potentially thousands of files in a single directory. If you've been around servers long enough, you know that directories become slower and slower as more files are added...

Redis: Better Session Storage for Magento October 28, 2015 by Scott Hebert

Source Link
Alistair
Alistair
  • 401
  • 4
  • 9

After some part time messing around with a copy of the site on a dev server, i finally got to the bottom of the lag issue. What I found totally infuriating is the smaller and far less powerful dev server would return pages faster! Which made no sense at all, as it was a mirror image of the site and db with the same OS and even the same versions of nginx php etc..

But then it hit me, what i didn't rsync over was the cache and sessions folders.. As I had an SSD webserver, i had allowed magento to use "file storage" for the cache and sessions. This had managed to accumulate hundreds of thousands of tiny files in mageDIR/var/session and /cache folders which i thought was no problem.. turns out it's a very big problem if left unchecked for a couple weeks.

Conclusion (for me, on my system) even with dedicated server with SSDs in raid, file storage is never the way to go. It would seem the speed of IO on the storage in this case wasn't an issue, i guess the way in which it handles huge amounts of tiny files is?

In any case Switching to a localhost redis-server for cache and sessions has reduced all lag and has been running very well for a month on the live machine. Page "wait" as reported by the speed test sites was in the 240ms area

Further savings were made when I switched ngx_pagespeed to memcached storage instead of file storage.

Now initial wait is down to 140ms odd plus 20ms odd per browser connection, but more importantly to me google no longer suggests I need to "reduce your server response time" which is great news as that warning from them is what sent me down this rabbit hole.