Performance issue with 1k to 10k websites/ stores

Question

I am trying to find a way to improve Magento performance when the amount of websites/ stores exceeds 1k, and my goal is around 10k. Here are some questions; any tips/ helps are extremely welcome!

1. Adding new websites/ stores is slow;
   I comment out $this->cleanModelCache() in _afterSave() in Mage_Core_Model_Abstract, and the situation seems better but is getting slower with increasing number of websites/ stores. And I don't know what would this affect the whole system in the future.

2. Api calls become slow.
   One of the main processes is to place order; my customized model deals with it by processing some data, and essentially using sales/quote model and sales/service_quote models. The process begins with Oauth. Both Oauth and placing order take longer when the number of website/ store grows, and the memory consumption seems larger. Does this have something to do with Mage loading the config xml, and the fact that config data gets bigger with increasing number of websites?

3. Opening up n98-magerun dev:console is taking longer; don't know the cause of it.

4. Saving configuration from admin panel takes longer; don't know how to improve it.

Is it possible to reconstruct the way Magento generate and load config data to lower its memory consumption? Is this one of the factors that cause performance issue for my situation?

Current Magento instance: Version = Magento EE 1.14.2.4;
Config cache on; other cache off;
Using Mysql 5.6 and MongoDB( for catalog_category_entity, catalog_product_entity, core_website);
number of websites = number of stores = number of views = 1024;
number of product = 4501;

Thank you all in advance!

Answer 1

One of the reasons it gets slow is because the configuration for each store is duplicated from /config/websites and /config/global and the code for doing that is not efficient in the least. Any setting change may end up causing several 10's of minutes, if not hours, of reduced performance and throughput. Making it more efficient will basically mean that Ben Marks will come after you... and not in a good way.

IF you are going to go down this route the easiest way would be to have 10k Magento installations and have some kind of a broker that delegates requests to the appropriate website. Though it will, of course, depend on what your actual use case is.

[added]

Depending on the use case you might be able to use categories as pseudo stores. You could technically then use layout XML to change the theming per store. But then you would run into the limitation of checkout. All stores would need to share the checkout.

Either way, 10k Magento stores is do-able, in that it is not impossible. But it will be a difficult road whatever path you choose.

Answer 2

You can try hacking the core and enabling website cache splits. You can try hacking the database and storing configuration information in memory. You can try replacing the configuration cache with something smarter - say caching the information from the xml files[which are static and apply to all websites and stores] while retrieving the override data dynamically.

I'm a server guy, so I'd go with mucking with the database. Especially since it is trivial to do.

If you have control over your database server: Rename the core_config_data table to core_config_data_offline Create a new core_config_data table using the MEMORY storage engine http://dev.mysql.com/doc/refman/5.7/en/memory-storage-engine.html Copy all the data from core_config_data_offline to core_config_data

Setup a cron job to check to see if core_config_data exists, if it does copy all the data from there to core_config_data_offline. If it does not, create it and copy everything from core_config_data_offline to core_config_data

Turn off the config cache. With the config cache turned on, you only get a performance boost for the first time config data is read from the database - after that it is in the cache and you suffer. On the downside the xml files are no longer cached, so you traded the performance hit of unserializing huge configuration data for the performance hit of parsing a bunch of xml files.

You may also want to experiment with changing the Mage/Core/Model/Config.php file and enable individual website caches. By default each store specific configuration data is cached individually. All the website configuration data is cached in one object.

Note that this is just for the configuration overrides[the admin settings]. So if you do all your configuration changes at the store level your already set. If your using "inherit from website" and making most of your store specific configuration changes at the site level - then the cache contains every website. By splitting it you can break it out much better. protected $_cacheSections = array( 'admin' => 0, 'adminhtml' => 0, 'crontab' => 0, 'install' => 0, 'stores' => 1, 'websites' => 0 );

to

protected $_cacheSections = array(
    'admin'     => 0,
    'adminhtml' => 0,
    'crontab'   => 0,
    'install'   => 0,
    'stores'    => 1,
    'websites'  => 1
);