Our admin side of our site has been running extremely slow when we're entering new products and saving them. We typically have 8 people entering products at once and we're being told that because of all the people in there adding stuff at once that is causing our store to run slowly.

This is unacceptable to me! We're still quite small. What do companies with 50 data entry people do???

We're kicking around the idea of getting a server here at our store to run a bare bones Magento installation on with very few extensions (our live store has quite a few) and then at the end of the day exporting the products out of that server into our live site. Uploading products via csv import has been going very fast for us but when we load them on the admin side that's when it gets slow.

Do any of you upload your new items like this???

(Our store has about 300,000 products)

  • It is perfectly fine if you'd do it that way, there is absolutely no reason to not use this. Especially if you are experiencing a slow website you should really be doing what you said in your post. – Jordy Mar 17 '15 at 19:31
  • We enter products straight into Magento through the admin. We usually have about 10 admins in at a time and a couple add new products. We did, however, spec our servers with this in mind. It may very well have something to do with your hardware configuration. – ddavidn Mar 17 '15 at 20:32
  • Our server is pretty beefy and when we watch the load on it we never see it go over 20%. My gut feeling is that there is some sort of server setting that isn't configured properly but I have no idea what it could be. We do have all of our indexes set to "Manual Index" as we've heard from quite a few people the "Update on Save" will slow us down quite a bit. Any ideas on where I could find out if it's a server setting?? – Dave Mar 17 '15 at 21:41

Editing directly in Magento with 300000 products will be slow on the Community Edition. This because with every save the indexer has to spin up and has to update the indexes. Unfortunately the indexers do not scale well beyond 100k in the CE version.

If you are running on Magento Enterprise Edition you have better indexers that will give you much better performance while saving products. In one of the most recent versions there is an option to always run the indexers when scheduled (instead on save) which should save a lot of performance.

If that doesn't suit your needs, you might consider using a seperate PIM to add all your products. One of the more recently announced products is http://www.akeneo.com/, maybe that could suit your needs.

If you want to go low tech, you might consider a simple Google Docs document to manage your products and import it with https://github.com/avstudnitz/AvS_FastSimpleImport and optionally using an importing framework like https://github.com/ho-nl/Ho_Import or https://github.com/AOEpeople/Aoe_Import.

| improve this answer | |
  • aoe displays the following information in their site: " Aoe_Import is not doing any save operations.save operations are performed by processors?" do you know what it means?. additionally, the same happens with HO_Import? thank you for your advice. – s_h Oct 31 '15 at 19:08

For a catalog that size, you need to use some kind of mass-importer (Magmi or some Magento extensions). Doing it by hand seems impossible to me if you wanted them all imported relatively soon.

I don't quite understand what you mean " exporting the products out of that server into our live site." That doesn't seem very sound.


OK, I see you're using an extension that uses Magento Dataflow, which is very slow. It's important to note that running multiple instances of the import will not hasten the import. If anything, it will make your site (front and back) run very slowly. Seeing that a product is taking 2-3 minutes to save, you should run one import at a time.

This is just a limitation of Dataflow. It's clunky and slow out of box. The extensions that use it more sophisticated but they can't get around the speed issue. If you don't have a custom integration in place, which some companies with ERP systems do, I suggest giving Magmi a try. It is several times faster than anything you do with Dataflow.

| improve this answer | |
  • We're not creating the new configurable product in Excel. We would be creating the new products on our server and then exporting them using an extension called"Export/Import bulk products MLX Store". We would use that same extension to upload them into the new site. – Dave Mar 17 '15 at 19:45
  • Didn't realize there was an extension that you were using. That's fine it's working. – musicliftsme Mar 17 '15 at 20:08
  • Well, I don't really know if I'd say it's working. We think we do it in theory but it seems like a overly complicated way to enter new products but we're pretty much out of ideas. Our data entry people are getting frustrated because they're waiting 2-3minutes for product info to save. – Dave Mar 17 '15 at 20:23
  • Updated my answer. – musicliftsme Mar 17 '15 at 20:30
  • We don't do multiple csv imports at one time. The 2-3 minutes to save is when we're manually entering a configurable product using Magento's Add a product feature. – Dave Mar 17 '15 at 21:57

Been there before with that data volume on the business side. So the first is your hosting and technical implementation for performance due to the indexing on save or reindex if direct database import. The second is the management and approval flow for product management.

The performance you need high quality architecture and preferably separate admin servers to offload from the web servers. You can use EE which solves quite a lot of this but you can also do the same with CE, you just really need to know what you are doing.

On the product manager point there are a few options, the simplest is a direct Excel to Magento import where data entry control a specific set of products. The second is a passthrough that take files, runs them through mapping and loads to Magento. The third and most complex is an external PIM, however that becomes more complex as you need to config the PIM for the products, batch the updates with only changes and then have sync connectors to Magento. We use all three types depending on the scenario having used the first to 70,000-100,000 products (you need 64bit Excel) and the latter two to 300,000-500,000.

You obviously have Magmi but that's a technical tool and you still have the reindex problem as it's better at data load which is more technical than data maintenance which is business focused. The main thing is that your situation is a business process problem, the technical can always be mitigated but it sounds like you need an architect to understand your process and tell you what to do. The ones we use via via take around 2-5days for an overview and core issue identification but you really don't want to know their chargeouts. It all comes down to how the pieces are added together for you budget, technical installation and business processes.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.