Assumptions:
1. Every merchant has each store/store view in magento
2. Every merchant product type they deal in defined for ex. Categories/nature of their products.
3. Attributes pre created in the system according to the nature of business.
4. Attribute set pre created in system.
5. There is abundance of resource in hardware ( CPU, RAM and Disk space)
6. The market place extension already installed like apptha unirgy etc.
I would like to work on most of indexing part r. As this is the one of beast needs to get out of this system.
Importing needs to be done through some fast tool to import which completely sits out of the magento and it bootloader processes.
I recommend MAGMI as I this is tried and tested tool and I have used with majority of my clients lile C&A marketing USA, Tolexo(India Mart ) India, Bigadda former reliance ecommerce site, shopelect.com ( Marketplace for Electical Goods B2B) India, Xwalker Inc. USA etc.
We are looking at the problem starting from core which is taking maximum turn around time in order to complete 1 merchant process.
Now the problem is:
For each product these many process must run.
Product A
Product Attributes
Product Prices
Catalog URL Rewrites
Product Flat Data
Category Flat Data
Category Products
Search Index
Stock Status
Tag Aggregation Data
I am trying here to offload
After that, clear all useless custom rewrites from the core_url_rewrite table.
Magento also creates rewrites for disabled and invisible products for sure we need to cut lot’s of corner for magento indexing process which is definitely an architectural issue of magento and where Java seems to be a best for these king of marketplace stores. In order to make it light weight we need reconsider the way re indexing works.
Breaking down the problem:
We are looking at majorly two main indexes everytime which needs to be figured out along the process:
Product, URL Rewrite, and Inventory Indexing are the major issues when running a catalog everytime.
Primarily the problem indexers are catalog_url which takes over 10 mins for 500 products. I try not to do it because it drives the cpu up to 400% and the site becomes unresponsive.
The catalog_product_price takes 5 minutes with 500 products, catalog stock is about the same. As you can appreciate it takes close to 12 hours to complete and then 12 hours later they are invalidated and round we go. I can't be the only store owner with this problem. –
We will work around this issue with userless core_url rewrites again.
Useless rewrites for products in category paths
For example, when a product with url 'product-123' is assigned to category 'cat-1/cat-2/cat-3', the following rewrites are created for each store view (taking the default .phtml url suffix):
• /product-123.html
• /cat-1/cat-2/cat-3/product-123.html
• /cat-1/cat-2/product-123.html, when category cat-2 has been configured as a layer navigation category.
• /cat-1/product-123.html, when category cat-1 has been configured as a layer navigation category.
After applying this rewrite and setting catalog/seo/product_use_categories to false, the number of system rewrites should drop significantly.
Product URL key clashes
When two products have the same url key, each reindex round adds a new custom rewrite.
For example, for two products with id's 123 and 456 with the same url key foo:
• Reindex round 1:
• System rewrite /foo-123.phtml > /catalog/product/view/id/123
• System rewrite /foo.phtml > /catalog/product/view/id/456
• Reindex round 2:
• Custom rewrite /foo-123.phtml > foo-124.phtml
• System rewrite /foo-124.phtml > /catalog/product/view/id/123
• System rewrite /foo.phtml > /catalog/product/view/id/456
• Reindex round 3:
• Custom rewrite /foo-123.phtml > foo-125.phtml
• Custom rewrite /foo-124.phtml > foo-125.phtml
• System rewrite /foo-125.phtml > /catalog/product/view/id/123
• System rewrite /foo.phtml > /catalog/product/view/id/456
To resolve this, make sure all product url keys are unique, for instance by adding the SKU to the url key.
After that, clear all useless custom rewrites from the core_url_rewrite table.
We will also look at the introduction of incremental indexing reduces the need to perform a full re-index and most indexing operations are now automated - saving you and your staff time and energy to focus on revenue-generating activities.
Re-indexing on these process with fast asynchronous cache.
Product Attributes
Product Prices
Product Flat Data
Category Flat Data
I would like to store and directly import the data from the sources directly into flat tables.
I make them them get created on the free load of very first indexing of the process and use the variable structure of different attribute and attribute sets.
So we have catalog and category flat tables pre-exists in the system.
The approach will be simpler.. We will going to indexing on the fly at the time of importing the products;
We will use magmi for reindxing asynchronously. Never wait merchant to hold for this long.
http://wiki.magmi.org/index.php?title=Magmi_Magento_Reindexer
So now using the structure for abundance of products.
Product import form MAGMI in to the entity tables and flat tables while RE indexing it on the fly.
Reducing search indexing time.
Search Index
I would like to have the indexers placed inside the full text search tool. Like sphix or elastic search.
Assuming there would be a state table which is directly storing the merchant sync updates related data. Which will send the notification to merchants about there data syn completion status.