There is no silver bullet solution. There are some workarounds, I suggest
Sonassi_Fastsearchindex - but that's specifically for catalog search.
Perhaps disabling index updates on save - scheduling to run overnight - will provide some relief? Combined with adding more caching - memcached, Redis, APC - and a full-page-cache like Varnish (if you're running CE) may get you started. If you plan on using Varnish, look at
Nexcess_Turpentine on github for a quick-start.
The indexing issues - specifically catalog_url_rewrites - are well known and documented in the community. Magento has handled these in the Enterprise version because these are the customers who are most adversely affected. Many EE customers have 10k+ products and multiple store views, websites, etc.
However, if you have a large catalog and a large number of attributes you may find yourself in the position that indexing will take a long period of time - specifically catalog_url_rewrite, product_flat -- in that case my suggestion is not to fix the index run time length but rather to offload some processing to allow the box to spend CPU cycles indexing rather than serving content.
The questions to ask yourself:
- Am I losing business due to indexing issues?
- Am I losing productivity due to indexing issues?
- Am I at risk of losing conversions or is my conversion rate suffering?
- Are my customers at risk of purchasing items out of stock that are a direct result of indexes being out of sync (inventory, etc.)
- Are my catalog pricing rules part of my core business and
- Is my on-site-search conversion rate higher than norm (8-10%), thus benefitting from better indexing?
There is no silver bullet solution to this particular issue - as a solution provider you should help your customer make the decision that will best improve sales and the business while keeping overhead costs low.
Offload catalog search and layered nav to Solr.
Scale horizontally. Add more Apache/nginx servers. More servers = more concurrent throughput. This isn't 1:1. Nexcess has a great whitepaper on performance and Apache configuration here: http://www.nexcess.net/magento-best-practices-whitepaper
And, if you opt to go with Varnish - remember: