Magento has a notoriously slow indexing engine that recently received a much-needed refresh in the Enterprise Edition 1.13 (though, at the time of this writing it is currently being overhauled for some issues with SEO).
In this answer, I have provided some direction on the types of issues you typically encounter when managing a site of this magnitude:
Is Magento the right platform for 1M products?
While your store may have 1M unique products, based on the number of pricing rules, stores, store views, tiered prices, etc., your indexes may have > 1M index entries. I have seen index processes run for 10+ hours. At this size, the flat catalog is not helpful, and in some documented cases, it actually negatively impacts performance.
At this scale, one would hope that you have the financial backing or annual volume to consider the Enterprise solution, as it addresses these indexing issues specifically. Otherwise, while it is possible to run a Community Edition store with a catalog of this size, all of your hardware needs scale horizontally -- meaning, you need larger and larger hardware to accomodate.
At this time, there are no fixes planned for indexing improvement in Community Edition.
There are some community modules which tout that they improve index speed - I have no personal experience with any of them:
Ecomdev has a freely-available indexer specifically for UrlRewrites: