If you add store and website views into the mix you'll see an even larger number of rows.
Unfortunately this is a by product of Magento's flexible pricing schema.
I'd recommend upgrading to 1.8.x CE or 1.13.x EE if you are not on the latest release, as some fixes have been made around this specific issue:
Catalog price rules are applied properly to customer groups.
What those specifics are I'm not entirely sure.
If upgrading isn't an option:
Here are a few things to help out with performance of Magento and Indexing via Asynchronous Indexing, Disabling unused Core Modules and Logging as well as a Class Path Cache. All of which should improve indexing performance with minimal cost/time, and without upgrading Magento.
Disable logging and any unused core modules you don't use:
Some modules to consider when needing more insight into the indexing/caching mechanisms:
Improve Index management in Magento. Enables Magento to process indexes asynchronously. I would highly recommend giving this a try as it will chunk up the data needing to be indexed and spawned across separate PHP processes. By Default Magento only uses one CPU Core when indexing. On a large multi-core system this helps tremendously with indexing performance.
Another free option available on GitHub that looks to be more up to date:
There is a paid for solution as well, as Aoe_Index hasn't been updated in over 4 years.
With as many files/folders that Magento has, this helps the Autoloader find the exact files its looking for instead of fseek's on your servers file system to locate the files. Not a direct solution to your problem but will greatly speed up Magento's performance.