I have magento installed on host gator everything was working fine.But suddenly they said my magento CPU usage is very high so they are suspending my account and recommended to me either buy dedicated server or resolve the magento cpu usage issue. Can any body tell me how to find out which part of my magento web site is taking so much cpu usage.
I think that it's the Magento part :)
Sincerely, though, Magento is not meant to be hosted on a shared-resource server and there is no magic bullet that will fix these 'issues'. However, I have had tremendous luck hosting very small shops at Amazon Web Services' EC2. At the time of this writing, their Spot instances run at about $115/year. That's a virtualized, dedicated environment.
Perhaps you're not devops and you don't want to manage the infrastructure/environment at all. I would check this list out of hosting partners with Magento and contact one of them: