I had this problem last month.
It was only our office network. Which triggered me thinking, ok, this might be our ip. Something is not getting pass through to our ip.
This link will explain little bit of it.
Then I contacted our hosting provider and also ISP with my trace rout.
While doing tracerout I saw that there was blockade for our ip while routing though different pipes. It was showing timeout.
To check trace route
Open your command and type following
tracert [your_ip_of_site or just domain.com]
This will give you an idea what's going on.
How to fix?
This part is very tricky, since it's not your magento having the problem, it's the network issue persist whether in your ISP or your hosting provider but they both deny of such.
For quick fix, you can ask your ISP to change your ip to something else. However you might be thinking, oh! wait what if my customer also having same issue as mine?
You simply need to contact your hosting provider and ISP to fix the issue.
Even my problem was resolved, no one said who's fault was it. I personally doubt it was my web hosting provider who's network routing was having issue.
You can also ask your web hosting provider to do
reverse trace route to your
local ip. This will help you to convince the problem persist in their end.