You need to profile and find where the actual bottlenecks are. If your TTFB (time to first byte) is 32 seconds (wowzers) it means that Magento is taking a long time to render the page to be delivered to the browser.
I wholly agree with @MageWH that the issue is likely that you're loading product swatches. However, I see that these are all media cache images and that leads me to believe that you're probably calling Mage::getModel('catalog/product') in a loop. This is likely the source of your problem.
Vinai Kopp has a writeup about why you wouldn't want to do this and potential ways to avoid it:
Magento's Expert Consulting Group (ECG) also has a writeup about best practices to apply (and antipatterns to avoid) when building your store:
Liberally employing some block caches and a full page cache will only mask your real problem here. I think you should use some tools from ECG to find other potential problems:
Best of luck!