David,
As we all know, there are a lot of different variables that go into determining Magento's page speed. It's important to know the number of visitors per day (total), the number of concurrent visitors at any given time, the number of products/sku's, and the type of hardware you're on. Having more insight into your environment will help big time in determining what the issue is. It'd be appreciated if you could provide a bit more information.
Also, Varnish may not be the best solution for your specific site. You can't just simply turn on Varnish and expect it to work. Though, if you're familiar with how to properly set up Varnish, it may be in your best interest to check out (https://github.com/nexcess/magento-turpentine). From Turpentine's github page:
"Turpentine is a full page cache extension for Magento that works with Varnish, a very fast caching reverse-proxy. By default, Varnish doesn't cache requests with cookies and Magento sends the frontend cookie with every request causing a (near) zero hit-rate for Varnish's cache. Turpentine configures Varnish to work with Magento and modifies Magento's behaviour to significantly improve the cache hit rate.
Note that while this extension is now considered stable, it is strongly recommended that it be tested on a development/staging site before deploying on a production site due to the potential need to add custom ESI policies for blocks added by other extensions."
Everyone who I know has used this has been blown away by what it can do. It may be worth checking out if your goal is to further increase page load time while using Varnish. At this point, it's hard to say where the issue is coming from as we don't know the details.
As another resource, I'd recommend checking out Nexcess' White Paper as it provides information about the best and most popular Magento configurations. You may download the Nexcess White Paper at (http://www.nexcess.net/magento-best-practices-whitepaper). It's incredibly thorough and provides some really amazing insight into Magento optimization that is hard to find anywhere else.
Also, as a last thought. Are you using a CDN?