What you are describing isn't default behaviour.
Typically as FPC is primed, when not logged in - or having done anything to make the session specific to you, the page load timeline tends to be:
--1.2s--------0.8s------------------0.6s-----------------------0.1s---------------
Uncached Mage default cache Partially cached in FPC Fully cached in FPC
What should happen is that a single page load primes the Magento cache and to a lesser degree, the Magento FPC cache. The subsequent load for the same content fully primes the FPC cache resulting in a 3rd page load for the same content to take around 0.1s.
But when you do something to make your session unique, eg, add-to-cart - then FPC exhibits slightly different behaviour and the render times do increase slightly.
--1.4s--------0.8s------------------0.6s---------------
Uncached Mage default cache Fully cached in FPC
It sounds like you've broken the FPC logic by make your amendments (hole-punches) by essentially causing FPC to behave like in my second example.
Ie. It doesn't serve any content from a global cache, but rather only content specific to your session
My suggestion would be to either install a clean EE store - and watch the behaviour there - or move the local/community directories and custom theme package directories (to force enterprise/default to load).