the answer is: no you can not build any decent application and keep page load in multiuser environment faster than 2 seconds.
you would say you can, yes, but only for a limited time.
you have to understand, that even if you have perfect code, light theme, simple products - your system load will grow proportionally to:
1 - frontend requests
2 - database writes
cache is not build for speed, but to lower your servers load and cut
on infrastructure costs. avoiding cache is the last thing you would do
if your are for profit.
performance is not only one request. all these fancy pagespeed metrics only for fun.
hiding errors and poor code with cache is another problem.
this is why you have to run staging setups. test all your changes and updates with
also many magento users trying to enable all the possible caches, but usually this makes site speed even worse. it depends.