Force quote_address instead of customer address when browser is refreshed Magento 2

Question

I am having hard time understanding how below code is getting shipping address data for display on checkout/#payment page. It displays correct address when we go from shipping to payment page but switches to the 1st address from user's address book when hit refresh on the payment page. I understand that the user addresses are coming from DefaultConfigProvider(vendor/magento/module-checkout/Model/DefaultConfigProvider.php) which is ultimately getting it from customer repository(customer_address_entity) table. My requirement is to display quote_address not the customer address even when browser has been refreshed knowingly or unknowingly.

<!-- ko foreach: getRegion('ship-to') -->
    <!-- ko template: getTemplate() -->
    <!--/ko-->
<!--/ko-->

Any help is greatly appreciated since I spent bunch of hours and still unable to find the root cause.

Thanks