We found the solution here: https://github.com/magento/magento2/issues/26647
We've overridden the vendor/magento/module-customer/view/frontend/layout/default.xml in our custom theme.
So, changing its structure during the upgrade caused the issue. Ours was missing the "sectionNamesProvider" argument tag for customer.section.config block.
<block name="customer.section.config" class="Magento\Customer\Block\SectionConfig"
<argument name="sectionNamesProvider" xsi:type="object">Magento\Customer\Block\SectionNamesProvider</argument>
We couldn't predict this. As Magento guys said, this is an issue that needs to be solved.
"Stuff like this happens very frequently in "patch" releases. We should somehow try to avoid this."