as @Suresh Chikani mentioned in the comment, breadcrumbs are displayed OOTB in those places(product page & category), if it's not then somebody removed it.
It could be removed in few ways
1) somebody removed the parent block in which the block containing your functionality was, in this case just "move" the block containing your desired functionality to a different existing block/container
2) somebody removed the block, you need to find the code and change it or add a new block as you done in your answer.
3) Somebody could rewrite the block class (Magento\Theme\Block\Html\Breadcrumbs) itself and override the toHtml function or any other that is responsible for displaying content.
Look in
di.xml
for
Magento\Theme\Block\Html\Breadcrumbs
if nothing is found then no one rewritten it.
4) Somebody could rewritten
vendor/magento/module-theme/view/frontend/layout/default.xml
Where the setup of breadcrumbs originally takes place.
https://devdocs.magento.com/guides/v2.3/frontend-dev-guide/layouts/layout-override.html
And going back to your question, if you want only add it only category & product page then you should add it to
catalog_product_view.xml
catalog_category_view.xml
not to default.xml