as @Suresh Chikani mentioned in the comment, breadcrumbs are displayed OOTB in those places(product page & category), if it's not then somebody removed it.
It could be removed in few ways
1) somebody removed the parent block in which the block containing your functionality was, in this case just "move" the block containing your desired functionality to a different existing block/container
2) somebody removed the block, you need to find the code and change it or add a new block as you done in your answer.
3) Somebody could rewrite the block class (Magento\Theme\Block\Html\Breadcrumbs) itself and override the toHtml function or any other that is responsible for displaying content.
if nothing is found then no one rewritten it.
4) Somebody could rewritten
Where the setup of breadcrumbs originally takes place.
And going back to your question, if you want only add it only category & product page then you should add it to
not to default.xml