Addition to Erfan's answer:

If there had been no products assigned to category before, there would be no products at all listed under "Category Products" tab. 

`Reset the Filter` - clicking it is a must even when you see blank list (no products assigned). It will then show all products in whole shop.