By default, you should see page 3 of your category and not page 1 of your category. Be honest, Magento has nothing to do with this behaviour, because the back-button of a particular browser has no special preference for a Magento site. :-)
It may sound confusing. To understand, you should know how back-button of a browser works. Every brwoser will keep a reference to the urls that you have visited previously. So the history Url stack will look like this in your above depicted case.
domain.com/some-product-in-category.html <<=== current_url
Now when you click on the back-button, according to the URL stacks, browser will redirect you to the url
domain.com/category?limit=24&p=3 for sure. If it is not happened like that, then that means
Or somehow,the previous link reference will be
domain.com/category?limit=24&p=1. The link that I have attached above gives you a clear such an example. Go through it.