Most of my categories are working fine and only loading 12 products per page. Two of my categories are loading in every product assigned to those categories causing a very high page load time.
The categories have the same relevant settings as other categories that are working correctly (Anchor -> Yes, Display: Products Only).
Products per page on grid default value is set to 12 and products per page on grid allowed values is 6, 12, 18.
List mode is set to Grid Only.
Flat catalog category and flat catalog product are set to no.
Update: I've traced through the codebase and found that this category is deciding to be set to isLoaded before calling _loadEntities() so it never calls the $this->getSelect()->limitPage($this->curPage, $this->_pageSize); however other categories get into this function because isLoaded is false. You can see this functionality in Magento\Eav\Model\Entity\Collection\AbstractCollection