I haven't done something similar but as I checked the code right now, you can create an after Plugin for this method \Magento\Shipping\Model\CarrierFactory::createIfActive or also this method \Magento\Shipping\Model\Shipping::isShippingCarrierAvailable, there you can add your validation and return true or false for the shipping method you allow or not.
Although I haven't tried it yet, maybe it will guide you to the right direction.