I can not open the admin panel and get a 404.
$this->_routerList does not contain the admin route.
Have a look at
\Magento\Framework\App\Http::launch. There is a line
$areaCode = $this->_areaList->getCodeByFrontName($this->_request->getFrontName());
which determines the area.
The admin route is only loaded, if the area is detected as adminhtml.
Digging down leads you to
There is a check whether the current URL (unsecure URL or custom_admin_url) matches the current host.
In cases where this does not match (for example if behind a proxy), this might fail.