There is several elements to consider here.
There is no "headless mode" switch in magento. If you do not want to use regular view layer you need to set it up in the vhost configuration. You will still need it though for access to admin panel, unless you are also working on admin UI in Angular.
I am not sure why do you mean be "remove" themes and controller. If you install magento via composer you will have it either way and you would have to remove them during each deploy. If you install magento to app/code then you will have to check everything during magento upgrade. There is also dependency-injection compilation that need to be done and some of those elements are needed there and may (not checked) cause an error.
Since they are not used then it is no harm to leave the file as they are, provided you will configure your vhost in a way not to access them. Basically you want to have
/admin (or your actual admin route) and
/rest routes proxying to magento2 while any other will trigger your Angular app
Everything depends on your application. You can have magento being setup on a subdomain (ie.
backend.yourdomain.com) or on the same domain but run via host configuration.
One important thing to consider is that if you make your app working only in browser and a request to server will only return app shell with general configuration then you will not be properly crawled by search engines. It is generally good thing to make it work in a way that a server request will produce the html which would be produced by the app in the browser. The current solutions that I know are doing that (VueStorefron and Deity, and probably FrontCommerce as well).
Another thing is Magento2 rest API is not generally ready for such use. You can easily do cart and checkout on that as endpoint require no authorization or customer token which has short live span (default 4h) and is created on login. However other endpoints (getting product list, product data, reviews, category data, cms blocks and pages) require usually authorization with admin privileges. If you want to do request directly from the browser you will need to provided that tokens (either admin token taken from
integration/admin/token endpoint or via oAuth). This will somehow be in the browser and therefore exposed to everyone. It will cause security risk as, as far as I know, Magento does not have read and write privileges for specific resources. Most of them work in a all or nothing. So if you expose those to everyone you will not only let them fetch product data but also modify it via a
PUT request. That is why both VueStorefront and Deity are adding server side layer in node to manage this connection without exposing credentials to the browser. This layer also work as a way to make server side rendering possible.