Out of the box, no, there's no way to exclude certain products from the sitemap generated by Magento's
Catalog -> Google Sitemap feature.
If I was going to go about doing this programmatically, modern versions of Magento (checked in the 1.7.x branch, this might be around in earlier/EE versions) use the following resource model class
to fetch a list of products.
$collection = Mage::getResourceModel('sitemap/catalog_product')->getCollection($storeId);
This is not a standard Magento CRUD model, and
getCollection does not return a collection object. Instead,
getCollection manually queries the database for these products.
If I was going to implement functionality that prevented certain products from showing up in the site map, I'd try to either
A class rewrite the
getCollection method which calls the
parent::getCollection, and then manually filters out any products from the array
A class rewrite on
_addFilter which calls the parent::_addFilter method, and then adds an additional WHERE clause(s) to the
_select to exclude the specific product(s). Sort of a hack, but it's the only method where you have access to the
_select object used to query the database. Ideally you'd want to have some sort of global/static flag so you only added your new WHERE clause(s) once.