So what your trying to achieve is difficult because of the way widgets are added to the page, when a widget is saved it's layout XML update is automatically generated HOWEVER the name that is generated is just an random hash generated in the file below.
$hash = Mage::helper('core')->uniqHash();
$xml .= '<block type="' . $this->getType() . '" name="' . $hash . '"' . $template . '>';
This is then later saved to the database for example in my case the random name was 9d10a0d6931d18586a907b201633d878 which you can see is in the below image.
So although it's probably not very practical to do so in theory there is a way you could do this without touching any code you could create the widget search for the update record in the database table
core_layout_update get your new block name and then do for example
<remove name="9d10a0d6931d18586a907b201633d878"/> However if you make any change to the widget the XML and name will refresh and you would need to update the name references once again. I tried unsetChild and that didn't seem to work I suspect that is due to the order in which the XML is processed for widgets.
I think you have two real options:
1) Add a new text_list to the left column, right column or anywhere else you would want to add widgets and then mabey remove them on a page. You can do this in your themes local.xml like this.
<block type="core/text_list" name="widgets_left" as="widgets_left" translate="label">
This will add a new option in the backend as in the image which you can then add widget instances too or just remove if you need to by name as you have the widgets_left name as a reference.
You can then do either on the below in the specific category you'd like them removed from:
2) Alternatively you could just add them in Via XML like below, but it sounds to me like you want as much control as possible via the CMS and why not for me widgets are underused.
<block type="cms/block" name="cms_left_block_name">
You will then have a reference fixed as "cms_left_block_name" which you can use to either perform a remove or unsetChild.