The quick and ugly way is to check the current customer's group and then set the theme programatically:

    Mage::getDesign()->setArea('frontend')
        ->setPackageName('your_package')
        ->setTheme('your_theme');

But this solution definitely misses some flexibility.

The more sophisticated way would be to create a customer group layout handels and then set custom themes there. This solution is inspired by [this article][1] by Atwix.

So first you have to observe `controller_action_layout_load_before` event:

    <events>
        <controller_action_layout_load_before>
            <observers>
                <customer_group_handle>
                    <class>module/observer</class>
                    <method>addCustomerGroupHandle</method>
                </customer_group_handle>
            </observers>
        </controller_action_layout_load_before>
    </events>

Then in your observer class implement `addCustomerGroupHandle` method:

    public function addCustomerGroupHandle(Varien_Event_Observer $observer)
    {
        if (Mage::helper('customer')->isLoggedIn()) {
            /** @var $update Mage_Core_Model_Layout_Update */
            $update = $observer->getEvent()->getLayout()->getUpdate();
            $groupId = Mage::helper('customer')->getCustomer()->getGroupId();
            $groupName = Mage::getModel('customer/group')->load($groupId)->getCode();
            $update->addHandle('customer_group_' . str_replace(' ', '_', strtolower($groupName)));
        }
     
        return $this;
    }

Note: The `str_replace` here is not foolproof so I suggest replacing it with a regular expression which will replace all non-alphanumeric characters with underscores and then trim leading and trailing underscores.

So now you can set a custom theme for any customer group through xml:

    <?xml version="1.0" encoding="UTF-8"?>
    <layout>
        <customer_group_wholesale>
            <reference name=”root”>
                <action method=”setTheme”><theme>modern</theme></action>
            </reference>
        </customer_group_wholesale>
    </layout>

  [1]: http://www.atwix.com/magento/magento-customer-group-layout-handle/