Credit memo is a default way of refunding in Magento.
Now, I've made one order with subtotal
45.0, shipping cost
5.0 and tax
The total of it
I need to have an invoice made, after which the status of the order moved to
processing, and at this point I could make a new credit memo.
Eventually I get this view
Which means that I have partial control over what is being refunded.
In this case all items are to be refunded, so the subtotal is 45, and the tax bound to it, normally, cannot be changed.
However, I can choose to refund shipping or not.
At it seems, your question is still not answered, and I will not pretend to know if the default credit memo entity can be used exactly for what you need.
But, it is good to keep in mind that credit memo can be created programatically, and the process will be very simmilar to making an order.
That given, then maybe it is possible to extend the credit memo to add new total named
Cancelation Fee that will subtract, as opposed to the addition that
Adjustment Fee already makes.
Needless to say you could do something like
$cancelationFee = 0.01 * $grandTotal;
$creditMemo->create(); (or whatever the method is that would create the memo)
So, this is just an idea that I had. I hope it helps.