After asking around this are the reasons I've got. Hope its satisfying for you, even though its probably what you expected.
1) The adminhtml interface was made with the intention to make it hard for merchants to totally screw up.
For that reason many features that are provided by Magento as a Framework are not possible through the user interface.
The reason why only products in the same attribute set are selectable as associated simple products for configurables is because that's what was in the specs.
As you said, it kind of makes sense that way.
2) Another reasons is thinking of the purpose of attribute sets. One of the reasons they exist is to reduce the number of attributes and options that have to be loaded and processed during a request. Applying this thinking to configurables makes sense, since it is a relatively resource expensive product type.
Because the backend was built in that way, the frontend logic for configurable never was expected to deal with simple products from different attribute sets.
Thats why the restrictions aren't implemented completely there.
Probably it would be easy to make configurables work with simples from different attribute sets. It just wasn't intended that way.
I didn't ask further and didn't look myself to see what code interaction exactly produced the strange result on the frontend. Probably its not important, since if I understood correctly you where asking for the reasoning behind the decision to exclude simple products from attribute sets different to the configurable, not the code explanation.