From what I have been able to see jQuery appears to be the js library of choice in nearly every web system/framework/cms. It also seems to be a more intuitive and powerful library but that is subjective I suppose.
Does prototype provide any functionality that jquery doesn't necessary for magento to do what it does?
Is there any performance advantages to using prototype?
Would it be a good idea to use prototype, instead of jquery, for custom js on the frontend of a magento site?
Aside from jQuery being lightly adopted during the development of Magento we must consider another aspect:
Magento was developed around 2007 at a time when TC39 was proposing EcmaScript 5 and it was not fully implemented or developed. Prototype.JS aimed to solve this issue by bringing ECMA5 features to legacy browsers which lacked support at the time. Prototype attempted to fill this need first and foremost and so it was adopted by the developers of Magento to bring ECMA5 features to the frontend framework being developed.
Another consideration is that one of the pillars of the Magento platform was extreme flexibility and extendability. Prototype provides the ability to have OOP-like features where you can create new objects that extend or inherit from other given objects and call the super class directly from child classes. It's quite powerful in that regard. Even today in jQuery this is possible to some extent with widgets but now we use native ECMA5 OOP features found in modern browsers.
Let's also consider AJAX. Google Maps and Gmail are largely considered to be the landmark uses of AJAX from that era, with Maps being launched just 2 years prior to Magento in 2005. AJAX was not terribly uncommon or painful to work with at the time but using a library gave you a standard API in 2007 to achieving cross-browser compatibility without extra complexity. Magento was very ambitious in its use of AJAX (for a free product!) for the checkout, admin panel, and therefore it adopted the best library for managing AJAX at the time. That library was Prototype.
If you know how to code with Prototype you might want to continue using it since it won't make you struggle with compatibility between JS libs.
If not, the struggle now is not that complex since jQuery can be easily included in a compatible way using noConflict() and work along with Prototype.
As for the advantages/performance, jQuery is pretty new and at the time Magento came out jQuery was still in diapers. Prototype is robust IMHO but since jQuery is now widely extended (and even included in Magento2) a lot of JS development is being done with it even in Magento.
