Is there a way to add a pick 3 max in the multiple select form in the custom options page? It's got required but not any other validation.
No, there isn't a way to define a minimum other than 1, or any maximum in Magento.
<select> by name, or apply the limitation to all of them.
As for back-end validation, you could always create an event observer and listen to
sales_quote_product_add_after and remove the product from the cart, send an error to the session and redirect back the product page if they select too many values. (There isn't a
sales_quote_product_add_before as far as I'm aware, which is annoying.)