To add form validation, you'll have to create the file to work on, and hook it to the Magento layout.
To follow best practices, you'll have to create your own module if you don't have already.
I'll start from bottom up, to evidentiate the core need:
Create a layout file
You can skip the mycustom folder if you'd like and name your js file as you want - but remember to change accordingly also in the next steps.
Now you need to reference the xml layout file you just created.
Create your module
<!-- this will look for the XML file in the app/design/adminhtml/default/default/layout folder -->
Again, you can also skip the folder and name the xml file as you want.
Now you need to reference your module.
and put your validation code in it.
For JS form validations you can check this answers: Custom form Validation - Magento
What to be aware of/check:
- Look into Configuration > Advanced > Advanced to see if your new module is listed and Enabled
- Log files in /var/log folder
- Typing errors
- Root of mentioned paths is /web/ of your Magento installation folder
More detailed version here: Add JS to Magento Sales Order