I need a button in my custom system.xml to do Ajax call. How can this be achieved? or how can I link a JS file into my system.xml?
frontend_model will be your friend to achieve your requirement.
Just look at Validate VAT Number example from store > configuration > general > Store Information.
In this example magento using below
Just look at this file, you will definitely get the idea how to achieve this.