Can someone explain on how to integrate a custom payment gateway. I have no clue on what has to be done. The payment gateway is set in tomcat server running in another port. what should i do ?
Seems like have to create a payment method. In onepage checkout your payment method will appear under the
Payment Method section. Once the user selects it and click on the
Place Order button the user should redirect to your payment gateway. After entering relevant details in the payment gateway the user should redirect back to success or unsuccess pages accordingly.
To achieve this best thing is to find an extension which can be customized according to your requirement. Unless you'll have to build it from scratch. For that hope this will help. http://www.excellencemagentoblog.com/magento-create-custom-payment-method
If you are want to communicate with new payment gateway ,first you need to create a payment method module (you can use above answer link) and decide whether it captures the amount or authorize based on that implement your payment method model flags.Once this is done you need communicate your payment gateway with HTTPCLIENT request and handle the response.
Here you can check the how magento handles authorize.net payment gateway
Hope this will help you get start