Send email to any of the spam detecting service like http://www.emailspamtest.com to check it spam rate and possible issues.
It can be wrong headers and encoding, stop words in the subject and the body, no text part in html emails, too many images comparing to the text size.
Also try to use smtp server instead of default mail function.
Edit:
Here is the full code to send transactional email in Magento:
public function sendTransactionalEmail()
{
// Transactional Email Template's ID
$templateId = 1;
// Set sender information
$senderName = Mage::getStoreConfig('trans_email/ident_support/name');
$senderEmail = Mage::getStoreConfig('trans_email/ident_support/email');
$sender = array('name' => $senderName,
'email' => $senderEmail);
// Set recepient information
$recepientEmail = '[email protected]';
$recepientName = 'John Doe';
// Get Store ID
$store = Mage::app()->getStore()->getId();
// Set variables that can be used in email template
$vars = array('customerName' => '[email protected]',
'customerEmail' => 'Mr. Nil Cust');
$translate = Mage::getSingleton('core/translate');
// Send Transactional Email
Mage::getModel('core/email_template')
->sendTransactional($templateId, $sender, $recepientEmail, $recepientName, $vars, $storeId);
$translate->setTranslateInline(true);
}
Hope this helps.