If you take a look at app/code/core/Mage/Captcha/etc/config.xml
you'll find:
<captcha translate="label">
<fonts>
<linlibertine>
<label>LinLibertine</label>
<path>lib/LinLibertineFont/LinLibertine_Bd-2.8.1.ttf</path>
</linlibertine>
</fonts>
Further in you'll find getFonts() in app/code/core/Mage/Captcha/Helper/Data.php
/**
* Get list of available fonts
* Return format:
* [['arial'] => ['label' => 'Arial', 'path' => '/www/magento/fonts/arial.ttf']]
*
* @return array
*/
public function getFonts()
{
$node = Mage::getConfig()->getNode(Mage_Captcha_Helper_Data::XML_PATH_CAPTCHA_FONTS);
$fonts = array();
if ($node) {
foreach ($node->children() as $fontName => $fontNode) {
$fonts[$fontName] = array(
'label' => (string)$fontNode->label,
'path' => Mage::getBaseDir('base') . DS . $fontNode->path
);
}
}
return $fonts;
}
With the above, adding a compatible .ttf font to your lib/
folder and adding the font option to the configuration XML nodes will add a new font to the available list.
Magento does use Zend Libraries Zend_Captcha, you'll find this in lib/Zend/Captcha
Keep in mind the LinLibertine font is used with PDF generation so removing it or replacing it may cause other issues.
Also I have not tested any of this but general observations but hopefully will lead you in the right direction.
Hope this helps!