First thing is first. You cannot upload images through AJAX.
You have to trick the browser (or the user).
take a look at how the category images are uploaded in app/design/adminhtml/default/default/template/catalog/category/edit/form.phtml
.
The idea is to have an iframe
in your DOM with and id (let's say 'trickIframe') and make that form submit to that iframe.
<iframe name="trickIframe" style="display:none; width:100%;" src="<?php echo $this->getJsUrl() ?>blank.html"></iframe>
<form target="trickIframe" id="some_id_here" action="<?php echo $this->getSaveUrl() ?>" method="post" enctype="multipart/form-data">
....
</form>
Then you can handle the post request in the save method and that method should return some javascript code for refresing (or replace page content or what ever) just like Mage_Adminhtml_Catalog_CategoryController::saveAction
does.
$this->getResponse()->setBody(
'<script type="text/javascript">parent.updateContent("' . $url . '", {}, '.$refreshTree.');</script>'
);