First step: Change the layout file
In your lotusbreath/onestepcheckout.xml files, replace* <lotusbreath_onestepcheckout_index_index>
handle to <checkout_cart_index>
.
So, now the checkout block will be render in cart.
Change <block type="lotusbreath_onestepcheckout/onepage" name="checkout.onepage" as="checkout.onepage">
to <block type="lotusbreath_onestepcheckout/onepage" name="checkout.onepage" as="checkout.onepage" after="-">
. The after
atributes is not necessary but I think is better checkout code be after cart block.
Now, we have the one step checkout in cart page, like this:

* Note: If you want to keep one step checkout page (/onestepcheckout/index), not replace <lotusbreath_onestepcheckout_index_index>
but make a copy and change the copied handle.
Second step: Create a modal/dialog
Now we need use a javascript lib to create a dialog/modal. The "OneStepCheckout from LotusBreath" uses jQuery, so there are many options to do it.
An html element to keep attention is <div class="lt-checkoutpage layout-2col " id="checkoutSteps">
. All elements of checkout are inside the #checkoutSteps
. Probably that ID is used as a reference in OneStepCheckout JavaScript. Put it in the modal.
To add a custom Js lib, use a action inside the <reference name="head"/>
:
<action method="addItem">
<type>skin_js</type>
<name>path_in_my_theme/to/MY_JS_LIB_FILE.js</name>
</action>
To get modal/dialog working, probably you will need some (or many) css modify.
I did a simple test with custom CSS & JS:

Third Step: Fix Css bugs and Ajaxs
After aplied dialog/modal, depending on the library used and changes made with css, you need fix js/css to get all checkout working again.
Fourth step: Replace the call-to-action (the buttons) to checkout.
Finally is necessary just change the call to action to checkout, removing links to original page and use javascript to display the dialog/modal.
You can use a onclick attributes on elements or use jQuery 'click' method in script tag or js file, like this:
jQuery('.my-element-class').click(function(){
//Code to display my dialog/modal
});