No need to write plugin for this,  we have to do it in xml level 

create a layout xml 

**app\code\MYNAME_SPACE\MY_MODULE\view\base\layout\catalog_product_prices.xml**

    <?xml version="1.0"?>
    <!--
    /**
     * Copyright © 2013-2017 Magento, Inc. All rights reserved.
     * See COPYING.txt for license details.
     */
    -->
    <layout xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/layout_generic.xsd">
        <referenceBlock name="render.product.prices">
            <arguments>
                <argument name="default" xsi:type="array">
                    <item name="prices" xsi:type="array">
                        <item name="final_price" xsi:type="array">
                            <item name="render_template" xsi:type="string">MYNAME_SPACE_MY_MODULE::product/price/final_price.phtml</item>
                        </item>
                    </item>
                </argument>
            </arguments>
        </referenceBlock>
    </layout>

Above xml will effect only for simple product if you need to modify the bundle product then 

    <argument name="default" xsi:type="array">

should be 

    <argument name="bundle" xsi:type="array">

or in case of a configurable product

    <argument name="configurable" xsi:type="array">

create a phtml that is 

**\app\code\MYNAME_SPACE\MY_MODULE\view\base\templates\product\price\final_price.phtml**

Copy final_price from core module and modify it as you needed 

to modify other price type you can refer 

**vendor\magento\module-bundle\view\base\layout\catalog_product_prices.xml**