HomTom just change the block name name="right.reports.product.viewed" fro this code
<block type="reports/product_viewed" name="right.reports.product.viewed" as="recently_viewed" template="reports/product_viewed.phtml" /> of product page .
As two blocks have same name to it.<default> handler is working also on all frontend page
Default handler is also remove Recently Viewed Products from product page view
So,change
<catalog_product_view translate="label">
<reference name="content">
<block type="reports/product_viewed" name="right.reports.onlyproduct.viewed" as="recently_viewed" template="reports/product_viewed.phtml" />
</reference>
</catalog_product_view>
name="right.reports.product.viewed" change to name="right.reports.onlyproduct.viewed"