My entire navigation block was being cached and any changes I was making did not reflect on the frontend at all unless the cache are being cleared. But other phtml files were fine. 

In my navigation block class I added the following code and it is now working as expected:

    protected function _construct()
    {
        parent::_construct();

        $this->addData(array(   'cache_lifetime' => null ));
    }

The following link helped me to fix the issue:
http://stackoverflow.com/a/25032002/1302297