Suppose in category page there are 24 products,so you will call the $product->getName() for all 24 products,and we know that for magic function,firstly it will search in this class after that its parent classes and then it will go for the magic function,so with this call we are losing our precious milliseconds,is there any way in magento to avoid magic function except directly calling $product->getData('name')?
This is called micro-optimization. There are plenty of other discussions on the topic, such as this Programmers SE question "Is micro-optimisation important when coding?".
Long story short, don't worry about these magic calls. Magento uses them all over the place, and just about everything else in Magento causes more of a performance hit than the magic calls.