Like @carsen.eldon mentioned, Magento does not do that. It works similarly with categories. I think that's why the replace import functionality might have been "invented".
In order for Magento to know whether to leave the images alone, or to update it, it would need to keep track of image metadata — I.e. a hash of the file, or the filesize in bytes, or some sort of metadata (exif) inside the actual image file.
I do strongly think that CSV import should be idempotent and declarative. As it stands now, importing the same file twice causes side effects.
There's a bugreport for it here: https://github.com/magento/magento2/issues/14398
Magento's import is quite lacking in many respects.
Edit: I've added my PR here: https://github.com/magento/magento2/pull/21146