IMO, you should prefer using plugins and "fallback" to observers when you can't modify some arbitrary functionality; either because it's isolated within a protected method (plugins do not work on protected or private methods), or because the observer provides access to an object that is otherwise inaccessible because of the nature of the code. The former is rather rare, but Magento explicitly passes objects into an observer.

IMO, you should prefer using plugins and "fallback" to observers when you can't modify some arbitrary functionality; either because it's isolated within a protected method (plugins do not work on protected or private methods), or because the observer provides access to an object that is otherwise inaccessible because of the nature of the code. The latter is rather rare, but Magento explicitly passes objects into an observer.