It depends on what your observer does and if it gets dispatched.
If it's heavy, it will have an impact.
You should define events only in the area that you need it.
For example, if you need something to happen when every page loads on frontend and you put it under the
global area then it will impact the admin also, because it will be dispatched for every admin page also. And you might not need that.
But if, for example, you are observing the event
page_block_html_topmenu_gethtml_before under the global area it will have no impact on the backend, because that event is not dispatched in the backend.
I choose this example, because in the
Mage_Catalog module this exact event is observed under the
global area. And this does not impact the frontend.
But because it's in the core, doesn't make it right.
As a general rule you should observe events only in the area you need it, because otherwise it might impact performance.