Yes, tht settings "Allowed Countries" allows you to restrict the purchase of products on your site only to those allowed countries. If New Zealand is the only country allowed, then someone who is adding in billing and shipping information will only be able to see the New Zealand options.
If you only restrict countries via the shipping options (like you have done), then the end user will see that there are no shipping options available to them during the checkout process.
However, this could go awry if you're offering free shipping for orders over a specific amount, for example.
So, yes, you'll want to restrict using the Allowed Countries option.
Many store owners will do this because international shipping and customers are too complex to deal with... especially if you have heavy items that are appropriate for local delivery but are not easily able to be put on an airplane.
Here's how this looks in Magento 1.9: (in this case I also have Netherlands listed as an allowed country using a multi-select):

And how it looks in Magento 2.1 with New Zealand as the default and allowed country: (same basic function, but newer UI)

I hope this helps!