I am getting ready to restructure my product database to use configurable products instead of custom options. I want the configurable products to be indexed by Google, but not the related simple products, since this would create duplicate content. If I have the products set as "Not Visible Individually" will this keep them from being crawled by Google or do I need to do something else?
I read a couple posts from people saying that google was crawling product pages that weren't visible and creating 404 erros so I'm wondering if that was the best practice is.