Two separate stores using one instance of Magento are hosted on two different domains.
Duplication of the actual websites themselves isn't a problem, but many other URLs are duplicated across the two domains e.g. the XML sitemaps for each domain are available on both domains:
https://www.cakesworld.com/sitemaps/baconsworld/sitemap.xml (lists all the URLs on baconsworld.com) https://www.baconsworld.com/sitemaps/cakesworld/sitemap.xml (lists all the URLs on cakesworld.com)
Also, many of these types of URLs exist, e.g. on cakesworld.com, duplicates with baconsworld as a folder (also existing on the same domain with the correct cakesworld folder:
https://www.cakesworld.com/static/version1557229/frontend/Dev/baconsworld/en_GB/Magento_Theme/templates/breadcrumbs.html https://www.cakesworld.com/static/version1556876/frontend/Dev/baconsworld/en_GB/Magento_Ui/templates/modal/modalcustom.html https://www.cakesworld.com/static/version1559139/frontend/Dev/baconsworld/en_GB/jstranslation.json
Googlebot regularly crawls these URLs. They can be blocked in robots.txt, but is it possible to prevent them from existing? Our developers say it isn't - it's something to do with the knockout.js Ajax elements and associated static.html files used as a template, and it's not possible to prevent them from being generated.