0

Two separate stores using one instance of Magento are hosted on two different domains.

E.g.

  • cakesworld.com
  • baconsworld.com

Duplication of the actual websites themselves isn't a problem, but many other URLs are duplicated across the two domains e.g. the XML sitemaps for each domain are available on both domains:

https://www.cakesworld.com/sitemaps/baconsworld/sitemap.xml (lists all the URLs on baconsworld.com)
https://www.baconsworld.com/sitemaps/cakesworld/sitemap.xml (lists all the URLs on cakesworld.com)

Also, many of these types of URLs exist, e.g. on cakesworld.com, duplicates with baconsworld as a folder (also existing on the same domain with the correct cakesworld folder:

https://www.cakesworld.com/static/version1557229/frontend/Dev/baconsworld/en_GB/Magento_Theme/templates/breadcrumbs.html
https://www.cakesworld.com/static/version1556876/frontend/Dev/baconsworld/en_GB/Magento_Ui/templates/modal/modalcustom.html
https://www.cakesworld.com/static/version1559139/frontend/Dev/baconsworld/en_GB/jstranslation.json

Googlebot regularly crawls these URLs. They can be blocked in robots.txt, but is it possible to prevent them from existing? Our developers say it isn't - it's something to do with the knockout.js Ajax elements and associated static.html files used as a template, and it's not possible to prevent them from being generated.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.