0

I have recently replatformed from a custom PHP build to Magento 2.2.4, during the go-live process we changed from www to non-www versions of the site. I've updated Search Console to make the preference to the non-www version of the site.

Now I'm getting the following notice:

enter image description here

Magento is using a stock version of the robots.txt file as seen here current robots.txt

Everything within Google says the root domain is able to be indexed: enter image description here

The default domain is also not www it's the non-www version. enter image description here

I also verified that the site has Index,Follow enabled.

Does anyone have any thoughts as to why this maybe occurring?

Improve this question

1 Answer 1

Reset to default
1

It's something in the robots.txt

You could just delete it as it will not hurt anything to have it or not have it.

People think the robots.txt makes a difference in SEO - only if you have robotted out pages you want indexed. Otherwise it makes no difference on 99.9% of sites. So delete it and fetch in SC - that should go away.

Improve this answer
7
  • I thought about that, I just wish I knew the cause. Everything on the old site ended with *.php so I was thinking it was that entry, but it wouldn’t surprise me if it was a bug either. If I were to try removing things slowly what should I try first?
    – Rob
    Commented Jun 19, 2018 at 11:28
  • It's one step, delete this: witsendmosaic.com/robots.txt Who knows what line in there is causing this, as Magento likely is auto generating any part of your page in any number of these robotted out sections. Is your site like a million pages? If no, just delete it, you don't need to robot out anything.
    – Josh Bachynski
    Commented Jun 20, 2018 at 19:58
  • It's about 15k pages, I'm just nervous as the business has an "SEO guy" and he'll probably have his way if I don't figure out the reason why it's being blocked and removing the robots.txt might make him uncomfortable.
    – Rob
    Commented Jun 22, 2018 at 15:53
  • I removed everything that was not a folder, or a specific directive and requested a fetch in SC. If this doesn't work, I'll remove the robots.txt entirely. :crosses_fingers:
    – Rob
    Commented Jun 23, 2018 at 12:32
  • Submit the new robots txt in search console and fetch the index page and you should see the change right away if there is one. Trust me, the robots txt on a 15k block is NOTHING.. SEOs hold on to this as an illusion of what is important in SEO, it's really not. I am doing an online SEO 101 course Thursday i can explain more then.
    – Josh Bachynski
    Commented Jun 26, 2018 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.