0

I have a weird problem. I am able to update my robots.txt however for some reason, whenever I click on www.mysite.com/robots.txt, the following code would always be at the end:

User-agent: *
Disallow:
Crawl-delay: 10

I could change any other stuff and it would show instantly. But this code at the end just appears from nowhere! I don't see it when I download the robots.txt from my file manager.

1 Answer 1

0

I'd suggest to temporary rename your robots.txt on the server - if you still can access it after verifying that the file is definitely not preset in your webroot anymore your robots.txt seems to be auto-updated/generate from either

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.