I have configured Homepage, Contact us, etc. to be monitored by uptimerobot.com.
But randomly it causes 404 error. And after clearing the cache it fixes the issue (but reappears later).
Note that 404 happens to only those pages which are configured to be tracked by uptimerobot.com
FYI, I am using
- Magento CE 2.3.1
- FPC (Filesystem)
I believe it has something to with the request headers sent uptimerobot in order to access those pages.
"HEAD /s HTTP/1.1" 200 0 "https://www.magepsycho.com/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
Have anyone faced such issue with some kind of crawlers/bots?
NOTE: The issue is not occurring when the cache is disabled.