Looks like this is caused by Magento 2 full page cache bug in muti-store views. I have disabled the full page cache from env.php. It seems to fixes the issue. Have to debug more to find out.
Will update you guys soon.
Updates. found the issue. This is because Magento 2 is not handling HTTP HEAD requests. When a bot hits the website with HEAD type request, Magento returns 404 and caches it. The solution as below:
From e3890bfda2d67508df76ce00cdd675c60743ace5 Mon Sep 17 00:00:00 2001
From: Siyu Qian <[email protected]>
Date: Fri, 7 Jun 2019 15:17:40 +1200
Subject: [PATCH] BMQ-54 Full page cache issue
---
.../Plugin/PreventCachingHead404ResultsPlugin.php | 37 ++++++++++++++++++++++
app/code/MageBinary/Theme/etc/di.xml | 3 ++
2 files changed, 40 insertions(+)
create mode 100644 app/code/MageBinary/Theme/Plugin/PreventCachingHead404ResultsPlugin.php
diff --git a/app/code/MageBinary/Theme/Plugin/PreventCachingHead404ResultsPlugin.php b/app/code/MageBinary/Theme/Plugin/PreventCachingHead404ResultsPlugin.php
new file mode 100644
index 0000000..a16ad17
--- /dev/null
+++ b/app/code/MageBinary/Theme/Plugin/PreventCachingHead404ResultsPlugin.php
@@ -0,0 +1,37 @@
+<?php
+namespace MageBinary\Theme\Plugin;
+
+class PreventCachingHead404ResultsPlugin
+{
+ /**
+ * @var \Magento\Framework\App\Request\Http
+ */
+ protected $request;
+
+ /**
+ * PreventCachingHeadRequestsPlugin constructor.
+ * @param \Magento\Framework\App\Request\Http $request
+ */
+ public function __construct(\Magento\Framework\App\Request\Http $request)
+ {
+ $this->request = $request;
+ }
+
+ /**
+ * Prevent caching HEAD requests that return a 404 result
+ * @param \Magento\Framework\App\PageCache\Kernel $subject
+ * @param callable $proceed
+ * @param \Magento\Framework\App\Response\Http $response
+ * @return void
+ */
+ public function aroundProcess(
+ \Magento\Framework\App\PageCache\Kernel $subject,
+ callable $proceed,
+ \Magento\Framework\App\Response\Http $response
+ ) {
+ if ($this->request->isHead() && $response->getHttpResponseCode() == 404) {
+ return;
+ }
+ return $proceed($response);
+ }
+}
\ No newline at end of file
diff --git a/app/code/MageBinary/Theme/etc/di.xml b/app/code/MageBinary/Theme/etc/di.xml
index 5dfebc7..7df8e7a 100644
--- a/app/code/MageBinary/Theme/etc/di.xml
+++ b/app/code/MageBinary/Theme/etc/di.xml
@@ -16,4 +16,7 @@
</type>
<preference for="Magento\Framework\Url\QueryParamsResolver" type="MageBinary\Theme\Url\QueryParamsResolver" />
<preference for="Magento\CatalogWidget\Model\Rule\Condition\Product" type="MageBinary\Theme\CatalogWidget\Rule\Condition\Product" />
+ <type name="Magento\Framework\App\PageCache\Kernel">
+ <plugin name="prevent_caching_404_head_results" type="MageBinary\Theme\Plugin\PreventCachingHead404ResultsPlugin" />
+ </type>
</config>
--
2.10.5
php bin/magento cache:clean full_page