I am trying to edit search engine robots from
content and theme configuration but no option to edit.
Magento 2.2.5 Any luck ?
If you navigate to
Store > Configuration > Catalog > XML Sitemap
and scroll down to
Search Engine Submission Settings, you’ll find a setting that enables you to add the sitemap: directive to your
If you enable this, the URL towards your main sitemap
index file (the one that contains just two URL-s, the URLs of your actual URL sitemap and image sitemap) will be added to the
robots.txt file that’s dynamically generated on your website and if you visit
yourstore.com/robots.txt you’ll see.
Content > Design > Configuration
In here you can open an accordion section titled “Search Engine Robots”.
PRO TIP: If you’re having trouble saving the configuration at this step locally, try this fix.
Content -> Design -> Configuration
In the grid you will see a row which has only the website name no store view information, open that grid and you add the robots data.
If it's still not updating delete the robots.txt file from the server and try to access it again from a browser. It will be re-created