It is commonly recommended to use "Meta noindex, follow" over robots.txt to disallow pages from being indexed by google ... for example http://www.magestore.com/magento-2-tutorial/most-extreme-magento-2-seo-mistakes/
Now my question is where can I configure that in Magento 2.1?!
You can configure the option by following below path in Magento 2.x admin.
Stores -> Configuration -> General (Left side) -> Design (section) -> Search Engine Robots -> Default Robots (Select)-> Select
INDEX, FOLLOW
NOINDEX, FOLLOW
INDEX, NOFOLLOW
NOINDEX, NOFOLLOW
Save the configuration as you need.
You can configure this option by [STORE VIEW] specific.
If you want configure for specific request.
Event used "layout_generate_blocks_after"
Assume you know how to create a basic module and add below code as an event observer
/app/code/Myvendor/Mymodule/etc/frontend/events.xml
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">
<event name="layout_generate_blocks_after">
<observer name="learning_custom_event_obs" instance="Myvendor\Mymodule\Observer\Test" />
</event>
</config>
Inside your observer /app/code/Myvendor/Mymodule/Observer/Test.php
namespace Myvendor\Mymodule\Observer;
use \Magento\Framework\Event\Observer;
use \Magento\Framework\Event\ObserverInterface;
class Test implements ObserverInterface {
protected $request;
protected $layoutFactory;
public function __construct(
\Magento\Framework\App\Request\Http $request,
\Magento\Framework\View\Page\Config $layoutFactory
) {
$this->request = $request;
$this->layoutFactory = $layoutFactory;
}
public function execute(Observer $observer) {
$fullActionName = $this->request->getFullActionName();
if ($fullActionName == "catalog_product_view"){
$this->layoutFactory->setRobots('NOINDEX,NOFOLLOW');
}
}
}
/app/code/Myvendor/Mymodule/Observer/Test.php
Add your request ($fullActionName) or action name. I configured for product view page for reference which is working fine.
Hope this helps.
Content > Design > Configuration > Edit > Search Engine Robots > Default Robots
– Xenocide8998
Feb 12 '18 at 16:14
You can find information about robots.txt directives in the SEO Best Practices section of the Magento user guide. Topics include: Using a Sitemap, Sitemap Configuration, and Search Engine Robots.
Our user guides are written for merchants, with a focus on the Admin and storefront. Stop by magento.com and see Resources > Documentation for quick links to all our user guides.
You can still use layout updates for this as in Magento1 but the XML is a little different.
For insance, in the admin, go to a CMS Page. Then go to the design tab, in the layout updates add this:
<head>
<meta name="robots" content="NOINDEX,NOFOLLOW"/>
</head>
This will update the robots tag for the current page. You can do the same for all meta tags out their in the same way.
Keep in mind that if there already is a <head> tag in your layout update, you just add the <meta> part to the existing <head> tag.
Please correct the XML data and try again. Element 'head': This element is not expected. Expected is one of ( referenceContainer, container, update, move ). Line: 1
– EpixRu
May 3 '18 at 3:47
in 2.1.12 this is now managed from
Content > Design > Configuration > edit Global (or per store) >
Search engine robots > Default Robots
It is not possible to do this yet and it has been added as a feature request to Magento 2.
Here I am sharing the code, Suppose when the product disables from the backend then at the time automatically set "NOINDEX,NOFOLLOW" in SEO tab.
But this code works only those conditions if someone using Mageplaza SEO module Because I had used the Mazeplaza Seo module for the category & catalog both.
Follow Stepwise process -:
i)You have to create "events.xml" in /app/code/Vajor/Seo/etc/adminhtml Where Vajor -> Namespace, Seo -> Module name
events.xml
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">
<event name="catalog_product_save_before">
<observer name="vajor_seo_observer_seo" instance="Vajor\Seo\Observer\Seo" />
</event>
</config>
ii) /app/code/Vajor/Seo/Observer/Seo.php/
<?php
namespace Vajor\Seo\Observer;
class Seo implements \Magento\Framework\Event\ObserverInterface
{
private $logger;
public function __construct(\Psr\Log\LoggerInterface $logger){
$this->logger = $logger;
}
public function execute(\Magento\Framework\Event\Observer $observer){
/** @type \Magento\Catalog\Model\Product $product */
$product = $observer->getProduct();
$product_status = $product->getStatus();
$_getMetatag_value = $product->getMpMetaRobots();
try{
if ($product_status == 2 && $_getMetatag_value == "INDEX,FOLLOW"){ /*Disable Product*/
$product->setMpMetaRobots("NOINDEX,NOFOLLOW");
$product->save();
/*$_getMetatag_value = $product->getMpMetaRobots();*/
}if ($product_status == 1 && $_getMetatag_value =="NOINDEX,NOFOLLOW"){ /*Enable Product*/
$product->setMpMetaRobots("INDEX,FOLLOW");
$product->save();
}
}catch (\Exception $e){
$this->logger->critical($e->getMessage());
}
}
}
?>
I have found two fixes for this issue... {I recommend Fix Two}
Fix One
Locate the file DataProvider.php and edit it.
Path: '{your site directory}/vendor/magento/module-theme/Model/Design/Config/DataProvider.php'
now locate line {155}
This controls the specific drop-down box.
You need to change a boolean value... true to false on line 179.
{before change}
/**
* Retrieve modified Search Engine Robots metadata
*
* Disable Search Engine Robots fields in case when current scope is 'stores'.
*
* @param string $scope
* @param array $fields
* @return array
*/
private function getSearchEngineRobotsMetadata($scope, array $fields = [])
{
if ($scope == \Magento\Store\Model\ScopeInterface::SCOPE_STORES) {
$resetToDefaultsData = [
'arguments' => [
'data' => [
'config' => [
'disabled' => true,
'is_disable_inheritance' => true,
],
],
],
];
$fields = array_merge($fields, ['reset_to_defaults' => $resetToDefaultsData]);
foreach ($fields as &$field) {
$field['arguments']['data']['config']['disabled'] = true;
$field['arguments']['data']['config']['is_disable_inheritance'] = true;
}
}
return $fields;
}
[AFTER CHANGE]
/**
* Retrieve modified Search Engine Robots metadata
*
* Disable Search Engine Robots fields in case when current scope is 'stores'.
*
* @param string $scope
* @param array $fields
* @return array
*/
private function getSearchEngineRobotsMetadata($scope, array $fields = [])
{
if ($scope == \Magento\Store\Model\ScopeInterface::SCOPE_STORES) {
$resetToDefaultsData = [
'arguments' => [
'data' => [
'config' => [
'disabled' => true,
'is_disable_inheritance' => true,
],
],
],
];
$fields = array_merge($fields, ['reset_to_defaults' => $resetToDefaultsData]);
foreach ($fields as &$field) {
$field['arguments']['data']['config']['disabled'] = false;
$field['arguments']['data']['config']['is_disable_inheritance'] = true;
}
}
return $fields;
}
your code should now look like the above. {Warning} making changes is not recommended to the Magento core. If you do make a change revert it after saving the value to the desired configuration.
Fix Two (Recommended)
Don't use the default store view to create your store! Create a new store view.
Configure as needed, as all options are enabled by default. {Unlike the default store view}.
Go to admin panel
Content
Configuration
locate
Global Main Website
Click Edit
Configure your site here!
I hope this helps... Many thanks to all contributors as you have helped me so much over the years, I hope this help you all and gives a little back.
For configure in specific layout, simplest way is add this meta in head tag of layout file
For example: catalogsearch_result_index.xml
<?xml version="1.0"?>
<page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" layout="2columns-left" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd">
<head>
<meta name="robots" content="NOINDEX,NOFOLLOW"/>
</head>
</body>
