It is commonly recommended to use "Meta noindex, follow" over robots.txt to disallow pages from being indexed by google ... for example http://www.magestore.com/magento-2-tutorial/most-extreme-magento-2-seo-mistakes/
Now my question is where can I configure that in Magento 2.1?!
You can configure the option by following below path in Magento 2.x admin.
Stores -> Configuration -> General (Left side) -> Design (section) -> Search Engine Robots -> Default Robots (Select)-> Select
INDEX, FOLLOW
NOINDEX, FOLLOW
INDEX, NOFOLLOW
NOINDEX, NOFOLLOW
Save the configuration as you need.
You can configure this option by [STORE VIEW] specific.
If you want configure for specific request.
Event used "layout_generate_blocks_after"
Assume you know how to create a basic module and add below code as an event observer
/app/code/Myvendor/Mymodule/etc/frontend/events.xml
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">
<event name="layout_generate_blocks_after">
<observer name="learning_custom_event_obs" instance="Myvendor\Mymodule\Observer\Test" />
</event>
</config>
Inside your observer /app/code/Myvendor/Mymodule/Observer/Test.php
namespace Myvendor\Mymodule\Observer;
use \Magento\Framework\Event\Observer;
use \Magento\Framework\Event\ObserverInterface;
class Test implements ObserverInterface {
protected $request;
protected $layoutFactory;
public function __construct(
\Magento\Framework\App\Request\Http $request,
\Magento\Framework\View\Page\Config $layoutFactory
) {
$this->request = $request;
$this->layoutFactory = $layoutFactory;
}
public function execute(Observer $observer) {
$fullActionName = $this->request->getFullActionName();
if ($fullActionName == "catalog_product_view"){
$this->layoutFactory->setRobots('NOINDEX,NOFOLLOW');
}
}
}
/app/code/Myvendor/Mymodule/Observer/Test.php
Add your request ($fullActionName) or action name. I configured for product view page for reference which is working fine.
Hope this helps.
Content > Design > Configuration > Edit > Search Engine Robots > Default Robots
– Xenocide8998
Feb 12 '18 at 16:14
You can find information about robots.txt directives in the SEO Best Practices section of the Magento user guide. Topics include: Using a Sitemap, Sitemap Configuration, and Search Engine Robots.
Our user guides are written for merchants, with a focus on the Admin and storefront. Stop by magento.com and see Resources > Documentation for quick links to all our user guides.
You can still use layout updates for this as in Magento1 but the XML is a little different.
For insance, in the admin, go to a CMS Page. Then go to the design tab, in the layout updates add this:
<head>
<meta name="robots" content="NOINDEX,NOFOLLOW"/>
</head>
This will update the robots tag for the current page. You can do the same for all meta tags out their in the same way.
Keep in mind that if there already is a <head> tag in your layout update, you just add the <meta> part to the existing <head> tag.
Please correct the XML data and try again. Element 'head': This element is not expected. Expected is one of ( referenceContainer, container, update, move ). Line: 1
– EpixRu
May 3 '18 at 3:47
in 2.1.12 this is now managed from
Content > Design > Configuration > edit Global (or per store) >
Search engine robots > Default Robots
It is not possible to do this yet and it has been added as a feature request to Magento 2.
Here I am sharing the code, Suppose when the product disables from the backend then at the time automatically set "NOINDEX,NOFOLLOW" in SEO tab.
But this code works only those conditions if someone using Mageplaza SEO module Because I had used the Mazeplaza Seo module for the category & catalog both.
Follow Stepwise process -:
i)You have to create "events.xml" in /app/code/Vajor/Seo/etc/adminhtml Where Vajor -> Namespace, Seo -> Module name
events.xml
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">
<event name="catalog_product_save_before">
<observer name="vajor_seo_observer_seo" instance="Vajor\Seo\Observer\Seo" />
</event>
</config>
ii) /app/code/Vajor/Seo/Observer/Seo.php/
<?php
namespace Vajor\Seo\Observer;
class Seo implements \Magento\Framework\Event\ObserverInterface
{
private $logger;
public function __construct(\Psr\Log\LoggerInterface $logger){
$this->logger = $logger;
}
public function execute(\Magento\Framework\Event\Observer $observer){
/** @type \Magento\Catalog\Model\Product $product */
$product = $observer->getProduct();
$product_status = $product->getStatus();
$_getMetatag_value = $product->getMpMetaRobots();
try{
if ($product_status == 2 && $_getMetatag_value == "INDEX,FOLLOW"){ /*Disable Product*/
$product->setMpMetaRobots("NOINDEX,NOFOLLOW");
$product->save();
/*$_getMetatag_value = $product->getMpMetaRobots();*/
}if ($product_status == 1 && $_getMetatag_value =="NOINDEX,NOFOLLOW"){ /*Enable Product*/
$product->setMpMetaRobots("INDEX,FOLLOW");
$product->save();
}
}catch (\Exception $e){
$this->logger->critical($e->getMessage());
}
}
}
?>
