It is commonly recommended to use "Meta noindex, follow" over robots.txt to disallow pages from being indexed by google ... for example http://www.magestore.com/magento-2-tutorial/most-extreme-magento-2-seo-mistakes/

Now my question is where can I configure that in Magento 2.1?!

You can configure the option by following below path in Magento 2.x admin.

Stores -> Configuration -> General (Left side) -> Design (section) -> Search Engine Robots -> Default Robots (Select)-> Select

  1. INDEX, FOLLOW

  2. NOINDEX, FOLLOW

  3. INDEX, NOFOLLOW

  4. NOINDEX, NOFOLLOW

Save the configuration as you need.

You can configure this option by [STORE VIEW] specific.

If you want configure for specific request.

Event used "layout_generate_blocks_after"

Assume you know how to create a basic module and add below code as an event observer

/app/code/Myvendor/Mymodule/etc/frontend/events.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">
        <event name="layout_generate_blocks_after">
        <observer name="learning_custom_event_obs" instance="Myvendor\Mymodule\Observer\Test" />
    </event>
</config>

Inside your observer /app/code/Myvendor/Mymodule/Observer/Test.php

namespace Myvendor\Mymodule\Observer;

use \Magento\Framework\Event\Observer;
use \Magento\Framework\Event\ObserverInterface;

class Test implements ObserverInterface {

protected $request;

protected $layoutFactory;

public function __construct(
    \Magento\Framework\App\Request\Http $request,
    \Magento\Framework\View\Page\Config $layoutFactory
) {
   $this->request = $request;
   $this->layoutFactory = $layoutFactory;
  }



    public function execute(Observer $observer) {

    $fullActionName = $this->request->getFullActionName();

     if ($fullActionName == "catalog_product_view"){
          $this->layoutFactory->setRobots('NOINDEX,NOFOLLOW');
       }
    }
}

/app/code/Myvendor/Mymodule/Observer/Test.php

Add your request ($fullActionName) or action name. I configured for product view page for reference which is working fine.

Hope this helps.

  • But this will do it for the whole website. I don't want that. I want to "NoIndex" a specific page in my magento website in a try to solve SEO layered navigation problem – abutmah Jul 24 '16 at 9:36
  • @abutmah Check the updated answer, better to change the question to " Set NOINDEX, NOFOLLOW on particular action or request in Magento 2" hope to cache better – Krishna ijjada Jul 24 '16 at 11:54
  • 3
    in 2.2.2 now found in Content > Design > Configuration > Edit > Search Engine Robots > Default Robots – Xenocide8998 Feb 12 at 16:14

You can find information about robots.txt directives in the SEO Best Practices section of the Magento user guide. Topics include: Using a Sitemap, Sitemap Configuration, and Search Engine Robots.

Our user guides are written for merchants, with a focus on the Admin and storefront. Stop by magento.com and see Resources > Documentation for quick links to all our user guides.

You can still use layout updates for this as in Magento1 but the XML is a little different.

For insance, in the admin, go to a CMS Page. Then go to the design tab, in the layout updates add this:

<head>
    <meta name="robots" content="NOINDEX,NOFOLLOW"/>
</head>

This will update the robots tag for the current page. You can do the same for all meta tags out their in the same way.

Keep in mind that if there already is a <head> tag in your layout update, you just add the <meta> part to the existing <head> tag.

  • This does not work in Magento2.2.3, you get an error that says Please correct the XML data and try again. Element 'head': This element is not expected. Expected is one of ( referenceContainer, container, update, move ). Line: 1 – EpixRu May 3 at 3:47

It is not possible to do this yet and it has been added as a feature request to Magento 2.

in 2.1.12 this is now managed from

Content > Design > Configuration > edit Global (or per store) >

Search engine robots > Default Robots

Here I am sharing the code, Suppose when the product disables from the backend then at the time automatically set "NOINDEX,NOFOLLOW" in SEO tab.

But this code works only those conditions if someone using Mageplaza SEO module Because I had used the Mazeplaza Seo module for the category & catalog both.

Follow Stepwise process -:

i)You have to create "events.xml" in /app/code/Vajor/Seo/etc/adminhtml Where Vajor -> Namespace, Seo -> Module name

events.xml

<?xml version="1.0"?>
    <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">

       <event name="catalog_product_save_before">
           <observer name="vajor_seo_observer_seo" instance="Vajor\Seo\Observer\Seo" />
       </event>

    </config>

ii) /app/code/Vajor/Seo/Observer/Seo.php/

<?php
namespace Vajor\Seo\Observer;

class Seo implements \Magento\Framework\Event\ObserverInterface
{

        private $logger;

        public function __construct(\Psr\Log\LoggerInterface $logger){
            $this->logger = $logger;       

        }

        public function execute(\Magento\Framework\Event\Observer $observer){

             /** @type \Magento\Catalog\Model\Product $product */
                 $product = $observer->getProduct();
                 $product_status = $product->getStatus();           
                 $_getMetatag_value = $product->getMpMetaRobots();

            try{

                    if ($product_status == 2 && $_getMetatag_value == "INDEX,FOLLOW"){                /*Disable Product*/
                            $product->setMpMetaRobots("NOINDEX,NOFOLLOW");
                            $product->save();
                            /*$_getMetatag_value = $product->getMpMetaRobots();*/

                }if ($product_status == 1 && $_getMetatag_value =="NOINDEX,NOFOLLOW"){              /*Enable Product*/
                            $product->setMpMetaRobots("INDEX,FOLLOW");
                            $product->save();

                }

            }catch (\Exception $e){
                    $this->logger->critical($e->getMessage());
            }    

      }

  }


?>

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.