8

It is commonly recommended to use "Meta noindex, follow" over robots.txt to disallow pages from being indexed by google ... for example http://www.magestore.com/magento-2-tutorial/most-extreme-magento-2-seo-mistakes/

Now my question is where can I configure that in Magento 2.1?!

Improve this question

8 Answers 8

Reset to default
21

You can configure the option by following below path in Magento 2.x admin.

Stores -> Configuration -> General (Left side) -> Design (section) -> Search Engine Robots -> Default Robots (Select)-> Select

  1. INDEX, FOLLOW

  2. NOINDEX, FOLLOW

  3. INDEX, NOFOLLOW

  4. NOINDEX, NOFOLLOW

Save the configuration as you need.

You can configure this option by [STORE VIEW] specific.

If you want configure for specific request.

Event used "layout_generate_blocks_after"

Assume you know how to create a basic module and add below code as an event observer

/app/code/Myvendor/Mymodule/etc/frontend/events.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">
        <event name="layout_generate_blocks_after">
        <observer name="learning_custom_event_obs" instance="Myvendor\Mymodule\Observer\Test" />
    </event>
</config>

Inside your observer /app/code/Myvendor/Mymodule/Observer/Test.php 

namespace Myvendor\Mymodule\Observer;

use \Magento\Framework\Event\Observer;
use \Magento\Framework\Event\ObserverInterface;

class Test implements ObserverInterface {

protected $request;

protected $layoutFactory;

public function __construct(
    \Magento\Framework\App\Request\Http $request,
    \Magento\Framework\View\Page\Config $layoutFactory
) {
   $this->request = $request;
   $this->layoutFactory = $layoutFactory;
  }



    public function execute(Observer $observer) {

    $fullActionName = $this->request->getFullActionName();

     if ($fullActionName == "catalog_product_view"){
          $this->layoutFactory->setRobots('NOINDEX,NOFOLLOW');
       }
    }
}

/app/code/Myvendor/Mymodule/Observer/Test.php

Add your request ($fullActionName) or action name. I configured for product view page for reference which is working fine.

Hope this helps.

Improve this answer
4
  • But this will do it for the whole website. I don't want that. I want to "NoIndex" a specific page in my magento website in a try to solve SEO layered navigation problem
    – abutmah
    Jul 24, 2016 at 9:36
  • @abutmah Check the updated answer, better to change the question to " Set NOINDEX, NOFOLLOW on particular action or request in Magento 2" hope to cache better
    – Krishna ijjada
    Jul 24, 2016 at 11:54
  • 6
    in 2.2.2 now found in Content > Design > Configuration > Edit > Search Engine Robots > Default Robots
    – Xenocide8998
    Feb 12, 2018 at 16:14
  • Good answer. +1
    – Shoaib Munir
    Mar 14, 2019 at 18:27
2

You can still use layout updates for this as in Magento1 but the XML is a little different.

For insance, in the admin, go to a CMS Page. Then go to the design tab, in the layout updates add this:

<head>
    <meta name="robots" content="NOINDEX,NOFOLLOW"/>
</head>

This will update the robots tag for the current page. You can do the same for all meta tags out their in the same way.

Keep in mind that if there already is a <head> tag in your layout update, you just add the <meta> part to the existing <head> tag.

Improve this answer
1
  • This does not work in Magento2.2.3, you get an error that says Please correct the XML data and try again. Element 'head': This element is not expected. Expected is one of ( referenceContainer, container, update, move ). Line: 1
    – EpixRu
    May 3, 2018 at 3:47
2

in 2.1.12 this is now managed from

Content > Design > Configuration > edit Global (or per store) >

Search engine robots > Default Robots

Improve this answer
1

You can find information about robots.txt directives in the SEO Best Practices section of the Magento user guide. Topics include: Using a Sitemap, Sitemap Configuration, and Search Engine Robots.

Our user guides are written for merchants, with a focus on the Admin and storefront. Stop by magento.com and see Resources > Documentation for quick links to all our user guides.

Improve this answer
1

For configure in specific layout, simplest way is add this meta in head tag of layout file

For example: catalogsearch_result_index.xml

<?xml version="1.0"?> <page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" layout="2columns-left" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd"> <head> <meta name="robots" content="NOINDEX,NOFOLLOW"/> </head> </body>

Improve this answer
0

It is not possible to do this yet and it has been added as a feature request to Magento 2.

Improve this answer
3
0

Here I am sharing the code, Suppose when the product disables from the backend then at the time automatically set "NOINDEX,NOFOLLOW" in SEO tab.

But this code works only those conditions if someone using Mageplaza SEO module Because I had used the Mazeplaza Seo module for the category & catalog both.

Follow Stepwise process -:

i)You have to create "events.xml" in /app/code/Vajor/Seo/etc/adminhtml Where Vajor -> Namespace, Seo -> Module name

events.xml

<?xml version="1.0"?>
    <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">

       <event name="catalog_product_save_before">
           <observer name="vajor_seo_observer_seo" instance="Vajor\Seo\Observer\Seo" />
       </event>

    </config>

ii) /app/code/Vajor/Seo/Observer/Seo.php/

<?php
namespace Vajor\Seo\Observer;

class Seo implements \Magento\Framework\Event\ObserverInterface
{

        private $logger;

        public function __construct(\Psr\Log\LoggerInterface $logger){
            $this->logger = $logger;       

        }

        public function execute(\Magento\Framework\Event\Observer $observer){

             /** @type \Magento\Catalog\Model\Product $product */
                 $product = $observer->getProduct();
                 $product_status = $product->getStatus();           
                 $_getMetatag_value = $product->getMpMetaRobots();

            try{

                    if ($product_status == 2 && $_getMetatag_value == "INDEX,FOLLOW"){                /*Disable Product*/
                            $product->setMpMetaRobots("NOINDEX,NOFOLLOW");
                            $product->save();
                            /*$_getMetatag_value = $product->getMpMetaRobots();*/

                }if ($product_status == 1 && $_getMetatag_value =="NOINDEX,NOFOLLOW"){              /*Enable Product*/
                            $product->setMpMetaRobots("INDEX,FOLLOW");
                            $product->save();

                }

            }catch (\Exception $e){
                    $this->logger->critical($e->getMessage());
            }    

      }

  }


?>
Improve this answer
0

I have found two fixes for this issue... {I recommend Fix Two}

Fix One

Locate the file DataProvider.php and edit it.

Path: '{your site directory}/vendor/magento/module-theme/Model/Design/Config/DataProvider.php'

now locate line {155}

This controls the specific drop-down box.

You need to change a boolean value... true to false on line 179.

{before change}

/**
 * Retrieve modified Search Engine Robots metadata
 *
 * Disable Search Engine Robots fields in case when current scope is 'stores'.
 *
 * @param string $scope
 * @param array $fields
 * @return array
 */
private function getSearchEngineRobotsMetadata($scope, array $fields = [])
{
    if ($scope == \Magento\Store\Model\ScopeInterface::SCOPE_STORES) {
        $resetToDefaultsData = [
            'arguments' => [
                'data' => [
                    'config' => [
                        'disabled' => true,
                        'is_disable_inheritance' => true,
                    ],
                ],
            ],
        ];
        $fields = array_merge($fields, ['reset_to_defaults' => $resetToDefaultsData]);
        foreach ($fields as &$field) {
            $field['arguments']['data']['config']['disabled'] = true;
            $field['arguments']['data']['config']['is_disable_inheritance'] = true;
        }
    }
    return $fields;
}

[AFTER CHANGE]

/**
 * Retrieve modified Search Engine Robots metadata
 *
 * Disable Search Engine Robots fields in case when current scope is 'stores'.
 *
 * @param string $scope
 * @param array $fields
 * @return array
 */
private function getSearchEngineRobotsMetadata($scope, array $fields = [])
{
    if ($scope == \Magento\Store\Model\ScopeInterface::SCOPE_STORES) {
        $resetToDefaultsData = [
            'arguments' => [
                'data' => [
                    'config' => [
                        'disabled' => true,
                        'is_disable_inheritance' => true,
                    ],
                ],
            ],
        ];
        $fields = array_merge($fields, ['reset_to_defaults' => $resetToDefaultsData]);
        foreach ($fields as &$field) {
            $field['arguments']['data']['config']['disabled'] = false;
            $field['arguments']['data']['config']['is_disable_inheritance'] = true;
        }
    }
    return $fields;
}

your code should now look like the above. {Warning} making changes is not recommended to the Magento core. If you do make a change revert it after saving the value to the desired configuration.

Fix Two (Recommended)

Don't use the default store view to create your store! Create a new store view.

Configure as needed, as all options are enabled by default. {Unlike the default store view}.

Go to admin panel

Content

Configuration

locate

Global Main Website

Click Edit

Configure your site here!

I hope this helps... Many thanks to all contributors as you have helped me so much over the years, I hope this help you all and gives a little back.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.