We’re rewarding the question askers & reputations are being recalculated! Read more.
18

You can configure the option by following below path in Magento 2.x admin. Stores -> Configuration -> General (Left side) -> Design (section) -> Search Engine Robots -> Default Robots (Select)-> Select INDEX, FOLLOW NOINDEX, FOLLOW INDEX, NOFOLLOW NOINDEX, NOFOLLOW Save the configuration as you need. You can configure this option by [...


12

BOT & SESSION I'm not sure Bot needs sessions and the answer may be different for every bot for sure they don't store a cookie so it is no possible to re-use the old session matching the session ID stored in the client cookie. How other E-Commerce fix this Prestashop folks revolved the problem in this way: Assigning to the bots the old sessions using ...


7

Writing robots.txt is tricky. Robots defines the architecture of your website for crawlers. Normally i use the following robots for my website # Website Sitemap Sitemap: www.example.com/sitemap.xml User-agent: Googlebot-Image Disallow: / Allow: /media/catalog/product/ # Crawlers Setup User-agent: * # Directories User-agent: * Disallow: /app/ Disallow: /...


6

I reckon Inchoo gives a good overview as well, with different options and examples to learn from Inchoo’s recommended Magento robots.txt boilerplate: # Google Image Crawler Setup User-agent: Googlebot-Image Disallow: # Crawlers Setup User-agent: * # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /errors/ ...


4

It seems your Nginx or Apache is set differently. Try locating the file in var directory. If you have SSH access, run below command to see if there is a robots file on server. Check if there is a robots file on server locate -c robots.txt If the result found is greater than 0, it means the file exists on the server. Next, you can run below command to find ...


3

Simply, create a file robots.txt in the root path of your installation like this: # Sitemap Sitemap: http://example.com/sitemap.xml # Google Images User-agent: Googlebot-Image Disallow: / Allow: /media/catalog/product/ Allow: /media/wysiwyg/ # Crawlers Setup User-agent: * # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /...


3

To prevent products from being crawled by Google you can set the meta robots to NOINDEX,NOFOLLOW. If you want to do this for all Simple products in your catalog you can add this to your local.xml file: <PRODUCT_TYPE_simple> <reference name="head"> <action method="setRobots"><value>NOINDEX,NOFOLLOW</value></action&...


3

Basically that's it. I do something similar to change the type of banner rotator on some special category pages and the entries look like this.


3

See solution from here as copy below to fix your issue and prevent bots crawling through your layered navigation Denying layered navigation for crawlers and fix SEO issues caused by the huge number of layered navigation URLs can be done by using PRG Pattern. This works like a charm, i. e. not changing the UX regarding Layered Navigation and 100%...


3

Edit cms page in that Design > Layout Update XML add like below: <head><meta name="robots" content="NOINDEX,NOFOLLOW"/></head>


3

Remove them from robots.txt, and instead make sure that they are not even accessible from the web, because they should not! Files that can be removed: /install.php /LICENSE.html /LICENSE.txt /LICENSE_AFL.txt /STATUS.txt Files that should be protected /cron.php /cron.sh /error_log (if it exists) If you use Apache with the latest default .htaccess file ...


2

Some technical notes. Empty lines According to the original robots.txt specification, records must be separated by (at least) one empty line, and records must not contain empty lines. So this means that you should add a new line after Baiduspider’s Disallow: / (the first record ends there): User-agent: Baiduspider Disallow: / User-agent: * Disallow: /...


2

The site should remain in NOINDEX mode until you're absolutely ready to go live. In my experience a site being indexed by Google early cannot hurt you other than to bring traffic that is simply unable to checkout to your store. That can cause bad perception issues in the future. Additionally there are a bunch of factors that go into your pagerank including ...


2

If Google does not crawl your site, nobody find you :) It is vise to restrict some areas like 1. layered navigation 2. search and advanced search 3. paginated content Also you can disallow some user agents (bots and crawlers) in the .htaccess file. Taken from stackoverflow: You need to have mod_rewrite enabled. Placed it in .htaccess in that folder. If ...


2

Creating and configuring robots.txt is very easy. You can create it with simple text editor. find the best example here http://www.robotstxt.org/robotstxt.html that how to create robots.txt file. Once you have created Robots.txt file it is supposed to submit at root directory of your webstore. For an example if your store domain is www.example.com you ...


2

There are dozens of different bots and not all of them play nice, but you have handful of tools at your disposal: robots.txt has crawl-delay command. This makes sure that crawler waits at least specified number of seconds before making next query to the server. Google Webmaster Tool provides an option to set crawl rate. Obviously it only works for Googlebot....


2

To block those kind of URLs with robots.txt you could use the following lines: Disallow: /*?dir=* Disallow: /*&dir=* Disallow: /*?mode=* Disallow: /*&mode=* Disallow: /*?order=* Disallow: /*&order=* Disallow: /*?p=* Disallow: /*&p=* This will tell user-agents not to access all URLs which contain ascending/decending sorting, grid/list mode, ...


2

The code you have in your robots.txt file is spot on. Submitting the urls in your Google Webmaster Tools account for removal from search index will be the best way to prevent them showing. There are a number of other urls and prefixes you need to think about too. The below is more or less optimum Magento robots.txt file default entries: # This file is to ...


2

in 2.1.12 this is now managed from Content > Design > Configuration > edit Global (or per store) > Search engine robots > Default Robots


2

You should try something like this because what you did is too restrictive. User-agent: Yandex Disallow: / User-agent: baiduspider Disallow: / User-agent: AhrefsBot Disallow: / User-agent: BLEXBot Disallow: / User-Agent: *


2

robots.txt dont work as expected anymore. use something simple like in server config: APACHE: RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot|Yahoo) [NC] RewriteCond %{QUERY_STRING} ^clothing_size=([0-9]*) [NC] RewriteRule .* - [G] NGINX: if ($http_user_agent ~* "Baiduspider|Googlebot|bingbot|Yahoo|YandexBot") { set $layered A; } if ($...


1

You can still use layout updates for this as in Magento1 but the XML is a little different. For insance, in the admin, go to a CMS Page. Then go to the design tab, in the layout updates add this: <head> <meta name="robots" content="NOINDEX,NOFOLLOW"/> </head> This will update the robots tag for the current page. You can do the same ...


1

You can find information about robots.txt directives in the SEO Best Practices section of the Magento user guide. Topics include: Using a Sitemap, Sitemap Configuration, and Search Engine Robots. Our user guides are written for merchants, with a focus on the Admin and storefront. Stop by magento.com and see Resources > Documentation for quick links to ...


1

The official docs are your friend. On the admin sidebar, tap Stores. Then under Settings, choose Configuration. In the panel on the left under General, choose Design. Expand the Search Engine Robots section. Enter your values in the text box or click Reset to Default to auto-populate Magento's recommendations. Flush your caches.


1

Fra,you have save magento session data at database,You need to change it to files from local.xml(app/etc/) by <session_save><![CDATA[files]]></session_save> from <session_save><![CDATA[db]]></session_save> That will resolved issue.


1

If you installed Magento 1.x in the shop directory, you can use the robots.txt from this answer which you already found and prefix all those entries with /shop. i.e. /app/ => /shop/app/ However, this message seems to indicate a different problem: Fail Disallow: /app/ - Allowed (doesn't work) It seems like you have a different application installed at ...


1

while robots.txt is good, but i guess search index will be maintained as it is. so you have to login to google webmaster tools and remove those urls, just if you need them gone. also do not count on robots.txt and robot's good behavior, add something like this to your .htaccess: RewriteEngine on RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot|Yahoo|...


1

I reckon Inchoo gives a good overview as well, with different options and examples to learn from Inchoo’s recommended Magento robots.txt boilerplate: # Google Image Crawler Setup User-agent: Googlebot-Image Disallow: # Crawlers Setup User-agent: * # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /...


1

It's not a bot. This means that you have yandex metrika added to your source code. This is the same as google analytics. It could be that some extension is adding it to your magento's output Look for this piece in your source code: new Ya.Metrika You can search through the files of your magento installation for this piece of code to find which extension ...


1

All very good suggestions and we have come up with a solution that works very well. The challenge is to tell Google not to do something without upsetting its algorithm to much. Basically we want to slow down Google without being penalized in the end. Good points about the search functionality on the site, this is for customers and definitely not for bots. ...


Only top voted, non community-wiki answers of a minimum length are eligible