Questions tagged [robots.txt]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
2 answers
171 views

How can I have different robots.txt files for each store in a Magento installation with multiple stores?

I have a Magento installation with multiple stores, and I need to have different robots.txt files for each store. Currently, I have a single robots.txt file in the root directory of my Magento ...
Vipin Maurya's user avatar
0 votes
1 answer
35 views

How to add specific categories in robots.txt file in magento 2

I want to stop some categories from being crawled. So i decided to add the category paths in robots.txt file. But i am not sure about the syntax. How can i achieve this. Thanks
Mani's user avatar
  • 383
1 vote
1 answer
125 views

Remove search results indexed pages on Google

Google Search console is indexing thousands of Magento search results like these : How do I get rid of these pages and avoid Google to show it in the search results ? Will avoiding these URLs in ...
bnjmandre's user avatar
1 vote
3 answers
474 views

Adobe Magento Cloud how to add robots.txt

What are the best ways to add the robots.txt file to adobe magento cloud environment
Mohamed Abbas's user avatar
0 votes
1 answer
45 views

Conflict with google indexing

Google Search throws a problem with a link that is not being indexing I search the cause and is for a rule of robotx.txt that block the catalog Disallow: /catalog/ So since the link path starts with ...
ramon colmenares's user avatar
2 votes
3 answers
1k views

How can I Set Up & Configure Robots.txt in Magento 2?

I'm trying to set up the robots.txt on our Magento store. Actually, I do not really have any idea how to do that, so I need the help of any SEO or an experienced person who can help me with this.
Kuldeep Sharma's user avatar
0 votes
1 answer
67 views

Blocking .ru Bots

I've been getting a number of bots with an email that ends with .ru - I was hoping to stop them from creating customer accounts on our system. I've seen a few SQL functions that will remove existing ...
Jeffrey Marks's user avatar
1 vote
0 answers
183 views

Magento 2: Custom Instructions of Robots.txt is not working

We are facing an issue and need your guidance to solve that. The issue is that we have updated the custom instructions for robots.txt file as below User-agent:* Disallow: But in the frontend, its ...
MagentoDev's user avatar
2 votes
1 answer
99 views

Hits from google bot to urls ending with /customer/customer

When we tail -f access.log, we can see a lot of PLP requests coming from googlebot that contains customer/customer/contactus in it. For example requests are coming like below catalog/product/view/id/...
Manaf P M's user avatar
  • 223
0 votes
1 answer
124 views

Magento 1 remove Disallow: /tag/ in robot.txt

My robots.txt contain any code and this code Disallow: /tag/ Is it okay for SEO to delete this command?
user3485709's user avatar
0 votes
1 answer
124 views

robots.txt Disallow: /checkout/ still myweb.com/checkout/cart/ Google still crawls

robots.txt Disallow: /checkout/ still, myweb.com/checkout/cart/ Google still crawls, I have set robots from admin. Thank You!
Danes's user avatar
  • 81
3 votes
0 answers
460 views

Magento 2.3.2. Multistore robots.txt wrong sitemap URL

A client has a Magento 2.3.2 multistore with 5 stores. For convenience I name them store 1/5. The store has created 5 sitemaps in magentoroot/sitemap/{storename}/sitemap.xml. The robots.txt I assume ...
JerGrun's user avatar
  • 343
0 votes
1 answer
680 views

Magento 2.3 robots.txt file

Situation is: we have no robots.txt file in this project. It is not in root folder. It is not in /pub folder. But... If we browse ourdomain/robots.txt it is loaded & printed (containing just the ...
Raul Sanchez's user avatar
  • 3,276
0 votes
1 answer
147 views

Magento 2 Sign Static Files Showing CSS/JS 404 Error in SEO Reports

I am using Version Number In File Paths in Magento 2, but it's showing 404 errors in SEO reports after run Magento deploy command because of deploy command change version number every time I have ...
Banna Online's user avatar
  • 1,059
0 votes
1 answer
35 views

Magneto 1.9: checkout strange behavior

Very often I observe customers adding to cart then moving to the checkout then disappearing. It looks like a bot or maybe competitors searching for our products and trying to hit us with fake ...
Rob D. A.'s user avatar
  • 867
1 vote
1 answer
424 views

How we can noindex,nofollow the category-product-url?

I want to noindex,nofollow only those product urls in which category is coming in url. For example : I have product abc.html and it is assigned to 2 categories so it can open by three ways. 1. ...
Magento Dev's user avatar
  • 1,264
0 votes
2 answers
2k views

Magento 2.3.0 robots.txt file location in server directory

I am trying to delete a disallow condition from robots.txt file. This is not located in root or /pub directories. I can't change it from admin panel because seems like Magento 2.3.0 has a bug on ...
Khaled Bin Amir's user avatar
1 vote
0 answers
2k views

magento 2 robots.txt file is not creating in root directory after enabling settings

Recently we have deployed the website to live and after when SEO ask to put the robots.txt file in root directory for search engine crawl. We have checked setting in Content > Design > ...
akgola's user avatar
  • 2,917
0 votes
1 answer
207 views

How to set meta robots

I want to change my Magento 1 site so that all of the products pages are Noindex, follow. Does anyone know how to do this please?
abdul wahab's user avatar
0 votes
1 answer
693 views

Google still crawls url while i disallow from robots

I will go straight to the point. Recently I did several fixes in my robots.txt file, and I've added several disallows like these: Disallow: /review/ Disallow: /catalog/product/view/ the problem is ...
akalogiros's user avatar
0 votes
1 answer
419 views

Where in the database is robots.txt content saved?

According to M2 docs. You do not have to generate a robots.txt because it generates on demand and stores the contents in the database. It does not create a file, but you can view the content in your ...
vitoriodachef's user avatar
1 vote
1 answer
927 views

How to detect and Modify robots.txt inside Magento

How do I modify and ensure that the magento site has robots.txt please? I am checking if the site has robots.txt as it is being flagged by our SEo tool . see screenshot - http://prntscr.com/k07bvl ...
Tony's user avatar
  • 11
2 votes
1 answer
973 views

what is wrong with robots.txt this file?

Hi Please check my Robots file. Many things are blocked, I don't think we should add Disallow: / User-agent: * Crawl-delay: 10 Allow: / #Disallow: / ###################################### ######## ...
Rachna Thakur's user avatar
0 votes
2 answers
2k views

Magento 2 : Favicon icon is not showing for Magento root txt files

I have manually created robots.txt file and pasted it in the Magento root folder. when I search for robots.txt file in the browser, It's not showing favicon icon. I thought I have missed something, ...
Siva's user avatar
  • 1,120
4 votes
2 answers
5k views

Location of robots.txt file in Magento 2.2

Where can I find the location of robots.txt file in the Magento root? Checked pub folder also.
sandip's user avatar
  • 3,954
1 vote
2 answers
1k views

Robots.txt for Magento 2.1+ - where did it go?

Where is the actual robots.txt file like Magento 1.9.x if it exists? My site URL /robots.txt does not work. The official dev docs for this http://docs.magento.com/m2/ce/user_guide/marketing/search-...
camdixon's user avatar
  • 560
0 votes
1 answer
73 views

unable to remove some codes from robots.txt

I have a weird problem. I am able to update my robots.txt however for some reason, whenever I click on www.mysite.com/robots.txt, the following code would always be at the end: User-agent: * Disallow:...
James Tsai's user avatar
1 vote
2 answers
3k views

Magento 2 created robots.txt/sitemap.xml in `http/` but not `pub/`

I have created a robots.txt and sitemap.xml files using Magento UI but when trying to access www.mydomain.com/robots.txt or www.mydomain.com/sitemap.xml it gives me 404 not found. I found that robots....
Greg's user avatar
  • 2,929
0 votes
1 answer
4k views

Magento 1.9 robots.txt seo

I think i have a problem with robots.txt in magento 1.9.3.4 In Search Console at Index Status of https version i see this: Is it a issue? and if yes, how can i solve it? http: Robots.txt # Website ...
Pantelis's user avatar
  • 109
0 votes
1 answer
71 views

Traffic Bandwidth Issues

I'm having several issues with my site, I checked AWStats and it is showing that robots have used 18gb bandwidth this month (usual for other months is 150mb). I was told by my server hosts to alter ...
sarah's user avatar
  • 1
1 vote
3 answers
138 views

Removing cron.php, cron.sh, ... from robots.txt - reasons, and is there *any* influence on the shop somehow?

We are currently evaluating to optimize our robots.txt file. Our SEO Manager requested to remove specific stuff that Magento puts there by default. Those values are: -# Files -Disallow: /cron.php -...
Max's user avatar
  • 1,834
1 vote
1 answer
939 views

Magento2: Set NOINDEX, NOFOLLOW for 404 page not found

How to set NOINDEX, NOFOLLOW for 404 page. I've added below xml from admin no-route page. <reference name="head"> <action method="setRobots"><value>NOINDEX,NOFOLLOW</...
Kul's user avatar
  • 1,777
1 vote
3 answers
2k views

googlebot and layered navigation

I am having trouble with googlebot crawling through our layered navigation. I get numerous logs like: GET /shop/dresses?clothing_size=250&color=247&dir=desc&limit=15&mode=list&...
Pragman's user avatar
  • 407
1 vote
1 answer
253 views

robots.txt and website path

We are running magento but we need to have the Store View Code in our url i.e: www.mydomain.com/shop/ if shop is our store view code. Do I need to reflect this in our robots.txt. For example at the ...
Pragman's user avatar
  • 407
0 votes
1 answer
3k views

How to correctly setup robots.txt to block out bad bots but allow Google, Yahoo, Bing and other good bots to access and index the site?

We were having some problems with performance a few months ago and my hosting company told me this: "You have also many hits from all kinds of bots, and when they have too many requests at the ...
Allysin's user avatar
  • 250
0 votes
2 answers
661 views

Magento Website Crawl Issue - A description for this result is not available because of this site's robots.txt

I am working on this site and some how (I don't know why) Google stopped crawling the home page URL (http://www.indusuno.com/) When I search for the site in Google, it shows "A description for this ...
Indus Uno's user avatar
0 votes
1 answer
49 views

multi site development site showing in google

I have been testing a multi store environment on Magento 1.9 in the same install that I have my primary live site, and have realized that the content is being indexed twice by Google. Can I create a ...
Brian Maher's user avatar
2 votes
2 answers
2k views

"magento2/sitemap.xml" is always and automatically added to my robots.txt

I don't know how and why I find the following in my robots.txt Sitemap: http://www.[mySite].com/magento2/sitemap.xml worse the link opens to a "404 page not found". I have tried to remove it from ...
Whaaaaat's user avatar
  • 423
8 votes
8 answers
25k views

Meta noindex, follow in Magento 2.1

It is commonly recommended to use "Meta noindex, follow" over robots.txt to disallow pages from being indexed by google ... for example http://www.magestore.com/magento-2-tutorial/most-extreme-magento-...
abutmah's user avatar
  • 81
0 votes
1 answer
672 views

multisite magento and robots.txt

In robots.txt can be only one directive host. How right use directive host in robots.txt in magento multisite configuration with many domains?
Boulatov's user avatar
2 votes
0 answers
410 views

Site search spam

Our site has been receiving lots of spammy searches, for random keywords, and these results have started to show in Google: I have found that sites are linking to these searches: Our robots.txt ...
Andy Speed's user avatar
1 vote
2 answers
386 views

Webmaster tool robots.txt fetch failed, fresh magento 2 web site

Problem is google webmaster tools cant fetch my magento2 web site, My backend settings, I want to use secure base url so that reason I chanced base url as a https, but I think it doesn't matter ...
Whatatimetobealive's user avatar
5 votes
1 answer
4k views

Web Crawler creates a lot of sessions

I recently noted that the table core_session is growing very fast ( +400MB ) I'm quite sure this is something abnormal. I have truncated it but after a couple of days, It is again very big. If I ...
Fra's user avatar
  • 6,965
1 vote
0 answers
160 views

Managing out of control robot sessions with Redis

I've been having major issues with robots leaving sessions files. Over 100 sessions / second. At some point Redis just stopped accepting the sessions and Magento went to writing them back to disk. ...
innerspace's user avatar
1 vote
1 answer
284 views

Robots.txt - Magento not in root folder

I'm trying to update my robots.txt file for magento installed in a folder public_html/shop/. I've updated my robots.txt and .htaccess file based on solutions provided on below post: Google Ignore'...
user5699516's user avatar
1 vote
2 answers
2k views

Google Ignore's Magento's robots.txt

I have the following in my robots.txt for months now: Disallow: /catalogsearch/ Disallow: /catalogsearch/* Disallow: /webshop/catalogsearch/* Disallow: /webshop/catalogsearch/ My webshop url ...
TechnicalJoe's user avatar
0 votes
1 answer
1k views

How is my robots.txt? [duplicate]

Since I haven't ever made a robots.txt before, I am little nervous submitting one for a fully working webshop. Does anyone see anything critical in the following, or should I be able to submit it as ...
user2806026's user avatar
1 vote
1 answer
581 views

Yandex bot in my website

I have noticed that when I'm surfing my website, where it appears the Waiting for www.whatever.com... on the left-bottom corner from google chrome it appears suddenly a Waiting for mc.yandex.com... ...
danibg's user avatar
  • 317
1 vote
3 answers
2k views

High CPU load due to Google crawling search terms

I'm experiencing a much higher than normal CPU load on my server since a few days mainly caused by excessive search term crawling by Google on my internal Magento search engine. This almost ...
Badger's user avatar
  • 463
0 votes
1 answer
305 views

Duplicate title tags on non crawl pages

favorite This question already has an answer here: Duplicate title tags in Google Webmaster Tools 1 answer I have set up my robot.txt file so that google doesn't crawl limit pages (how many products ...
Adam Allen's user avatar
  • 2,205