Questions tagged [robots.txt]
The robots.txt tag has no usage guidance.
62
questions
0
votes
1
answer
13
views
How to add specific categories in robots.txt file in magento 2
I want to stop some categories from being crawled. So i decided to add the category paths in robots.txt file. But i am not sure about the syntax. How can i achieve this. Thanks
- 131
1
vote
0
answers
67
views
Remove search results indexed pages on Google
Google Search console is indexing thousands of Magento search results like these :
How do I get rid of these pages and avoid Google to show it in the search results ?
Will avoiding these URLs in ...
- 11
1
vote
2
answers
136
views
Adobe Magento Cloud how to add robots.txt
What are the best ways to add the robots.txt file to adobe magento cloud environment
- 588
0
votes
0
answers
20
views
Magento 2: How to prevent product meta description and tags in sitemap?
I am setting up the robots.txt for sitemap and I want to restrict the product meta description and meta tags for the web crawlers but I don't know which instructions I need to put in robots.txt file. ...
- 512
0
votes
1
answer
38
views
Conflict with google indexing
Google Search throws a problem with a link that is not being indexing
I search the cause and is for a rule of robotx.txt that block the catalog
Disallow: /catalog/
So since the link path starts with ...
2
votes
3
answers
597
views
How can I Set Up & Configure Robots.txt in Magento 2?
I'm trying to set up the robots.txt on our Magento store. Actually, I do not really have any idea how to do that, so I need the help of any SEO or an experienced person who can help me with this.
- 416
0
votes
1
answer
54
views
Blocking .ru Bots
I've been getting a number of bots with an email that ends with .ru - I was hoping to stop them from creating customer accounts on our system.
I've seen a few SQL functions that will remove existing ...
1
vote
0
answers
151
views
Magento 2: Custom Instructions of Robots.txt is not working
We are facing an issue and need your guidance to solve that. The issue is that we have updated the custom instructions for robots.txt file as below
User-agent:*
Disallow:
But in the frontend, its ...
- 512
2
votes
1
answer
78
views
Hits from google bot to urls ending with /customer/customer
When we tail -f access.log, we can see a lot of PLP requests coming from googlebot that contains customer/customer/contactus in it. For example requests are coming like below
catalog/product/view/id/...
- 223
0
votes
1
answer
111
views
Magento 1 remove Disallow: /tag/ in robot.txt
My robots.txt contain any code and this code
Disallow: /tag/
Is it okay for SEO to delete this command?
- 49
0
votes
1
answer
119
views
robots.txt Disallow: /checkout/ still myweb.com/checkout/cart/ Google still crawls
robots.txt
Disallow: /checkout/
still, myweb.com/checkout/cart/ Google still crawls,
I have set robots from admin.
Thank You!
- 81
3
votes
0
answers
415
views
Magento 2.3.2. Multistore robots.txt wrong sitemap URL
A client has a Magento 2.3.2 multistore with 5 stores. For convenience I name them store 1/5. The store has created 5 sitemaps in magentoroot/sitemap/{storename}/sitemap.xml.
The robots.txt I assume ...
- 343
0
votes
1
answer
575
views
Magento 2.3 robots.txt file
Situation is: we have no robots.txt file in this project. It is not in root folder. It is not in /pub folder. But... If we browse ourdomain/robots.txt it is loaded & printed (containing just the ...
- 3,144
0
votes
1
answer
130
views
Magento 2 Sign Static Files Showing CSS/JS 404 Error in SEO Reports
I am using Version Number In File Paths in Magento 2, but it's showing 404 errors in SEO reports after run Magento deploy command because of deploy command change version number every time
I have ...
- 1,045
0
votes
1
answer
33
views
Magneto 1.9: checkout strange behavior
Very often I observe customers adding to cart then moving to the checkout then disappearing. It looks like a bot or maybe competitors searching for our products and trying to hit us with fake ...
- 867
1
vote
1
answer
380
views
How we can noindex,nofollow the category-product-url?
I want to noindex,nofollow only those product urls in which category is coming in url.
For example : I have product abc.html and it is assigned to 2 categories so it can open by three ways.
1. ...
- 1,234
0
votes
2
answers
2k
views
Magento 2.3.0 robots.txt file location in server directory
I am trying to delete a disallow condition from robots.txt file. This is not located in root or /pub directories.
I can't change it from admin panel because seems like Magento 2.3.0 has a bug on ...
1
vote
0
answers
2k
views
magento 2 robots.txt file is not creating in root directory after enabling settings
Recently we have deployed the website to live and after when SEO ask to put the robots.txt file in root directory for search engine crawl.
We have checked setting in Content > Design > ...
- 2,877
0
votes
1
answer
185
views
How to set meta robots
I want to change my Magento 1 site so that all of the products pages are Noindex, follow.
Does anyone know how to do this please?
0
votes
1
answer
580
views
Google still crawls url while i disallow from robots
I will go straight to the point.
Recently I did several fixes in my robots.txt file, and I've added several disallows like these:
Disallow: /review/
Disallow: /catalog/product/view/
the problem is ...
0
votes
1
answer
335
views
Where in the database is robots.txt content saved?
According to M2 docs.
You do not have to generate a robots.txt because it generates on demand and stores the contents in the database. It does not create a file, but you can view the content in your ...
- 3,390
1
vote
1
answer
872
views
How to detect and Modify robots.txt inside Magento
How do I modify and ensure that the magento site has robots.txt please?
I am checking if the site has robots.txt as it is being flagged by our SEo tool . see screenshot - http://prntscr.com/k07bvl
...
- 11
2
votes
1
answer
948
views
what is wrong with robots.txt this file?
Hi Please check my Robots file. Many things are blocked, I don't think we should add Disallow: /
User-agent: *
Crawl-delay: 10
Allow: /
#Disallow: /
######################################
######## ...
0
votes
2
answers
1k
views
Magento 2 : Favicon icon is not showing for Magento root txt files
I have manually created robots.txt file and pasted it in the Magento root folder. when I search for robots.txt file in the browser, It's not showing favicon icon. I thought I have missed something, ...
- 1,124
4
votes
2
answers
5k
views
Location of robots.txt file in Magento 2.2
Where can I find the location of robots.txt file in the Magento root?
Checked pub folder also.
- 3,884
0
votes
2
answers
1k
views
Robots.txt for Magento 2.1+ - where did it go?
Where is the actual robots.txt file like Magento 1.9.x if it exists?
My site URL /robots.txt does not work.
The official dev docs for this http://docs.magento.com/m2/ce/user_guide/marketing/search-...
- 550
0
votes
1
answer
69
views
unable to remove some codes from robots.txt
I have a weird problem. I am able to update my robots.txt however for some reason, whenever I click on www.mysite.com/robots.txt, the following code would always be at the end:
User-agent: *
Disallow:...
- 341
1
vote
2
answers
3k
views
Magento 2 created robots.txt/sitemap.xml in `http/` but not `pub/`
I have created a robots.txt and sitemap.xml files using Magento UI but when trying to access www.mydomain.com/robots.txt or www.mydomain.com/sitemap.xml it gives me 404 not found.
I found that robots....
- 2,889
0
votes
1
answer
4k
views
Magento 1.9 robots.txt seo
I think i have a problem with robots.txt in magento 1.9.3.4
In Search Console at Index Status of https version i see this:
Is it a issue? and if yes, how can i solve it?
http:
Robots.txt
# Website ...
- 109
0
votes
1
answer
67
views
Traffic Bandwidth Issues
I'm having several issues with my site, I checked AWStats and it is showing that robots have used 18gb bandwidth this month (usual for other months is 150mb).
I was told by my server hosts to alter ...
- 1
1
vote
3
answers
137
views
Removing cron.php, cron.sh, ... from robots.txt - reasons, and is there *any* influence on the shop somehow?
We are currently evaluating to optimize our robots.txt file. Our SEO Manager requested to remove specific stuff that Magento puts there by default.
Those values are:
-# Files
-Disallow: /cron.php
-...
- 1,824
1
vote
1
answer
918
views
Magento2: Set NOINDEX, NOFOLLOW for 404 page not found
How to set NOINDEX, NOFOLLOW for 404 page.
I've added below xml from admin no-route page.
<reference name="head">
<action method="setRobots"><value>NOINDEX,NOFOLLOW</...
- 1,737
1
vote
3
answers
2k
views
googlebot and layered navigation
I am having trouble with googlebot crawling through our layered navigation. I get numerous logs like:
GET /shop/dresses?clothing_size=250&color=247&dir=desc&limit=15&mode=list&...
- 407
1
vote
1
answer
251
views
robots.txt and website path
We are running magento but we need to have the Store View Code in our url i.e:
www.mydomain.com/shop/ if shop is our store view code.
Do I need to reflect this in our robots.txt.
For example at the ...
- 407
0
votes
1
answer
2k
views
How to correctly setup robots.txt to block out bad bots but allow Google, Yahoo, Bing and other good bots to access and index the site?
We were having some problems with performance a few months ago and my hosting company told me this:
"You have also many hits from all kinds of bots, and when they have too many requests at the ...
- 164
0
votes
2
answers
626
views
Magento Website Crawl Issue - A description for this result is not available because of this site's robots.txt
I am working on this site and some how (I don't know why) Google stopped crawling the home page URL (http://www.indusuno.com/)
When I search for the site in Google, it shows "A description for this ...
- 11
0
votes
1
answer
49
views
multi site development site showing in google
I have been testing a multi store environment on Magento 1.9 in the same install that I have my primary live site, and have realized that the content is being indexed twice by Google. Can I create a ...
- 11
2
votes
2
answers
1k
views
"magento2/sitemap.xml" is always and automatically added to my robots.txt
I don't know how and why I find the following in my robots.txt
Sitemap: http://www.[mySite].com/magento2/sitemap.xml
worse the link opens to a "404 page not found".
I have tried to remove it from ...
- 423
8
votes
8
answers
24k
views
Meta noindex, follow in Magento 2.1
It is commonly recommended to use "Meta noindex, follow" over robots.txt to disallow pages from being indexed by google ... for example http://www.magestore.com/magento-2-tutorial/most-extreme-magento-...
- 81
0
votes
1
answer
667
views
multisite magento and robots.txt
In robots.txt can be only one directive host.
How right use directive host in robots.txt in magento multisite configuration with many domains?
- 41
2
votes
0
answers
365
views
Site search spam
Our site has been receiving lots of spammy searches, for random keywords, and these results have started to show in Google:
I have found that sites are linking to these searches:
Our robots.txt ...
- 21
1
vote
2
answers
347
views
Webmaster tool robots.txt fetch failed, fresh magento 2 web site
Problem is google webmaster tools cant fetch my magento2 web site,
My backend settings,
I want to use secure base url so that reason I chanced base url as a https, but I think it doesn't matter ...
5
votes
1
answer
4k
views
Web Crawler creates a lot of sessions
I recently noted that the table core_session is growing very fast ( +400MB ) I'm quite sure this is something abnormal.
I have truncated it but after a couple of days, It is again very big.
If I ...
- 6,945
1
vote
0
answers
152
views
Managing out of control robot sessions with Redis
I've been having major issues with robots leaving sessions files. Over 100 sessions / second. At some point Redis just stopped accepting the sessions and Magento went to writing them back to disk. ...
- 11
1
vote
1
answer
261
views
Robots.txt - Magento not in root folder
I'm trying to update my robots.txt file for magento installed in a folder public_html/shop/.
I've updated my robots.txt and .htaccess file based on solutions provided on below post:
Google Ignore'...
- 13
1
vote
2
answers
2k
views
Google Ignore's Magento's robots.txt
I have the following in my robots.txt for months now:
Disallow: /catalogsearch/
Disallow: /catalogsearch/*
Disallow: /webshop/catalogsearch/*
Disallow: /webshop/catalogsearch/
My webshop url ...
- 11
0
votes
1
answer
1k
views
How is my robots.txt? [duplicate]
Since I haven't ever made a robots.txt before, I am little nervous submitting one for a fully working webshop. Does anyone see anything critical in the following, or should I be able to submit it as ...
- 319
1
vote
1
answer
551
views
Yandex bot in my website
I have noticed that when I'm surfing my website, where it appears the Waiting for www.whatever.com... on the left-bottom corner from google chrome it appears suddenly a Waiting for mc.yandex.com... ...
- 317
1
vote
3
answers
2k
views
High CPU load due to Google crawling search terms
I'm experiencing a much higher than normal CPU load on my server since a few days mainly caused by excessive search term crawling by Google on my internal Magento search engine. This almost ...
- 463
0
votes
1
answer
300
views
Duplicate title tags on non crawl pages
favorite
This question already has an answer here:
Duplicate title tags in Google Webmaster Tools 1 answer
I have set up my robot.txt file so that google doesn't crawl limit pages (how many products ...
- 2,205