Questions tagged [robots.txt]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
14 views

Blocking .ru Bots

I've been getting a number of bots with an email that ends with .ru - I was hoping to stop them from creating customer accounts on our system. I've seen a few SQL functions that will remove existing ...
0
votes
0answers
15 views

SEO settings for magento 2.4

In google I see my store title have the following design: Store name: more text I've tried to look for where this colon is but can't find it or change it to something else. Where is this done? Also ...
0
votes
0answers
11 views

Commerce Edition - Website Restrictions Enabled and robots.txt

Is it expected functionality that a website with restriction mode set to "Website Closed" will cause the router to exceed it's 100 loop limit when trying to reach /robots.txt? I'm seeing ...
1
vote
0answers
34 views

Magento 2: Custom Instructions of Robots.txt is not working

We are facing an issue and need your guidance to solve that. The issue is that we have updated the custom instructions for robots.txt file as below User-agent:* Disallow: But in the frontend, its ...
2
votes
1answer
49 views

Hits from google bot to urls ending with /customer/customer

When we tail -f access.log, we can see a lot of PLP requests coming from googlebot that contains customer/customer/contactus in it. For example requests are coming like below catalog/product/view/id/...
0
votes
1answer
23 views

Magento 1 remove Disallow: /tag/ in robot.txt

My robots.txt contain any code and this code Disallow: /tag/ Is it okay for SEO to delete this command?
0
votes
1answer
63 views

robots.txt Disallow: /checkout/ still myweb.com/checkout/cart/ Google still crawls

robots.txt Disallow: /checkout/ still, myweb.com/checkout/cart/ Google still crawls, I have set robots from admin. Thank You!
2
votes
0answers
197 views

Magento 2.3.2. Multistore robots.txt wrong sitemap URL

A client has a Magento 2.3.2 multistore with 5 stores. For convenience I name them store 1/5. The store has created 5 sitemaps in magentoroot/sitemap/{storename}/sitemap.xml. The robots.txt I assume ...
0
votes
1answer
138 views

Magento 2.3 robots.txt file

Situation is: we have no robots.txt file in this project. It is not in root folder. It is not in /pub folder. But... If we browse ourdomain/robots.txt it is loaded & printed (containing just the ...
0
votes
1answer
36 views

Magento 2 Sign Static Files Showing CSS/JS 404 Error in SEO Reports

I am using Version Number In File Paths in Magento 2, but it's showing 404 errors in SEO reports after run Magento deploy command because of deploy command change version number every time I have ...
0
votes
1answer
29 views

Magneto 1.9: checkout strange behavior

Very often I observe customers adding to cart then moving to the checkout then disappearing. It looks like a bot or maybe competitors searching for our products and trying to hit us with fake ...
1
vote
1answer
175 views

How we can noindex,nofollow the category-product-url?

I want to noindex,nofollow only those product urls in which category is coming in url. For example : I have product abc.html and it is assigned to 2 categories so it can open by three ways. 1. ...
0
votes
2answers
1k views

Magento 2.3.0 robots.txt file location in server directory

I am trying to delete a disallow condition from robots.txt file. This is not located in root or /pub directories. I can't change it from admin panel because seems like Magento 2.3.0 has a bug on ...
1
vote
0answers
1k views

magento 2 robots.txt file is not creating in root directory after enabling settings

Recently we have deployed the website to live and after when SEO ask to put the robots.txt file in root directory for search engine crawl. We have checked setting in Content > Design > ...
0
votes
1answer
118 views

How to set meta robots

I want to change my Magento 1 site so that all of the products pages are Noindex, follow. Does anyone know how to do this please?
0
votes
1answer
329 views

Google still crawls url while i disallow from robots

I will go straight to the point. Recently I did several fixes in my robots.txt file, and I've added several disallows like these: Disallow: /review/ Disallow: /catalog/product/view/ the problem is ...
0
votes
1answer
115 views

Where in the database is robots.txt content saved?

According to M2 docs. You do not have to generate a robots.txt because it generates on demand and stores the contents in the database. It does not create a file, but you can view the content in your ...
1
vote
1answer
718 views

How to detect and Modify robots.txt inside Magento

How do I modify and ensure that the magento site has robots.txt please? I am checking if the site has robots.txt as it is being flagged by our SEo tool . see screenshot - http://prntscr.com/k07bvl ...
2
votes
1answer
671 views

what is wrong with robots.txt this file?

Hi Please check my Robots file. Many things are blocked, I don't think we should add Disallow: / User-agent: * Crawl-delay: 10 Allow: / #Disallow: / ###################################### ######## ...
0
votes
1answer
828 views

Magento 2 : Favicon icon is not showing for Magento root txt files

I have manually created robots.txt file and pasted it in the Magento root folder. when I search for robots.txt file in the browser, It's not showing favicon icon. I thought I have missed something, ...
3
votes
2answers
3k views

Physical location of robots.txt file in Magento 2.2

Where can I find the location of robots.txt file in the Magento root? Checked pub folder also. Any one helps.
0
votes
2answers
1k views

Robots.txt for Magento 2.1+ - where did it go?

Where is the actual robots.txt file like Magento 1.9.x if it exists? My site URL /robots.txt does not work. The official dev docs for this http://docs.magento.com/m2/ce/user_guide/marketing/search-...
0
votes
1answer
41 views

unable to remove some codes from robots.txt

I have a weird problem. I am able to update my robots.txt however for some reason, whenever I click on www.mysite.com/robots.txt, the following code would always be at the end: User-agent: * Disallow:...
1
vote
2answers
3k views

Magento 2 created robots.txt/sitemap.xml in `http/` but not `pub/`

I have created a robots.txt and sitemap.xml files using Magento UI but when trying to access www.mydomain.com/robots.txt or www.mydomain.com/sitemap.xml it gives me 404 not found. I found that robots....
0
votes
1answer
4k views

Magento 1.9 robots.txt seo

I think i have a problem with robots.txt in magento 1.9.3.4 In Search Console at Index Status of https version i see this: Is it a issue? and if yes, how can i solve it? http: Robots.txt # Website ...
0
votes
1answer
54 views

Traffic Bandwidth Issues

I'm having several issues with my site, I checked AWStats and it is showing that robots have used 18gb bandwidth this month (usual for other months is 150mb). I was told by my server hosts to alter ...
1
vote
3answers
116 views

Removing cron.php, cron.sh, … from robots.txt - reasons, and is there *any* influence on the shop somehow?

We are currently evaluating to optimize our robots.txt file. Our SEO Manager requested to remove specific stuff that Magento puts there by default. Those values are: -# Files -Disallow: /cron.php -...
1
vote
1answer
676 views

Magento2: Set NOINDEX, NOFOLLOW for 404 page not found

How to set NOINDEX, NOFOLLOW for 404 page. I've added below xml from admin no-route page. <reference name="head"> <action method="setRobots"><value>NOINDEX,NOFOLLOW</...
1
vote
2answers
1k views

googlebot and layered navigation

I am having trouble with googlebot crawling through our layered navigation. I get numerous logs like: GET /shop/dresses?clothing_size=250&color=247&dir=desc&limit=15&mode=list&...
1
vote
1answer
221 views

robots.txt and website path

We are running magento but we need to have the Store View Code in our url i.e: www.mydomain.com/shop/ if shop is our store view code. Do I need to reflect this in our robots.txt. For example at the ...
0
votes
1answer
2k views

How to correctly setup robots.txt to block out bad bots but allow Google, Yahoo, Bing and other good bots to access and index the site?

We were having some problems with performance a few months ago and my hosting company told me this: "You have also many hits from all kinds of bots, and when they have too many requests at the ...
0
votes
2answers
487 views

Magento Website Crawl Issue - A description for this result is not available because of this site's robots.txt

I am working on this site and some how (I don't know why) Google stopped crawling the home page URL (http://www.indusuno.com/) When I search for the site in Google, it shows "A description for this ...
0
votes
1answer
27 views

multi site development site showing in google

I have been testing a multi store environment on Magento 1.9 in the same install that I have my primary live site, and have realized that the content is being indexed twice by Google. Can I create a ...
2
votes
1answer
1k views

“magento2/sitemap.xml” is always and automatically added to my robots.txt

I don't know how and why I find the following in my robots.txt Sitemap: http://www.[mySite].com/magento2/sitemap.xml worse the link opens to a "404 page not found". I have tried to remove it from ...
8
votes
8answers
20k views

Meta noindex, follow in Magento 2.1

It is commonly recommended to use "Meta noindex, follow" over robots.txt to disallow pages from being indexed by google ... for example http://www.magestore.com/magento-2-tutorial/most-extreme-magento-...
0
votes
1answer
452 views

multisite magento and robots.txt

In robots.txt can be only one directive host. How right use directive host in robots.txt in magento multisite configuration with many domains?
1
vote
0answers
180 views

Site search spam

Our site has been receiving lots of spammy searches, for random keywords, and these results have started to show in Google: I have found that sites are linking to these searches: Our robots.txt ...
1
vote
2answers
302 views

Webmaster tool robots.txt fetch failed, fresh magento 2 web site

Problem is google webmaster tools cant fetch my magento2 web site, My backend settings, I want to use secure base url so that reason I chanced base url as a https, but I think it doesn't matter ...
5
votes
1answer
3k views

Web Crawler creates a lot of sessions

I recently noted that the table core_session is growing very fast ( +400MB ) I'm quite sure this is something abnormal. I have truncated it but after a couple of days, It is again very big. If I ...
1
vote
0answers
132 views

Managing out of control robot sessions with Redis

I've been having major issues with robots leaving sessions files. Over 100 sessions / second. At some point Redis just stopped accepting the sessions and Magento went to writing them back to disk. ...
1
vote
1answer
218 views

Robots.txt - Magento not in root folder

I'm trying to update my robots.txt file for magento installed in a folder public_html/shop/. I've updated my robots.txt and .htaccess file based on solutions provided on below post: Google Ignore'...
1
vote
2answers
1k views

Google Ignore's Magento's robots.txt

I have the following in my robots.txt for months now: Disallow: /catalogsearch/ Disallow: /catalogsearch/* Disallow: /webshop/catalogsearch/* Disallow: /webshop/catalogsearch/ My webshop url ...
0
votes
1answer
1k views

How is my robots.txt? [duplicate]

Since I haven't ever made a robots.txt before, I am little nervous submitting one for a fully working webshop. Does anyone see anything critical in the following, or should I be able to submit it as ...
1
vote
1answer
457 views

Yandex bot in my website

I have noticed that when I'm surfing my website, where it appears the Waiting for www.whatever.com... on the left-bottom corner from google chrome it appears suddenly a Waiting for mc.yandex.com... ...
1
vote
3answers
2k views

High CPU load due to Google crawling search terms

I'm experiencing a much higher than normal CPU load on my server since a few days mainly caused by excessive search term crawling by Google on my internal Magento search engine. This almost ...
0
votes
1answer
298 views

Duplicate title tags on non crawl pages

favorite This question already has an answer here: Duplicate title tags in Google Webmaster Tools 1 answer I have set up my robot.txt file so that google doesn't crawl limit pages (how many products ...
1
vote
1answer
451 views

magento shop down by bots crawling

I have a problem with my magento store. Every now and then when a bot crawls my shop, the website goes down. I have made changes to robots.txt and .htacces. Also varnish is installed and a full page ...
1
vote
3answers
2k views

robots.txt allow crawling of category/product pages and js/css/images

I'm new to configuring robots.txt and trying to understand how to best configure for our Magento EE install. Right now, I have the following: ### bots should have full access to category/product url-...
0
votes
2answers
2k views

Properly setting up the robots.txt for magento

I want to remove the pages from search engine by using robots.txt to not to index such pages. The repeated URL's Containing the same content appear as website.com/category.html?dir=asc&mode=list&...
3
votes
3answers
141 views

Should I temporarily turn Indexing and Bots off?

My Magento store has been live on my server for around 10 days. The website isn't quite ready and will take another week before all of the pages, categories and products are what I want. After ...