Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [robots.txt]

The tag has no usage guidance.

0
votes
1answer
27 views

Magento 2.3.0 robots.txt file location in server directory

I am trying to delete a disallow condition from robots.txt file. This is not located in root or /pub directories. I can't change it from admin panel because seems like Magento 2.3.0 has a bug on ...
0
votes
0answers
15 views

magento 2 robots.txt file is not creating in root directory after enabling settings

Recently we have deployed the website to live and after when SEO ask to put the robots.txt file in root directory for search engine crawl. We have checked setting in Content > Design > ...
0
votes
1answer
23 views

How to set meta robots

I want to change my Magento 1 site so that all of the products pages are Noindex, follow. Does anyone know how to do this please?
0
votes
0answers
20 views

robots.txt and sitemap.xml site.webmanifest

Hello im having touble to load files robots.txt and sitemap.xml site.webmanifest form https://example.com/robots.txt for example, since magento root directory is set under /pub through nginx.conf....
0
votes
1answer
57 views

Google still crawls url while i disallow from robots

I will go straight to the point. Recently I did several fixes in my robots.txt file, and I've added several disallows like these: Disallow: /review/ Disallow: /catalog/product/view/ the problem is ...
0
votes
1answer
29 views

Where in the database is robots.txt content saved?

According to M2 docs. You do not have to generate a robots.txt because it generates on demand and stores the contents in the database. It does not create a file, but you can view the content in your ...
0
votes
0answers
38 views

No Index For Layered Navigation on Magento

I'm trying to make no indexed my layered navigation pages on my magento store. I follow this guide http://trafficmotion.com/blog/index.php/2014/03/20/noindexing-filtered-product-listing-magento/ and ...
1
vote
1answer
274 views

How to detect and Modify robots.txt inside Magento

How do I modify and ensure that the magento site has robots.txt please? I am checking if the site has robots.txt as it is being flagged by our SEo tool . see screenshot - http://prntscr.com/k07bvl ...
2
votes
1answer
242 views

what is wrong with robots.txt this file?

Hi Please check my Robots file. Many things are blocked, I don't think we should add Disallow: / User-agent: * Crawl-delay: 10 Allow: / #Disallow: / ###################################### ######## ...
0
votes
1answer
219 views

Magento 2 : Favicon icon is not showing for Magento root txt files

I have manually created robots.txt file and pasted it in the Magento root folder. when I search for robots.txt file in the browser, It's not showing favicon icon. I thought I have missed something, ...
2
votes
2answers
622 views

Physical location of robots.txt file in Magento 2.2

Where can I find the location of robots.txt file in the Magento root? Checked pub folder also. Any one helps.
0
votes
2answers
196 views

Robots.txt for Magento 2.1+ - where did it go?

Where is the actual robots.txt file like Magento 1.9.x if it exists? My site URL /robots.txt does not work. The official dev docs for this http://docs.magento.com/m2/ce/user_guide/marketing/search-...
0
votes
1answer
34 views

unable to remove some codes from robots.txt

I have a weird problem. I am able to update my robots.txt however for some reason, whenever I click on www.mysite.com/robots.txt, the following code would always be at the end: User-agent: * Disallow:...
1
vote
1answer
1k views

Magento 2 created robots.txt/sitemap.xml in `http/` but not `pub/`

I have created a robots.txt and sitemap.xml files using Magento UI but when trying to access www.mydomain.com/robots.txt or www.mydomain.com/sitemap.xml it gives me 404 not found. I found that robots....
0
votes
1answer
3k views

Magento 1.9 robots.txt seo

I think i have a problem with robots.txt in magento 1.9.3.4 In Search Console at Index Status of https version i see this: Is it a issue? and if yes, how can i solve it? http: Robots.txt # ...
0
votes
1answer
37 views

Traffic Bandwidth Issues

I'm having several issues with my site, I checked AWStats and it is showing that robots have used 18gb bandwidth this month (usual for other months is 150mb). I was told by my server hosts to alter ...
1
vote
3answers
79 views

Removing cron.php, cron.sh, … from robots.txt - reasons, and is there *any* influence on the shop somehow?

We are currently evaluating to optimize our robots.txt file. Our SEO Manager requested to remove specific stuff that Magento puts there by default. Those values are: -# Files -Disallow: /cron.php -...
1
vote
1answer
356 views

Magento2: Set NOINDEX, NOFOLLOW for 404 page not found

How to set NOINDEX, NOFOLLOW for 404 page. I've added below xml from admin no-route page. <reference name="head"> <action method="setRobots"><value>NOINDEX,NOFOLLOW</...
1
vote
2answers
302 views

googlebot and layered navigation

I am having trouble with googlebot crawling through our layered navigation. I get numerous logs like: GET /shop/dresses?clothing_size=250&color=247&dir=desc&limit=15&mode=list&...
1
vote
1answer
177 views

robots.txt and website path

We are running magento but we need to have the Store View Code in our url i.e: www.mydomain.com/shop/ if shop is our store view code. Do I need to reflect this in our robots.txt. For example at the ...
0
votes
1answer
2k views

How to correctly setup robots.txt to block out bad bots but allow Google, Yahoo, Bing and other good bots to access and index the site?

We were having some problems with performance a few months ago and my hosting company told me this: "You have also many hits from all kinds of bots, and when they have too many requests at the ...
0
votes
2answers
272 views

Magento Website Crawl Issue - A description for this result is not available because of this site's robots.txt

I am working on this site and some how (I don't know why) Google stopped crawling the home page URL (http://www.indusuno.com/) When I search for the site in Google, it shows "A description for this ...
0
votes
1answer
19 views

multi site development site showing in google

I have been testing a multi store environment on Magento 1.9 in the same install that I have my primary live site, and have realized that the content is being indexed twice by Google. Can I create a ...
1
vote
1answer
755 views

“magento2/sitemap.xml” is always and automatically added to my robots.txt

I don't know how and why I find the following in my robots.txt Sitemap: http://www.[mySite].com/magento2/sitemap.xml worse the link opens to a "404 page not found". I have tried to remove it from ...
2
votes
6answers
9k views

Meta noindex, follow in Magento 2.1

It is commonly recommended to use "Meta noindex, follow" over robots.txt to disallow pages from being indexed by google ... for example http://www.magestore.com/magento-2-tutorial/most-extreme-magento-...
0
votes
1answer
217 views

multisite magento and robots.txt

In robots.txt can be only one directive host. How right use directive host in robots.txt in magento multisite configuration with many domains?
1
vote
0answers
135 views

Site search spam

Our site has been receiving lots of spammy searches, for random keywords, and these results have started to show in Google: I have found that sites are linking to these searches: Our robots.txt ...
0
votes
1answer
232 views

Webmaster tool robots.txt fetch failed, fresh magento 2 web site

Problem is google webmaster tools cant fetch my magento2 web site, My backend settings, I want to use secure base url so that reason I chanced base url as a https, but I think it doesn't matter ...
4
votes
2answers
2k views

Web Crawler creates a lot of sessions

I recently noted that the table core_session is growing very fast ( +400MB ) I'm quite sure this is something abnormal. I have truncated it but after a couple of days, It is again very big. If I ...
1
vote
0answers
82 views

Managing out of control robot sessions with Redis

I've been having major issues with robots leaving sessions files. Over 100 sessions / second. At some point Redis just stopped accepting the sessions and Magento went to writing them back to disk. ...
0
votes
1answer
119 views

Robots.txt - Magento not in root folder

I'm trying to update my robots.txt file for magento installed in a folder public_html/shop/. I've updated my robots.txt and .htaccess file based on solutions provided on below post: Google Ignore'...
0
votes
2answers
1k views

Google Ignore's Magento's robots.txt

I have the following in my robots.txt for months now: Disallow: /catalogsearch/ Disallow: /catalogsearch/* Disallow: /webshop/catalogsearch/* Disallow: /webshop/catalogsearch/ My webshop url ...
0
votes
1answer
924 views

How is my robots.txt? [duplicate]

Since I haven't ever made a robots.txt before, I am little nervous submitting one for a fully working webshop. Does anyone see anything critical in the following, or should I be able to submit it as ...
1
vote
1answer
336 views

Yandex bot in my website

I have noticed that when I'm surfing my website, where it appears the Waiting for www.whatever.com... on the left-bottom corner from google chrome it appears suddenly a Waiting for mc.yandex.com... ...
1
vote
3answers
1k views

High CPU load due to Google crawling search terms

I'm experiencing a much higher than normal CPU load on my server since a few days mainly caused by excessive search term crawling by Google on my internal Magento search engine. This almost ...
0
votes
1answer
292 views

Duplicate title tags on non crawl pages

favorite This question already has an answer here: Duplicate title tags in Google Webmaster Tools 1 answer I have set up my robot.txt file so that google doesn't crawl limit pages (how many products ...
1
vote
1answer
348 views

magento shop down by bots crawling

I have a problem with my magento store. Every now and then when a bot crawls my shop, the website goes down. I have made changes to robots.txt and .htacces. Also varnish is installed and a full page ...
1
vote
2answers
2k views

robots.txt allow crawling of category/product pages and js/css/images

I'm new to configuring robots.txt and trying to understand how to best configure for our Magento EE install. Right now, I have the following: ### bots should have full access to category/product url-...
0
votes
2answers
1k views

Properly setting up the robots.txt for magento

I want to remove the pages from search engine by using robots.txt to not to index such pages. The repeated URL's Containing the same content appear as website.com/category.html?dir=asc&mode=list&...
3
votes
3answers
129 views

Should I temporarily turn Indexing and Bots off?

My Magento store has been live on my server for around 10 days. The website isn't quite ready and will take another week before all of the pages, categories and products are what I want. After ...
0
votes
1answer
143 views

where to add meta tag to stop robots/spiders in magento site

In magento site, 32 gb is wasted by "Not viewed traffic includes traffic generated by robots, worms, or replies with special HTTP status codes." Googlebot alone is using 29.41 gb. I wanted to avoid ...
0
votes
3answers
248 views

Google feed verse Google Crawling

We have our simple products listed at catalog in order to have them available in our Google shopping feed. Is there a way to remove these products from being crawled by google bot so they don't show ...
0
votes
1answer
676 views

How to prevent Magento admin url being indexed from Google?

Is there any safe way apart from the robots.txt to prevent the Magento admin url being indexed by Google.
0
votes
2answers
690 views

Creating robots dot text

I am currently trying to create a robots dot text file to improve my SEO. I am not quite sure were to start. I have read a few tutorials and stay don't feel comfortable making one. Can someone point ...
1
vote
2answers
4k views

Magento robots.txt for multi store

I am planning to use the following robots.txt: ## robots.txt for Store   ## GENERAL SETTINGS   ## Enable robots.txt rules for all crawlers User-agent: *   ## Crawl-delay parameter: number of seconds ...
4
votes
1answer
5k views

NOINDEX,NOFOLLOW for selected category pages

I'm reading this, and it says that you can add NOINDEX,NOFOLLOW directives using page handles on files like local.xml. What I want to do is to include these directives for a certain selected set of ...
0
votes
2answers
1k views

Need advice for my robots.txt file

I've noticed my indexed pages are dropping from Google like a rock. I'm reviewing everything that was changed in the last month. I noticed in my Google Webmaster Tools that there were some ...
7
votes
4answers
17k views

How do I configure Robots.txt in Magento?

I'm trying to setting up de robots.txt on our Magento store. Actually, I have no idea of the best way to set up the robots.txt.