0

I think i have a problem with robots.txt in magento 1.9.3.4

In Search Console at Index Status of https version i see this: enter image description here

Is it a issue? and if yes, how can i solve it?

http: enter image description here

Robots.txt

# Website Sitemap
Sitemap: https://domain/sitemap.xml
 
# Crawlers Setup
User-agent: *
#Crawl-delay: 10
 
# Allowable Index
# Mind that Allow is not an official standard
Allow: /*?p=
Allow: /index.php/blog/
Allow: /catalog/seo_sitemap/category/
# Allow: /catalogsearch/result/
Allow: /media/catalog/
 
# Directories
Disallow: /404/
Disallow: /app/
 
Disallow: /cgi-bin/
Disallow: /downloader/
 
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
# Disallow: /media/
Disallow: /media/captcha/
# Disallow: /media/catalog/
#Disallow: /media/css/
#Disallow: /media/css_secure/
Disallow: /media/customer/
Disallow: /media/dhl/
Disallow: /media/downloadable/
Disallow: /media/import/
#Disallow: /media/js/
Disallow: /media/pdf/
Disallow: /media/sales/
Disallow: /media/tmp/
Disallow: /media/wysiwyg/
Disallow: /media/xmlconnect/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
#Disallow: /skin/
Disallow: /stats/
Disallow: /var/
 
# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product/gallery/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
 
# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /get.php # Magento 1.5+
 
# Paths (no clean URLs)
#Disallow: /*.js$
#Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?SID=
Disallow: /rss*
Disallow: /*PHPSESSID
2
  • Can you add a copy of your robots.txt to check what urls are you blocking, have into mind if you are blocking non-seo urls but at the same time you are serving content with non-seo urls then the amount of page blocked will be a lot for bots Oct 20 '17 at 6:15
  • i don't know if it safe to provide a link to site here
    – Pantelis
    Oct 20 '17 at 10:56
1

this line will block crawlers to get your images from cms pages and blocks: Disallow: /media/wysiwyg/

and if you have a page the search terms page and/or a cloud of most search terms this will also cause many blocks to crawlers: Disallow: /catalogsearch/

All the rest looks fine, about the url, it's fine, just check that your product and category pages have SEO friendly urls (something like my-nice-product.html instead of .../catalog/product/view/... or /women.html instead of .../catalog/category/view/...)

2
  • I think it is ok, right? diapasonhifi.gr so.... i must to comment this lines, i understand right? #Disallow: /media/wysiwyg/ #Disallow: /catalogsearch/
    – Pantelis
    Oct 20 '17 at 11:09
  • Yes Pantelis, comment first line, all the rest looks fine and you haven't a search terms page or cloud so it's fine Disallow: /catalogsearch/ Oct 20 '17 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.