what is wrong with robots.txt this file?

Question

Hi Please check my Robots file. Many things are blocked, I don't think we should add Disallow: /

User-agent: *
Crawl-delay: 10
Allow: /
#Disallow: /

######################################
######## Goole Shopping Images #######
######################################

User-agent: Googlebot
Disallow:

User-agent: AdsBot-Google
Disallow:

User-agent: Googlebot-Image
Disallow:

#######################################
############### SITEMAP ###############
#######################################

Sitemap: https://www.hifix.co.uk/sitemap.xml


#######################################
################ PAGES ################
#######################################


#######################################
##### MAGENTO DIRECTORIES & FILES #####
#######################################

##### Directories ##### 

Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /lib/
Disallow: /magento/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /stats/
Disallow: /var/

##### Paths (clean URLs) #####

Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product/gallery/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

##### Files #####

Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt

#######################################
######## QUERY STRING BLOCKER #########
#######################################


#Disallow: /*?*


#######################################
########### SCREAMING FROG ############
#######################################

User-agent: Screaming Frog SEO Spider
Allow: /
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /*.png$
Disallow: /*.bmp$
Disallow: /*.xml$
Disallow: /*.css$
Disallow: /*.js$

Please suggest, what is wrong with this file?

Don't know who added Disallow:

I think disallow should not be in here as I want Googlebot to index images and pages

Answer 1

Here is what I have for the top of my robots.txt file

The crawl delay on yours may need to also be put to 30. This is ignored by Googlebot, but other bots follow it. Allowing Google to go first.

#****************************************************************************
# robots.txt
#     : Robots, spiders, and search engines use this file to detmine which 
#       content they should *not* crawl while indexing your website.
#     : This system is called "The Robots Exclusion Standard."
#     : It is strongly encouraged to use a robots.txt validator to check
#       for valid syntax before any robots read it!
#
# Examples:
#
# Instruct all robots to stay out of the admin area.
#     : User-agent: *
# Crawl-delay: 30
#     : Disallow:   /admin/
#
# Restrict Google and MSN from indexing your images.
#     : User-agent: Googlebot
# Crawl-delay: 30
#     : Disallow:   /images/
#     : User-agent: MSNBot
# Crawl-delay: 30
#     : Disallow:   /images/
#****************************************************************************

## robots.txt for Magento Community and Enterprise

## GENERAL SETTINGS

## Enable robots.txt rules for all crawlers
User-agent: *

## Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
## Set a custom crawl rate if you're experiencing traffic problems with your server.
Crawl-delay: 30