1

I have disallowed? SID in my robots.txt but Google still likes to show them in search results.

2 Answers 2

2

You should not put this in your robots.txt. It is better to do this in the search console from Google.

There is a function there for parameters under

Crawl -> URL Parameters -> Add Parameter

Source: Click here

And if you don´t want these to come up even there is this setting also

Go to Admin -> System -> Configuration -> Web -> Session Validation Settings 

disable config Use SID on Frontend = No

This could also be needed in your case to make it work with the .htaccess

RewriteEngine on 
RewriteCond %{HTTP_HOST} !^www\.singhcycle\.com$ [NC]
RewriteRule ^(.*)$ http://www.singhcycle.com/$1 [R=301,L]
4
  • What I find it strange Google bot is respecting the other url parameters its only with ?SID Feb 4, 2016 at 9:48
  • Can you try and see if it works that way? I´ve also made an udpate at the bottom to my answer
    – seb
    Feb 4, 2016 at 14:13
  • This is what search console say for SID."From parameters that Googlebot discovered on your site here is a list of parameters that appear not to change the content of your pages. Google will ignore these. (If you're certain that this is incorrect, you can specify how you want these parameters to be handled.)" I don't want should be the appropriate way to handle is parameter. Feb 4, 2016 at 14:33
  • SID is already added automatically to search console. I think I need to tell how to handle this parameter. Feb 4, 2016 at 14:37
4

The line Disallow: ?SID (or Disallow: /?SID) will block URLs like:

  • http://example.com/?SID
  • http://example.com/?SIDD
  • http://example.com/?SID/foo

But it will not block URLs like:

  • http://example.com/foo?SID
  • http://example.com/foo/?SID

For the Googlebot, you can specify a wildcard (but this is not supported by all bots because it’s not part of the official robots.txt standard):

Disallow: /*?SID

This will block all URLs whose path starts with any string (indicated by *) followed by ?SID.


Your current robots.txt seems to include Disallow: /*?SID=, which would block URLs like http://example.com/foo?SID=bar. So this should work. The only issue I can see in your robots.txt is that you include new lines in a record, which is not allowed.

So instead of

# Crawlers Setup
User-agent: *

# Directories
Disallow: /404/

it would have to be

# Crawlers Setup
User-agent: *
# Directories
Disallow: /404/

(But it’s likely that the Googlebot handles it correctly anyway.)

1
  • I got your point. I would remove the white lines to stick with robots.txt standards. Feb 9, 2016 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.