Robots.txt

[osknockout 0]

Robots.txt is a file to keep robots from indexing
certain portions of the site.

I was looking at Xisto's robot.txt, here's a copy:

[/br]User-agent: *[br]Disallow: /stylesheets/[/br][br]User-agent: Googlebot[br]Disallow: /stylesheets/[/br]

One question. If you specify '*', then why would one need to
specify the Googlebot?

[alperuzi 0]

maybe its just redundant, or maybe some bots don't recognize the asterisk

[OpaQue 15]

Oh.. I was restricting google from indexing some files and then there was not any need. I simply forgot to remove that thing. However, I think I kept it because if I need to add somethign else, then I wont have to search everything back and look for the exact syntax. In short.. I was lazy :rolleyes:But Google does recognise more complex sets of instruction at robots.txt file than other bots. Examples are special wildcards which allows to filter urls .. etc.

[bjrn 0]

Oh.. I was restricting google from indexing some files and then there was not any need.

I simply forgot to remove that thing. However, I think I kept it because if I need to add somethign else, then I wont have to search everything back and look for the exact syntax. In short.. I was lazy

41302[/snapback]

Laziness is good. If you're lazy you'll always keep handy things around

 

I thought I could post a link to a quite nice robots.txt generator. It's quite simple; you enter whether bots are allowed or not, and then you can enter allowed/refused for a number of common bots if you want them to be different from the default. And then you can enter specific restricted directories.

 

Nothing complex, but nice. And there are examples and links to pre-made robots.txt files.

[osknockout 0]

Alright, just checking... I found almost the same thing at quite a few othersites like cnn.com, microsoft.com, and howstuffworks.comAre there any robots that don't recognize the asterisk?

[bjrn 0]

Alright, just checking... I found almost the same thing at quite a few other

sites like cnn.com, microsoft.com, and howstuffworks.com

 

Are there any robots that don't recognize the asterisk?

41438[/snapback]

As far as I know Googlebot is the only bot that recognises "*.gif" and such. Other bots don't understand the asterix.