Jump to content
xisto Community
Sign in to follow this  
osknockout

Robots.txt

Recommended Posts

Robots.txt is a file to keep robots from indexing
certain portions of the site.

I was looking at Xisto's robot.txt, here's a copy:

[/br]User-agent: *[br]Disallow: /stylesheets/[/br][br]User-agent: Googlebot[br]Disallow: /stylesheets/[/br]

One question. If you specify '*', then why would one need to
specify the Googlebot?

Share this post


Link to post
Share on other sites

Oh.. I was restricting google from indexing some files and then there was not any need. I simply forgot to remove that thing. However, I think I kept it because if I need to add somethign else, then I wont have to search everything back and look for the exact syntax. In short.. I was lazy :rolleyes:But Google does recognise more complex sets of instruction at robots.txt file than other bots. Examples are special wildcards which allows to filter urls .. etc.

Share this post


Link to post
Share on other sites

Oh.. I was restricting google from indexing some files and then there was not any need.

I simply forgot to remove that thing. However, I think I kept it because if I need to add somethign else, then I wont have to search everything back and look for the exact syntax. In short.. I was lazy :rolleyes:

41302[/snapback]

Laziness is good. If you're lazy you'll always keep handy things around :)

 

I thought I could post a link to a quite nice robots.txt generator. It's quite simple; you enter whether bots are allowed or not, and then you can enter allowed/refused for a number of common bots if you want them to be different from the default. And then you can enter specific restricted directories.

 

Nothing complex, but nice. And there are examples and links to pre-made robots.txt files.

Share this post


Link to post
Share on other sites

Alright, just checking... I found almost the same thing at quite a few othersites like cnn.com, microsoft.com, and howstuffworks.comAre there any robots that don't recognize the asterisk?

Share this post


Link to post
Share on other sites

Alright, just checking... I found almost the same thing at quite a few other

sites like cnn.com, microsoft.com, and howstuffworks.com

 

Are there any robots that don't recognize the asterisk?

41438[/snapback]

As far as I know Googlebot is the only bot that recognises "*.gif" and such. Other bots don't understand the asterix.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×
×
  • Create New...

Important Information

Terms of Use | Privacy Policy | Guidelines | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.