Robots.txt Introduction bots and spiders crawling the web

[NilsC 0]

Search engines look in the "root" directory for robots.txt. This file first tell the spider / bot (called "User-agent" from now on) what files it can harvest and the folders it can harvest from. This is called "The Robots Exclusion Standard".

 

The format (syntax) of the robots.txt file has to be followed. It consists of records that have 2 fields. The first is the "User-agent Line" the second is one or more "Disallow Line(s)"

 

Syntax is <field> ":" <value>

 

You should create the text in UNIX line ender mode. Good text editors or one of the Linux line editors work.

 

***WARNING***

Do not use your HTML editor to create the robots.txt field unless it have a "text mode" to edit in. Notepad can be used if your FTP client have

First we talk about the "User Agents"

 

We have all seen "Googlebot" who hangs out here enough to be a Moderator so we use that as an example

The useragent line specifies the bot that this record is for:

 

User-agent: Googlebot

 

or you can specify all bots/spiders using wildcard "*":

 

User-agent: *

 

There are lists of all known User-agents on the net and you can check your log files to find User-agents that hit your site.

 

Now comes the second part of the record, the "Disallow"

 

It may have one or more lines depending on how much restriction you need. It's called "Disallow" because you allow all useragents to harvest all files and folders unless you specify otherwise. This example tells useragents that they are not allowed to harvest from the file my_email.html:

 

Disallow: my_email.html

 

To tell the useragents that a directory is off limit you have to use this format:

 

Disallow: /cgi-bin/

 

Now useragents (conforming bots and spiders) will stay out of that directory.

 

Can I use wildcards when setting up the rules? In a way you can because if you use:

 

Disallow: /private

 

This would block /private.html and /private/index.html and any other files receding in the /private/ folder. Don't put disallow: alone on a line, because the blank space is interpreted as "good to go, no restrictions" unless that is what you are trying to do.

 

This brings us to the next step because we as "professional" web designers put comments on everything we do so we can figure out what was going on when we wrote this file. You can put comments in the robots/txt file by starting the comment with #. One word of caution, put the comment on a line by itself because a lot of useragents don't interpret white-space. The:

"Disallow: /images/ #donât need to see my wedding pictures".

Will not stop all useragents from harvesting from the /images/ directory because they read this as:

"Disallow: /image/#donâtneedtoseemyweddingpictures"

and will go on into the /images/ directory. So put the comment on the next line after the disallow statement.

 

A couple of examples to round out this tutorial:

 

In my web-site here at Xisto I use the simplest form of the robots.txt file because I use the web-site for testing concepts and don't need it harvested. You can see what it looks like here: http://forums.xisto.com/no_longer_exists/

 

#First exampleUser-agent: *Disallow: /#This keeps all robots out (the setting I use now)#Second exampleUser-agent: *Disallow:#This allows the useragent to harvest all files and folders.#Third exampleUser-agent: *Disallow: /images/Disallow: /cgi-bin/Disallow: /my_mail.html#This will block all boots from the files and folders inside#the images and cgi-bin folder. The file my_mail.html will#not be harvested.#Fourth exampleUser-agent: AnthillDisallow: /pricelist/#This will block the Anthill useragent from accessing your#pricelist folder and all files in that directory. (Anthill#is used to gather price-information automatically from online#stores. Support for international versions.)#Fifth exampleUser-agent: linklookerDisallow: /#This is a new bot, that is not registered so who know what#the data is collected for.

List on known bots and spiders with detailed information and description.

 

If you are allowing some bots but not all then make sure you list the bots you allow first then the deny.

#Allow googlebot, msnbot, and askjeeves to harvest all files and folders.User-agent: googlebotDisallow:User-agent: msnbotdisallow:User-agent: askjeevesDisallow:#The rest of the bots and spiders are blockedUser-agent: *Disallow: /

This are a few examples of how a simple robots.txt file is setup, there are a lot of complicated configurations out there and even some sites that have put the useragent and disallow statement backward.

 

There are a lot of good tutorials on robots.txt and you can do a search on Google to find them. Here is a link to TheBigCrawl â checking robots.txt files for errors.

If you took the time to create a robots.txt file, take the time to check it "Robots.txt checker

 

Hope this is to some help.

 

Nils

[OpaQue 15]

Also, MSN and GOOGLE Both support their own types of custom robots.txt commands which their crawler can recognise.One such example is wildcards. Google Bot understands then and can be used to ban a certain types of files or even URLs.You can checkout MSN Search and go to webmaster section to configure the robots.txt. There they have given instructions to configure it for MSN crawler. [ PS : This is a WONDERFUL TUTORIAL ]

[szupie 0]

Nice tutorial! I have a few questions:1. Do all bots read this file?2. If they do, do they all follow what you said in the file? or can they break the rules and ignore you file?3. Why not HTML editors?

[NilsC 0]

Do all bots read this file, Yes and No, all compliant bots and spiders read this file.

Same answer, if they are compliant they follow the rules.

Unless the HTML editor have a "text edit feature" that put "Unix line endings" at the end of the file bots and spiders may not be able to read. (EditPad Lite, freeware have a setting for this)

All bots are not like, there are bots / spiders that are not conforming. One example that I hate, they are called "spam harvesters" they do not follow the rules and will continue down all your pages collecting email addresses. That is why I'm a member of the "HoneyPot" project. Since conforming bots/spiders follow the rules and dont harvest from a file/directory that is on the "Disallow" list you know that only "BAD" bots will harvest data from the spamtrap. Looking in the logfiles you can now identify new User-Agents that are bad or identify themselves as users(browsers) An example is one of the most active spamharvesters that operate theit bots from hijacked computers in Netherland, their bots/spiders identify themself as "Netscape 3.0 Compatible (WhatsNew Robot)"and you can see details here Spam Harvest bot

 

Hope this helped.

Nils

[ewc21 0]

I wonder if putting the following in your HEAD tag will help<meta name="robots" content="follow, index">or should you just use the robots.txt on your root folder.Anyone?

[spacewaste1405241471 0]

I'm wondering what if i put my robot file inside the directory /examplewould it read only what is inside example...or would it read my whole site still?

[syhs89 0]

robots.txt is suppose to put in root folder...it might dun do the trick...and may i know other robots name like yahoo.com?what if i just wanna block googlebot?is itUser-agent: googlebotDisallow:/User-agent: *Disallow:????

[m3ch4 0]

Search engines look in the "root" directory for robots.txt. This file first tell the spider / bot (called "User-agent" from now on) what files it can harvest and the folders it can harvest from. This is called "The Robots Exclusion Standard".

qucik question: root directory refers to where you place your "index.html" file, correct?

[NilsC 0]

you can have index.html in any folder, Your root folder on this server is "public_html" and that's where the robots.txt file goes.

Yahoo bot is identified as "Inktomi Slurp",
http://forums.xisto.com/no_longer_exists/
https://help.yahoo.com/kb/search/SLN22600.html?impressions=true
and there is a list of bots here.

http://forums.xisto.com/no_longer_exists/

Nils

[m3ch4 0]

you can have index.html in any folder, Your root folder on this server is "public_html" and that's where the robots.txt file goes.

I'm sorry, I need a bit more clairification here ><'' (m3ch4 = newb =P)

if I create a site (not done yet, but below will be the basic map right now) and it looks like this...

system482.astahost.com (main page)
-includes "index.html"

->links to->system482.astahost.com/video_games/
-includes "index.html"
->links to->system482.astahost.com/personal/
-includes "index.html"
->links to->system482.astahost.com/blah_blah_blah/
-includes "index.html"

where is "public_html" exsist?

[NilsC 0]

When you are hosted you get access to cPanel, in cPanel there are several folders for the different options. When you upload your website, you upload it to a folder called "Public_html" this folder is also called your root html folder. So you put all the files hat belongs to this domain name "system482.astahost.com" in here, and when you get the account there is a default file (and others) called index.html in this folder. Anything you put in this folder is hotlinked to your www folder so you can get to your site by typeng

http://system482.astahost.com/or'>http://system482.astahost.com/or http://system482.astahost.com/index.htmlor'>http://system482.astahost.com/index.htmlor http://forums.xisto.com/no_longer_exists/ 
or http://forums.xisto.com/no_longer_exists/ 
index.html

There are private folder in there that you can only see if you are logged into cPanel or by ftp.

The robots.txt file goes in that directory "public_html" and you will see your site on the web.
spiders and robots search your

http://system482.astahost.com/

directory for a file called robots.txt and if you put it in a subdirectory it's ignored. so putting a robots.txt files in the video_games/ subdirectory will get it ignored.

Hope this helps.

Nils

[m3ch4 0]

Hope this helps.

 

Nils

<{POST_SNAPBACK}>

Answers everything!

 

Thanks a ton man! (I've yet to make my way through the Xisto cpanel, I have approval already, but I have a few other things to deal with before I can go full tilt on my site =S)

[mitchellmckain 0]

If you disallow everythingUser-agent: *Disallow: /this does not block the web crawlers from your index.html, right? It just blocks them from indexing the directory contents, right?