Do Google Crawlers Index Our Sites?

[TheJeffsta 0]

Do the Google crawlers pick up info about all/most of the websites hosted here? If they do how frequently do they scan through the servers? Do all hosted sites include a robot.txt thingy or .htaccess (forgot what one google looks at)?

lol, my last site got on it but then my host started stuffing up my site so i stopped using it.

This post has been edited by microscopic^earthling: Today, 07:41 PM

What was changed? Edited August 27, 2005 by TheJeffsta (see edit history)

[szupie 0]

Crawlers crawl through links of sites they have already indexed, so Xisto links to your site, then Google should be able to crawl to your site. If you don't want them to, you can specify it in the "robot.txt thingy" , but you have to write it yourself, it is not included by default. I think they go through a site once a day or something, but I think you can set it to something else.