How Searchengines Makes Their Bots?

[itssami 0]

Where as i know, when we submit a website to a search engine, they usually send a BOT to our website , which index all the pages and then adds it to the website.i have a stupid question that how do they make the spider bot i was just thinking about it Is there any other way to add website without a bot ?? Are there some search engines which dont use bots ? and they use some other way ??

[jlhaslip 4]

To find the ways to make a 'bot', use Google. (Google is your friend) I have seen instructions on php programming sites, BUT, they can be quite resource heavy, so I wouldn't use them on a Shared Server such as the Xisto Hosting. They are also prone to having lots of errors, so they can't be depended upon. This is something best left to professional coders. An amatuer Bot will be trouble waiting to happen.As for other methods to getting the available information off the increasing number of Web Locations and indexing them into a Searchable Database, I would think that if you discover a new method, Google would love to hear about it. Good luck in your search.

[Lyon2 0]

itssami, that's not a stupid question at all, very pertinent i may say so.Actually, i'm allways reading seo and search engines ebooks, at least the best ones, and i can't find a resource, ebook or something that can help me learn how to build a spider/boot to crawl the web for pages or links to pages and so on.I would really like help on this issue like itssami, so if someone has good information about this, i would apreciate it very much.

[anwiii 17]

Where as i know, when we submit a website to a search engine, they usually send a BOT to our website , which index all the pages and then adds it to the website.i have a stupid question that how do they make the spider bot i was just thinking about it Is there any other way to add website without a bot ?? Are there some search engines which dont use bots ? and they use some other way ??

search engines normally use bots so you're out of luck there, but directories like yahoo, or dmoz, etc....don't they have live people looking over your website to make sure you found the right directory and your site is up and running with no under construction page.

now although normally all search engines and directories take time to list because when you submit your site, you are last in the que for the bot to spider which takes up to 3 months now i hear since there are more than 3 billion webpages out there now(not sure on exact figure). 10 years ago, it was alot easier and faster to get listed.

now you have two options if you don't want to wait. one is to pay for submission like yahoo(although yahoo isn't a search engine) OR you want to use some strategy in allowing the search engines to find you easily. google is great about listing sites that never was submitted since the bots like to float around alot and find websites they never saw before.

how do you accoplish this second alternative? find someone who is already linked to google. and ask them to link to you. whether a big or small site, doesn't matter as long as that websites links are bot friendly. i recomend everyone to consider google site maps(even though i never used it before). i believe it's created as an .xml file extention? not sure of that so don't quote me. or creat your own site map that is mirrored in tree order according to how you set up your pages and make sure all your pages are linked together.

what this will do is when the bots revisit the website that is now linked to you, it will see your link and follow it. if it follows it and your site hasn't already been submitted, then google will automatically submit your site for you. now some website put the revisit metatag in their head. i believe this is a useless tag as google will revisit when it guesses how often you update for new content. so make sure you link to a page already indexed in google that always has new content every week or two.

i've heard of people getting listed this way the same day. but usually 1-2 weeks if the site linking to you has good content. ALSO....the even better news about this way of getting your site listed is that google loves new web pages it finds and some say(a rumor that i believe is true) will get you a higher listing than if you submitted your site directly to google either by hand or with an autosubmitter.

now Xisto is indexed by google. if you post a link to your site and use your keywords and a good description within your post in the appropriate sub forum, then you wont have to submit your site at all. the search engines will find you just like they will find your other posts that you post on trap 17. i've already seen my posts from her in the search engines accidently....

anyway, this information was given to the best of my knowledge right now. i haven't updated myself in the last 1-2 months with search engine placement and seo, but i'm sure if you take to heart what i've said, you'll have no problem getting listed....BUT! with a bot....but quicker than you would think and faster than alot of other people get listed if you really want to get listed without a bot though, go to yahoo and submit your site.....dmoz or aol. i think inktomi powers aol....but again, i know i might be off in some of my facts here.

wish ya luck with whatever you're trying to accomplish bud.

[Lyon2 0]

anwiii, sorry but i must correct you in somethings you're saying:Yahoo doesn't pay their team to look up the website, that's done with scripts, and in google is the same thing, imagine to visit more then 10 billion web pages, can you imagine how many workers will you need to look up the web page one by one?It matter who your site exchange or even get links from.It's better having 3 links from high popular websites in google that 100 links from not popular websites.

[amhso 0]

as someone said before, bots are really resource hogs. They search millions of sites a day, using a lot of CPU/mem. There isnt much point to making one. Except i remember on the news a few years ago, a kid in college made a more efficient bot/server than google had for a search engine...so i think google hired him.