Jump to content
xisto Community
Sign in to follow this  
GM-University1405241492

MSN Help Problem

Recommended Posts

I have a problem with MSN search, I was under the impression that it followed link on my site, but for some reason it is finding pages that don't exsist on my forums, or really wierd ones, it hasn't read any posts, it just checks the login and registration pages and all sorts of wierd things...I am sure that the group that the bots are treated as have acess to the forums to read them, but they still don't check posts, it just comes every day and see's if my users long-in differently now... :)

Share this post


Link to post
Share on other sites

Here is my robots.txt for my Vbulletin forum:

User-agent: * Disallow: /cgi-bin/ Disallow: /forums/attachment.php Disallow: /forums/avatar.php Disallow: /forums/editpost.php Disallow: /forums/member.php Disallow: /forums/member2.php Disallow: /forums/misc.php Disallow: /forums/moderator.php Disallow: /forums/newreply.php Disallow: /forums/newthread.php Disallow: /forums/online.php Disallow: /forums/poll.php Disallow: /forums/postings.php Disallow: /forums/printthread.php Disallow: /forums/private.php Disallow: /forums/private2.php Disallow: /forums/report.php Disallow: /forums/search.php Disallow: /forums/sendtofriend.php Disallow: /forums/threadrate.php Disallow: /forums/usercp.php Disallow: /forums/admin/ Disallow: /forums/images/ Disallow: /forums/mod/ Disallow: /admin/ Disallow: /cache/ Disallow: /albums/ Disallow: /gallery/ Disallow: /downloads/ Disallow: /images/ Disallow: /includes/ Disallow: /language/ Disallow: /manuel/ Disallow: /modules/ Disallow: /netpbm/ Disallow: /tmp/ Disallow: admin.php Disallow: auth.inc.php Disallow: banners.php Disallow: config.php Disallow: counter.php Disallow: download.php Disallow: footer.php Disallow: friend.php Disallow: header.php Disallow: index.php Disallow: links.php Disallow: mainfile.php Disallow: modules.php Disallow: reviews.php Disallow: sections.php Disallow: stats.php Disallow: top.php Disallow: topics.php Disallow: user.php Disallow: vbp_includes.php Disallow: vbpolls.php Disallow: viewtopic.php Disallow: voteinclude.php

Might help you set yours up for IPB.
Yes I have most of it disallowed but what is not disallowed the spider can crawl

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×
×
  • Create New...

Important Information

Terms of Use | Privacy Policy | Guidelines | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.