Adsense Crawler Errors

[Ahsaniqbalkmc 0]

Google adsesn is performing quite well on my website. Most of the time my ecpm is above $5, and I guess it is a good value. Although the niche I am working on doesn't have very relevant ads, but the ones displayed on my website perform average and thus have an average CTR. Recently I am having a little problem with adsense. Though the performance hasn't dropped nor has the relevancy, I am seeing a lot of crawler errors in my adsense account. The worrying part of it is that these errors keep on increasing with every passing day. They are currently about 101. The reason for these errors is stated to be ROBOTS.TXT file however, my ROBOTS file is configured to allow all sorts of bots to the website.I need to know some things about these crawler errors.Number 1: Obviously the first thing I need to know is that why I am getting these errors. There is nothing wrong with my robots.txt file and I am sure about it. I have posted on the official google forum but they didn't reply so I thought the next best place to get my answers is Xisto.Number 2: How can these errors affect the adsense performance on my website. As these errors are keeping on rising, I am worried about the performance of my adsense ads.Number 3: What are the possible options with me to get these errors solved.Thanks in advance to all those who spend their time to read this post and answer it. Check for my website in the signature.

[mahesh2k 0]

Few things to clarify, never disclose your CTR and eCPM on any public forum. The reason behind it is that mentioned in google TOS. So if you do that then they're likely to cancel your account. The reason they can find you is because you're using sites as signatures here and they can trace it. They have teams monitoring the term adsense ctr and ecpm. So i hope you don't publish your ecpm and ctr from now on. Crawler errors are usually because of the translator bot or the bot which failed to catch usercontent. Robots.txt from your site is likely to be fine but the issue is with their crawler. You can go to your adsense account and explicitly accept these crawlers access to your adsense ID sites. This way the error that causes this bug will likely to vanish. Crawler errors occur every now and then to almost every website. So it is beneficial for you to make sure that you don't get them everyday or too often. If you get it then there is something wrong on your server. Usercontent crawler is not issue as it only fetches the content and has nothing to do with ads. Transate bot also has nothing to do with your content and adsense earnings. You can do much without these and can simply ignore them as well. If these are the bots which are causing error then i think there is no need to do anything from your side.

[Ahsaniqbalkmc 0]

Thank you for your advice. I was thinking the displaying only click rate publicly can cause problems. I was not aware of the fact the effective cost should also not be disclosed. Here I find it relevant to ask whether the use of terms like average, above average, very low etc is ok or not. I don't think there is any problem in saying that the click rate on my website is low or is good etc.

Crawler errors occur every now and then to almost every website. So it is beneficial for you to make sure that you don't get them everyday or too often. If you get it then there is something wrong on your server. Usercontent crawler is not issue as it only fetches the content and has nothing to do with ads. Transate bot also has nothing to do with your content and adsense earnings. You can do much without these and can simply ignore them as well. If these are the bots which are causing error then i think there is no need to do anything from your side.

I don't get these errors regularly. I check on them every day and find that they keep on increasing and decreasing. Since the robots.txt is fine and the relevancy of ads is normal now, I don't think I should do something. I am afraid I would do some harm to my website as I did when there was an intrusion to my website through some script.

[mahesh2k 0]

This happens sometimes when crawler hits for the new updates on your pages. These crawlers are causing trouble for not only your site but many other sites as well. I found hubpages and infobarrel having issues with it as well. So that it looks normal that you're getting these errors. These crawlers are actually fetching content from your site and showing ads as well, so that's why it is recorded in your account. As for privacy of your adsense data. Some members from hubpages and infobarrel got their account crushed because of such issues. I think adsense team is too thick skinned to take such actions. If you're not sure about what to give in public, ask on their support forums. Still i'm not sure if they reply to support forum or not.

[Ahsaniqbalkmc 0]

This happens sometimes when crawler hits for the new updates on your pages. These crawlers are causing trouble for not only your site but many other sites as well. I found hubpages and infobarrel having issues with it as well. So that it looks normal that you're getting these errors. These crawlers are actually fetching content from your site and showing ads as well, so that's why it is recorded in your account.
As for privacy of your adsense data. Some members from hubpages and infobarrel got their account crushed because of such issues. I think adsense team is too thick skinned to take such actions. If you're not sure about what to give in public, ask on their support forums. Still i'm not sure if they reply to support forum or not.

So this means that I don't need to worry about the crawler errors. I have posted many times of Google's official forum about this issue but they didn't reply. All of the other issues that I posted on Google's official forum got good responses.
I read a lot of blog posts, forum entries, and comments etc about the harshness of google. I think it is a matter of past now. Google is much more softer nowadays. One reason of it can be the growing competition with bing but the fact is that Google is not as harsh with adsense publishers nowadays as it used to be. I think if I had made the mistake of publishing my adsense values publically two years ago, I would have been banned.

[mahesh2k 0]

Google webmaster and adsense forum is mobbed by bunch of eltists who think they know a lot of google. In reality they don't know anything at all. They only join that forum to harass newbies and want to take discussion to show off their elitist attitude. They usually don't reply to serious queries and issues but only join discussion when they find it's affecting their eltist ego. I hardly found anyone answering questions related to crawler. I got most of my answers from other IRC channels where some members were helping each other. You don't have to worry about crawler errors because they're shown on your google adsense and not webmaster dashboard. If you login to webmaster dashboard and found there are many errors then you should consider taking robots.txt for edit. I'm not going to comment on harshness of google because you never know what could happen tomorrow. I usually don't trust forum members because i know one forum where bunch of elitist notified google about this. So knowing all this i don't want anything like that to happen to someone who's sole earning is based on adsense. This is the reason i prefer caution and as i don't trust open forum, it's my natural reaction. You can take that action at your own risk, but i digress.

[Ahsaniqbalkmc 0]

Google webmaster and adsense forum is mobbed by bunch of eltists who think they know a lot of google. In reality they don't know anything at all. They only join that forum to harass newbies and want to take discussion to show off their elitist attitude. They usually don't reply to serious queries and issues but only join discussion when they find it's affecting their eltist ego. I hardly found anyone answering questions related to crawler. I got most of my answers from other IRC channels where some members were helping each other.

For me, Google's official forum is a great source of knowledge. This is possibly because I am not an expert yet and consequently I don't understand the tone of replies made by people there. But a man with such vast experience as you can not be wrong. However, I would still say that as far as I get something to learn from them, they are good for me.

I'm not going to comment on harshness of google because you never know what could happen tomorrow. I usually don't trust forum members because i know one forum where bunch of elitist notified google about this. So knowing all this i don't want anything like that to happen to someone who's sole earning is based on adsense. This is the reason i prefer caution and as i don't trust open forum, it's my natural reaction. You can take that action at your own risk, but i digress.

Your discussions tell of your experience in online business. I have had a lot to learn from you and still I am learning a lot. I think I should be more careful about discussions related to adsense. As you stated, for someone whose sole earning is adsense, getting kicked out is the worst nightmare.

You don't have to worry about crawler errors because they're shown on your google adsense and not webmaster dashboard. If you login to webmaster dashboard and found there are many errors then you should consider taking robots.txt for edit.

I logged into my Google Webmaster Tools account and found that I have no crawler errors, however I was stunned to see something else. You would remember the time when I was asking about the best software to design websites and you suggested me to go to a CMS instead of designing a website myself. At that time when I made all the changes and went to wordpress (a very fruitful decision indeed (thanks to you)), I made a few redirects. I copied the content from the old pages of my frontpage designed website and pasted them on my new wordpress based site and created corresponding redirects in the .htaccess file.Today when I logged in to my Google Webmaster Tools account, I am seeing some of those redirected pages shown to me as "404 Not Found". I don't understand what happened.

[mahesh2k 0]

I guess they're indexed pages from your old site and as you created backlinks to them, they're now appeared broken. You can do one thing -install '404 to start' plugin and redirect the traffic from crawler to the main page. This way you'll not get 404 error in next few months from google. This method should be followed when you move data from one wordpress blog to another. The reason this plugin works is because instead of redirecting it to 404 page, it goes to main page or any other page of your choice. Slowly even crawlers pick this up and take that dead page and search the index (main page) for available pages. I find this method much better than manually creating redirects for all broken pages. Most of the time it is hard to manage all that. If you deleted or removed or renamed old content then chances are there that you have to create redirection for all. If you fail to do that then surely there will be crawler errors. I think in order to avoid that , this plugin will help you for sure. Don't expect google crawler to get used to this anyway soon but by using temporary permalink you can permit google to divert and this will be in your favor because keeping dead pages or 404 error is not good for your site. Hope this plugin helps you address this issue.