Suggestions Needed For Latest Ycc Bot Maker Breakdown

[tansqrx 0]

This is not specifically geared toward Xisto because it is a plea for help for one of the programs that I am working on. I know there are a lot of smart people here so I decided to give it a shot. About a week ago I received word that version 1.2 of YCC Bot Maker had stopped working. This was not a huge surprise as Yahoo! continuously changes its registration process and YCC Bot Maker is very dependant on the data held in the registration page. I hoped to have a fix put out fairly quickly but this time I am stuck and have yet to find a solution. From what I can gather, Yahoo! has not changed any of the pages or any of the processes used during registration. One day it was working and the next it was broken. This leads to be believe that Yahoo! is doing some sort of new browser profiling or timing analysis. YCC Bot Maker uses the HttpWebRequest object built into .NET. By adding accept, referer, user agent, and others I can fairly approximate the look of IE7. I start Fiddler (http://www.telerik.com/fiddler) and make a side by side comparison of a session with IE7 and then YCC Bot Maker. The requests are almost identical except a few header locations are swapped. Here is the last request during the registration process using IE7 on the India server. The response shows a congratulations page for creating a new account.

POST /registration;_ylt=Amw.PhwB5E.stLUBnuRzxoSZ2PAI 
HTTP/1.0
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/x-silverlight, */*

Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
UA-CPU: x86
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)
Host: edit.india.yahoo.com
Content-Length: 807Connection: Keep-AlivePragma: no-cacheCookie: B=4dnko1h43ek4p&b=3&s=97u=5b178pd43ek5n&dracs=&t=x2F6Q4.MzeGZRlr35zVb1DkywcQjtqkCMrdWMjcL5jX.V6Y1do4hqQ8aOF2HAHDw3jr30lHHDANXTIXGXaohokYjurJtWrgBEDE3ucfh2EOavo0VGwzarjnQ4VJVw_kLYlX4.XVpbLNai8H1BhVbwb8.iOvNBnMSdx1yaaDHdSh6zml1DMIEHZ143m0LGzz8Rxn6nnHe8JcWdRh1en0AJC.s9eYumSrm1taEZAoQ_SoCEt00C8MtUQ--%7EB&done=http%3A%2F%2Fmail.yahoo.com&last=&partner=yahoo_default&intl=us&src=ym&.scrumb=&jsenabled=0&preferredcontent=us&firstname=sadfdas&secondname=ewerq&gender=m&mm=2&dd=1&yyyy=1981&country=us&postalcode=58443&yahooid=dfk58443lkalsdfk&domain=yahoo.com&password=bobobob&passwordconfirm=bobobob&altemail=&secquestion=Who+was+your+childhood+hero%3F&secquestionanswer=dsfdsfadfs&cword=fw8nyvl&cdata=ftqaZeJZFelVDTGYXguGQO75qbdYepa6qbxfs5c2jew_iXlBPSPrIVt8DMAhTLjTWv7KGDDVEErJLca0hg--&showc=1&tos_agreed=y&IAgreeBtn=Create+My+Account

Here is the same output from YCC Bot Maker on the India server. This time I receive a âplease try this code insteadâ and a failed attempt. Additionally, if I make another request with the previous failed attempt using the new CAPTCHA, I get a successful account creation page.

You can see that the requests are exactly the same with the exception of user data and Yahoo! specific variables. I have no way of explaining this because from everything I know it should. I need your help getting the next version of YCC Bot Maker working. If you have suggestions or know of some new technology Yahoo! is using please let me know.

[faulty.lee 0]

I would really like to help. I did a same tool to login into Xisto and fetch the remaining point, so can I can keep an eye on it without loading the Xisto main page. Anyway, do you mind to show me the section of the coding which is doing the login? Or better still, strip the login coding out and put inside a mini project, so we can just try out that portion.

[tansqrx 0]

Thank you for your offer. Let me do some clarification. YCC Bot Maker is a program that signs up for Yahoo! accounts the same way you would if you used Internet Explorer or Firefox. One difference is that it is a VB.NET form that puts all the signup information in one spot and is much quicker than using a web browser. Another difference for this particular question is that YCC Bot Maker uses a series of HttpWebRequest objects and not the Webbrowser object (i.e. the Internet Explorer ActiveX control) to make the requests. I have all the code needed to create the HTTP requests and get valid responses back but I suspect that Yahoo! has added code on their servers to profile the type of âbrowserâ used ask for accounts. More specifically I have changed the UserAgent, Accept, AcceptLanguage, and UA-CPU properties of the HttpWebRequest to mimic Internet Explorer. Even with all of this the Yahoo! server some how knows that my program is not a regular web browser and rejects the request (more on this later).

Up until about a month ago YCC Bot Maker was working fine. Then over night I started getting a prompt to reenter the CAPTCHA. After a through review of the Yahoo! signup page I found that nothing had changed.

There is a series of three requests, get the mail login page which is used to grab the Yahoo! cookie, get the registration page where you actually enter the user data, and finally submit a HTML form POST with the user data and see if it was successful. The first two requests are still fine but when I send the POST, the account is denied and a prompt to enter a new CAPTCHA is shown. After playing around with the sequence I found that I can get a valid account if I reenter the second CAPTCHA and submit once again. An account is only granted if two successful CAPTCHAs are entered. Once again if I use Internet Explorer or FireFox, I do not have to enter the second CAPTCHA. Because I can finally get an account, but only after more verification, I have come to the conclusion that the Yahoo! server knows this is not actually Internet Explorer and adds extra validation.

My question is how does this technology work and how can I get around it.

NOTE:

Here is the code that I use to generate my HTTP requests. It is not really needed but I did put a lot of hard work into it and it is a common question asked. The full library is called HTTPHelper and can be downloaded at http://forums.xisto.com/no_longer_exists/. The original intent was to make up for many of the short comings in version 1.1 of the .NET Framework but it is still very valuable even with the latest version.

Here is an excerpt of a HTTP GET request. HTTPDataIn and HTTPDataOut are structures defined within the class and hold needed variables.

Dim iIndex As Integer

Try
HTTPDataOut.bUnsucessful = False
HTTPDataOut.strErrorCode = ""
httpRequest = CType(WebRequest.Create(HTTPDataIn.strServer), HttpWebRequest)

'proxy
If HTTPDataIn.bEnableProxy = True Then
Dim wpRequest As New WebProxy
Dim uriPrxy As New Uri(HTTPDataIn.piProxy.strServer + " linenums:0'>Public Shared Function HTTPGET(ByVal HTTPDataIn As HTTPDataIn) As HTTPDataOut Dim HTTPDataOut As New HTTPDataOut Dim httpRequest As HttpWebRequest Dim httpResponse As HttpWebResponse = Nothing Dim responseStream As Stream = Nothing Dim responseEncoding As Encoding Dim responseStreamReader As StreamReader = Nothing 'Dim cookieContainerTempOut As New CookieCollection Dim iIndex As Integer Try HTTPDataOut.bUnsucessful = False HTTPDataOut.strErrorCode = "" httpRequest = CType(WebRequest.Create(HTTPDataIn.strServer), HttpWebRequest) 'proxy If HTTPDataIn.bEnableProxy = True Then Dim wpRequest As New WebProxy Dim uriPrxy As New Uri(HTTPDataIn.piProxy.strServer + ":" + HTTPDataIn.piProxy.strPort) wpRequest.Address = uriPrxy If HTTPDataIn.piProxy.strUsername IsNot Nothing Then wpRequest.Credentials = New NetworkCredential(HTTPDataIn.piProxy.strUsername, HTTPDataIn.piProxy.strPassword) End If httpRequest.Proxy = wpRequest End If 'set the timeout If HTTPDataIn.bEnforceTimeout = True Then httpRequest.Timeout = HTTPDataIn.iTimeout * 1000 End If 'auto redirect If HTTPDataIn.bAllowAutoRedirect = True Then httpRequest.AllowAutoRedirect = True Else httpRequest.AllowAutoRedirect = False End If 'accept If HTTPDataIn.strAccept.Length > 0 Then httpRequest.Accept = HTTPDataIn.strAccept End If 'Referer If HTTPDataIn.strReferer IsNot Nothing Then httpRequest.Referer = HTTPDataIn.strReferer End If httpRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us") httpRequest.Headers.Add("UA-CPU", "x86") 'useragent If HTTPDataIn.strUserAgent.Length > 0 Then httpRequest.UserAgent = HTTPDataIn.strUserAgent End If 'httpRequest.KeepAlive = True 'Cookies httpRequest.CookieContainer = New CookieContainer() If HTTPDataIn.cookieCollection IsNot Nothing Then For iIndex = 0 To HTTPDataIn.cookieCollection.Count - 1 httpRequest.CookieContainer.Add(HTTPDataIn.cookieCollection) Next End If httpRequest.ProtocolVersion = HttpVersion.Version10 'Console Logging Out LoggingOut(HTTPDataIn, httpRequest, HTTPDataIn.cookieCollection, True) httpResponse = CType(httpRequest.GetResponse(), HttpWebResponse) responseStream = httpResponse.GetResponseStream() responseEncoding = System.Text.Encoding.GetEncoding("utf-8") HTTPDataOut.strResponseCode = httpResponse.StatusCode.ToString 'Return header information Dim colHeaders As New System.Net.WebHeaderCollection colHeaders = httpResponse.Headers HTTPDataOut.strLocation = colHeaders("Location") HTTPDataOut.strEncoding = colHeaders("Content-Encoding") HTTPDataOut.strServer = httpResponse.ResponseUri.OriginalString 'Return cookies Dim ccTemp As CookieCollection = httpResponse.Cookies 'This is required because of a "bug" in the cookies class. Its not really a bug 'because it is RFC complient but not all cookies will be set in some cases. An 'example of this is http://www.aol.com/. When sending cookies to other subdomains such as 'new.aol.com the cookie will not be sent even though the PATH variable is set to '/' If HTTPDataIn.bSetCookieToRoot = True Then Dim strSplit() As String For iIndex = 0 To ccTemp.Count - 1 strSplit = ccTemp.Item(iIndex).Domain.Split("."c) 'Use only the domain, i.e. .aol.com ccTemp.Item(iIndex).Domain = "." + strSplit(strSplit.Length - 2) + "." + strSplit(strSplit.Length - 1) strSplit = Nothing Next End If HTTPDataOut.cookieContainer = ccTemp 'Decode if gzip is used If HTTPDataOut.strEncoding = "gzip" Then Dim gzipDLL As New ICSharpCode.SharpZipLib.GZip.GZipInputStream(responseStream) responseStreamReader = New StreamReader(gzipDLL, responseEncoding) Else responseStreamReader = New StreamReader(responseStream, responseEncoding) End If If Not responseStreamReader Is Nothing Then HTTPDataOut.strResponse = responseStreamReader.ReadToEnd Else HTTPDataOut.strResponse = "" HTTPDataOut.strErrorCode = "Null Response" HTTPDataOut.bUnsucessful = True End If 'Console Logging In LoggingIn(HTTPDataIn, HTTPDataOut, httpResponse) httpResponse.Close() 'handle thrown exceptions Catch ex As WebException WebExceptionHandling(ex, HTTPDataOut) Catch ex As InvalidOperationException HTTPDataOut.strResponse = "" HTTPDataOut.strErrorCode = "Invalid Operation Exception" HTTPDataOut.bUnsucessful = True Catch ex As UriFormatException HTTPDataOut.strResponse = "" HTTPDataOut.strErrorCode = "Uri Format Exception" HTTPDataOut.bUnsucessful = True Catch ex As NullReferenceException HTTPDataOut.strResponse = "" HTTPDataOut.strErrorCode = "Null Reference Exception" HTTPDataOut.bUnsucessful = True Catch ex As Exception HTTPDataOut.strResponse = "" HTTPDataOut.strErrorCode = "General Exception" HTTPDataOut.bUnsucessful = True Finally If Not httpResponse Is Nothing Then httpResponse.Close() End If If Not responseStream Is Nothing Then responseStream.Close() End If If Not responseStreamReader Is Nothing Then responseStreamReader.Close() End If End Try Return HTTPDataOut End Function
Here is a simple call using HTTPHelper, in this case requesting the Yahoo! mail signin page.

strSignupServer = HTTPStringFind(_HTTPDataOut.strResponse, "<p class=""yreglgsub""><a href=""", """>")

requestSigninPage()
End If
Catch ex As Exception
MessageBox.Show(ex.ToString)
End Try
End Function linenums:0'>Private Function requestMailPage() As Boolean _HTTPDataIn.strServer = strRegionalServer + "/config/mail" _HTTPDataIn.bAllowAutoRedirect = True _HTTPDataIn.cookieCollection = Nothing Try _HTTPDataOut = HTTPGET(_HTTPDataIn) If _HTTPDataOut.bUnsucessful = True Then BeginInvoke(_cmd_BeforeFailDeleg, New Object() {_strLoginName, _strPassword, _HTTPDataOut.strErrorCode}) Else _strReferer = _HTTPDataOut.strServer _cookieHomepage = _HTTPDataOut.cookieContainer 'Signup server _strSignupServer = HTTPStringFind(_HTTPDataOut.strResponse, "<p class=""yreglgsub""><a href=""", """>") requestSigninPage() End If Catch ex As Exception MessageBox.Show(ex.ToString) End Try End Function

[faulty.lee 0]

I look into that in the coming few days.One i think i can suggest is using packet capturing tools, maybe wireshark(free) or commview(my favourite), so study the diff in the data exchanged during the login process.I've got wireshark installed, will try that do. I'll post the result here.

[tansqrx 0]

I have relied heavily on capture tools to produce what I have now. I primarily use Wireshark but most of the Yahoo! pages are SSL so Wireshark is useless. There is a way to decode SSL packets with Wireshark but you have to have the server private key which I donât think Yahoo! is willing to give up anytime soon. The next best thing is Fiddler (http://www.telerik.com/
fiddler) which is an Internet Explorer sniffer. It adds a certificate to the system that acts like a man in the middle attack and you are able to see all of the SSL traffic coming into and out of Internet Explorer. It does have a downside in that you can not see the entire packet but only the TCP payload which is usually enough. The best part is that it is free and the most powerful of this class of tool.Currently my httpWebRequest mimics Internet Explorer in all of the HTTP fields but not the order. The only thing left that I can see is that the COOKIE field is placed ahead of the KEEP-ALIVE field. No matter what I do I can not change the order to place the COOKIE at the end just before the /r/n/r/n. I suppose the httpWebRequest class automatically places the keep alive at the end during the automatic generation of the request headers and the placement is hard coded. From a specification standpoint this should not matter but this could be the thing that Yahoo! is punishing me for.My next task is to create my own httpWebRequest class where I can have finer control of the headers. This will be a lot of extra work but it looks like thatâs the only way to get around this problem. I have also been exploring using the .NET Webbrowser object to just script my way through the registration process but that has opened a whole new set of problems and I am not happy with the results. Even when using the unmanaged ActiveX control I keep getting scripting error pop-ups and it is about 10 times slower than using my current method. I have successfully created accounts using this method but it was not painless and there is much work to be done before this is a viable method. This just once again shows that there is some fundamental difference between using httpWebRequest and Internet Explorer to create Yahoo! accounts.P.S. I done some extra exploratory work this weekend to compare the results from HTTPHelper to Internet Explorer when requesting https://www.google.de/?gfe_rd=cr&ei=BwkjVKfAD8uH8QfckIGgCQ&gws_rd=ssl (because of the SSL problem mentioned earlier). When the captures from Wireshark are run through diff there is no difference at all. The packets are identical down to the smallest detail.

[faulty.lee 0]

Looks like you've done most of the thing i can think of doing. Doesn't look like i can help much here. Anyway i'll still give it a try.I got a few more "crazy" suggestion. Well, I'ven't touch anything from your side yet, so not sure if this is workable., and it might sound stupid.1. Login to yahoo without SSL? I've tried turning off ssl suport then it won't allow me to login into yahoo. Not sure if there's way to disable the requirement so "older" browser would work. That way you can capture and compare the packet properly.2. When you're trying doing a packet comparison, due to the fact that ssl encrypted the data, you can't see it's content. But, isn't the public key that's used to encrypt the data is viewable in the browser. If not the browser also won't be able to encrypt the data. I know it's useless to have just the public key, then again you can use the same key to encrypt the data you generated, and see if it resulted to the same output. This might be tedious, considering you need to try a few different timestamp to get it right.3. Save the login page of yahoo, modified the post action, point to a non ssl localhost php page, to capture what it's trying to post. Then you can see if it's the same as what you expected.I think the 3rd one might be more feasible. This sounds like a HUGE workaround to solve a very simple problem, which have yet to find another better solution :)EDIT: i wonder if YPOP! is still working?

Edited June 3, 2008 by faulty.lee (see edit history)