FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

More Wierdness

 
Post new topic   Reply to topic     Forum Index -> Mango
View previous topic :: View next topic  
Author Message
Meldryn



Joined: 06 Apr 2005
Posts: 23

PostPosted: Sun Apr 10, 2005 10:27 pm    Post subject: More Wierdness Reply with quote

Hi again!

Now that the linking problems are resolved, I'm getting runtime errors too It seems that some of the internal asserts in the library are failing. Heres the code:


Code:
module StockBot.WebBot;
import mango.http.client.HttpClient;
import mango.http.server.HttpHeaders;
import mango.io.Stdout;

static this()
{
}

public char[] DownloadString(char[] url)
{
      // callback for client reader
        void sink (char[] content)
        {
         return content;
        }

        // create client for a GET request
        auto HttpClient client = new HttpClient (HttpClient.Get, url);

        // setup a Host header
        client.getRequestHeaders.add (HttpHeader.Host, client.getUri.getHost());
      client.getRequestHeaders.add(HttpHeader.Accept, "*/*");
      client.getRequestHeaders.add(HttpHeader.AcceptLanguage, "en-us");
      client.getRequestHeaders.add(HttpHeader.Connection, "Keep-Alive");
      client.getRequestHeaders.add(HttpHeader.UserAgent, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.40607)");

        // make request
        client.open (80000);

      Stdout.put(client.getResponseHeaders);

        // check return status for validity
        if (client.isResponseOK)
      {
         // extract content length (be aware of -1 return, for no header)
         int length = client.getResponseHeaders.getInt (HttpHeader.ContentLength);
         if (length < 0)
            length = int.max;
         
         
         // display remaining content
         client.read (&sink, length);
      }

}


Which is passed something like "http://www.google.com/" for example. This pops up the error box saying: AssertError Failure WebBot(46), where 46 is the last line in that code. But the console has the correct header information. Curious and puzzling... so any help is appreciated.
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Sun Apr 10, 2005 11:16 pm    Post subject: Reply with quote

A few things:

- I know not what this error box is that you speak of. Is it a DFL thing?

- I don't suppose the "return content;" for a void function has anything to do with it?

- Can you isolate the function as a console application only (no DFL) and succeed there?

- Can you run it through a debugger?

It's almost impossible for me to guess what the problem might be, given the number of unknowns, but can verify that what you're doing ought to operate correctly.

Idea Doh! You don't have a return statement in DownLoadString().

D v120 now issues a HALT instruction for missing returns and for missing switch-defaults. This is very likely to be the problem.

Note that the "return content;" is within a nested function ~ it is not the return for DownloadString(). That nested function, called sink(), may be called multiple times before all the returned data is actually recieved, since it is invoked asynchronously to the receipt of socket-data from the website. You need to gather up all the content within sink(), and then return the whole thing at the end of DownloadString().

For very simple cases, you might just have a local variable, perhaps called, 'returnedContent' which you simply set to the argument provided in the sink() callback. You'd then return the 'returnedContent' from DownloadString() instead.

For more complex scenarios, you'd likely return the IBuffer instance (where all the socket data is being fed into) from DownloadString() instead. Said IBuffer is returned from client.open(). For such cases, do not call client.read() since that will, obviously, eat all the data instead. Client.read() is provided simply as a means of convenience.
Back to top
View user's profile Send private message
Meldryn



Joined: 06 Apr 2005
Posts: 23

PostPosted: Mon Apr 11, 2005 12:13 am    Post subject: Reply with quote

Thanks for all the suggestions... I am a C# programmer mainly, and havent had to learn some of the subtlties of other languages. Although, some of those things are pretty obviously wrong. Embarassed Can you point me to a tutorial for setting up a debugger with D? Like I said, I have always worked within an IDE with built-in build-debug functionality.

Thanks, Kyle
Back to top
View user's profile Send private message
Meldryn



Joined: 06 Apr 2005
Posts: 23

PostPosted: Mon Apr 11, 2005 12:18 am    Post subject: Reply with quote

Quote:
Fixed the obvious problems, and now its working as expected. Thanks very much for all the help and for a well written, very helpful library


Well... It WAS. Now I get things like "truncated response" and "4Invalid UTF-8 Sequence" Prolly something equally obvious... Dont spend more time if you dont want to... Sad Basically, I'm trying to port my StockBot app which is in C# to D for speed considerations, and the http library is a huge factor in the performance. How would you implement using your library a method to retrieve a url like http://finance.yahoo.com/q/ks?s=LEND Where the program will be issuing as many concurrent requests at that url. (with the LEND changed to another from a list of stock symbols) Like I said, the C# app suffers from crippling performance issues from the lack of efficiency in the WebClient class in the .NET libraries. Even with async requests, it fails to complete a full run of the dataset in enough time for my hopes/expectations. Would it be feasible/better to open a socket and issue all the requests at that one open connection? I have almost no idea about how things are "supposed" to work with http, although I'm pretty good at hacking something together... Smile Let me know if you have any suggestions.

Kyle
Back to top
View user's profile Send private message
Meldryn



Joined: 06 Apr 2005
Posts: 23

PostPosted: Mon Apr 11, 2005 12:35 am    Post subject: Reply with quote

I fixed my last problem by switching back to the default timeout. I would still appreciate your comments on the design of what I talked about in the last post. Thanks!

Kyle
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon Apr 11, 2005 12:35 am    Post subject: Reply with quote

Meldryn wrote:
Thanks for all the suggestions... I am a C# programmer mainly, and havent had to learn some of the subtlties of other languages. Although, some of those things are pretty obviously wrong. Embarassed Can you point me to a tutorial for setting up a debugger with D? Like I said, I have always worked within an IDE with built-in build-debug functionality.

Thanks, Kyle

Wiki4d apparently has some suggestions for debug tools. I use MSVC, but others have success with other (free) MS tools.
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon Apr 11, 2005 1:04 am    Post subject: Reply with quote

Meldryn wrote:
Quote:
Fixed the obvious problems, and now its working as expected. Thanks very much for all the help and for a well written, very helpful library


How would you implement using your library a method to retrieve a url like http://finance.yahoo.com/q/ks?s=LEND Where the program will be issuing as many concurrent requests at that url. (with the LEND changed to another from a list of stock symbols) Like I said, the C# app suffers from crippling performance issues from the lack of efficiency in the WebClient class in the .NET libraries. Even with async requests, it fails to complete a full run of the dataset in enough time for my hopes/expectations. Would it be feasible/better to open a socket and issue all the requests at that one open connection? I have almost no idea about how things are "supposed" to work with http, although I'm pretty good at hacking something together... Smile Let me know if you have any suggestions.

If you are dependent upon a back-end web site, then you will typically be bottlenecked by the response time from it. There are many different ways to alleviate that, so I'll note just a couple:

Arrow You can go in and extend the mango.http.client to support HTTP pipelining ~ probably find lots of info on that via Google. However, that's non-trivial, and it's only a partial solution since you're still at the mercy of the back-end server anyway -- there's just less latency involved. The server you're hitting might be busy with other clients, so it's no panacea.

Arrow Often the best thing to do is find a way to cache the results from the back-end. For example, what duration should a stock value be considered 'valid' within your program? One second? One Minute? One Millisecond? If it's somewhere in the range of the former two, then you should perhaps consider hitting the sever only when the timeout has expired, and cache the result locally for the (presumeably multiple) calculations your program performs, between timeout periods. Hitting the back-end each and every time is often complete overkill. But then, I really don't know your design requirements. Mango has all the pieces to do fairly sophisticated caching, but you can probably get away with just having an AA of stock-values, keyed by the ticker-name, and somehow coupled with a timeout value for each. If you're using Mango, then System.getMillisecs() will be your friend.

These issues are not really inherent to any given HTTP library; it's much more involved than that. For instance, you might fire up multiple client threads to hit the back-end server as hard as you can. The server(s) still may not give you the priority you want, or you might slow the server(s) down with too many concurrent requests; or it might actually see you as a Denial of Service bot, and ignore you Twisted Evil

HTTP is, generally, not high performance. In fact, it's downright pedestrian. Performance was never a primary design consideration for B.Lee, and rightly so. One has to be a bit creative to get around such constraints.
Back to top
View user's profile Send private message
Meldryn



Joined: 06 Apr 2005
Posts: 23

PostPosted: Mon Apr 11, 2005 1:26 am    Post subject: Reply with quote

Thanks for the suggestions, one question I have is how to send a query string, cause it seems that sending it as part of the url doesnt return the right page. Also, do you have any sort of im program? This is kinda slow, if you have the time some time. Thanks again.

Kyle
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon Apr 11, 2005 1:04 pm    Post subject: Reply with quote

Meldryn wrote:
Thanks for the suggestions, one question I have is how to send a query string, cause it seems that sending it as part of the url doesnt return the right page. Also, do you have any sort of im program? This is kinda slow, if you have the time some time. Thanks again.

You hit a bug; one introduced quite a while back, it seems. Get these files via SVN (on dsource) and you'll be good to go:

mango/http/client/HttpClient.d
mango/io/Uri.d

BTW: you can use client.getRequestParams() to retrieve an object that supports adding/formatting query parameters. Sometimes that's more convenient than building them into the URI directly.

- Kris
Back to top
View user's profile Send private message
Meldryn



Joined: 06 Apr 2005
Posts: 23

PostPosted: Mon Apr 11, 2005 1:18 pm    Post subject: Reply with quote

Downloaded the 1.3 package and that didnt fix it.
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon Apr 11, 2005 1:29 pm    Post subject: Reply with quote

Meldryn wrote:
Downloaded the 1.3 package and that didnt fix it.

Pardon the confusion. I meant go to the SVN repository on dsource, and get those two specific files. If you don't have an SVN client, then you could copy and paste the code via the dsource repository viewer for Mango, over here: http://svn.dsource.org/svn/projects/mango/trunk/mango/
Back to top
View user's profile Send private message
Meldryn



Joined: 06 Apr 2005
Posts: 23

PostPosted: Mon Apr 11, 2005 2:33 pm    Post subject: Reply with quote

I did that first... still didnt work, do I have to recompile the library from the current source?
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon Apr 11, 2005 2:46 pm    Post subject: Reply with quote

Meldryn wrote:
I did that first... still didnt work, do I have to recompile the library from the current source?

Just use Build, as you've been doing (no need to rebuild libs if build is referencing the Mango source files). But you have to ensure you put those two files in the appropriate place:

- Get the two files again (from SVN)
- Place Uri.d into the mango/io directory
- Place HttpClient.d into the mango/http/client directory
- Use Build to compile your app, with the new files from SVN
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Mango All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group