| View previous topic :: View next topic |
| Author |
Message |
brad
Site Admin
Joined: 22 Feb 2004
Posts: 490
Location: Atlanta, GA USA
|
 Posted: Wed Apr 14, 2004 4:27 pm Post subject: HTTP Client help |
 |
|
Kris,
I'm trying to get the contents of a web page back into a char[] or out to the console, or something. I've added a unit test of my own to run on Beta 3. Can you tell me why I don't see anything on the console? And, how I could get this into a char[] or other object so I can parse out interesting content?
I guess what I'm asking for, is a bit of a tutorial on how this IO stuff works.
In the long run, I'd like to take the results and combine it with certain predictions my friends made at the beginning of the baseball season, and have a servlet that serves them pages of "our group's standings". The servlet will be called from a server request, and will in turn, go to sportsline.com, get the results of the page I'm trying to call in my unittest, parse the results, see how everyone is doing compared to their predictions, and send that content back to their browser.
| Code: | void testSportsline ()
{
Buffer buf = new Buffer(256);
char[] host = "cbs.sportsline.com";
char[] page = "/mlb/standings";
char[] request = "";
InternetAddress addr = new InternetAddress(host, 80);
// connect to the server
SocketConduit socket = new SocketConduit ();
socket.connect (addr);
// reset buffer for another go around
buf.clear ();
// would need this if we were using Reader/Writer IO
// buf.setConduit(socket);
// send a request. This will be serviced by our IProvider
request ~= "GET " ~ page ~ " HTTP/1.1\n";
request ~= "Host: " ~ host ~ "\n";
printf("?.*s\n",request);
buf.append (request);
socket.write (buf);
// wait for something to be returned (you can set a timeout)
socket.read (buf);
// dump response to the console
Stdio.stdout.write (buf);
// close and destroy socket (the latter will do both)
socket.close ();
delete socket;
} |
Are there other objects in Dsc that I should be using, like request object, or what am I missing?
Thx
_________________
I really like the vest! |
|
| Back to top |
|
 |
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Wed Apr 14, 2004 8:51 pm Post subject: |
|
|
What you're doing is fine in general Brad, but I spot a couple of issues:
1) You fell into the HTTP trap of not sending enough \n. The HTTP spec requires a completely blank line to terminate the request. You should try the following:
| Code: | buf.append ("GET ")
.append (page)
.append (" HTTP/1.1")
.append ("\n\n");
|
2) You might consider making the buffer bigger. I'd recommend trying socket.createBuffer(), part of the IConduit interface, to give you a buffer of an approriate size. Also, you *may* wish to use seperate buffers for input and output, though that's not strictly necessary for this type of thing.
Here's a few notes to keep in mind;
1) appending directly to a buffer like this is nice and simple. You will perhaps want to use one of the Writers at some point, if you want to perform any kind of formatted output (with numbers, for example).
2) You may need to support cookies for the target host. If so, then you're into new territory. Dsc.server supports cookies and headers, and both are seperated out into distinct classes such that they can be reused for doing "client-side" requests (what you are attempting) versus server-side acceptance (what the servlet engine does). However, I don't have any examples of that yet. If you need to do client-side cookies for this project, I'll help you out asap.
Finally: to turn the recieved data into a char array, call buf.toString(), or take a gander at IBuffer.d for more options.
Hope that helps!
- Kris
Last edited by kris on Thu Apr 15, 2004 3:47 pm; edited 2 times in total |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Wed Apr 14, 2004 8:56 pm Post subject: |
|
|
(Edit: Removed invalid reference to cr)
Regarding a "request" object: I will be putting one together that ties in headers, cookies, and all that stuff. Perhaps next week.
- Kris
Last edited by kris on Thu Apr 15, 2004 11:25 am; edited 1 time in total |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Wed Apr 14, 2004 11:09 pm Post subject: |
|
|
While we're on this topic, you might also consider taking a look at dsc.server.utils.Uri for constructing the path portion of the GET request. A Uri object will 'escape' various reserved characters appropriately on your behalf (e.g. turning <space> into ?20). Uri has a toString() method but it is also an IWritable object, so it can be passed directly to one of the Writers for output (including Stdout if you want to see the content). In your particular case, you'd probably use a MutableUri (from the same module).
You might also take advantage of HttpOutputParams, which can help with contructing a query string (if you need such things). This class is also an IWritable, so it's really simple to shove it through a Writer too.
Likewise, you might also use HttpOutputHeaders to construct outgoing headers, and HttpInputHeaders to strip and index returned headers.
- Kris |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Thu Apr 15, 2004 3:15 pm Post subject: |
|
|
Here's some of the various ways to access data returned from a Conduit (socket, in this case) via a Buffer. Each of these examples display the results on Stdout. Is this the kind of thing you were looking for Brad?
| Code: | /*******************************************************************************
Shows a variety of ways in which to access data returned from
a client-side socket connection.
*******************************************************************************/
void testHttpClient ()
{
/***********************************************************************
Bulk copy from input to Stdout
***********************************************************************/
void readBulk (IConduit conduit)
{
// stream directly to another conduit
conduit.copyTo (Stdio.stdout);
}
/***********************************************************************
Unwound version of the above, where we explicitly move
data between endpoints using a buffer
***********************************************************************/
void readBulkExplicit (IConduit conduit)
{
// create a buffer for transferring content
IBuffer buffer = conduit.createBuffer ();
// stream the data from file to stdout. This is probably as fast as
// one can practically make operations of this type.
while (conduit.read (buffer) != conduit.Eof)
Stdio.stdout.write (buffer);
// ensure the writer actually wrote everything
assert (Stdio.stdout.flush (buffer));
}
/***********************************************************************
Read input a line at a time from the input. You can access
the line content via method Token.toString()
***********************************************************************/
void readLines (IConduit conduit)
{
// create a Token and bind it to both the conduit and a line-tokenizer
CompositeToken line = new CompositeToken (Tokenizers.line, conduit);
// read data a line at a time. Method next() returns false when no more
// delimiters are found. Note there may be an unterminated line at eof
while (line.next() || line.getLength())
Stdout.put(line).cr();
}
/***********************************************************************
Read input a char at a time. This will read in big chunks
of data from the input, but spit them out one character at
a time. While this is a lot faster then reading one char
at a time from the OS (the typical approach), this style
of reading is not recommended. Instead, try the 'chunked'
approach (below).
***********************************************************************/
void readChars (IConduit conduit)
{
// create a reader on the conduit
Reader reader = new Reader (conduit);
// read a single character at a time, until we run out of them.
// An exception will eventually be thrown because we're using
// the 'formatted' data approach
try {
while (true)
{
char c;
reader.get (c);
Stdout.put (c);
}
} catch (IOException x){}
}
/***********************************************************************
Read big chunks of data, and process them as such. Each read
operation will eat as much as there is from the input, up to
the size limit of the buffer.
***********************************************************************/
void readChunks (IConduit conduit)
{
IBuffer buffer = new Buffer(1024);
while (conduit.read (buffer) != conduit.Eof)
{
Stdout.put (buffer.toString());
buffer.clear ();
}
}
/***********************************************************************
unwound version of the above, where we setup buffer space
on the stack (instead of the heap). We need to explicitly
ask the buffer how much data is actually available. One
could use the return value from read() as the number of
bytes available instead (in this particular case), or
use buffer.toString() as in the above case
***********************************************************************/
void readChunksExplicit (IConduit conduit)
{
char[1024] content;
IBuffer buffer = new Buffer (content);
while (conduit.read (buffer) != conduit.Eof)
{
Stdout.put (content[0..buffer.readable()]);
buffer.clear ();
}
}
/***********************************************************************
Open a web-site connection, and process the returned page
***********************************************************************/
// create a socket and connect it to Walter's site
SocketConduit sc = new SocketConduit();
sc.connect (new InternetAddress("www.digitalmars.com", 80));
// construct a (flushing) writer and bind it to the socket
Writer w = new DisplayWriter (new FlushBuffer (256, sc));
// send HTTP request
w.put ("GET /d/intro.html HTTP/1.1")
.cr ()
.put ("Host: www.digitalmars.com")
.cr ()
.cr ();
// set read-timeout to 1 second (avoids stalling for ever!)
sc.setTimeout (1_000_000);
//readBulk (sc);
//readBulkExplicit(sc);
//readLines(sc);
//readChars(sc);
//readChunks(sc);
readChunksExplicit(sc);
}
|
|
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Thu Apr 15, 2004 3:39 pm Post subject: |
|
|
... and here's an example where the returned headers are parsed out:
| Code: | /*******************************************************************************
Extract client-side headers from an HTTP reply
*******************************************************************************/
void testHttpClient2 ()
{
// create a socket and connect it to Walter's site
SocketConduit sc = new SocketConduit();
sc.connect (new InternetAddress("www.digitalmars.com", 80));
// construct a (flushing) writer and bind it to the socket
Writer w = new DisplayWriter (new FlushBuffer (256, sc));
// send HTTP request
w.put ("GET /d/intro.html HTTP/1.1")
.cr ()
.put ("Host: www.digitalmars.com")
.cr ()
.cr ();
// set read-timeout to 1 second (avoids stalling for ever!)
sc.setTimeout (1_000_000);
// create an input buffer
IBuffer buffer = sc.createBuffer();
// extract all headers
HttpInputHeaders headers = new HttpInputHeaders();
headers.parse (buffer);
// display parsed headers -- these should match HttpHeader.HttpHeaders
foreach (HeaderElement header; headers)
Stdout.put (header.name.value)
.put (header.value)
.cr ();
// display remaining content
while (sc.read (buffer) != sc.Eof)
{
Stdout.put (buffer.toString());
buffer.clear ();
}
} |
Note: this example uncovered a wee bug in Buffer.toString(), where it would include the start of the buffer even if it had already been read. Fixed for Beta-4 ... |
|
| Back to top |
|
|
brad
Site Admin
Joined: 22 Feb 2004
Posts: 490
Location: Atlanta, GA USA
|
| Posted: Thu Apr 15, 2004 4:31 pm Post subject: |
|
|
I'm sure I'll be able to get farther with all of the posts on this page. It seems every time I want to do something in D (for fun), I pay dearly the next three days at work. I'm thinking I just need to be independently wealthy. That would solve a bunch of things.
_________________
I really like the vest! |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Thu Apr 15, 2004 7:00 pm Post subject: |
|
|
... and here's an example of how HttpClient is likely to be setup and used (will be in Beta-4):
| Code: | /*******************************************************************************
Poke an HTTP server, using HttpClient.
*******************************************************************************/
void testHttpClient3()
{
IBuffer buffer;
IConduit conduit;
char[] host = "www.digitalmars.com";
char[] path = "/d/intro.html";
// create client for a GET request
HttpClient client = new HttpClient (HttpClient.Get);
// setup request path, plus a Host: header
client.getUri.setPath (path);
client.getOutputHeaders.add (HttpHeaders.Host, host);
// make request; extract conduit from returned buffer
buffer = client.request (new InternetAddress (host, 80));
conduit = buffer.getConduit;
// extract content length (be aware of -1 return, for no header)
int length = client.getInputHeaders.getInt (HttpHeaders.ContentLength);
if (length < 0)
length = int.max;
// display all returned headers
Stdout.put (client.getInputHeaders);
// display remaining content
while (length > 0 && conduit.read(buffer) != conduit.Eof)
{
length -= buffer.readable;
Stdout.put (buffer.toString);
buffer.clear();
}
client.close();
} |
This example takes advantage of the D syntax regarding "methods as fields"; there's a distinct lack of unecessary parenthesis. |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Tue May 04, 2004 1:51 am Post subject: |
|
|
There's an example of HttpClient in action over here: http://svn.dsource.org/svn/projects/mango/trunk/example/servlets.d
Scroll about half way down to class PingThread; that may be a useful technique for you Brad (the background thread approach) if multiple users are gonna' be checking your proxy site ... |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
