[JJR]

Please feel free to correct any inaccuracies in this post...

Per Kris's request, this is an update concerning the status of the DSC Beta 3 on Linux after a fair bit of testing, particularly in the server component.

Some Background:
----------------------

I'm running Gentoo Linux 1.4 with kernel 2.6.3. My computer is a Compaq Presario X1050 with a Pentium-M 1.4 GHz CPU and 256 MB of memory. I've also been testing the DSC.server component on a Desktop System, and Athlon XP 1400+ with 512 MB of ram and the same linux version. Both systems use NPTL (Native Posix Thread Library; this doesn't seem to influence anything currently, but I thought I'd mention it since it is a variation from normal linux systems).

I've been using various distributions of Linux for several years off and on, but am not an experienced programmer on this platform, nor can I be considered an expert linux system user, though I'm acceptably comfortable with its operation, setup, and some administration. Low level socket/network programming and technical understanding has not been my forte. But I learned a lot on this one... mostly due to a very persistant and knowledgeable DSC developer who demonstrated some impressive skills at remote bug hunting .

DSC.server Platform Testing:
----------------------------------

As Kris noted earlier, testing win32 DSC.server on WinXP HE has helped weed out a number of issues on that platform. It now appears to be running quite stably. The goal then has been to get the Linux version running comparably with equivalent configuration. We soon found that Linux had its own peculiarities that needed to be addressed in order for any progress to be made.

DSC.server on Linux:
-------------------------

The default unittest for DSC.server is testHttpServer() found in unittest.d. Currently this funtion sets up a server with testServer() which then calls the function testClient(). testClient() goes through 100000 iterations each of which creates a client connection to the server, sends a request, recieves a reply from the server (some html data), and then closes/deletes the connection. On WinXP HE, we succeeded in getting this stress test of connections to work. The settings were initally set so that the server operated on 1 thread (though the default had been 20... we tested lower to weed out possible issues ).

Summary:

Linux didn't fair too well with the same settings. The program would either segfault, quit because of inability to connect, or complain about socket access failure. Kris found a few possible solutions to these problems. One was to modify one of the socket options attributes: SO_LINGER. Another was to modify some D techniques (using struct/union combination) for better C function compatibility (setsockopt() arguments). He also realized that linux may very well be running out of file handles when such a high number of sockets were being created. He tried implementing ways to reduce file handle consumption by improving reuse management. We also considered ways of modifying intrinsic linux system limits: these include global file handle maximum and per-process file handle maximum adjustments. I discuss these issues in detail in this post.

The Issues in Detail:
------------------------

First there was SO_LINGER. Initially we tried adjusting this socket option by using a setLingerPeriod(0) call in order to improve the ability to recycle sockets. It was unclear whether this made any improvement.

Gradually, while testing, it became apparent that most of the errors that were occurring when running dsctest were less due to variations in certain socket option settings and more to do with how soon I ran tests after the previous one. Kris was seeing a pattern too. He started investigating file handle limitations on Linux because he understood their requirements as it pertained to sockets and a very high resource consumption in his test case. I also noticed that, if I slightly delayed the iterations with console output, the system was somtimes able to complete the whole 100000 trys with only consistant period errors (but these didn't end the program). We decided to limit iterations on the Linux tests to see how far along the tests could get before faulting. This could indicate when the handle resources were being exhausted. As it turned out, I could only get to 6100 iterations before problems started. All these observations contributed to the theory that linux file handles were inadequate for the test case. Kris looked up some data on the web and identified some details that pertained to file handle limits on Linux.

There are two file handle (aka file descriptors) limits in linux: one is a global (or system wide limit) and one is a per-process limit.

The global limit varies on distributions. On mine it was set to 26950. Kris found instructions showing how this limit may be increased or decreased depending on system requirements (web servers, etc). All that the sys admin had to do is include a "echo 120000 > /proc/sys/fs/file_max" on most linux systems (120000 could be almost any other number within reason).

The per-process limit is probably the more critical of the two as pertaining to DSC.server. Kris figured this was what was limiting the actual number of iterations that dsctest could perform without error (the default limit on Linux is only 1024). Once again Kris came across two possibilities for modifying the per-process limit. One was the use of the bash shell command "ulimit -n." Yet it wasn't clear if this instruction had actual control over the per-process limit adjustment even as "root". I found that the system "man" entry said that few OSes supported changing the values using this command. We decided to give it a go anyway to see if it fixed the 6100 iteration limit of dcstest. I could execute "ulimit -n 65536" and then execute "ulimit -n" again to show whether the value had been changed. The system returned the changed value. Whether it was actually doing anything deep down inside was unknown. I soon found that changing these values had no effect on furthering dsctest beyond 6100 iterations. So our conclusion was that "ulimit" did not seem to be changing the actual internal per-process limit from 1024.

There was one other method for changing per-process file handles. This was to change a #define value in limits.h within the linux source and then perform a recompile of the kernel. This has not been tried yet, but appears to be last remaining test to see if the Linux version can be made to run comparably to the windows version of DSC.server. As of now, dsc.server runs but is much more limited then the windows version. I would love to see this barrier removed.

This is a rather long-winded, boring overview of what has been performed so far with linux and DSC.server. As a result of this barrage of testing, I think even the linux version of DSC.server beta 3 has been improved upon (other issues have been discovered and fixed). Yet DSC still needs to be thoroughly tested on other linux distributions.

Any expert advice from linux gurus would definitely be desirable to help improve linux performance.

Oh and thanks for actually reading this through, if you did

- John

[jcc7]

[kris]

A very big "thank you" to John for his insights, painstaking methodology, and for sticking it out through that process.

The seemingly innocuous little HTTP test wreaks havoc upon the OS socket strata: When running under WinXP-HE on that 1.4Ghz Centrino laptop, it creates, uses, and destroys 200,000 sockets (100,000 client-side, 100,000 server-side) in ~29 seconds

- Kris

[JJR]

Oops, I forgot to mention that...

Performance on Win XP HE was indeed incredible, 100,000 iterations, 1 thread:

29 seconds to complete on Centrino, as Kris mentioned

45 seconds on the Desktop Athlon XP 1400+

I'd like to see someone show off this libraries capabilities in a server app.

[csauls]

I'll see if I can talk to one of my Gentoo-worshipping buddies about this... It'd be nice to find a way around this without expecting a kernal rebuild. (Then again, one friend of mine rebuilds his practically every day... so I doubt he'd be bothered.)
_________________
Chris Nicholson-Sauls

[JJR]

To get another opinion would be a great asset. I've actually already tried recompiling the kernel with the modification in src/linux/include/kernel/limits.h.

Unfortunately it had no affect on the test when I reran it.

I don't know what more can be done... Maybe your friend can think of somehting creative.

[csauls]

Well, I did get ahold of one buddy... he actually recommended the 'echo ###### > /proc/sys/fs/file-max' bit right off. Figures. Also offered the idea of running ulimit/limit as root though, using su... something like:

su - root
ulimit -n 16384 # if root uses sh
limit descriptors 16384 # if root uses csh
su - your_user_name
program_to_run
_________________
Chris Nicholson-Sauls

[brad]

Guys,

Is this still an issue?

I'm running the very latest /trunk version of Mango on Linux, compiled into a small program. It's basically unittest.d, stripped of everything except testServletEngine() and whatever other functions it needs.

It compiles fine (using Build ) but I'm getting the error:

[kris]

Apparently you have unittest enabled (via the compiler flags), and one of Chris Miller's unittests is failing. I think that's one really, really, old unittest ... cos' there's no way it would have ever passed. Hmmm. Guess I've never run -unittest on socket.d before.

While we're on the subject; the unittest.d file in Mango is poorly named. It's not the same as D unittest at all, since it doesn't actually have any of those. It should be called something else instead. When it was written, there were some things it needed to do which D-unittest could not handle ~ hence the alternate path.

Hope that helps;

- Kris

[brad]

[kris]

Ahem ... I'm afraid that is the Mango version of socket.d ... I changed as little as possible from the original file.

[Derek Parnell]

[brad]

[kris]

[brad]

yes, I've been watching...