View previous topic :: View next topic |
Author |
Message |
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Thu Oct 20, 2005 10:01 pm Post subject: |
|
|
Progress
Got a working prototype for the map parser needed for the in-situ loader. It now resolves all symbols (including the pesky implib style ones) on the map to their in-memory counterparts. I'm one step closer to runtime linking.
What's next is sorting all that stuff out into discrete modules by namespace. I'm considering lumping them into the same global namespace as is used for the C/C++ imports and calling it a day. However, I think this is going to have serious ramifications for efficency during runtime, but if thats the only impact then I may just document that note and move on.
I can now see that the runtime linker will have to follow the following steps for most applications:
1) bootstrap the loaders (automatic)
2) load the in-situ module (map file)
3) load the platform library (phobos)
From here the application can discretely link in a new library/module by loading and linking each of the modules dependencies until all dependencies in the chain are resolved. The operation fails/throws if any module in this chain cannot be fully resolved... that's where a dynamic module loader comes in handy, as it can search for needed dependencies.
I think that's it. The rest is all down to performance, which is going to take some serious profiling to get right.
Aside: This was a nice little exercise to get back into the swing of things, as I spent entirely too much time in XML/webdev land. Its amazing how much one language set can rewire your brain, making for a tough adjustment period to work in yet another. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Fri Oct 21, 2005 7:38 pm Post subject: |
|
|
Working In-Situ Loader
I cannot take credit for the idea of parsing a .map file and using it for reflection. However, I can say with confidence that the technique has a new lease on life in this project.
What is currently in the SVN repository is a very straightforward .map file parser that meshes with the rest of the DDL library structure. As you'd expect this means that 'bless' and 'ddlinfo' also work as expected with .map files.
The trick here is that the standard .map file just doesn't have enough info, so you need the verbose version. Just compile your program with -L" -Map" and DDL will take care of the rest.
There is certainly room for improvement here. For starters, DDL only needs less than half the info that is provided in the file. It could also stand to be compressed to further reduce its footprint. Also, the current in situ loader doesn't sort symbols into its separate modules at all: something that could become a performance problem later on.
I may wind up rolling a file format (.situ) that will be a compressed, module-sorted, minimal data symbol file in order to address these problems. Should I go down this path, it will require an additional tool to build the file, but will remain optional (a trait I want to maintain with the DDL utilities).
Code: | private import ddl.all;
import std.stdio;
import std.c.windows.windows;
static uint cafebabe = 0xDECAFBAD;
void main(){
DynamicLibrary lib = loadDDL("test.map");
assert(lib);
writefln("?d exports total",lib.getExports.length);
ExportSymbol exp = lib.getExport("_D4test8cafebabek");
writefln("?s (?0.8X) == ?0.8X",demangleSymbol(exp.name),exp.address,*cast(uint*)(exp.address));
} |
Output: Code: | 1717 exports total
uint test.cafebabe (00421B50) == DECAFBAD |
_________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sat Oct 22, 2005 7:51 pm Post subject: |
|
|
In Situ Tool
I went ahead with the idea from my last post and created a .situ file format that contains the bare essentials from the verbose map file. After adding compression, the resulting file is 1/10 the size of the original.
I also neglected to 'add' the insitu project directory. This is now in the repository.
The 'insitu.exe' util is also available in the downloads section for DDL. You can also get the source in the SVN repository. Since .map and .situ are new file types to the ddl project, support has been compiled into the other utils as well.
A Note To Reflection System Prototype Authors
The addition of these classes, means that it is no longer necessary to use a DMD front-end to perform some kinds of reflection. While not perfect for determining everything (struct and class layouts, field names, parameter names, etc), it works well for class names, namespaces, methods; anything that is exportable.
Once DMD has a full-scale reflection system, I expect .map file parsing to dissapear from DDL completely. However, there's no reason why a DMD front-end-based reflection system couldn't replace this mode in the meantime. Please contact me if you have such a system: I may be able to add proper support for it in this project. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sun Oct 23, 2005 5:40 pm Post subject: |
|
|
Quick Update
I added a primitive linking example (test9) under the /test area in the project. The ddl utils were also moved to the new /utils subdirectory, which should make the project organization a little easier to understand.
All the needed tools and technical reccomendations are almost in place for runtime linking under win32. The only major stumbling block I found today was that sometimes, optlink likes to declare lines like this in a .map file:
Code: | 0000:00000000 Abs __ModuleInfo_3std5stdio 00000000 |
This is even if the following appears in the executable source:
Code: | private import std.stdio;
void main(){
//...
writefln("hello world");
} |
This is a big problem. Note that the above snippet from the .map file is the only reference to std.stdio; none of its members appear in the .map. This happens seemingly at random, depending on how the source is laid out, and is difficult to reproduce.
As near as I can tell, the 'Abs' refers to an 'absolute address' and is meant for the linker to resolve at a later point. The kicker is that this can sometimes show up when in reference to modules that really are a part of the runtime, just that their exports are unavailable in the map.
Since using phobos is assumed to be a part of the eventual runtime linking process, these 'Abs' modules will simply be loaded at runtime instead. The drawback here, is that without knowing what will trigger an 'Abs' module mapping, there's no telling if static data won't be bound correctly. It also raises the potential for TypeInfo mishaps and the rest.
The only workaround I can concieve is to reccomend using purely runtime linked executables, based from a stub loader, if I cannot keep the above from ocurring. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Sun Oct 23, 2005 6:29 pm Post subject: |
|
|
I think your DDL project is rather important, so I'd like to raise a concern sooner rather than later:
pragma wrote: | Since using phobos is assumed to be a part of the eventual runtime linking process ... |
Can't this dependency on phobos be made pluggable (or versioned) instead, such that one is able to use alternate libraries? I mean, Ares (and Mango) makes a point of not linking phobos. Looking at the (runtime) code there seems to be some file input used, along with some text formatting for exceptions. Would you be open to isolating those things within a 'utility' module of sorts? Or some other resolution?
Seems to me that you're constructing a fundamental platform for D programming -- particularly server-side development -- so it would be nice to be agnostic
Also, I see that all loaders are instantiated as part of a static bootstrap ctor, which requires static linking on each of those modules ... would you consider supporting a means to avoid that, since oftentimes a developer will be concerned with only one type of loader? I mean, you might use the core loader to dynamically load file-handlers; as the need arises (necessitating just one 'default' handler to be statically linked). Kinda' fun to have to loader dynamically load itself |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sun Oct 23, 2005 8:09 pm Post subject: |
|
|
Thank you for your feedback Kris.
kris wrote: | I think your DDL project is rather important, so I'd like to raise a concern sooner rather than later:
pragma wrote: | Since using phobos is assumed to be a part of the eventual runtime linking process ... |
Can't this dependency on phobos be made pluggable (or versioned) instead, such that one is able to use alternate libraries? I mean, Ares (and Mango) makes a point of not linking phobos. Looking at the (runtime) code there seems to be some file input used, along with some text formatting for exceptions. Would you be open to isolating those things within a 'utility' module of sorts? Or some other resolution? |
Ahh.. I didn't mean to say that DDL would be permanently bound to phobos. Granted, I've leaned quite heavily on the stream classes and whatnot for loading/saving of things. Ultimately I plan on making this platform-library agnostic once I get past the prototype stage. For right now, the phobos dependencies will have to stand, as there's just so much more to test and develop at the moment.
Quote: |
Seems to me that you're constructing a fundamental platform for D programming -- particularly server-side development -- so it would be nice to be agnostic |
Also, it may not look like it in the code, but I've made a mental note to keep Ares in scope for the long run. Right now, its all about getting past that all-important runtime linking stage of the project.
I'll add a milestone to trac for eliminating internal dependencies on phobos... I don't know why I didn't add that earlier.
Quote: | Also, I see that all loaders are instantiated as part of a static bootstrap ctor, which requires static linking on each of those modules ... would you consider supporting a means to avoid that, since oftentimes a developer will be concerned with only one type of loader? I mean, you might use the core loader to dynamically load file-handlers; as the need arises (necessitating just one 'default' handler to be statically linked). Kinda' fun to have to loader dynamically load itself |
You're right. Now that I think aobut it, the DDL runtime really has been gaining some weight lately... and there's no real point in having 100? of it linked all the time (however in some cases, you'll need most of it). Besides, there may be uses for these classes beyond what your or I have imagined thus far.
My original goal was to standardize the file extensions to the loader types in a centralized way, but I suppose these can be pushed out to the loaders themselves. And i like the idea of the DDL runtime loading itself... something that until today wasn't completely possible. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Mon Oct 24, 2005 4:36 pm Post subject: |
|
|
Awesome |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Wed Oct 26, 2005 10:23 am Post subject: |
|
|
Progress - The Short, Short Version
Hit a few nasty bugs last night, and probed deeper into what dynamic linking really involves with D.
Progress - The Long Version (or How I Learned to Love Segments)
First, I discovered that the nonsense with "Abs" was really due to modules that the linker deemed weren't needed in the final executable. When I did some tests, the modules labeled "Abs" were in fact not used at all. So the in situ loader has been modified to cull these entries out of the map.
Then I did some linking tests, and found that the OMFBinary class has some deficiencies when dealing with DGROUP (this was documented, but long since forgotten) and walter's funky use of .asm and segment names (see internal/minit.asm) to achieve a centeralized module listing. The silent killer, was a symbol known as __nullext which is defined as merely '0'.
Why was this a problem? I think OMF wants to give any public symbol an address, no matter what. Near as I can tell, __nullext was defined as simply living in the bottom of DGROUP (0:0), and was never intended to be used directly, but instead as a "null external"; this would make it a link-time constant of zero. Of course this was making my linker go haywire, as the size of segment zero was always zero thus making any attempt to return the address of __nullext a showstopper.
So all I need to do is really translate the address of 0:0 to (void*)(0) which is what it would have been were I not using "physical" addresses (okay, relative to the logical address of the current program.. but who's counting?) and had proper segments to begin with.
I also made some adjustments to get DGROUPed segments to glom together when resolving addresses. Its not likely to be used much, but I think it could help the OMFBinary a little more accurate when it comes to DGROUP logical addressing.
As it so happens, there's a similar problem in snn.lib. I'm pretty sure that these enhancements will make this lib linkable too.
There is of course one caveat to all this. If any module, or set of modules, is intended to work with data as it is linked into given segments, that module will behave as nothing was linked in at all; this may be a failure-prone situation.
For example, minit.asm expects external ModuleInfo instances to be dumped into a special segment, which makes them easily retrievable by a single function or even a base segment address; this has the effect of making the linker build the global ModuleInfo[] for the application even though its parts are contributed from N other modules. Clever, eh?
Runtime linking effectively destroys this capability, which is fine given what DDL does. However, if there are any other bits out there, expecting this capability, they're going to break. The problem is that there just isn't any feasable way to reconcile segments like this without creating one sprawling, ever-growing, segmented model of the application that reamains friendly to garbage collection. So something has to give.
Fortunately, all of this kind of code exists in the platform libraries: phobos.lib and snn.lib. Half of that problem is already solved, with the other half on the way. Ultimately, D Modules don't rely on this behavior, so the only remaining problem is with custom coded bits (from other languages) contributed to DDL's linker -- I that can be handled with documentation. AFAIK, its not an often-used technique, and can be worked around in any number of ways.
My Backup Approach: any part that needs to be segment-assigned-at-link can be simply forced in situ by clever use of imports inside DDL. The linker can handle things from there. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Sat Nov 05, 2005 3:18 pm Post subject: Linker Prototype |
|
|
Status
It links.
In the SVN repo, test.d represents the earliest working cut of an honest-to-god runtime linker. The code looks like hell, and is in dire need of optimization, but it manages to load test/testmodule.obj and run some methods in there just fine.
Code: | //test.d -- linker cut for brevity
alias uint function(uint a,uint b) AddFunction;
char[] addFunctionSignature = "_D4test10testmodule3addFkkZk";
alias void function() HelloWorldFunction;
char[] helloWorldFunctionSignature = "_D4test10testmodule10helloWorldFZv";
void main(){
// pull in the in situ module (all the currently compiled-in support)
addLibraryToXRef(loadDDL("test.map"));
// get runtime libraries
addLibraryToXRef(loadDDL("phobos.lib"));
addLibraryToXRef(loadDDL("snn.lib"));
// load in the test module
DynamicLibrary testModule = loadDDL("testmodule.obj");
// perform a dynamic link for this module
linkLibrary(testModule);
// add test
AddFunction addFunc = cast(AddFunction)(testModule.getExport(addFunctionSignature).address);
writefln("add: 42+69 = ?d",addFunc(42,69));
// hello world test
HelloWorldFunction helloWorld = cast(HelloWorldFunction)(testModule.getExport(helloWorldFunctionSignature).address);
helloWorld();
}
|
Code: | //output
TestClass initalized
testmodule initalized
add: 42+69 = 111
TestClass created
A = 1
Hello DDL World!
testmodule destroyed
|
To contrast and compare, the testmodule.d is here: http://svn.dsource.org/projects/ddl/trunk/test/testmodule.d As you can see, the dependencies for the module are in fact being provided at runtime, such that the calls to writefln() and printf() work correctly.
It should be noted that the module constructor is in fact being run at link-time. This may be changed later on. Also, the static destructor for the module is also left hanging out there, and will require a "manual" call if it is to ever be run (the GC knows nothing about it).
The OMF parser made yet another push toward maturity as I had trouble with snn.lib last time out (see previous post). It now works perfectly, and is the most spec-conformant rendition of this parser yet.
If you try running this thing, it will take its time performing a conservative link between the X-ref (cross-reference) libraries and the provided one for linkage. I'm assuming the problem is in the linker, but it may very well be in the OMF library: the profiler will tell all.
Either way, I plan on investigating some strategies (better hashtables, Boyer-Moore string matching and Bloom Filters to name a few) to accelerate and refine the runtime linking algorithm. Over the horizon, will be testing to confirm how this interacts with the GC, which will be critical for projects like DSP. _________________ -- !Eric.t.Anderton at gmail
Last edited by pragma on Thu Nov 17, 2005 12:36 pm; edited 1 time in total |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Sat Nov 05, 2005 5:07 pm Post subject: |
|
|
Awesome
I'll be interested to see what the profiler turns up. |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Tue Nov 15, 2005 10:07 pm Post subject: |
|
|
Status
The current go-round with the project has been spent on promotional work.
I've put in some decent work on the website-to-be, which should be going live once the codebase nears the refactoring phase. I should probably schedule this activity in on Trac when I get the chance.
There's been considerably more buzz in the newsgroup about the project, which is pretty good seeing as how no official effort to promote DDL has been made yet. I find this very encouraging. But it has set in my mind the necessity for a FAQ, even if we're still mired somewhere in the Beta stage of the project.
I can't remember if I've mentioned this here before, but the project is now on the dsource Trac test area. Feel free to check it out.
http://trac.dsource.org/projects/ddl
More to follow soon. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
AgentOrange
Joined: 21 Jul 2005 Posts: 61
|
Posted: Wed Nov 16, 2005 12:59 am Post subject: |
|
|
Ive just checked in code that should load COFF .obj and .lib files but it hasnt been fully tested yet. |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Wed Nov 30, 2005 9:13 pm Post subject: |
|
|
Status
FYI, the Symbol mangler sub-thread has been relocated to its own thread purely for the sake of organization:
http://www.dsource.org/forums/viewtopic.php?t=1107
Please feel free to post all over the anarchy that is this forum. I'll keep things in line as needed.
d -> XML -> XSL -> HTML
The doc generator is maturing nicely, but it is still lacking a few things (documentation in the DDL code being one of them!). Its basically just a set of batch scripts and xsl transforms, the former could easily be redone as a bash script for linux.
I have considered redoing all this as a heavyweight tool using a compiled XSL lib as to avoid having everyone grab AltovaXML or their $XSLT_ENGINE_OF_CHOICE.
But every time I consider going heavyweight, I feel as though too much leverage is lost with the 'openness' of the script-set version. Since documentation should be configurable on a project-to-project basis, perhaps its not a bad idea after all.
Mangler, Linker Work and Refactoring
Anyway, I took a break from XSL hell to kick around a revised linker design. I also merged Don's new code in from his forum post and plan on folding it into the DDL suite.
The mangler template members to Module probably won't appear until after the scheduled refactoring in Jan, simply because there's no good place to put them until that's taken care of. I'll add this point, and others, to the list of things to be rafactored in the library. To those of you who have been kicking around with what's in SVN, I'm open to suggestions and comments on this matter. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
pragma
Joined: 28 May 2004 Posts: 607 Location: Washington, DC
|
Posted: Thu Dec 01, 2005 11:36 am Post subject: |
|
|
Quickie
I left this out of last-night's status: The Milestones for Refactoring and ELF/COFF support have been swapped. I did this due to the growing need for a revision of the current mechanics of the library as a whole. I'm a firm believer in refactoring a project as you go, and the earlier you schedule things like this in, the less likely you move twoard a complete rewrite.
Also, with the doc generator on the way, I need to schedule time for documentation... lots of time. _________________ -- !Eric.t.Anderton at gmail |
|
Back to top |
|
|
kris
Joined: 27 Mar 2004 Posts: 1494 Location: South Pacific
|
Posted: Thu Dec 01, 2005 12:10 pm Post subject: |
|
|
pragma wrote: | I have considered redoing all this as a heavyweight tool using a compiled XSL lib as to avoid having everyone grab AltovaXML or their $XSLT_ENGINE_OF_CHOICE.
But every time I consider going heavyweight, I feel as though too much leverage is lost with the 'openness' of the script-set version. Since documentation should be configurable on a project-to-project basis, perhaps its not a bad idea after all. |
It would be really nice to have a working "out of the box" system. Is it feasable to provide both options? |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|