FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Status Log
Goto page Previous  1, 2, 3, 4 ... 10, 11, 12  Next
 
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries
View previous topic :: View next topic  
Author Message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Oct 20, 2005 10:01 pm    Post subject: Reply with quote

Progress

Got a working prototype for the map parser needed for the in-situ loader. It now resolves all symbols (including the pesky implib style ones) on the map to their in-memory counterparts. I'm one step closer to runtime linking.

What's next is sorting all that stuff out into discrete modules by namespace. I'm considering lumping them into the same global namespace as is used for the C/C++ imports and calling it a day. However, I think this is going to have serious ramifications for efficency during runtime, but if thats the only impact then I may just document that note and move on.

I can now see that the runtime linker will have to follow the following steps for most applications:

1) bootstrap the loaders (automatic)
2) load the in-situ module (map file)
3) load the platform library (phobos)

From here the application can discretely link in a new library/module by loading and linking each of the modules dependencies until all dependencies in the chain are resolved. The operation fails/throws if any module in this chain cannot be fully resolved... that's where a dynamic module loader comes in handy, as it can search for needed dependencies.

I think that's it. The rest is all down to performance, which is going to take some serious profiling to get right.

Aside: This was a nice little exercise to get back into the swing of things, as I spent entirely too much time in XML/webdev land. Its amazing how much one language set can rewire your brain, making for a tough adjustment period to work in yet another.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Oct 21, 2005 7:38 pm    Post subject: Reply with quote

Working In-Situ Loader

I cannot take credit for the idea of parsing a .map file and using it for reflection. However, I can say with confidence that the technique has a new lease on life in this project.

What is currently in the SVN repository is a very straightforward .map file parser that meshes with the rest of the DDL library structure. As you'd expect this means that 'bless' and 'ddlinfo' also work as expected with .map files.

The trick here is that the standard .map file just doesn't have enough info, so you need the verbose version. Just compile your program with -L" -Map" and DDL will take care of the rest.

There is certainly room for improvement here. For starters, DDL only needs less than half the info that is provided in the file. It could also stand to be compressed to further reduce its footprint. Also, the current in situ loader doesn't sort symbols into its separate modules at all: something that could become a performance problem later on.

I may wind up rolling a file format (.situ) that will be a compressed, module-sorted, minimal data symbol file in order to address these problems. Should I go down this path, it will require an additional tool to build the file, but will remain optional (a trait I want to maintain with the DDL utilities).

Code:
private import ddl.all;

import std.stdio;
import std.c.windows.windows;

static uint cafebabe = 0xDECAFBAD;

void main(){
   DynamicLibrary lib = loadDDL("test.map");
   
   assert(lib);
   
   writefln("?d exports total",lib.getExports.length);
   
   ExportSymbol exp = lib.getExport("_D4test8cafebabek");
   writefln("?s (?0.8X) == ?0.8X",demangleSymbol(exp.name),exp.address,*cast(uint*)(exp.address));
}


Output:
Code:
1717 exports total
uint test.cafebabe (00421B50) == DECAFBAD

_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sat Oct 22, 2005 7:51 pm    Post subject: Reply with quote

In Situ Tool

I went ahead with the idea from my last post and created a .situ file format that contains the bare essentials from the verbose map file. After adding compression, the resulting file is 1/10 the size of the original.

I also neglected to 'add' the insitu project directory. This is now in the repository.

The 'insitu.exe' util is also available in the downloads section for DDL. You can also get the source in the SVN repository. Since .map and .situ are new file types to the ddl project, support has been compiled into the other utils as well.

A Note To Reflection System Prototype Authors

The addition of these classes, means that it is no longer necessary to use a DMD front-end to perform some kinds of reflection. While not perfect for determining everything (struct and class layouts, field names, parameter names, etc), it works well for class names, namespaces, methods; anything that is exportable.

Once DMD has a full-scale reflection system, I expect .map file parsing to dissapear from DDL completely. However, there's no reason why a DMD front-end-based reflection system couldn't replace this mode in the meantime. Please contact me if you have such a system: I may be able to add proper support for it in this project. Smile
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Oct 23, 2005 5:40 pm    Post subject: Reply with quote

Quick Update

I added a primitive linking example (test9) under the /test area in the project. The ddl utils were also moved to the new /utils subdirectory, which should make the project organization a little easier to understand.

All the needed tools and technical reccomendations are almost in place for runtime linking under win32. The only major stumbling block I found today was that sometimes, optlink likes to declare lines like this in a .map file:

Code:
 0000:00000000  Abs  __ModuleInfo_3std5stdio     00000000


This is even if the following appears in the executable source:

Code:
private import std.stdio;
void main(){
  //...
  writefln("hello world");
}


This is a big problem. Note that the above snippet from the .map file is the only reference to std.stdio; none of its members appear in the .map. This happens seemingly at random, depending on how the source is laid out, and is difficult to reproduce.

As near as I can tell, the 'Abs' refers to an 'absolute address' and is meant for the linker to resolve at a later point. The kicker is that this can sometimes show up when in reference to modules that really are a part of the runtime, just that their exports are unavailable in the map.

Since using phobos is assumed to be a part of the eventual runtime linking process, these 'Abs' modules will simply be loaded at runtime instead. The drawback here, is that without knowing what will trigger an 'Abs' module mapping, there's no telling if static data won't be bound correctly. It also raises the potential for TypeInfo mishaps and the rest.

The only workaround I can concieve is to reccomend using purely runtime linked executables, based from a stub loader, if I cannot keep the above from ocurring.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Sun Oct 23, 2005 6:29 pm    Post subject: Reply with quote

I think your DDL project is rather important, so I'd like to raise a concern sooner rather than later:

pragma wrote:
Since using phobos is assumed to be a part of the eventual runtime linking process ...

Can't this dependency on phobos be made pluggable (or versioned) instead, such that one is able to use alternate libraries? I mean, Ares (and Mango) makes a point of not linking phobos. Looking at the (runtime) code there seems to be some file input used, along with some text formatting for exceptions. Would you be open to isolating those things within a 'utility' module of sorts? Or some other resolution?

Seems to me that you're constructing a fundamental platform for D programming -- particularly server-side development -- so it would be nice to be agnostic Wink

Also, I see that all loaders are instantiated as part of a static bootstrap ctor, which requires static linking on each of those modules ... would you consider supporting a means to avoid that, since oftentimes a developer will be concerned with only one type of loader? I mean, you might use the core loader to dynamically load file-handlers; as the need arises (necessitating just one 'default' handler to be statically linked). Kinda' fun to have to loader dynamically load itself Cool
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Oct 23, 2005 8:09 pm    Post subject: Reply with quote

Thank you for your feedback Kris. Smile

kris wrote:
I think your DDL project is rather important, so I'd like to raise a concern sooner rather than later:

pragma wrote:
Since using phobos is assumed to be a part of the eventual runtime linking process ...

Can't this dependency on phobos be made pluggable (or versioned) instead, such that one is able to use alternate libraries? I mean, Ares (and Mango) makes a point of not linking phobos. Looking at the (runtime) code there seems to be some file input used, along with some text formatting for exceptions. Would you be open to isolating those things within a 'utility' module of sorts? Or some other resolution?


Ahh.. I didn't mean to say that DDL would be permanently bound to phobos. Granted, I've leaned quite heavily on the stream classes and whatnot for loading/saving of things. Ultimately I plan on making this platform-library agnostic once I get past the prototype stage. For right now, the phobos dependencies will have to stand, as there's just so much more to test and develop at the moment.

Quote:

Seems to me that you're constructing a fundamental platform for D programming -- particularly server-side development -- so it would be nice to be agnostic Wink


Also, it may not look like it in the code, but I've made a mental note to keep Ares in scope for the long run. Right now, its all about getting past that all-important runtime linking stage of the project.

I'll add a milestone to trac for eliminating internal dependencies on phobos... I don't know why I didn't add that earlier. Wink

Quote:
Also, I see that all loaders are instantiated as part of a static bootstrap ctor, which requires static linking on each of those modules ... would you consider supporting a means to avoid that, since oftentimes a developer will be concerned with only one type of loader? I mean, you might use the core loader to dynamically load file-handlers; as the need arises (necessitating just one 'default' handler to be statically linked). Kinda' fun to have to loader dynamically load itself Cool


You're right. Now that I think aobut it, the DDL runtime really has been gaining some weight lately... and there's no real point in having 100? of it linked all the time (however in some cases, you'll need most of it). Besides, there may be uses for these classes beyond what your or I have imagined thus far.

My original goal was to standardize the file extensions to the loader types in a centralized way, but I suppose these can be pushed out to the loaders themselves. And i like the idea of the DDL runtime loading itself... something that until today wasn't completely possible.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon Oct 24, 2005 4:36 pm    Post subject: Reply with quote

Awesome Cool
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Oct 26, 2005 10:23 am    Post subject: Reply with quote

Progress - The Short, Short Version

Hit a few nasty bugs last night, and probed deeper into what dynamic linking really involves with D.

Progress - The Long Version (or How I Learned to Love Segments)

First, I discovered that the nonsense with "Abs" was really due to modules that the linker deemed weren't needed in the final executable. When I did some tests, the modules labeled "Abs" were in fact not used at all. So the in situ loader has been modified to cull these entries out of the map.

Then I did some linking tests, and found that the OMFBinary class has some deficiencies when dealing with DGROUP (this was documented, but long since forgotten) and walter's funky use of .asm and segment names (see internal/minit.asm) to achieve a centeralized module listing. The silent killer, was a symbol known as __nullext which is defined as merely '0'.

Why was this a problem? I think OMF wants to give any public symbol an address, no matter what. Near as I can tell, __nullext was defined as simply living in the bottom of DGROUP (0:0), and was never intended to be used directly, but instead as a "null external"; this would make it a link-time constant of zero. Of course this was making my linker go haywire, as the size of segment zero was always zero thus making any attempt to return the address of __nullext a showstopper.

So all I need to do is really translate the address of 0:0 to (void*)(0) which is what it would have been were I not using "physical" addresses (okay, relative to the logical address of the current program.. but who's counting?) and had proper segments to begin with.

I also made some adjustments to get DGROUPed segments to glom together when resolving addresses. Its not likely to be used much, but I think it could help the OMFBinary a little more accurate when it comes to DGROUP logical addressing.

As it so happens, there's a similar problem in snn.lib. I'm pretty sure that these enhancements will make this lib linkable too.

There is of course one caveat to all this. If any module, or set of modules, is intended to work with data as it is linked into given segments, that module will behave as nothing was linked in at all; this may be a failure-prone situation.

For example, minit.asm expects external ModuleInfo instances to be dumped into a special segment, which makes them easily retrievable by a single function or even a base segment address; this has the effect of making the linker build the global ModuleInfo[] for the application even though its parts are contributed from N other modules. Clever, eh?

Runtime linking effectively destroys this capability, which is fine given what DDL does. However, if there are any other bits out there, expecting this capability, they're going to break. The problem is that there just isn't any feasable way to reconcile segments like this without creating one sprawling, ever-growing, segmented model of the application that reamains friendly to garbage collection. So something has to give.

Fortunately, all of this kind of code exists in the platform libraries: phobos.lib and snn.lib. Half of that problem is already solved, with the other half on the way. Ultimately, D Modules don't rely on this behavior, so the only remaining problem is with custom coded bits (from other languages) contributed to DDL's linker -- I that can be handled with documentation. AFAIK, its not an often-used technique, and can be worked around in any number of ways. Smile

My Backup Approach: any part that needs to be segment-assigned-at-link can be simply forced in situ by clever use of imports inside DDL. The linker can handle things from there.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sat Nov 05, 2005 3:18 pm    Post subject: Linker Prototype Reply with quote

Status

It links. Cool

In the SVN repo, test.d represents the earliest working cut of an honest-to-god runtime linker. The code looks like hell, and is in dire need of optimization, but it manages to load test/testmodule.obj and run some methods in there just fine.

Code:
//test.d -- linker cut for brevity
alias uint function(uint a,uint b) AddFunction;
char[] addFunctionSignature = "_D4test10testmodule3addFkkZk";

alias void function() HelloWorldFunction;
char[] helloWorldFunctionSignature = "_D4test10testmodule10helloWorldFZv";

void main(){   
   // pull in the in situ module (all the currently compiled-in support)
   addLibraryToXRef(loadDDL("test.map"));
      
   // get runtime libraries
   addLibraryToXRef(loadDDL("phobos.lib"));
   addLibraryToXRef(loadDDL("snn.lib"));
      
   // load in the test module
   DynamicLibrary testModule = loadDDL("testmodule.obj");

   // perform a dynamic link for this module
   linkLibrary(testModule);
      
   // add test
   AddFunction addFunc = cast(AddFunction)(testModule.getExport(addFunctionSignature).address);
   writefln("add: 42+69 = ?d",addFunc(42,69));
   
   // hello world test
   HelloWorldFunction helloWorld = cast(HelloWorldFunction)(testModule.getExport(helloWorldFunctionSignature).address);
   helloWorld();
}


Code:
//output
TestClass initalized
testmodule initalized
add: 42+69 = 111
TestClass created
A = 1
Hello DDL World!
testmodule destroyed


To contrast and compare, the testmodule.d is here: http://svn.dsource.org/projects/ddl/trunk/test/testmodule.d As you can see, the dependencies for the module are in fact being provided at runtime, such that the calls to writefln() and printf() work correctly. Wink

It should be noted that the module constructor is in fact being run at link-time. This may be changed later on. Also, the static destructor for the module is also left hanging out there, and will require a "manual" call if it is to ever be run (the GC knows nothing about it).

The OMF parser made yet another push toward maturity as I had trouble with snn.lib last time out (see previous post). It now works perfectly, and is the most spec-conformant rendition of this parser yet.

If you try running this thing, it will take its time performing a conservative link between the X-ref (cross-reference) libraries and the provided one for linkage. I'm assuming the problem is in the linker, but it may very well be in the OMF library: the profiler will tell all.

Either way, I plan on investigating some strategies (better hashtables, Boyer-Moore string matching and Bloom Filters to name a few) to accelerate and refine the runtime linking algorithm. Over the horizon, will be testing to confirm how this interacts with the GC, which will be critical for projects like DSP.
_________________
-- !Eric.t.Anderton at gmail


Last edited by pragma on Thu Nov 17, 2005 12:36 pm; edited 1 time in total
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Sat Nov 05, 2005 5:07 pm    Post subject: Reply with quote

Awesome Cool

I'll be interested to see what the profiler turns up.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Nov 15, 2005 10:07 pm    Post subject: Reply with quote

Status

The current go-round with the project has been spent on promotional work.

I've put in some decent work on the website-to-be, which should be going live once the codebase nears the refactoring phase. I should probably schedule this activity in on Trac when I get the chance.

There's been considerably more buzz in the newsgroup about the project, which is pretty good seeing as how no official effort to promote DDL has been made yet. I find this very encouraging. But it has set in my mind the necessity for a FAQ, even if we're still mired somewhere in the Beta stage of the project.

I can't remember if I've mentioned this here before, but the project is now on the dsource Trac test area. Feel free to check it out.

http://trac.dsource.org/projects/ddl

More to follow soon. Smile
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
AgentOrange



Joined: 21 Jul 2005
Posts: 61

PostPosted: Wed Nov 16, 2005 12:59 am    Post subject: Reply with quote

Ive just checked in code that should load COFF .obj and .lib files but it hasnt been fully tested yet.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Nov 30, 2005 9:13 pm    Post subject: Reply with quote

Status

FYI, the Symbol mangler sub-thread has been relocated to its own thread purely for the sake of organization:

http://www.dsource.org/forums/viewtopic.php?t=1107

Please feel free to post all over the anarchy that is this forum. I'll keep things in line as needed. Wink

d -> XML -> XSL -> HTML
The doc generator is maturing nicely, but it is still lacking a few things (documentation in the DDL code being one of them!). Its basically just a set of batch scripts and xsl transforms, the former could easily be redone as a bash script for linux.

I have considered redoing all this as a heavyweight tool using a compiled XSL lib as to avoid having everyone grab AltovaXML or their $XSLT_ENGINE_OF_CHOICE.

But every time I consider going heavyweight, I feel as though too much leverage is lost with the 'openness' of the script-set version. Since documentation should be configurable on a project-to-project basis, perhaps its not a bad idea after all.

Mangler, Linker Work and Refactoring
Anyway, I took a break from XSL hell to kick around a revised linker design. I also merged Don's new code in from his forum post and plan on folding it into the DDL suite.

The mangler template members to Module probably won't appear until after the scheduled refactoring in Jan, simply because there's no good place to put them until that's taken care of. I'll add this point, and others, to the list of things to be rafactored in the library. To those of you who have been kicking around with what's in SVN, I'm open to suggestions and comments on this matter.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Dec 01, 2005 11:36 am    Post subject: Reply with quote

Quickie

I left this out of last-night's status: The Milestones for Refactoring and ELF/COFF support have been swapped. I did this due to the growing need for a revision of the current mechanics of the library as a whole. I'm a firm believer in refactoring a project as you go, and the earlier you schedule things like this in, the less likely you move twoard a complete rewrite.

Also, with the doc generator on the way, I need to schedule time for documentation... lots of time.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Thu Dec 01, 2005 12:10 pm    Post subject: Reply with quote

pragma wrote:
I have considered redoing all this as a heavyweight tool using a compiled XSL lib as to avoid having everyone grab AltovaXML or their $XSLT_ENGINE_OF_CHOICE.

But every time I consider going heavyweight, I feel as though too much leverage is lost with the 'openness' of the script-set version. Since documentation should be configurable on a project-to-project basis, perhaps its not a bad idea after all.

It would be really nice to have a working "out of the box" system. Is it feasable to provide both options?
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries All times are GMT - 6 Hours
Goto page Previous  1, 2, 3, 4 ... 10, 11, 12  Next
Page 3 of 12

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group