FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Status Log
Goto page Previous  1, 2, 3 ... , 10, 11, 12  Next
 
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries
View previous topic :: View next topic  
Author Message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Wed Jul 26, 2006 10:16 am    Post subject: Reply with quote

thanks Wink
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Jul 28, 2006 11:04 am    Post subject: Reply with quote

Status

Still working on the correct way to resolve fixups within OMF modules.

Here's the background:

The infuriating thing about all this is that the fixup table works by basing fixup values on Extern addresses or Segment base addresses. These are referenced by a single TargetIndex and a flag indicating either a segment index or an extern index.

Now Externs can be generated by four different record types: PUBDEF (sometimes), EXTDEF, CEXTDEF, and COMDAT. As the specification doesn't dictate how to map a single TargetIndex number onto four sets of records, I'm left to hack around to interpret the right approach.

So far I've eliminated the following mappings:
- ddl 1.1 - which was mostly right, but a damn lucky first try
- Parse-order of these four record types
- PUBDEFs - EXTDEFs - CEXDEFs - COMDATs
- EXTDEFs - CEXTDEFs - COMDATs - PUBDEFs

Thankfully, phobos.lib is a nice big, fat library, so I can look for patterns in the data. As it turns out the WKEXT records are a strong indicator thanks to them defaulting each reference to an external ModuleInfo to __nullext. WKEXTs also use the FIXUPPs TargetIndex ordering, so they're doulbly useful. I spent a lot of time over the last two days correlating the skew in Extern references with the sizes of the various other record sets.

Just last night, I narrowed it down to the mapping being all CEXTDEFs followed by all EXTDEFs. This is very bizarre as not only are the CEXTDEF records an extension to the original spec (typically, new means last), but the index mapping only works if I include the empty 'zero index' placeholder (EXTDEFs start at 1 everywhere in the spec) in the EXTDEF set. I'm guessing that COMDAT's are next in the set, as this lets the load and link run to completion.

Also, I'm still not 100? sure that PUBDEFs even factor in, only because they're only supposed to if the 'make this public external' COMENT subrecord is present - which it's not.

So what's wrong?

I'm still having trouble getting linktest1 to work past linking. Binding and calling add() in the test yields an Access Violation, so I'm still reviewing both the extern lookup and the fixup algorithms to find the cause. Push comes to shove, I'll have to dump parts of memory in the log and backtrack what the proper fixup addresses should have been. I know I'm close at this point, so I'm taking my time to make sure I don't have any more missteps.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Jul 31, 2006 8:57 am    Post subject: Reply with quote

Status

SVN now contains a working runtime linker and OMF loader implementation that at a minimum, works with linktest1.d. The test itself is a blend of the old implementation, plus all the various tests I added to it as I went along debugging things.

While this hardly makes a thorough test suite, it does bode well for things.

Code:

C:\home\pragma\src\ddl\trunk>test\linktest1
Starting.
Loading test module
add: 42+69 = 111
printf: 42CE0C test printf: 42CE0C
verifyPrintf: true
printf: 42CE0C getPrintf: 42CE0C
testString: 11BDD80 [11 11BDD6A]
getTestString: 11BDD80
Hello DDL World!

hello world
getWritefln: 11BD917 (42137C)
hello world2

TestClass created with no parameters
A = 1
TestClass created with no parameters
A = 1
TestClass created with parameter 31415
A = 31415
In externally callable function, A=1
In externally callable function, A=31415
Done.


linktest1 presently tests the following use scenarios:
- symbol field statically bound to an extern (printf in this case)
- function that returns address of said field
- function that returns an extern's value (printf again)
- function that calls an extern (printf)
- function that returns a D extern (writefln)
- function that calls a D extern (writefln)
- function that creates a class
- class constructor that calls a member function
- call to member function, on class created in external module

While this seems like overkill, its all needed to cover fixups in the data and text areas, that are either segment-relative, or self-relative address values. To make matters more complicated, the class creation is also dependent upon proper vtable initalization, and ModuleInfo initalization.

The only alarming point in the data above, is that there are *two* valid locations for writefln - I think it may be duplicating the stdio.d module due to some fault in the symbol lookup scheme. It's supposed to be resolving it from the insitu lib, but that doesn't seem to be the case. However, the program doesn't seem to really care in this case, and emits via the duplicated writefln just fine.

For extra credit, you'll see that the output stream also works fine even though printf, writefln, and mango.io.Stdout are all being used. While this was completely incedental, it helps underscore the fact that the dynamic lib is indeed being properly meshed with the runtime.

The new design features a the proposals for subtypes of ExportSymbols, a more spec-compliant OMFBinary implementation, and *far* better debugging support on OMFModule (has a data-dump for segment images).

What took so long?

There were many, many subtleties to handling OMF data that I simply did not anticipate when starting the project over a year ago. These included, but were not limited to:

- Tons of legacy cruft that has to be accounted for, even if to only throw an exception.
- A complete dirth of good, and up to date, reference implementations online to follow as a guide. Of those I found, none have a license compatible with this project.
- FIXUPP records having an "extern index" ordering that is largely undocumented, especially when taking the extended OMF records into account.
- FIXUPP record documentation makes mention of the previous "enumerated data" record, with respect to things like the current segmentIndex and offset. It fails to mention that the FIXUPP destination offset adds this to the previously parsed data record's effective address at fixup time - failing to do this was causing fixup locations above 1K in any given segment to fail. Sad
- WKEXT records referencing the "extern index" ordering much like FIXUPP records.
- Extended OMF records like CEXTDEF and COMDAT featuring minimal documentation as to how they modify (and in essence, redefine) the behavior the older record types. This especially goes for FIXUPP with respect to calculating the "extern index" and how COMDAT apparently also falls into the same category as LIDATA and LEDATA with rspect to finding a destination fixup address.

There's probably more, but that was the worst of it - it took a huge amount of hacking and analyzing loaded segment images byte-by-byte to figure out what was going on. At one point, I was even decompiling portions of those images by hand (goes with the HLA research I did recently) to make sure that the fixup targets were being resolved correctly.

Next Steps?

Should the next series of tests go well, the project timeline will be revised to feature the following:

- Test out the linker and to find the reason behind std.stdio being duplicated, rather than being implicitly linked in.
- Run another series of tests featuring templates, and template instances that have been duplicated across modules.
- Get the PathLoader online again so ddl can be more easily taken on by the masses.
- Redo the ZipLoader.
- Publish RC1
- Finish ELF support (possibly earlier)
- Publish RC2
- Readyness review for release

Edit

(note to self):

I just realized that there is a rather nasty bug in place, revolving around COMDAT continuations. As a result, fixups will only apply within the first 1K of any given COMDAT area. The solution will involve recording the proper composite offset from a COMDAT continuation, and the related record(s) before it.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon Jul 31, 2006 4:45 pm    Post subject: Reply with quote

Awesome! I'll try to break it again Twisted Evil
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Jul 31, 2006 5:01 pm    Post subject: Reply with quote

kris wrote:
Awesome! I'll try to break it again Twisted Evil


Fantastic.

FYI, you'll get more info during a debug build, including the new-and-improved segment dumps (complete with HEX and ASCII output) and Linker activity.

//yea, I know I'm long overdue for using Logger.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Aug 01, 2006 9:59 pm    Post subject: Reply with quote

Status

I dusted off the old PathLoader and gave it another go. As it turns out it was very close to the mark as it was, as it only took about 30 minutes of tweaking and testing to get it working correctly. This is now in SVN, along with pathtest1 and pathtest2.

FYI, the test suite isn't 100? done yet, as I still haven't tested loading discrete libs/objects based on paths deeper than the mapped root path.

Using these, in theory at least, along with the registry should go something like this:
Code:

auto registry = new DefaultRegistry();
auto dmdPath = new PathLibrary("path\to\dmd\lib",registry);
auto dmPath = new PathLibrary("path\to\dm\lib",registry);

registry.loadAndRegister("this.situ"); // this part hasn't changed
registry.register(dmdPath);
registry.register(dmPath);


The net result is that internal and external references to phobos, winsock, snn.lib and the rest are automatically loaded via their respective libraries and resolved accordingly.

Performance

There is a small speed issue with the above that has mostly to do with the fact that OMFLibrary doesn't employ any kind of lazy loading scheme. This means that the *entire* root lib space is loaded into memory, from the point of PathLibrary creation. Fortunately, this is very possible to install.

Overall, it looks like any app using DDL is going to have to account for some kind of 'warm up' period, much like we see with Java and .NET applications. It takes time to parse, load and bind entire libs in this fashion, so its no suprise that it hogs time and space to accomplish this. Please keep this in mind when developing for DDL - even with lazy loading, you have to pay one way or the other. Wink
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
BenjiSmith



Joined: 05 Jun 2004
Posts: 2
Location: Salt Lake City, Utah

PostPosted: Tue Aug 01, 2006 11:20 pm    Post subject: Reply with quote

I don't know if you guys will remember me, but I used to be active on the D newsgroup a few years ago.

I've been lurking for more than two years now, watching the progress of the D language and community. And, as far as I'm concerned, DDL is (by far) the most important remaining component necessary to increase widespread interest in D.

It looks like it's really coming together beautifully.

Thanks for all your hard work.

--benji smith
Back to top
View user's profile Send private message Send e-mail
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Aug 02, 2006 5:34 am    Post subject: Reply with quote

Thank you Benji. Its posts like yours that make this all the more worthwhile! Very Happy
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Aug 02, 2006 12:50 pm    Post subject: Reply with quote

Status

I added some bugfixes which amounts to some overall improvements to the linking process as a whole. The debug output of OMFModule and the linker have improved a little bit as well.

The linker now invokes the module initalizers *after* the entire link pass has been completed. This includes cases where a failure to link has been raised - sometimes a link pass can cause subordinate modules to be fully linked.

OMFModule now correctly interprets the addresses of resolved externs as such for the fixup pass. This involved delaying the interpretation of an extern index until fixup time, rather than before.

Outstanding Issues

For whatever reason, I'm getting some very odd behavior when the linktests attempt to call writefln, even though its being properly linked in dynamically. The only time this doesn't cause any problems is when stdio is imported by the host application.

Another issue, possibly related, is that exceptions thrown from within a linked lib aren't being handled correctly. It looks like the SEH is going to slow me down a bit while I figure all this out:

Code:
public void exceptionTest(){
   printf("Throwing...");
   throw new Exception("From - test.testmodule.exceptionTest()");
   printf("You shouldn't be here. :(\n");
}

public void exceptionTest2(){
   try{
      exceptionTest();
   }
   catch(Exception e){
      printf("That worked alright.\n");
   }
   printf("Done with exceptionTest2\n");
}


In pseudocode, the results of calling these various ways from the host executable, are thus:
Code:
//works as expected
exceptionTest();

// Works as expected
try{
    exceptionTest();
}catch(Exception e){
    Stdout.println("Exception Test Passed. ?s",e.toString());
}

//Access Violation
exceptionTest2();

// yields: "Exception Test Passed. Access Violation"
try{
    exceptionTest2();
}catch(Exception e){
    Stdout.println("Exception Test Passed. ?s",e.toString());
}


Weird stuff.

Edit

Apparently, everything from OMFModule on up is doing what it's supposed to be doing. Doing a side-by-side comparison of the memory areas of in-situ exception tests, and dynamic ones, very quickly showed some data discrepancies. Upon further inspection, I found that (somehow) there's either a subtle extern index mapping problem or I'm droping bits (read: left something out) somewhere between parse and link. The problem is that it keeps using externs #18 and #19 when it should be using whatever _except_list is mapped to. This would explain the bizarre Exception handling behavior.

The reason behind this could very well be that my FIXUPP parsing kung-fu is still in need of some more work. It looks like there's more neuances that need to be handled here with respect to interpreting fixup data. I already noticed that I dropped the target displacement field, which may explain why stdio has been so cranky.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Thu Aug 03, 2006 1:13 am    Post subject: Reply with quote

BenjiSmith wrote:
I don't know if you guys will remember me, but I used to be active on the D newsgroup a few years ago.

I've been lurking for more than two years now, watching the progress of the D language and community. And, as far as I'm concerned, DDL is (by far) the most important remaining component necessary to increase widespread interest in D.

Absolutely Smile
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Thu Aug 03, 2006 1:14 am    Post subject: Reply with quote

pragma wrote:
Overall, it looks like any app using DDL is going to have to account for some kind of 'warm up' period, much like we see with Java and .NET applications. It takes time to parse, load and bind entire libs in this fashion, so its no suprise that it hogs time and space to accomplish this. Please keep this in mind when developing for DDL - even with lazy loading, you have to pay one way or the other. Wink

That's what SplashScreens are for Smile
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Aug 03, 2006 11:54 am    Post subject: Reply with quote

Status

Fixed the exception handling problem. As I suspected, it came down to my interpretation of the spec, which was *almost* clear on how these extern references are to be resolved.

I'll spare everyone the boring details and direct you to the following notes page, should you be interested. The section on FIXUPP records is where it comes together in plain english:

http://www.dsource.org/projects/ddl/wiki/DevNotes/OMFLinkerNotes

Bringing You More Zeroes, in More Places

As evidence that the _except_list extern is being resolved correctly, here is the data dump from both the local and dynamic versions of the same routine. The asterisk denotes the address of origin.

The zeroes are the fixup values for _except_list, which were being populated with locations to other routines by mistake. The actual reference is "FS[0]" which is dead-on for using SEH.

Code:
Remote Data: [13D4D8A]
 | 
  [13D4D80]  8D 04 FF 50 E8 0F 7A 04 FF C3*55 8B EC 6A FF 64 |  ...P..z...U..j.d
  [13D4D90]  8B 15 00 00 00 00 68 CE 4D 3D 01 52 64 89 25 00 |  ......h.M=.Rd.?.
  [13D4DA0]  00 00 00 83 EC 08 83 EC 04 53 56 57 C7 45 FC 00 |  .........SVW.E..
  [13D4DB0]  00 00 00 E8 AD FF FF FF C7 45 FC FF FF FF FF 8B |  .........E......
  [13D4DC0]  4D F4 64 89 0D 00 00 00 00 5F 5E 5B C9 C3 B8 00 |  M.d......_^[....
  [13D4DD0]  32 3F 01 E9 3C 78 04 FF 68 D8 CD 43 00 E8 7E 7C |  2?..<x..h..C..~|
  [13D4DE0]  04 FF 83 C4 | ....

Local Data: [402038]
 | 
  [402030]  64 A7 01 00 C3 CC CC CC*55 8B EC 6A FF 64 8B 15 |  d.......U..j.d..
  [402040]  00 00 00 00 68 7C 20 40 00 52 64 89 25 00 00 00 |  ....h| @.Rd.?...
  [402050]  00 83 EC 08 83 EC 04 53 56 57 C7 45 FC 00 00 00 |  .......SVW.E....
  [402060]  00 E8 AA FF FF FF C7 45 FC FF FF FF FF 8B 4D F4 |  .......E......M.
  [402070]  64 89 0D 00 00 00 00 5F 5E 5B C9 C3 B8 98 30 43 |  d......_^[....0C
  [402080]  00 E9 8E A5 01 00 CC CC 55 8B EC 6A FF 64 8B 15 |  ........U..j.d..
  [402090]  00 00 00 00 | ....


BTW, the code to generate this kind of output is available in ddl.Util. For the Mango challenged, sprint() is only really used within the dataDumper() routine to convert a byte into a hex string - it doesn't take much to replace that part. Wink
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Aug 04, 2006 8:39 am    Post subject: Reply with quote

Status

Things are looking quite well for DDL for Windows. The mule/host example compiles and runs as expected, as do all of the tests thus far. I think I may turn my attention to documentation while I root around for more bugs in the implementation. New graphics for the wiki may also be a part of this pass too, time permitting. Wink

Enki

I recently gave Enki a shot in the arm. It now has rudimentary unicode support, base parser type support, and a parser library to keep all the goodies I've been working on. The grammars in that library are definately works in progress, but they'll mature as they pull Enki by the nose into a mature state. At a minimum, you can always expect enki itself to be the most mature aspect of this subproject.

Contribtions for the Enki parser library are welcome.

Thanks goes to BCS for keeping up with the changes to DMD and publishing patches to Enki when I didn't have time to do so. His feedback and suggestions are merged in with the current revision in SVN.

Documentation is still lacking, as is an official 1.2 release. These will come after I've had time to test and verfy the latest changes.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Aug 13, 2006 8:31 pm    Post subject: Reply with quote

Status

I'm moving steadly ahead on the documentation front by continuing to assemble an Enki-based D frontend.

Enki

Enki is really shaping up to be a solid tool. I'm going to continue working on refining D parsing and along the way, we should see 1.2 drop once I flush out any remaining bugs. I'm already seeing tons of room for improvement, that may have to wait for an Enki 2.0 (breaking changes in grammar, API, etc). Several new features have been added, as well as many changes and bugfixes to the engine itself, so checking out SVN is a *must* for new developers if you haven't done so already.

As far as the D Tokenizer goes, its not as fast as I would like - even with full optimization enabled. When profiling the tokenizer against tokenizing itself, the ResultT constructor, keyword parsing and setError were in the top three for total function time consumed. I was concerned that those lookups were going to slow things down, so I may adopt a different strategy next time around (probably just rely on the xref and leave it at that). I'm also looking at the codegen to see if there are other ways to speed things up.

As for setError(), I've considered installing a mode that won't bother to try and provide error information, to leave things in the hands of the developer. This way, there's not as much noise generated when you really want to do stuff like this:

Code:
 Foo ::= Bar | @setFatalError!("expected Bar");


This way, the D parser can stop dead at a given point while parsing, rather than trying every possible way through the grammar like it does now. Its that behvaior that I think is responsible for why the Tokenizer is so slow: Walter's frontend may be a bit more intellegent in terms of how its picking what parse path to go down, and how that is optimized.

I also have a ddoc parser patiently awaiting testing after I get the tokenizer worked out. Wink

Edit

After reviewing the DMD frontend, I have come to the conclusion that the lexer doesn't adhere one-to-one with the specification as presented on the DM website. I noticed a few other things as well:

- Treating keywords as identifiers, from the outset, can accelerate identification of keywords via lookup after the fact.
- The DMD frontend doesn't use a hashtable for keyword lookup?!
- The lexer converts parsed values (floats and integers) to types on the fly, rather than waiting for a semantic pass

But explicit-error mode is definately the way to go. If it turns out to be more useful, I may wind up meshing the current error code with some operators to give a truely powerful error handling setup.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Aug 16, 2006 12:28 pm    Post subject: Reply with quote

Status

DDL has been quiet lately, pending some review, testing and documentation work.

Meanwhile, H3r3tic has been testing out the unoffical DDL 1.2B and has stuffed my inbox with helper code, tutorial bits and helper utilities. I look forward to folding his contributions into what will become a very user-friendly DDL release!

Enki

I've been experimenting with improving the code generator such that it can operate more efficently and provide a way to support different output languages. At this point, I think this is something that is probably best left to a 2.0 release, as it will force some minor changes to Enki's grammar that may not be 100? backwards compatible to 1.2.

A preview of what I have in mind is lurking in SVN... somewhere. Wink

The drive to support different languages is simple: sometimes, we all get stuck in a position where D isn't a good idea, but having a generated parser is still a nice-to-have. In a perfect world, Enki will support Javascript, CFScript (broken ECMAscript), PHP, Java and possibly others. For now, I'll leave most of those other parsers for contributors and other engineers to handle - or likely just myself when I get around to them.

With that, I'll be working towards wrapping up Enki 1.2 pending a few minor enhancements and improvements.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries All times are GMT - 6 Hours
Goto page Previous  1, 2, 3 ... , 10, 11, 12  Next
Page 11 of 12

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group