FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Status Log
Goto page Previous  1, 2, 3 ... 9, 10, 11, 12  Next
 
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries
View previous topic :: View next topic  
Author Message
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Tue May 09, 2006 11:08 pm    Post subject: Reply with quote

pragma wrote:
The problem with getting the linker to generate a map file, is that DMD likes to use 'gcc' as the linker under Linux, and it doesn't handle the "-L" flag correctly for everything. Build 2.10 would seem like the right way to go on this, thanks to Derek breaking out the link step separately. However, last time I tried this version, it wouldn't behave as nicely as 2.09 did. I'll make another pass on this, so I can get things working correctly.


Actually using "gcc" to control the linker is fairly standard practice in the Linux world; it knows how to pass the appropriate flags to the linker. Trying to handle ld directly is considered a no-no is most instances (from what I understand). You can do it if you know what you are doing, but using gcc to call ld is the preference.

-JJR
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed May 10, 2006 8:26 am    Post subject: Reply with quote

JJR wrote:
pragma wrote:
The problem with getting the linker to generate a map file, is that DMD likes to use 'gcc' as the linker under Linux, and it doesn't handle the "-L" flag correctly for everything. Build 2.10 would seem like the right way to go on this, thanks to Derek breaking out the link step separately. However, last time I tried this version, it wouldn't behave as nicely as 2.09 did. I'll make another pass on this, so I can get things working correctly.


Actually using "gcc" to control the linker is fairly standard practice in the Linux world; it knows how to pass the appropriate flags to the linker. Trying to handle ld directly is considered a no-no is most instances (from what I understand). You can do it if you know what you are doing, but using gcc to call ld is the preference.

-JJR


Wow, what a strange convention - but it does make sense, the more I think about it. I guess I'll have to hack away on this one through gcc instead to figure out where the problem is.

Thanks! Smile
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon May 15, 2006 10:56 am    Post subject: Reply with quote

Status

No code update this time. I'm hard at work sifting through the ELF and COFF code, making my best effort to come up with a solution that will work for these two file types, against the current DDL design.

In a nutshell - it looks like ELF support may be pushed back yet again, in favor of getting relocatable file support (akin to what OMF suport does) as well as dynamic shared-object support. It's an interesting specification, and as Lars has given the project a great head-start in this area, I'm still learning the spec myself which is eating up a lot of time.

I think it bears mentioning that Walter has shed some light in the DNG regarding the exception handling mechanisms and the ABI in D. Basically the D ABI relies on whatever mechanism the native compiler for a given platform provides (where possible). So this confirms what I've been told, that there's no way to load windows-based D binaries on linux and vice-versa, in a runtime manner.

However, I don't think the differences cited exclude the possibility of implementing trampolines to convert from one exception format to the other. This would require a *lot* more muscle in DDL's linker, which is something that is way out beyond the horizon (v2.0 perhaps).

Enki

(The short-short version: I took the long way around to devising a potential ddoc generator solution that doesn't rely on DMD)

I mentioned enki before around here, but at the time it was mostly theory and two broken attempts to build it. I'm happy to say that the third time is the charm, and I'm well on my way to putting a serious layer of polish on this side-project.

I hit a point recently where I was re-doing the same thing over and over again when tackling new grammars and writing text parsers to support them. I spent a serious amount of time reading up about Peri's amazing LanguageMachine and thought that, while it is a grand project, it doesn't suit my needs very well. I also found the DGrammar project to hold some promise, but it fell into disuse some time ago. Obviously, there's a good deal of momentum behind the idea, going back to lexx and yacc, but where was my D-specific solution to make parsers easier to write?

My gut instinct was that generalized, table-based grammar engines are great for conversion tasks, but require gobs of scripting support for any serious processing in response to parsing events. As D is already a wonderfully generalized grammar, well suited for backend work, I set out to devise a parser frontend generator that will use a more targeted grammar instead. So far, so good.

Being very familar with how to work with good 'ole EBNF, I decided to embrace this as the core grammar for Enki. I added some extensions to allow for annotations, so that each production will trigger a function call or the creation of a class instance, which is perfect for generating typed parse trees, or event-based. What's more, is that these annotations also make use of typed bindings, that can help funnel production results into these backend events.

Code:

# Typical EBNF rule for "hello world"

Test ::= "hello" ws "world";

# When Foo is satisfied, call a user-defined function in the backend.

Foo  = String myfunc()  ::= "foo" | "bar";

# Create an instance of Gorf, passing along results from other productions
# Depending on which is satisfied, either a or b will be null/unset

Gorf = new Gorf(a,b) ::= Test:a | Foo:b;

# Create a line-by-line listing of an entire file, and cram it into a "MyFile"
# ":~" concatenates results into the binding
# "eoi" desgnates "end of input", "eol" designates "end of line"

File = new MyFile(String[] lines) ::= {({any} eol):~lines} eoi;


The above presently works, and is happily generating about 10-11 lines of parser code per line of Annotated EBNF, which includes error handling support. The code isn't always 100? efficent, but it is human-readable, as I wanted developers to be able to manually tweak the code if need be.

Said generated parser contains everything needed to interface with a stock parser base class, which provides any custom terminal expressions like "letter", "digit", "any", "eoi" , "eol", "sp", "ws" and so forth. This was done as these are very common terminal expressions, and can be represented in D far more efficently than an interpretation of EBNF could ever be. As such, there is nothing keeping the developer from adding custom hooks and terminal expressions should speed or maintenance be an issue.

And if all that wasn't enough, Enki's parser frontend is 100? self-hosted. In fact, I'm using the current revision to extend the current design even further:
Code:

# parse a delimiter and use it as a dynamic terminal ("." notation)
# returns the binding 'value' directly (presently supported)

StringExpr
  = String value
  ::= ("\"" | "'"):delim {any}:value .delim;

# pass a value forward, much like a template in D

Foo ::= letter:x Bar!(x);
Bar(y) ::= .y .y .y;

# iteration limits
Elipsis ::= "."<3,3>;
Elipsis2 ::= "."<3>;
HexNumber ::= hexdigit<1,4>;
ZeroOrMore ::=  foobar<0,3>;

#negation (not sure how useful this is)
# Foo passes if and only if Bar fails
Foo ::= !Bar;


I've left iteration counts/limits and such out for now, as I cannot yet devise a satisfactory syntax for such a thing.

Anyway, once I'm happy with the feature set, you all can expect to see it here as a partial replacement to the current ddoc mess that is in SVN right now.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Jun 04, 2006 7:09 am    Post subject: Reply with quote

Status

Working hard on tracking down a nasty OMF bug. I now have a pattern and a mode of failure pinned down, and am working on finding the root cause to solve this particular bug.

As far as the project schedule goes, it'll need to be drastically revised. I'm still toying with the notion of releasing 1.0 w/o ELF support in favor of the community getting an earlier crack at using DDL - documentation included. With that, I'm still experimenting with Enki to create some better ddoc tools. So far, I have a D lexer prototype, which should be adequate for feeding back into a secondary Enki parser for the actual D and ddoc parsing passes. Twisted Evil

OMF - Segment Relative Fixup Bug
H3r3tic wrote me about an issue regarding getting the latest source to load correctly under the latest D compiler. I did a *lot* of tracing and hacking to come to the conclusion that the code still isn't handling Segment relative fixups correctly.

Fortunately there's only one case of this particular problem. Essentially, the fixup record of this type is defined as an offset into a particular (virtual) segment, and an amount to increment that address by. For whatever reason, the fixup segment offset is being corrupted or parsed incorrectly, which caused the fixup to point into non-segment controlled space.

It just so happens that this usually points into the space controlled by the fixup records themselves, which led to some *very* strange data. I added an assert to help catch this case, which has led to stopping the debug output hundreds of lines earlier.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Jun 05, 2006 10:41 am    Post subject: Reply with quote

Status

Working on wrapping up this bugfix for Tom and company. It seems that I had data that was pointed to slices of the input buffer, rather than discrete portions of allocated memory (Embarassed). I should have a fix for this published soon.

Enki

Outside that, I've pushed Enki a little farther forward, allowing for more interesting ways to implement parsers. One issue came up with implementing a D lexer, that required a novel solution:

Code:
Keyword
  = bool addKeywordToken(tok)
  ::= "public":tok | "private":tok | "protected":tok;


This works well, but when the parsed token (tok) is passed to addKeywordToken(), the method will then have to sift tok through a swtich to yield an enum value like Tok_Public or somesuch. So I needed a specialized way to get Enki to allow for some kind of inline conversion or assignment. I came up with a 'literal' operator that allows for direct assignment of a D identifier to a binding:

Code:
Keyword
  = bool addKeywordToken(uint tok)
  ::= "public" @Tok_Public:tok |
       "private" @Tok_Private:tok |
       "protected" @Tok_Protected:tok;


While a little more verbose, the result is the parser is now leveraged much more directly, and the tokens can be converted directly into any arbitrary value. The only drawback is that like with any predefined terminals, Enki cannot verify if the identifier used after '@' is valid - so the developer is trusted to understand what's wrong when the generated parser fails to compile.

There's another subtle hole in the grammar that was pretty easy to cover thanks to some other newly coded extensions. The problem is with ambiguity:

Code:
Token ::= Keyword | Identifier;


Keywords have to co-exist with Identifiers such that a valid identifier may begin with a keyword. As all keywords happen to be valid identifiers, the onus is on the Keyword rule to somehow tell the difference.

Now Keyword, as defined earlier, is actually incorrect since it doesn't exclude the possiblity of things like 'publicfoo' or 'charvar'. This is easily solved by using Enki's negation operator ('!') and its test operator ('/').

Code:
Keyword
  = bool addKeywordToken(uint tok)
  ::= "public" /!IdentifierChar @Tok_Public:tok |
       "private" /!IdentifierChar @Tok_Private:tok |
       "protected" /!IdentifierChar @Tok_Protected:tok;


... where /!IdentifierChar has the effect of "test for anything but IdentifierChar", which is perfect for making sure that any given keyword is not a part of an identifier.

The next addition will probably be a 'literal test operator' probably represented by '&'. This way, parse expressions will pass D identifiers directly to 'terminal()', so the enum-based lex data can be used. terminal(uint) could be defined such that it inspects a token array rather than raw char data, which again is a part of allowing split lexer/parser designs.

Code:
ProtectionAttr ::= &Tok_Public | &Tok_Private | &Tok_Protected;

_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Jun 05, 2006 8:54 pm    Post subject: Reply with quote

Status

Looks like I was further from the mark than anticipated. The problem is now firmly within the realm of communal data records and how to reconcile them during a link. I'm not 100? sure what needs to be done to get these to work correctly, but I'm going to sleep on it tonight and study the OMF spec to gain some deeper insight on how a linker typically handles this. I'm going to do my best to avoid creating an API change here, but it may be inevidable.

Enki

Edit: Enki 1.0 is now released!

http://www.dsource.org/projects/ddl/wiki/Enki

Download and Sourcecode:
http://svn.dsource.org/projects/ddl/trunk/enki/
http://svn.dsource.org/projects/ddl/downloads/enki.1.0.win32.bin.zip
http://svn.dsource.org/projects/ddl/downloads/enki.sdk.1.0.src.zip

The short version: build 'bootstrap' to create the self-hosting EnkiParser.d and accompanying enki.bnf file. While all the source is provided, inspecting the source and the .bnf file together will give a rather deep insight as to how this tool works.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Jun 07, 2006 9:24 am    Post subject: Reply with quote

Status

DDL Development is still on hold. Confused

Lars and I are working on fixing the current problem with common data between modules and how this plays with both OMF and ELF support. It looks solvable, but is going to require some very careful planning to cover all the edge-cases present in this problem space. For such a simple feature, it creates an amazing amount of trouble.

In short, the new way of thinking is to make symbols carry some semantic information as to their 'strength' (weak,local,global) and take responsibility away from the discrete module types where linking is concerned. This then puts the responsibility on the linker, as it has to figure out how to resolve symbols of several different types instead of just one as before.

Enki and ddoc

Enki 1.0 is out, and 1.1 is on the way. As I work on the ddoc lexer, and implementing Walter's grammar on the digitalmars.com/d page, Enki gets pulled along for the ride. It has been a wonderful catharsis while I ponder how to fix the current crop of challenges for DDL.

The hardest part of this exercise has been figuring out how to properly Lex a D sourcefile so it can be parsed quickly. My Token struct currently looks like an amalgam of a variant implementation and a factory. Thanks to the latest improvments to Enki, I'm now making heavy use of enums to represent tokens.

The next challenge is to see what it takes to keep multiple char-width support in scope for the lexer. I know that it will need BOM detection similar to what DMD uses, but how will the terminals be processed? I'm assuming that it'll have to convert char data on the fly to compare, but maybe doing a wholesale conversion to dchar first is a better idea?

Anyway, after all this is said and done, I can easily see an Enki 2.0 being hatched that has all these D-isms folded back into its grammar. Things like parsing numeric types or multi-byte strings and such will be very easy to do at that point.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Jun 13, 2006 8:33 pm    Post subject: Reply with quote

Status

DDL is on hold pending design changes. I'm looking into refining the API, specifically for DynamicLibrary and DynamicModule to accomodate the changes to symbol handling. It's my hope that these changes will not only make for a smarter linker, but for easier to implement binary handlers as well.

Enki

Enki is now updated to 1.1 and will probably stay that way for a while. This update includes some bugfixes for codegen problems that a few folks found. Most notably, was a problem with binding type resolution, that caused multiple binding definitions to be output to the generated file. A more subtle problem had to do with the order of operations surrounding the rule predicate, expression and declaration sets of binding parameters.

The final set of improvements have been for the '&' and '@' operators. '&' is now included and will trigger use of the following identifier as a terminal, passed to the terminal() method. The '@' works as it always has, but now accepts parameters, similar to how rules take parameters. The only major difference here is that bindings passed to a '@' (literal) expression must be implicitly castable to the parameter types for that method.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Jun 27, 2006 8:49 pm    Post subject: Reply with quote

Status

I'm posting before I get this next stint underway.

I'm folding in the next set of API changes to pave the way to weak linking, as hinted in the last few posts. I outlined the planned changes in the Wiki, under the new DevNotes section:

http://www.dsource.org/projects/ddl/wiki/DevNotes/LinkerRevision1.2b

The changes are pervasive, but should result in a more streamlined API and module design. Ultimately, this will allow for shared symbol support to finally work correctly.

Enki

BCS has been helping out quite a bit over in the DNG, and has provided two patches for Enki already. One wonders what he's working on, but whatever it is, it's almost certainly pushing Enki to its limits.

Folding in his changes are next on my TODO list along with fixing a few other subtle bugs I came across while building an Enki-based XML parser. As another example of Enki working as designed, said parser works well, compiles fairly compactly and is a cinch to maintain.

The XML parser experiment prompted me to consider setting up at least an example grammar section, that will contain Enki grammars that support common standards like XML, XPath, XSL, etc.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Jul 11, 2006 12:47 pm    Post subject: Reply with quote

Status

No code commit today, just blogging what I've been up to.

I decided that refactoring the OMF parser was long overdue, and was needed in order to usher in the next round of changes. In doing so I uncovered a few subtle bugs in the code, and have already gained new insights into how OMF functions.

The new design uses a two-pass technique rather than a single-pass. This was done to allow for a purely-OMF based parser design that doesn't manipulate DDL-level constructs. The nice thing is that now its a library that should see more usefulness beyond DDL itself. Its also *much* easier to maintain now, and the debug info is greatly clarified. I think this more than justifies the loss in efficency, as the old rendition was just too hard to grok despite being a little faster.

Now OMFBinary, concentrates on decompressing and decoding the data in the object file, and presents it in a way that is easily manipulated by external code like OMFModule. OMFModule takes the OMF records as exposed by the new OMFBinary, and transforms them into segment images plus Weak, Strong and Extern ExportSymbols. It'll still rely on OMFBinary's Fixup table which is fine, since there's almost no good way to re-interpret that data without loosing one's sanity.

Why was this done? A few of the record types cannot be properly resolved into symbols with valid address information without first having 100? of the OMF file's data.

Anyway, the inital debug results at the parsing and symbol interpretation level are very encouraging. So far it can digest single .obj files again, and I'm winding up to tackle .lib files and implibs next.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Jul 18, 2006 7:01 am    Post subject: Reply with quote

Status

SVN now contains some incremental, if not dirty, improvements along the lines of this refactoring. The next update should bring things back to stability, but linking will likely still have some bugs in it.

Rebuilding OMF was the best thing for the library thus far. I was able to find even more corner cases in the specification, and why things weren't working before. We now have a model by which ?99.9 of all symbols encoded in those binaries can be handled in a compliant way.

So what about that ?00.1? Well, the problem is that there are a handful of runtime symbols that act like pointers to the start of a given segment. So they are handed to the OMF loader/linker with a defined location, yet the location (segment) has zero bytes allocated to it. This works fine as a metadata representation of a runtime model, but absolutely stinks for runtime linking.

The solution? Slap 'Extern' on them and move on - they resolve to a null address anyway. So far, there are only about 4 or so such symbols that exist, and they're *always* compiled into the current executable, so the linker should be able to make this work out as it always has.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Jul 19, 2006 1:36 pm    Post subject: Reply with quote

Status

SVN now contains a stable OMF build, with fixes to the ddl API and utilities.

I am still in the process of testing ddlinfo and other utils, but so far things look very promising. I may still have to revise OMFModule a little bit to prevent so many symbols from being posted as 'Weak'. Presently, I'm not even touching the WKEXT records, so its possible that DMD/OPTLINK is relying on the presence of these to indicate a 'true' Weak symbol. Sadly, the OMF spec seems to be lacking in this area. I'll try a few things to see which interpretation looks correct - its possible that OPTLINK really does treat every method definition as a weak symbol after all.

At a minimum, the Extern set of symbols looks 100? correct, which was the original problem in the old loader.

Next up: testing the new link process. Followed by catching up to DMD 0.136.

Enki

Greetz to BCS.

BCS has pretty much been the unofficial maintainer for Enki by posting patches and updates in the DNG. While merging these into 1.2 is still pending, I plan on doing it as soon as time allows.

Also, future versions of Enki will come with a set of grammars that are intended to be useful on other projects. I already have an XML grammar, and am working on D parse/lex grammars as well. Contributions for this "grammar library" are welcome!
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Jul 25, 2006 8:18 pm    Post subject: Reply with quote

Status

After beating my head against my desk for a few days straight, I discovered a bug with the fixup processing in the OMF loader. It seems that extern references are skewed by the number of cextdef instances in a given module - or something to that effect. I'm presently tearing apart my code and comparing it to the OMF specification for FIXUPP records. Hopefully, I'll figure out what is wrong with the current algorithm.

HLA

More ramblings about projects barely related to the task at hand. Wink

I placed some notes out on the Wiki, under Community, for a High-level-assembler concept. The idea of runtime code-generation, without use of a separate compiler, seems to be within reach. In digging deeper into the IA32 and AMD64 instruction set documentation, I've found myself tossing around some code that would look quite at home in an ASM compiler.

My god, has backwards compatibility made this specification a complete and utter mess. OMF looks sane by comparison.

I think the lowest rung of an eventual HLA design will in fact be a nearly full-blown ASM compiler, that is fed a parsetree or pseudocode from parts higher up in the chain. Granted that's overkill, but I figure that if I'm in the neighborhood, I might as well give it a shot.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Wed Jul 26, 2006 1:50 am    Post subject: Reply with quote

Hope it's OK to post here?

I'd like to understand the value-proposition of HLA better ~ I would have thought that binding Lua, some other interpreted high-level language, or the D compiler itself would be both more attractive and advantageous to users? Especially in light of the dynamic reflection and native binding offered by ddl?

- Kris
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Jul 26, 2006 7:36 am    Post subject: Reply with quote

kris wrote:
Hope it's OK to post here?


Absolutely it is. I just tend to push the longer sub-threads out to seperate ones, to keep things organized. Its always cool to post here.

Quote:

I'd like to understand the value-proposition of HLA better ~ I would have thought that binding Lua, some other interpreted high-level language, or the D compiler itself would be both more attractive and advantageous to users? Especially in light of the dynamic reflection and native binding offered by ddl?

- Kris


It depends on the application though, doesn't it? Smile

Its a longshot from here (and its certainly of lower priority than other stuff I have on the table), but I'd like to have library support for native code generation, so we can do things that the current crop of tools and bindings can't touch.

in short, I think there's value in having runtime proxy generation for things like RMI or CORBA. There's also value in just having an open toolkit that is just ripe for use in a compiler. Most alarmingly, there's nothing setting the stage for 64bit D or even secure variants of the language. As raw code generation is really one of the biggest stumbling blocks for those kinds of work, I think it's seriously worth looking into. Smile

But you're also right: such a thing can't replace the ease-of-use that bound scripting languages have for certain tasks. Even for things like DSP, where your code is already written in D for the most part, invoking the compiler is simply easier to do.

So its really more of a low-level tool, for low-level tasks.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries All times are GMT - 6 Hours
Goto page Previous  1, 2, 3 ... 9, 10, 11, 12  Next
Page 10 of 12

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group