FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Status Log
Goto page Previous  1, 2, 3 ... 7, 8, 9, 10, 11, 12  Next
 
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries
View previous topic :: View next topic  
Author Message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Feb 03, 2006 3:54 pm    Post subject: Reply with quote

larsivi wrote:
As mango.io is now in Ares, I think we should consider relying on that. We don't need mango.io copies all over the place, dependency on ares will come at some point in any case. Still, we'll need other parts of mango though.


Agreed. We can leave mango out of the DDL SVN for now (or forever) until we get things tightened up. We're not at the deployment/packaging stage yet anyway.

The matrix of library parts is much more complicated, thanks to supporting .zip. As a phobos component, wrapped around a C .lib file, it makes things every bit as complicated as using Mango. I wish I had a quick answer for all this, but it looks like the packaging and dependencies for DDL will be resolved with time.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Fri Feb 03, 2006 4:00 pm    Post subject: Reply with quote

I've branched out the elf development, btw. Won't commit more elf stuff to the trunk for a while.

Creating branches was a piece of cake Smile
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sat Feb 04, 2006 5:15 pm    Post subject: Reply with quote

larsivi wrote:
I've branched out the elf development, btw. Won't commit more elf stuff to the trunk for a while.

Creating branches was a piece of cake Smile


Yea, branching is one thing that SVN does very well. Feel free to use that as you 'sandbox' as you continue to hack away on ELF. Just be prepared for a very ugly merge when you're done. Wink

FYI, the mango-izing of things is coming along nicely. I think you'll find the changes refreshing.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Feb 06, 2006 4:22 pm    Post subject: Reply with quote

Status

Still plugging away at Mango refactoring. The refactoring thread has been useful for working some issues out with Kris; it would appear that I still have much to learn about everyone's favorite D library.

Lars is also finding the DDL architecture to be somewhat deficient when composing a static linker. The idea is to have a tighter testbed for manipulating the ELF loading components. I have a gut feeling that this may result in some interface changes that will allow all of the loaders to participate in compile-time linking as well as run-time.

Ideas

I read a pretty strong post out on the DNG regarding the need for stack traces, and line number info in debug builds. Regan voiced his opinion from the standpoint that as a customer-facing issue, it is the only way to get very vital feedback from ailing software. I was reminded that the notion of runtime feedback like this (sans debugger) has implications beyond just us developers on our own machines.

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/32890

So I thought about why we are where we are and thought that perhaps Walter is hesitant to stuff so much symbol data into a given program build. That in turn reminded me of the size concens I had with static binding with DDL.

And then it hit me: DDL might be poised to help solve this problem.

It ocurred to me that the stack, at any point during execution can be 'inspected' for return address information. Each CP value pushed on the stack would have to correlate to a function or method in any given Library's symbol tables. The data is all there: just dig down the stack until you hit _Dmain(), capture all the symbol names along the way and perform the action at the point of exception (read: a custom exception class).

However, this project is a good ways off from accomplishing a stack trace. DDL is deficient for this task in a few ways. First off, it's not capturing debug information, so line-number reporting isn't possible. Second the current data model is streamlined for symbol lookup by name, not by address. Third... we're not done with 1.0 yet! Wink

As it so happens, the lookup problem is hand-in-hand with the reason why we don't have a full reflection suite hooked up to DDL either. Also, the lack of debug info parsin has been of interest to AgentOrange, as he's crafting a debugger of sorts. So both of these were going to be done at some point anyways. Stack Tracing should fall out of all this eventually, I just don't know when.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Feb 21, 2006 8:19 pm    Post subject: Reply with quote

Edit: Dirty code commit in SVN now. Just about everything has been touched in this pass, with much more to do.

Status

The refactoring task is certainly taking up what little precious free time I've had this month. In response, I've pushed the 1.1B deadline back another four weeks, to allow enough time to get everything done right.

Don't even ask me about how testing is going... ugh. This is like orthodonture as applied to programming: slow and painful -- but look at that smile!

Overall, the changes made have already begun to transform how I look at handling I/O tasks within DDL. I've made heavy use of typedefs and custom Reader classes which has led to much cleaner code in places. Its also opened the door for further refactorings that will take this "type driven" mode of coding to a very tidy conclusion.

In a nutshell I've converted a lot of code that looks like this:
Code:
uint foo = cursor.getDWord();

To this:
Code:
uint foo;
reader.get(foo);


Practical Asthetics

As all the reader calls are uniform (.get() in this case), your eye naturally follows the portions that are different from line to line. This gives you a near instant recognition of what the current parsing task is, from a purely semantic point of view:

Code:
//from OMFBinary.d
      reader.get(flags);      
      reader.get(attributes);
      reader.get(alignment);      
      reader.get(dataOffset); // enum data offset field is ignored (always zero)
      reader.get(typeIndex); // type index is thrown out
      reader.get(grpIdx);
      reader.get(segIdx);


... so if you really have to know how wide each field is, just look at their declarations instead. That kind of information is no longer noise on the line. Its a small change, but I can already see how its going to make things easier to maintain.

I've stayed away from the whisper or C++ stream (>>) syntax for now, but another pass may see that change too.

Reader Extensions

After asking around, I found that adding getAll() and hasMore() methods to DDLReader was the easiest way to adapt DDL to use Mango. It certainly breaks the paradigm of the Reader class itself (Kris maintains that its really for conversion of the conduit's data), but it's really damn handy. This goes especially for when you want to pass your Reader class around all your parsing methods.

However, that may not matter past 1.0. I'm starting to realize that via custom IReadable classes, structs and typedefs, one could fold the entire parser *into* the Reader class itself.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
larsivi
Site Admin


Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway

PostPosted: Wed Feb 22, 2006 3:42 pm    Post subject: Reply with quote

Hmm, I'm either blind or tired, but I can't seem to find a way to set the file position to read.

The ELF format promises nothing about where to find data in the file, and not even if all the bytes contain anything sensible. So when using Stream, I used lot's of seek to get to the points where I knew there was data. Don't DDLReader have a similar concept?

Seems like Buffer has the method I'm looking for, skip(int), but it needs to be exposed somehow.
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Feb 22, 2006 4:21 pm    Post subject: Reply with quote

larsivi wrote:
Hmm, I'm either blind or tired, but I can't seem to find a way to set the file position to read.


Get used to that feeling. Smile Mango is a great library, but it forces you to rethink how you approach I/O tasks at a very fundamental level. On the surface it appears to look like every other lib out there, but in practice it is a very different beast.

Quote:

The ELF format promises nothing about where to find data in the file, and not even if all the bytes contain anything sensible. So when using Stream, I used lot's of seek to get to the points where I knew there was data. Don't DDLReader have a similar concept?

Seems like Buffer has the method I'm looking for, skip(int), but it needs to be exposed somehow.


Actually, skip() does some of what's needed, but it only applies within the context of what data is stored in its buffer. Its not a good replacement for a proper seek(). However, if you can guarantee that the buffer contains all of your data, then there's no harm in using it.

The only thing I've found for seeking with buffered reads is this:
Code:
assert(reader.isSeekable);
auto seekable = cast(ISeekable)(reader.getBuffer.getConduit);
seekable.seek(amount);


Which admittedly isn't that bad, it's just verbose. All you need to do is keep that seekable object on hand along with your reader, when you're in a seek-heavy section, if you don't want to repeat the overhead of that cast to ISeekable.

I've used this trick in the OMF code that's in SVN now.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Mar 08, 2006 8:37 pm    Post subject: Reply with quote

Status

I've made some small progress on the ongoing refactoring of the codebase. So far, the simplest of all the filetypes (.ddl) is now parsable via ddlinfo.

I need to spend some more time making some proper test harnesses for the various types, so problems will be easier to isolate. I also have to make the debug-level output much more uniform - this will be sent through the Mango logging infrastructure once that is put in place.

I have not upgraded to DMD 0.149, so the current codebase is technically 2-3 DMD revisions behind. I wanted to get a little more utility and stability out of the library first before folding in those changes as well. I don't forsee this being much more trouble than replacing 'bit' with 'bool', but there's bound to be some other changes brought on by this.

ArgParser

While its not a huge change, it does make life easier for using the various DDL utilities. The ArgParser will now pass an ordinal on to the default argument binding, which makes short work of tracking how many default/standard args are passed.

Here's an excerpt from ddlinfo.d:
Code:
ArgParser parser = new ArgParser(delegate uint(char[] value,uint ordinal){
      if(ordinal > 0) throw new DDLException("Invalid argument ?s",value);
      filename = value;
      return value.length;
   });
         
   parser.bind("-", "r",delegate void(){
      rawOutput=true;
   });
   
   parser.bind("-", "a",delegate void(){
      displayAttribs=true;
   });
   
   parser.bind("-", "s",delegate void(){
      displaySymbols=true;
   });
   
   // parse the arguments
   parser.parse(args[1..$]);


(If that's not readable enough, you can also use bindDefault() instead of the constructor syntax for establishing the default binding.)

This way the utils can now enforce minimum and maximum number of arguments in addition to all the typical behavior. This opens things up so that the default args are merely order dependent, rather position dependent. So in the case of ddlinfo, as long as there's only one default arg, it can appear in any order on the command line, regardless of where the switches are placed:

Code:
ddlinfo -asr foobar.ddl
ddlinfo -a foobar.ddl -r
ddlinfo foobar.ddl -r -as
ddlinfo -r -a -s foobar.ddl

_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Mar 14, 2006 9:13 pm    Post subject: Reply with quote

Status

Changeset 161 is online. This constitutes the best of the refactoring effort so far, with the DDL and InSitu loaders at almost 100? capability. Next stop: OMF testing and bugfixes.

Ideas

I'm seriously considering using DParser as the backbone for a doc generation tool. So far, I'm quite impressed with the quality of the codebase and the wonderful job AgentOrange has done translating Walter's C code over to D. I'm still having some small issues with getting it to build correctly, but that's for another thread.

Also, I've toyed around with php-based newsreaders in an effort to find something more capable than the current software in use over on digitalmars.com. My primary gripes with that suite is the navigation falls apart once you're looking at an article (can't see the thread side-by-side with the article), you can't cancel a post, there's zero profile management and attachments simply don't work.

While I understand that usenet works fundamentally different than your typical web forum, there's no reason why we can't close the gap.

The pnews project holds out some promise, but requires some special PHP support that my host doesn't provide. The result is something that is about the same quality as what's on DM, only with profile management, post deletion and phpBB integration. There are several others out there in this niche, with varying capability, but this one stands out as the most complete and hackable.

http://www.djsolaries.com/pnews/

Rolling a custom solution doesn't seem to be that bad: PEAR has a very complete NNTP wrapper that handles all the nuts-and-bolts. I might just go this route as a self-reward after the current DDL milestone is done. Wink

Along the same lines, I'm tempted to throw together a D news portal of some kind that aggregates *everything* related to D. Something that showcases the current goings on in the newsgroups with RSS feeds from the dsource Trac, blogs and whatever content I can manage to scrape together. I don't think we have quite the critical mass for a digg style approach, so it'll rely on some kind of automation to get the job done.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sat Mar 18, 2006 8:02 pm    Post subject: Reply with quote

Status

Gave the OMF classes some much needed attention this time around. While they pass a very minimalistic test, I can say that applying mango to the core set of binary handlers has worked well. My next stop is to fix some of the more broken parts of OMF, and take a few notes on how to improve things for post 1.0.

I'm upgrading to DMD v0.149 after this post, so future updates will be valid to compile against this compiler release.

Edit

It looks like my schedule of things to do, which is partially governed by the tickets out on trac, is as follows:

- Finish testing the refactored OMF library
- Fix the OMF LIDATA record issues
- Resolve #37 - Symbol Resolution Bug
- Resolve #34 - Bless/nested class bug
- Resolve #23 - Implib handling (add test cases to verify)
- Sweep remaining TODO's out of OMFBinary in order to satisfy #16

Things that are likely to get dropped from this milestone:

- #14 - Complete documentation (sorry guys, there's simply too much to be done - it could easily slip past release as well)

Things that won't be fixed until the next milestone:

- Refactoring COFF and Ar support
- Finishing ELF
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
jcc7



Joined: 22 Feb 2004
Posts: 657
Location: Muskogee, OK, USA

PostPosted: Sun Mar 19, 2006 11:21 pm    Post subject: PHP-Based Newsreader Reply with quote

pragma wrote:
Also, I've toyed around with php-based newsreaders in an effort to find something more capable than the current software in use over on digitalmars.com.
Great!

pragma wrote:
My primary gripes with that suite is the navigation falls apart once you're looking at an article (can't see the thread side-by-side with the article), you can't cancel a post, there's zero profile management and attachments simply don't work.
The navigation is pretty horrible. Attachments are definitely a pain, but they actually can be decoded from what the current web interface provides. Also, the web interface often garbles code snippets that people try to post in plain text.

pragma wrote:
While I understand that usenet works fundamentally different than your typical web forum, there's no reason why we can't close the gap.
Absolutely.

pragma wrote:
The pnews project holds out some promise, but requires some special PHP support that my host doesn't provide. The result is something that is about the same quality as what's on DM, only with profile management, post deletion and phpBB integration. There are several others out there in this niche, with varying capability, but this one stands out as the most complete and hackable.

http://www.djsolaries.com/pnews/
Wow. It looks great!

pragma wrote:
Rolling a custom solution doesn't seem to be that bad: PEAR has a very complete NNTP wrapper that handles all the nuts-and-bolts. I might just go this route as a self-reward after the current DDL milestone is done. Wink

Along the same lines, I'm tempted to throw together a D news portal of some kind that aggregates *everything* related to D. Something that showcases the current goings on in the newsgroups with RSS feeds from the dsource Trac, blogs and whatever content I can manage to scrape together. I don't think we have quite the critical mass for a digg style approach, so it'll rely on some kind of automation to get the job done.
Sounds like some cool ideas.
Back to top
View user's profile Send private message AIM Address
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Mar 22, 2006 11:19 am    Post subject: Reply with quote

Status

I stamped a few more OMF issues out last night, and pressed on to get linktest1.d to compile and run under the new codebase. In a debug run, I managed to get things to display debug output through most of loading phobos.lib, only to have it fail with an I/O error message, presumably from the bowels of win32 itself. In short, it was complaining about not having enough memory or somesuch for the operation. This may be an artifiact of how Mango is interfacing with win32, but I can't be sure of that.

So I tried a different approach: pre-buffering the file rather than relying on conduit/buffered I/O. This failed promptly, and at the time I didn't understand why. I now know that I made some rather sloppy assumptions in my Reader extensions that assume the presence of a conduit: pre-buffering eliminates the conduit from the picture, so this complicates things. I think I've identified all the major sticking points, but there's bound to be other gremlins in there someplace.

I'm starting to keep an eye on memory consumption, at least from the point-of-view of the task monitor. So far, I'm not seeing the best performance, as the act of loading seems to guzzle memory pretty badly. I'm guessing that its the generation of temporaries and buffer operations that are causing some of this. However, after binaries are loaded up, the in-use pool drops pretty fast as the GC catches up with things. I'm optimistic that after things are profiled, I'll find out exactly where all that RAM is going.

Also I stumbled upon a bug in DMD 149 and 150. It looks like passing a boolean to a variadic function, in a debug build, generates an external reference to a non-existant TypeInfo; the result is that your binary won't link. I suspect that this has something to do with the recent change from bit-to-bool. Its out in the 'puremagic' bugzilla if anyone's interested.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kylefurlong



Joined: 16 Nov 2005
Posts: 29

PostPosted: Wed Mar 22, 2006 6:29 pm    Post subject: Reply with quote

Just out of curiosity, what was the goal of this refactoring? What value was added?
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Mar 22, 2006 6:55 pm    Post subject: Reply with quote

kylefurlong wrote:
Just out of curiosity, what was the goal of this refactoring? What value was added?


I gather you're referring to the Mango refactoring? In short: its more about having superior plumbing, and overcoming the limitations of the original design. Smile

The major value added is from the flexibilty offered by Mango. The original Cursor/Buffer design was totally dependent upon pre-loading the binary files - we can now more or less stream them in via conduits and buffers if we so choose. Also, the various widgets in DDL are now a little more useful outside the context of DDL, as they accept more Mango-like components for I/O.

That flexibilty plays into yet another benefit: appeal to contributors. The model now has more than enough muscle to do things in the various parsers that I never thought would be needed.

Also, backing Mango has made porting between D implementations a lot easier as Ares has already made a move to adopt Mango as it's I/O library, and Mango is already ported to Phobos. So now DDL gets cross-implementation support for free.

As for the other refactoring goals, they were all focused on tightening the screws before the next major layer of functionality is put into place. Smile
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Mar 23, 2006 9:47 am    Post subject: Reply with quote

Status

No code commit yet - still debugging.

I continued my debugging pass on the refactored code, and managed to scare up a few more bugs lurking in bless and the DDL seralizer portions of the codebase. It looks like my getPosition() and seek() implementations are not functioning correctly within DDLWriter; the binaryStart field of the .ddl file format is being calculated incorrectly.

As for the mysterious memory problem, I managed to track that down to my console (cmd.exe); there was just too much output. As luck would have it, loading phobos.lib ran to completion and generated a hefty 6MB log file when I piped the output to a file. Keep in mind that the very same binary caused the showstopping error message, so I'm confident that this was a console problem (or possibly a problem with how Mango uses the console).

By this I've concluded that DDL is past due for using Mango logging support as raw console output can just up and fail under certain conditions. Not to mention that CrimsonEditor likes to bomb out when I load such massive files. At least the Mango logging package is there to be used. I'd be *way* behind schedule by this point if I had to roll my own.

BTW, the other major bug I'm working on is restoring support for the OMF LIDATA record, which has been commented out for some time. Its a sticky piece of code, since the record type uses a form of RLE to store symbol data. The spec is especially unforgiving in its description.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries All times are GMT - 6 Hours
Goto page Previous  1, 2, 3 ... 7, 8, 9, 10, 11, 12  Next
Page 8 of 12

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group