FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Status Log
Goto page Previous  1, 2, 3, ... 10, 11, 12  Next
 
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries
View previous topic :: View next topic  
Author Message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Aug 18, 2005 6:31 am    Post subject: Reply with quote

Status

Made minimal progress last night toward an actual loader-like component. I went about using the current omfloader.d as a template and began trimming the implementation back to the minimum set needed for 32 bit binaries. This is proving tricky since I want to minimize time and space used by this class, but OMF seems to assume that you're at a lower level than where D is.

An ideal implementation would throw COMDAT and LEDATA chunks into an open space at the top of the heap, then mark the used portion as allocated and slap the base address page into a segment register. While this works great in 16bit, single-threaded environments, it doesn't fly here. Instead, I'm in the game of doing array appends to build up data, which forces a lot of reallocation and rewriting thanks to COW. I could pre-allocate in larger chunks, say 32-64k at a time, but this seems a bit wasteful.

I hit a small roadblock at the fixups section, as it still isn't 100? clear to me how segment-relative and non-segment relative fixups interact with the fixup target, frame and offset to create an actual address. For the most part, it makes sense, but the web in general is light on details. I'll have to run some more trival examples through omfviewer (previously omfloader) to probe at what exactly means what.

I'm pretty sure that even after I get that "right", its still where the majority of my debugging is going to take place. Either the fixups work right, or you load, link and then segfault/GPF without any clue as to what went wrong. I'm looking forward to that. Wink
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sat Aug 20, 2005 10:46 pm    Post subject: Reply with quote

Status

It still feels like its too early to claim success, but my last run through with my testbed (however crude) certainly felt like one.

I grappled today with getting the OMF interals to cooperate correctly. What I was missing was an implied relationship between EXTDEF records and PUBDEF /COMDEF records; EXTDEF associates with the two latter record types by name. Also, it would seem that COMDEF plays the same role as LEDATA with repsect to FIXUPP records (this wasn't documented). Anyway, it all became clear to me after I drew a relationship diagram and traced the data around while double-checking what was on my monitor.

The last bit came into place when I finally understood how the fixups are supposed to work. Generally, speaking there are two basic flavors: self-relative and absolute fixups. The former being what JMP and CALL instructions need to use (constant to add to IP). It took a while to get the relative fixups to work correctly, as apparently the forumla is "target-dest-4" with the "-4" being crucial as this offsets the pointer by the width of the field being written.

Anyway, I can now get an OMF to resolve its internals and spit out what symbols are unresolved (dependencies) and what symbols are defined (publics). For basic arithmetic functions, I can bind to a symbol and call to get the result of the calculation. I can also bind to static symbols with the same effect. Cool

The next step is to bridge the gap between the loader and an actual DDL interface. There are many steps involved in this stage:

    - "Reverse Binding" by which existing (and not-yet existing) symbols are supplied *to* the loader (resolving dependencies).
    - Bootstrapping whatever moduleinfo may be present in the module.
    - Probing to determine what module/namespace is supported by this run at a binary file.


Time to get some sleep.

Edit: I don't know what's up with dsource's clock, but it doesn't have the right time. The time is currently 3:21AM EST
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Sun Aug 21, 2005 4:56 am    Post subject: Reply with quote

Looks like you're making some progress, Eric. Smile

A thought just occurred to me: in the process of porting dmd to linux, Walter had to convert the code generator to spit out the elf format on that platform. I wonder if it would be useful to use that same system on win32... ie request Walter to add his elf object generation support on the win32 platform too. It may be of little use at this point, but perhaps it could help you pave the way for bigger and better ddl modifications. (albeit, his current elf support doesn't know how to do shared objects or PIC which are so critical to the whole dynamic library goal).

I must be in la la land.

-JJR
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Aug 21, 2005 8:40 am    Post subject: Reply with quote

JJR wrote:
Looks like you're making some progress, Eric. Smile

A thought just occurred to me: in the process of porting dmd to linux, Walter had to convert the code generator to spit out the elf format on that platform. I wonder if it would be useful to use that same system on win32... ie request Walter to add his elf object generation support on the win32 platform too. It may be of little use at this point, but perhaps it could help you pave the way for bigger and better ddl modifications. (albeit, his current elf support doesn't know how to do shared objects or PIC which are so critical to the whole dynamic library goal).

I must be in la la land.

-JJR


Interesting, but I thought that ELF support was due to using GCC as a backend? I didn't think that the front-end code had anything to do with binary-level output. Smile

Anyway I'm *definately* going to write an ELF loader as well -- the format is far superior to OMF in a multitude of ways, plus it just has better documentation so it should be easier to support. You're right about it paving the way to bigger things: the cross-OS benefits of the format are too big to ignore.

As for shared objects and PIC, I think all you need to do is use the 'dl' tool that comes with binutils. AFAIK, the default mode for ELF output from GCC is not position independent, so the tool is needed to undo that.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Sun Aug 21, 2005 11:53 am    Post subject: Reply with quote

I think JJR is talking about Walter's version of the linux compiler, rather than David Friedman's GDC. You'd probably still need to bind Win32 DLLs, so there's likely no escaping the OMF hell.
Back to top
View user's profile Send private message
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Sun Aug 21, 2005 12:51 pm    Post subject: Reply with quote

Yeah. I was talking of Walter's version on linux. I got my terminology mixed up (ie. code generator). His back end produces *.o files. I assume they have to be elf format to be linked in by the linux linker. I imagine Walter had to study the elf spec somewhat to generate these files in that format.

I'm not sure what you mean about using binutils for creating shared objects. To make linux shared libs with gcc, you have to first compile the the code with the PIC flag (-fpic) set for each object file, and then run gcc again with directions to create the shared lib with all the object files (gcc -shared ...)

Perhaps you are referring to the steps required to make shared libs on windows with gcc? If so, I guess I was off on another tangent... my apologies.

Sorry If I'm confusing the issue. I'm probably not fully understanding what you're talking about.

-JJR
Back to top
View user's profile Send private message
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Sun Aug 21, 2005 2:04 pm    Post subject: Reply with quote

kris wrote:
I think JJR is talking about Walter's version of the linux compiler, rather than David Friedman's GDC. You'd probably still need to bind Win32 DLLs, so there's likely no escaping the OMF hell.


Very true, Kris. I guess there would be no escaping OMF stuff no matter how you look at it.

-JJR
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Aug 21, 2005 5:20 pm    Post subject: Reply with quote

JJR wrote:
kris wrote:
I think JJR is talking about Walter's version of the linux compiler, rather than David Friedman's GDC. You'd probably still need to bind Win32 DLLs, so there's likely no escaping the OMF hell.


Very true, Kris. I guess there would be no escaping OMF stuff no matter how you look at it.

-JJR


Yea, its tough to avoid. At the very least, Walter managed to draw the line at 32-bit support only: its made a lot of the fixup and address calculations easier to grasp. If we're talking about full-on win32 object support, then we really need to start talking about COFF as well. Other C compilers more readily generate that format, and more .lib files in distribution are formatted that way as well.

At the end of the day, it doesn't really matter to terribly much; I was pretty much counting on having to support all three types (COFF, OMF, ELF) from the very start. Wink
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Aug 22, 2005 7:31 am    Post subject: Reply with quote

Status

Made some sundry changes to the OMF parser, and integrated it into an actual library suitable for a DDL project architecture. One can now use a generic interface to load the object, and the correct loader will be invoked to handle the creation of a "DynamicLibrary" class. The DL in turn can be interrogated for its exports, for late binding. Hooks are also provided for providing symbols for binding as dependencies.

To complete the OMF parser, I still need to include .lib support so that I can load things like "phobos.lib" for binding.

For DDL as a whole, there are still a whole host of outstanding issues that need resolution. I have a feeling that the notion of bootstrapping modules and resolving dependencies at runtime will be far more bug-prone than the binary file parser modules. Time will tell.

I've decided that its time to splinter off from DSP and place this in a proper project. I've put the request out in the "Project Ideas" forum, so hopefully Brad will pick it up and place a new repository out there. If so, that will pretty much mark the end of this thread, and all news regarding DDL will go into the appropriate forum. Very Happy
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Sep 04, 2005 6:43 pm    Post subject: Reply with quote

Status

I'm back from my business trip, and so I'm back in the lab ready to get cracking on some code.

I scribbled down plenty of notes last week regarding how loading and linking should work. For now, I'm keeping DSP in mind as a test-case as it's the most aggressive and demanding scenario for use I can think of.

In order of dependency, here are the facets that need to be covered:

    - library validation
    - library caching
    - search path management (local FS and URL based)
    - runtime linking across modules
    - runtime compilation


(DSP would just slap the DSP preprocessor on top of this stack)

I've tried to work this into a clean, template-free class hierarchy and it refused to yield a typical class tree. This is largely due to the validator and the search path handlers can have variations in behavior, so they need to be pluggable. I discovered that if these are pulled to the side, the rest fall into a neat hierarchy.

Code:
interface LibraryValidator{}
class NoValidator : LibraryValidator{}
class DateTimeValidator : LibraryValidator{}

interface LibrarySearchPath{}
class NoSearchPath : LibrarySearchPath{}
class FileSearchPath : LibrarySearchPath{}
class URLSearchPath : LibrarySearchPath{}

class Loader{
   LibrarySearchPath searchPath;
}
class CachedLoader : Loader{
   LibraryValidator validator;
}
class Linker : CachedLoader{}
class Compiler : Linker{
  char[] compilerPath;
}


In a nutshell, the validator is used to determine when to purge a file from the cache and the search path supports searching and loading of a binary file from a given path. The compiler class at the bottom would be a runtime hook for DMD if it is installed on the machine; it would feed-back into the loader scheme by immediately loading whatever the compiler outputs.

(DSP would simply specialize the Compiler with a preprocessor step for DSP code, and would use the DateTimeValidator and FileSearchPath 'plugins')

I also drew up some other notes for pre-compiled XML, and a runtime Emit interface... fun stuff. I think I'll leave those on the back-burner for now until I can get the rest of all this off my plate.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Sep 05, 2005 2:51 pm    Post subject: Reply with quote

Status

Things are going very well. The OMF loader can now consume phobos.lib, including the .cpp binaries for recls (Thank you Mr. Wilson). I also have stub-outs for debug information that I may get around to finishing at a later point.

The recls stuff was miserable thanks to lots<of<templates>>.inTheSource(). There is an undocumented extension being used for identifiers that exceed 254 characters in length. This took several hours in a hex editor and many, many printf() statements to figure out how to handle. The fixed code is essentially reverse-engineered, and so noted in the code. Test case #2 supports this, and tests things with a suitably long ident.

I stumbled across an odd behavior with returning collections from members in D. Apparently, an array of classes cannot be implicitly casted to an interface type on return without generating buggy code. I may report this as a bug if I cannot find a precedent on the bugs list.

The library has a formal test suite that will grow alongside the code as new cases are discovered. I've opted for a separate file tree instead of using embedded unit tests and such.

The proposed class tree from my last post is now stubbed out in the repository. This was essential to get right as the next series of steps involves getting full-on runtime linking to work correctly. Plus, module info constructors and destructors also need to be hooked and executed appropriately. I anticipate this to be the hardest part yet, as bad function pointers and fixups are notoriously hard to debug.

When I get around to developing the habit, I'll be using the doc format proposed by Walter on the digitalmars site. Who knows, we may get doc generation Real Soon Now (tm).

Last, but not least, AgentOrange has pledged his support and is looking into throwing together a COFF loader for the project. I am very grateful to his and the support of others for this project. Thanks everybody!
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Sep 06, 2005 7:32 pm    Post subject: Reply with quote

Status

Made some incremental improvements to the overall library, most notably in the handling of externals, publics and dependencies for OMF binaries. Test4 now produces the following results:

Code:
Test 4 Modules (1):
test\test4.d
Dependencies (16):
__vtbl_9ClassInfo
__init_11TypeInfo_Aa
__nullext
_D6object6Object8opEqualsFC6ObjectZi
__d_array_bounds
__d_assert
_D6object6Object8toStringFZAa
_D6object6Object5printFZv
__ModuleInfo_3ddl3all
_D6object6Object6toHashFZk
__init_10TypeInfo_k
_D6object6Object5opCmpFC6ObjectZi
__Class_6Object
_D3ddl20DynamicLibraryLoader7loadDDLFAaZC3ddl14DynamicLibrary14DynamicLibrary
__ModuleInfo_3std5stdio
_D3std5stdio8writeflnFYv

Exports (13):
__arguments_AaAa
_D4test5test45test4FZv
_array_4test5test4
__arguments_Aa
__arguments_k
_D4test5test41fC4test5test48Foobar_i6Foobar
__init_4test5test48Foobar_i6Foobar
__Class_4test5test48Foobar_i6Foobar
_D4test5test47varTestFYv
_assert_4test5test4
__vtbl_4test5test48Foobar_i6Foobar
__arguments_Aak
__ModuleInfo_4test5test4


It may look like complete garbage to the untrained eye, but I can assure you that this is every indication that all is working exactly as it should be. What's neat is that one can deduce all kinds of information just by cracking these symbols. Its practically half-way to a reflection interface. Cool

I fleshed out the Loader hierarchy a bit more tonight. This is not intended to be a functional cut, but rather an inital stab at how the interfaces flesh out. So far, I've noticed how useless filenames are in this context; I need to redo everything with filenames *and* module namespaces in mind. I also need a quick-and-dirty way to correlate modules to namespaces (I think searching exports for "__ModuleInfo" will be the way to go).
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Sep 11, 2005 4:01 pm    Post subject: Reply with quote

Status

I now have a working symbol demangler that works wonders on the symbols produced by the current library. I am presently coordinating with James Dunne regarding the matter as it is based on the demangler.d source in the bindings project.

For grins, I stepped ahead and worked on the 'bless' utility that I mentioned some time ago in another thread. It works flawlessly, and can create a .ddl file that embeds another binary of an arbitrary type. The header is perfect for pre-loading, as it contains the bare essentials: what namespaces and what imports comprise the given library. This is essential for turbo-charging any library caching scheme for runtime linking.

Code:
C:\dev\ddl\trunk>bless bless.obj
module bless
import std.string
import ddl.all
import std.file
import std.stream
import std.stdio
import ddl.ddl.DDLBinary
Created 'bless.ddl'


I've also taken some liberties with the design and augmented the Loader scheme to allow loading via streams as well as via filenames. This was done to facilitate loading ddl-based binaries via recursion. It also allows for load-by-filename binaries to delay loading until their details are actually needed; this is both a time and space saving technique which may become important for performance reasons. Later down the road, I may decide to remove it if the gains aren't worth the extra code.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Sep 13, 2005 4:36 pm    Post subject: Reply with quote

Status

Tonight marks a huge update to the repository.

The library now has support for the new DDL file type. This is a special type that unifies all the other supported object types by wrapping them in a convienent header. That header is important as it not only identifies the contents of the file, but also expedites the planned library search and linking processes. For the curious, the BNF of a ddl looks like this:

Code:

   ddlfile ::= header embeddedFile
   header ::= version binaryStart binaryExtension definedNamespaces importedModules
   version ::= 0x00010000
   binaryExtension ::= string
   definedNamespaces ::= count string(count)
   importedModules ::= count string(count)
   
   count ::= uint
   binaryStart ::= uint
   string ::= length char(length)
   length ::= uint   
   
   embeddedFile ::= ubyte?   


DDL's are created by using the bless tool. I must give some credit to Kris for helping hatch this idea a few weeks back.

Code:
C:\dev\ddl\trunk>bless
Bless - DDL wrapper utility - V1.1
Copyright (C) 2005 Eric Anderton
Documentation: http://www.dsource.org/projects/ddl

Creates a .ddl file that wraps the provided object file.
The new file name will adopt the name of the provided
object file with a 'ddl' extension by default.

Usage:
  bless <object file> { -switch }

  -v       verbose output
  -x       (curse) extract object file from .ddl
  -n       no-create (for test purposes)
  -e<ext>  force ddl binary extension to type 'ext'
  -f<file> output to file 'file' (overrides default name)

  Note: The -e option is ignored when -x is present.


Supported Object Extensions:
ddl lib obj omf omflib


This is in the repository now. Just get the latest code, 'build bless.d' and off you go. Smile

(although it may look like it, bless will not let you insert a ddl into a ddl. Although the runtime will sure as hell try to support such a file anyway.)

The ddlinfo tool is has also been added to the set.

Code:
C:\dev\ddl\trunk>ddlinfo
DDL Info - DDL information utility - V1.0
Copyright (C) 2005 Eric Anderton
Documentation: http://www.dsource.org/projects/ddl

Displays information about a ddl or object file.

Usage:
  ddlinfo <object file> { -switch }

  -e<ext>  interpret the file as type 'ext' instead.

Supported Object Extensions:
ddl lib obj omf omflib


Go ahead and let 'er rip on phobos.lib. Twisted Evil

Notes

I'm considering adding some kind of strong-name or UUID to DDLs so that they can be reliably identified in some fashion; this will require library/object authors to embed hints in their code to use properly. I"m also seriously considering adding a CPU architecture hint as well. This will change the role of bless somewhat, which leads me to think that it should be named to 'ddltool' or something more general purpose (if less inspired). But for now, the current scheme should work well, but expect things to change in the near future.

TODO

I'm planning on modifying the ddl interfaces yet again to change the reponsibilities of the DynamicLibrary. Presently, it tries to give information about its subordinate modules, which hasn't proved practical. Instead, I'll change this to yield supported namespaces and imported module names (sound familar?). This should help the planned linking process, but in actuality may have to change a few more times before this gets done right.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Oct 11, 2005 8:50 am    Post subject: Reply with quote

Setback

Yesterday marked the end of a long trail of setbacks, the last of which nearly cost me all of my data.

First off, I moved into a newer, cleaner, quieter apartment. At a marginal additional expence per month, the result should be a happier Pragma as well as a happier Mrs. Pragma and ultimately: more productive creative time. DDL, as well as my other projects will benefit nicely from all this.

Second, misplacing your operating system disks and CDKeys once your OS-installations go sour is never a good thing. I can tell you this: Win2k evaluation version is great right up until that nasty 120-day limitation. Sad

Third, when building a near cutting-edge system, pay close attention to the compatibility of your BIOS, hard-drive and your operating system of choice. In this case, win2k (SP0) runs great on a Shuttle XPC (SN85 mainboard), but will refuse to see past the first 130GB of my SATA disk (230GB total). The remainder can be formatted only after the OS is up and running on that first-half of the disk, after a registry fix and nForce driver installation. What's worse? Try re-installaing on the fully-formatted disk: it can't be done and will cause the win2k installer to fail, or the resulting boot image to fail.

(I thought I was lucky enough to have a Knoppix 3.3 liveCD lying around, but this too would spout 'invalid block device' when mounting that NTFS monster. Apparently, this combination of parts is just too new for such an old liveCD.)

The solution: mounting a bevy of legacy drives onto my shuttle to create a 'lifeboat' of sorts so I may back-up my data elsewhere; then start over from scratch on that disk. Ick. My entire weekend fell into this format-and-shuffle routine. I still have much work to do before my development machine is back at 100?.

It was all very reminiscent of a similar battle I had with a 40GB UDMA drive which was just a smidge too advanced for my mainboard at the time. I found it quite ironic that this very drive is what saved the day this time around. If there's a greater justification for keeping older equipment around, I haven't found it.

So why worry?

Sure, the code is all in the SVN repo here on dsource. But there's much on those two computers of mine that you haven't seen yet.

Mostly, it comes down to some great new web graphics for a project web I'm going to set up sometime in the next month. DDL especially can use the publicity push as there's much in the kit already that can be used; yet I've all but failed in explaining how to use it. This has to change.

Secondly, I've taken steps to revamp my Cilantro blog enigne to be a minimalistic XML masterpiece, that requires zero database support: I've done this so DDOC generated files will mesh perfectly with blog/web content via a simple upload. It will also mean that I can stop using the forum here as a blog, and get a nicer web presence up for you folks. It also means that I plan on syndicating my posts out to the forums anyway, for convienience.

Back on Trac

Lastly, I've been working on DDL, heading toward the next milestone. Those of you who have dsource accounts (you may want to be logged in before you view... its still buggy) can view the staging area for the DDL Trac: http://trac.dsource.org/projects/ddl/

Enjoy.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DDL - D Dynamic Libraries All times are GMT - 6 Hours
Goto page Previous  1, 2, 3, ... 10, 11, 12  Next
Page 2 of 12

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group