| View previous topic :: View next topic |
| Author |
Message |
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
 Posted: Tue Dec 27, 2005 6:13 pm Post subject: |
 |
|
| Regarding the issue with .lib files, I think they're generally less attractive for dynamic linking. For example, one may end up loading symbols for the entire lib just to get one module linked. I suspect the zip-file approach is more compatible with DDL, just as it is with Java. Just the support for paths within a zipfile should alone be worthy of migration. I guess I'm thinking that lib files are a bit 'creaky' at this point, and could probably be unravelled and then rewrapped in something better (if at all) ? |
|
| Back to top |
|
 |
kylefurlong
Joined: 16 Nov 2005
Posts: 29
|
| Posted: Tue Dec 27, 2005 6:45 pm Post subject: |
|
|
I agree with kris. What I would look for from ddl eventually is making something like the Java jar system or the .NET assembly system possible for D. But that is just becuase I dont come from a C/C++ background and the whole obj/lib/header/linking thing is alien to me, so obviously this is your project eric, take it where you want.  |
|
| Back to top |
|
|
AgentOrange
Joined: 21 Jul 2005
Posts: 61
|
| Posted: Tue Dec 27, 2005 11:49 pm Post subject: |
|
|
Im not sure what you guys are refering to? Because thats exactly what dynamic modules do.
| Quote: | | For example, one may end up loading symbols for the entire lib just to get one module linked. |
What does this mean? A DLL contains the entire 'library' and you only resolve which symbols you need. I fail to see the problem you guys seem to be discussing... |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Wed Dec 28, 2005 12:18 am Post subject: |
|
|
| pragma wrote: | | The only drawback I can think of here, is that .lib style binaries are going to have no such facility as there's no way to tell which internal .obj should speak for the whole library. In that case, wrapping as a .ddl is the only way to go for big files like that |
This is the concern I was referring to. My point is simply that full support for .lib may be overkill, since they're perhaps not such a good fit for dynamic loading. |
|
| Back to top |
|
|
larsivi
Site Admin
Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway
|
| Posted: Wed Dec 28, 2005 1:29 am Post subject: |
|
|
| kris wrote: | | pragma wrote: | | The only drawback I can think of here, is that .lib style binaries are going to have no such facility as there's no way to tell which internal .obj should speak for the whole library. In that case, wrapping as a .ddl is the only way to go for big files like that |
This is the concern I was referring to. My point is simply that full support for .lib may be overkill, since they're perhaps not such a good fit for dynamic loading. |
And this is because static libs ain't designed for dynamic linking, whereas the dynamic libs are  What this will mean for the ELF side of things, I'm not sure of yet, but objects intended for dynamic linking have symbol sections at least named differently than those intended for static linking and also other sections that won't be found in static libs. I suppose that dynamic linking without the relocatable info might be a mean task, otherwise there wouldn't be so much fuzz about it |
|
| Back to top |
|
|
AgentOrange
Joined: 21 Jul 2005
Posts: 61
|
| Posted: Wed Dec 28, 2005 2:09 am Post subject: |
|
|
| Yeah I thought thats why we are converting libs into dynamic libraries called DDLs? |
|
| Back to top |
|
|
larsivi
Site Admin
Joined: 27 Mar 2004
Posts: 453
Location: Trondheim, Norway
|
| Posted: Wed Dec 28, 2005 2:47 am Post subject: |
|
|
| AgentOrange wrote: | | Yeah I thought thats why we are converting libs into dynamic libraries called DDLs? |
Yes, of course |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Wed Dec 28, 2005 8:53 am Post subject: |
|
|
Guys, thank you all for your input.
| kris wrote: | | pragma wrote: | | The only drawback I can think of here, is that .lib style binaries are going to have no such facility as there's no way to tell which internal .obj should speak for the whole library. In that case, wrapping as a .ddl is the only way to go for big files like that |
This is the concern I was referring to. My point is simply that full support for .lib may be overkill, since they're perhaps not such a good fit for dynamic loading. |
Well, the only thing that .lib files don't do so well is just metadata (which is really a strap-on/nice hack kind of situation anyway) and they carry some additional overhead, depending on their size and layout.
Phobos.lib and snn.lib come to mind here. Ideally, I'd have these broken down into .obj files, wrapped as .ddls for version info and leave it at that. Of course, that's the deluxe, "best practice" way to proceed. Other developers may choose the quick-and-dirty solution simply for RAD purposes or to circumvent issues with distribution and licensing regarding certain libs.
With respect to how DDL is architectured, supporting both OMF and COFF lib formats do not cost a whole heck of a lot. The formats are easily detectable, and leverage the code for the respective .obj parsers internally. In that light, full .lib support isn't overkill since its already here, and works very well. It just falls short a tad when compared to a .ddl file or a simple .obj with embedded metadata (as proposed). That's expected since its basically a primitive, and older wrapper type.
So Kris is right: it's not a good idea to use plain-ol .lib files directly unless you can avoid it, or are okay with their drawbacks. For the record, I don't think its worthwhile to drop .lib support, when it does have its uses. I am open to suggestions on how to improve it.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Wed Dec 28, 2005 9:15 am Post subject: |
|
|
| kris wrote: | | Regarding the issue with .lib files [...] Just the support for paths within a zipfile should alone be worthy of migration. |
I'm all about including .jar-style support for this thing. My only concern is that there is no zip compression lib in Ares (which has been a concern ever since I added .zip compression to the insitu format).
Seeing as how we need a solution here, should we co-opt the zip lib in phobos or seek out something else?
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Wed Dec 28, 2005 9:48 am Post subject: |
|
|
| kylefurlong wrote: | | I agree with kris. What I would look for from ddl eventually is making something like the Java jar system or the .NET assembly system possible for D. But that is just becuase I dont come from a C/C++ background and the whole obj/lib/header/linking thing is alien to me, so obviously this is your project eric, take it where you want. |
Thanks for your support Kyle. I'm coming from every place you mention, and I too want to have just such a system in place.
Just think of .obj files as .class files that contain code/data/symbols for a whole module (one .d source file) rather than just one class. An .obj both declares symbols it has (exports) and symbols it needs (dependencies) in order to be useable.
The .lib file is just a pile of .objs, with some glue to hold it all together, only without the directory structure that a .jar provides. It does, however, have a dictionary of sorts, which can be used to avoid un-needed parsing and loading (this is not exploited in DDL... yet).
Headers are needed to satisfy the compiler only, so it can make note of what symbols the resulting .obj needs in order to be linked (external dependencies). Its like saying "this is what this function/class/whatever looks like, and it's already compiled... honest." This is largely due to the fact that the compiler cannot read .obj/.lib files directly, and only uses lib path information to pass onto the linker.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Wed Dec 28, 2005 11:55 am Post subject: |
|
|
| pragma wrote: | | kris wrote: | | Regarding the issue with .lib files [...] Just the support for paths within a zipfile should alone be worthy of migration. |
I'm all about including .jar-style support for this thing. My only concern is that there is no zip compression lib in Ares (which has been a concern ever since I added .zip compression to the insitu format).
Seeing as how we need a solution here, should we co-opt the zip lib in phobos or seek out something else? |
I think the license allows you to co-opt it (as part of the DDL package). |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Wed Dec 28, 2005 12:26 pm Post subject: |
|
|
| kris wrote: | | pragma wrote: | | Seeing as how we need a solution here, should we co-opt the zip lib in phobos or seek out something else? |
I think the license allows you to co-opt it (as part of the DDL package). |
Ack. Just took a look at it - it pulls in a good deal of source.
std.zip
std.zlib
etc.c.zlib
zlib.lib --> etc/c/zlib/*.c
At least its well-contained: I can repackage this under a separate dir under DDL for Ares compatibliity like you suggest. I guess I'll have to package the accompanying zlib.lib along with it, so building won't be such a problem.
Crafting a loader should be easy: it identifies quite succinctly via inspection.
| Code: | | if (data[i .. i + 4] == cast(ubyte[])"PK\x03\x04") |
How to reconcile the internal directory structure with DDL's current understanding of library layouts is going to be the hard part. This may require an architecture change.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Thu Dec 29, 2005 1:05 pm Post subject: |
|
|
Status
Comitted a bunch of sundry changes and refactoring steps to the repository last night.
Things are really moving along in the forums of all places, as we have new threads popping up all over the place. If I haven't said it before, than I'll say it here: it is simply wonderful that the community has come out to comment, contribute, and critique like this. While I don't have a lot of FOSS experience, I can't imagine this is supposed to work any other way as it is getting results fast. Thanks everybody!
I think the current milestone is rapidly becomming a misnomer: Refactoring. Technically, the term is pretty broad in definition, and the tickets I had assigned myself in Trac reflect the cannon of typical refactoring tasks one would exepect. What has changed is the new support that is slated to be brought on board:
- .zip archives as 'libraries'
- possible java-style resource loading (thanks to .zip)
- path-based loading
- in-code metadata
- possible ELF loader (at least a unified COFF .lib/.ar loader) --> *way* ahead of schedule: go Lars!
Since there's no offical milestone for these, I may have to alter the project timetable, depending on how long all this takes. However, the addition of these capabilities has pressed the need for further refactoring, so even if this stuff is put off after 1.1B, this milestone is going to be dynamite. 
"Lazy" loading
I've tossed the term "lazy loading" around quite a bit when describing the behavior of various parts of DDL. I've come to learn that this actually breaks down into two forms of lazyness.
a) Delaying the actual read of the file into memory; parsing is performed immediately after reading.
b) Delaying the parsing step of binary loading: the raw file sits around in memory until this phase.
In a perfect world, everything would fall under (a). However, some formats like .zip or the various archive formats don't lend themselves to this behavior very well and work faster under (b) instead (at the cost of memory). In crafting parts of DDL, I've tried to be very aware of time/space mangement and have had to make compromises like this in order to keep things moving along. It is my hope that once the architecture is more stable, I can go back and profile things to see how best to improve the situation.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
Don Clugston
Joined: 05 Oct 2005
Posts: 91
Location: Germany (expat Australian)
|
| Posted: Fri Dec 30, 2005 2:53 am Post subject: DMD 0.142 and std.asserterror.d |
|
|
I can't get it to link with DMD 0.142. It prints a "LinkException: cannot resolve std.asserterror.d".
Which is interesting because that was the only part of Phobos that changed for DMD 0.142. Is there a bug in phobos.lib, or do I have to change something?
I'm using linktest3, I changed main() to:
void main(){
writefln("starting.");
auto reg = new DefaultRegistry();
Linker linker = new Linker(reg);
linker.loadLinkAndRegister("phobos.lib");
writefln("Done.");
} |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Fri Dec 30, 2005 9:18 am Post subject: Re: DMD 0.142 and std.asserterror.d |
|
|
| Don Clugston wrote: | I can't get it to link with DMD 0.142. It prints a "LinkException: cannot resolve std.asserterror.d".
Which is interesting because that was the only part of Phobos that changed for DMD 0.142. Is there a bug in phobos.lib, or do I have to change something?
I'm using linktest3, I changed main() to:
void main(){
writefln("starting.");
auto reg = new DefaultRegistry();
Linker linker = new Linker(reg);
linker.loadLinkAndRegister("phobos.lib");
writefln("Done.");
} |
Yikes. You probably don't want to use loadLinkAndRegister() in conjunction with the full platform lib like that, as that operation tries to link every single module in phobos.lib against what is in the linker's set of registered libs. The action will fail, since there's no registered libs to link it against. 
Seeing as how there's no tutorial section in the wiki (yet), let me explain in depth what's going on here.
The Linker
The linker supports three basic operations: load, register and link. Loading is the simple action of creating a DyanmicLibrary instance from a file, which is done largely through the LoaderRegistry. Registration is where the linker holds onto a DynamicLibrary instance to use for any linking operations. Linking is where a loaded DynamicLibrary's dependencies are resolved against any exports in the linker's registered libs.
The Linker class supplies several methods that allow for discrete load, register and link operations as well as some composite operations. The composite operations (like loadAndRegister() or loadLinkAndRegister()) do what they say, in the order they say it. They're provided as a convienence as well as a point for syncronization when DDL becomes more threadsafe.
During a link, the linker attempts to resolve every symbol dependency of every module in the library that is to be linked. For each such module, the linker searches through the registered set of libraries (in registration order), for any other module that exports the needed symbol. Should that module have outstanding dependencies, then that module in turn is also linked via the same process. Once a given module's dependencies are resolved, the module's init routine (if one exists) is run.
How to Link a Module
The reccomended way to use the linker is outlined in linktest1.d, presently in SVN. Here's a stripped-down example/tutorial:
| Code: | auto linker = new Linker(new DefaultRegistry());
linker.loadAndRegister("insitu.ddl");
linker.loadAndRegister("phobos.lib");
linker.loadAndRegister("snn.lib"); |
Lets break this down, as there's a lot going on in those four lines.
First and foremost: create a linker with your registry of choice. We're using the DefaultRegistry since it has loading support for everything built in. We won't get into using a custom registry here. anyway, what this does is establish a linker that can load any DDL supported binary file we throw at it.
Next, you want to load the in-situ module first so that everything defined in your .exe can be used for linking. This is generated by taking your .map file (use -L" -map" when building your app) and handing it to insitu.exe. The example above has "insitu.ddl"... you can simply pass it to bless.exe to make it a proper .ddl or leave it as xxx.situ: it doesn't matter.
 One caveat: always make sure that your in-situ module is up to date after building your program.
The next step is to bring in any platform libraries. This is done by calling loadAndRegister() for both phobos and snn (win32 only). The only reason why you'd want to load either of these is in the case your .exe doesn't have everything that any dynamically loaded modules need. In theory, if your .exe has 100? of both phobos.lib and snn.lib compiled into it, then you wouldn't need to load and register either of these -- just as long as you load that in-situ module!
Now you're all set up and ready to go. Lets load a module!
| Code: | | auto myModule = linker.loadAndLink("mymodule.ddl"); |
The file "mymodule.ddl" could be a wrapped lib, or an obj file. You can also just use a .lib or .obj directly; it doesn't matter.
The loadAndLink() operation does just that: it loads the module, and then links it against the modules we registered earlier. Provided that mymodule doesn't need anything not already compiled into your .exe, or in phobos for that matter, it will link just fine.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
