| View previous topic :: View next topic |
| Author |
Message |
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
 Posted: Thu Dec 01, 2005 2:26 pm Post subject: |
 |
|
| kris wrote: | | pragma wrote: | I have considered redoing all this as a heavyweight tool using a compiled XSL lib as to avoid having everyone grab AltovaXML or their $XSLT_ENGINE_OF_CHOICE.
But every time I consider going heavyweight, I feel as though too much leverage is lost with the 'openness' of the script-set version. Since documentation should be configurable on a project-to-project basis, perhaps its not a bad idea after all. |
It would be really nice to have a working "out of the box" system. Is it feasable to provide both options? |
Well, with sripts or not, I'm still keeping the stylesheets external to the app, so there's almost no difference. All the real work is done by the stylesheet engine; the main app/script just calls it a whole bunch of times against each stylesheet needed. To that end 90? of my effort has been in coding raw XSLT/XPATH code thus far.
The only gains I had my sights set on were fewer line-command arguments, and less using XSLT to generate the xref and all-modules scripts. Other than that, it's exactly the same deal. Even custom configuration can be handled 100? through XSLT and shell-scripts.
So to answer your question: it will be an out-of-the-box solution. I've just got to find a sane division between the standard kit and the custom configuration parts. I'm almost there now.
Anyway, I've also done a little more research into this today and found out that Sablotron is stuck at the XSLT1.0 level, and Altova isn't ported to *nix (although it does have a COM interface). To make matters worse, everything else FOSS in the XSL arena is written in Java.
So in order to go with a non-shell-script launched solution (.d application), I'd probably have to redo the stylehsheets with XSLT1.0... yuck.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
 |
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Thu Dec 01, 2005 3:53 pm Post subject: |
|
|
| Ack! |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Thu Dec 01, 2005 10:48 pm Post subject: |
|
|
Say that a few more times and you'll be caught up with me! 
I stubbed out the DOM 2.0 and XPath 2.0 interfaces months ago, under DSP, to work along with my XML parser. Now I'm starting to see how far ahead of the game that was: there are frighteningly few XPath and XSLT 2.0 implementations out there.
But good news, I found that Xalan has a C++ port, so things might be looking up.
http://xml.apache.org/xalan-c/build_instruct.html
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Thu Dec 01, 2005 11:01 pm Post subject: |
|
|
Status
I cleaned up the source tree a bit by moving all the test related stuff into /test. I contributed some helper scripts I had lying around, in case anyone is interested in playing along. They basically help with keeping the utilities in sync with source changes, as well as reducing the number steps needed to compose a test app.
Most importantly, buildtest ties the creation of the in-situ ddl file in with the compilation of any given test app, so you can't forget it. This relies on a few of the ddl utils, so make sure you run buildutils first.
Compile-time Goodness Baked Right in!
I made a slight architecture change (ahead of schedule) to incorporate this nifty little do-dad: DynamicLibrary is now a class instead of an interface and sports template methods for Don's Symbol Mangler.
The proof is in test/linktest2.d which should do all the talking for me:
| Code: | void main(){
writefln("starting.");
Linker linker = new Linker();
writefln("adding lib: test\\linktest2.ddl");
linker.add(loadDDL("test\\linktest2.ddl"));
writefln("adding lib: phobos.lib");
linker.add(loadDDL("phobos.lib"));
writefln("adding lib: snn.lib");
linker.add(loadDDL("snn.lib"));
DynamicLibrary testLibrary = loadDDL("test\\testmodule.obj");
linker.link(testLibrary);
auto addFun = testLibrary.getFunction!(uint,"test.testmodule.add",uint,uint)();
auto helloWorld = testLibrary.getFunction!(void,"test.testmodule.helloWorld")();
writefln("add: 42+69 = ?d",addFun(42,69));
helloWorld();
writefln("Done.");
} |
New Linker
This new test takes advantage of a new streamlined (read: simple) Linker that will become a much easier starting point for optimal linking than my old version. The old linker was an example of explosive premature optimization if there ever was one*. As it turns out I actually have more to worry about with my OMF parser and loader code than anything else. The code also relies on the DynamicLibrary's ability to cough up symbols quickly, so that too will become a big refactoring and optimization point.
Anyway if you read Linker.d, you'll see why I'm so psyched about all this.
(*I read www.thedailywtf.com regularily, so don't even think about it.  )
Misc
I closed a few tickets, and opened a whole bunch more for the next milestone. A few OMF changes also snuck their way into the changeset from who knows when: they're all bugfixes, I promise!
FYI, if your ticket/bug-report was closed, and the problem was not fixed, just open up a new one and I'll get right to it. 
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Thu Dec 08, 2005 9:14 am Post subject: |
|
|
Status
Did some work tigthening up the lib and documenting some more classes. This is not in SVN yet as I am still working on this task.
I did decide to depricate some of the stubbed-out interfaces for the loader, LibraryValidator and other such parts. For now, I want to reduce the working set to the absolute minimum for this milestone, so I can get to refactoring. After that phase, the next round of improvements should become apparent.
OMF Woes
After trying to dynamically bind the sieve.d example, the linker blew up claiming it couldn't find "__imp__GetThreadContext@8". As it turns out, this is partly due to the fact that I neglected to read deeper into the portion of the OMF spec regarding comment records. Apparently, this record type is co-opted to perfom a myriad of OMF extensions and record types, one of which is defining __imp__ symbols.
"__imp__" is a sort of magic-prefix that goes hand-in-hand with declspec(dllimport) or implib style linking. These are defined in libs that just wrap around a dll in some fashion or another. It seems simple, and it is, except I cannot find any information about how and where the actual dll load-and-bind ocurrs. Wierd.
Another problem is that I unknowingly blew-away __imp__ symbol support from the in-situ side. The example would have worked perfectly had I not taken that particular path earlier -- but the above problem would have reappeared at some later point. Dumb luck I guess.
So I'm off to do more research to figure out how to better support these kinds of imports.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Thu Dec 08, 2005 8:30 pm Post subject: |
|
|
Status
The OMF issue mentioned earlier has been added as a Trac ticket for the time being:
http://trac.dsource.org/projects/ddl/ticket/23
I don't see this as being a show-stopper yet, as most implib issues can be resolved via the insitu loader (compiled-into the main exe) for now. This is how I managed to get linktest3 to work correctly -- the insitu loader has been patched to address this correctly.
I'm not going to let this go quietly as the ability to dynamically load winsock via an implib would be nice for some applications. Of course the lack of an OS-based DDL loader makes it hard to improvise a work-around too.
Anyway, the ticket has more musings on the topic, and why this is going to be hard to solve.
Class Binding
Don threw the idea across the table about binding a class in an other thread , so I decided to give it a shot.
| Code: | // example of dynamic type binding and construction
auto typeTestClass = cast(ClassInfo)testLibrary.getExport("__Class_4test10testmodule9TestClass").address;
auto objTestClass = _d_newclass(typeTestClass);
auto ctorTestClass = cast(Object function(Object))testLibrary.getExport("_D4test10testmodule9TestClass5_ctorFZC4test10testmodule9TestClass").address;
ctorTestClass(objTestClass); |
Works like a charm. Just throw in a common known interface and cast away that Object to something more friendly.
Now all that's needed is a compile-time template mechanism for this and we'll have the start of something really nice.
Edit
The Runtime Linker milestone is pretty much in the bag. I decided to cut a lot of fat to make this happen largely due to the amount of refactoring and bugfixing that needs to go on. All those extra classes were going to compound things horribly.
Also, as DDL is an immature technology, a lot of the use idioms are still being defined. So I think pushing forward with a simple lib that exposes maximum flexibility (rather than impose particular useages like classpaths and library caching) is the way forward for now. After 1.0, I can re-visit a very stable DDL with additional goodies -- many of which will probably fall out of DSP.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Sun Dec 11, 2005 8:57 am Post subject: |
|
|
Status
I took a few extra minutes this time and rolled some scripts to create downloadable packages. The downloads page of the Wiki now reflects this:
http://trac.dsource.org/projects/ddl/wiki/Downloads
There's some overlap in the packages, depending on what you're after. If you want everything, just grab the SDK.
Off to refactor...
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Mon Dec 12, 2005 11:50 am Post subject: |
|
|
Status
I'm presently working the refactoring tickets over in Trac. If anyone has any grievances or issues with the current library design, now is the time to bring them up.
http://trac.dsource.org/projects/ddl/ticket/12
http://trac.dsource.org/projects/ddl/ticket/17
FYI, none of this is in SVN yet.
I've removed the loadDDL() functions in favor of a class-based approach. The problem, was that a flat function interface only forces the entire DDL runtime on the developer, which is something that I want to avoid. So I concocted the LoaderRegistry as a way to circumvent that.
The registry is a basic bucket that holds onto a set of library loaders. This, in turn, can be handed off to the linker or used directly to load libraries. The registry itself is passed onto any aggregate loader being used such that recursive loads become possible.
I rolled a very simple (10 line) DefaultRegistry, which extends the basic LoaderRegistry and plugs in all the standard DDL loaders automatically. This way, you get right back to basics even though there are more classes to manipulate than before.
| Code: | // replaces loadDDL("filename");
auto reg = new DefaultRegistry();
auto lib = reg.load("mylibrary.ddl");
// manipulate lib as you would have before |
Refactoring things to abstract away phobos/ares specific things is proving to be a real challenge. Right now, I'm in the process of migrating stdio and std.string functions into ddl.Utils.d. Streams are going to prove problematic -- I may wind up implementing the FileConduit interface, for the phobos side, just enough to satisfy the compiler.
What has proven most interesting is teaching the various loaders to participate in "type detection by inspection", which boils down to looking at the first n bytes of a file to see if its loadable by a given loader. So far, I have things set up to need no more than 8 bytes of the start of any such file. This means that filenames are now irrelevant where DDL is concerned.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Mon Dec 12, 2005 1:32 pm Post subject: |
|
|
Regarding files and so on ... it may be more to your benefit to just drag the entire file into a byte-array first? Then you could isolate all the I/O stuff cleanly with version()
I've ended up doing just that with the UnicodeFile class, with the rationale that probably 99.999? of relevant files are less than 10MB, and probably 95? are less than 2MB. The numbers are pulled out of the air, but at some point the otherwise beneficial goals of streaming become idealistic?
Just a thought. |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Mon Dec 12, 2005 5:18 pm Post subject: |
|
|
| kris wrote: | Regarding files and so on ... it may be more to your benefit to just drag the entire file into a byte-array first? Then you could isolate all the I/O stuff cleanly with version()
I've ended up doing just that with the UnicodeFile class, with the rationale that probably 99.999? of relevant files are less than 10MB, and probably 95? are less than 2MB. The numbers are pulled out of the air, but at some point the otherwise beneficial goals of streaming become idealistic?
Just a thought. |
Its a solid point. The majority of what DDL does with files involves taking slices of binary and text data directly from all over the file. All those slices probably cause the rest of the former file buffer to stick around anyway.
Then I guess the worst-case is not being able to handle a whole slew of files (unneeded file input and allocations), which is likely more a bug than anything else.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Wed Dec 21, 2005 12:43 pm Post subject: |
|
|
Status
The 1.0 Milestone is offically complete.
Things are progressing smoothly. The last of the refactored utils are up online, and work well with the revised source tree.
I anticipate the distinction between the cursor types and DDLFile to dissolve as I move forward - I keep wanting to use DDLFile like a stream. The various sub-parsers strewn around the library will probably fold into this as well, yielding something that looks a lot like Mango's FileConduit. This is with good reason as it will make transitioning to Ares easy.
I'm going to start pushing the envelope on DDL in order to create some solid examples for the SDK. The whole concept of DDL is difficult enough to convey, that having some succint code examples will help a lot with advertising the library's merits.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Wed Dec 21, 2005 10:42 pm Post subject: |
|
|
Bless
The bless utility has come a long way since the start of this project. One addition that is new under 1.1B, is support for attribute metadata on DDL files. The utility accepts individual attributes via the -a switch or a whole property file via the -p swtich. The latter, uses an .ini format to store all the name/value pairs:
| Code: | C:\dev\ddl\trunk>type root.attr
#
# Example DDL Properties File for use with bless
#
std.author = Eric Anderton
std.author.url = http://www.dsource.org/projects/ddl
std.filename = root.obj
std.copyright = Copyright (C) 2005
std.license = BSD
std.license.url = http://www.dsource.org/projects/ddl/wiki/License
std.support = Support, FAQ, API and more is available at our website
std.support.url = http://www.dsource.org/projects/ddl/wiki |
The attributes shown are just an example. The system supports completely arbitrary names and values.
Packaging this into the .ddl file is simple:
| Code: | | bless root.obj -proot.attr |
To check the results, just feed the .ddl file back into ddlinfo:
| Code: | C:\dev\ddl\trunk>ddlinfo root.ddl -a
filename: 'root.ddl'
type: 'DDL'
attributes:
std.author.url - http://www.dsource.org/projects/ddl
std.copyright - Copyright (C) 2005
std.author - Eric Anderton
std.support.url - http://www.dsource.org/projects/ddl/wiki
std.support - Support, FAQ, API and more is available at our website
std.license - BSD
std.filename - root.obj
std.license.url - http://www.dsource.org/projects/ddl/wiki/License |
Attributes are available programatically via the DynamicLibrary class. Not all DDL supported types supply attribute data, yet, but at least .ddl files do.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Thu Dec 22, 2005 12:37 am Post subject: |
|
|
^ awesome!
Do you have some ideas about how versioning might be handled using this? |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Thu Dec 22, 2005 9:39 am Post subject: |
|
|
| kris wrote: | ^ awesome!
Do you have some ideas about how versioning might be handled using this? |
Well, the .ddl file itself employs a version number right after the magic byte sequence, so there's no confusing different versions of .ddl file support.
As for the enclosed binary, there's nothing explicitly for versions so -- metadata to the rescue, right? I suppose one could roll a subclass of Linker that's a little smarter than the stock rendition, that is version aware.
| Code: | auto linker = new Linker(new DefaultRegistry());
linker.loadAndRegister("phobos.ddl","0.141"); |
Internally, the method could compare the attribute "std.version" to the value supplied, and would throw if they do not match.
Another thing is that the switches for bless can be used multiple times, with data from the latter switches overriding the former ones. So managing version numbers along with your standard attribs becomes very easy:
| Code: | | bless example.obj -pmetatdata.attr -astd.filename=example.obj -astd.version=1.0 |
So you can add your support and license info to everything using the same metadata file, and set the versions individually as appropriate.
Now the next step, of course, is to set things up to extract metadata from fields defined in the binary itself, so you could do all this from the sourcecode. 
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
pragma
Joined: 28 May 2004
Posts: 607
Location: Washington, DC
|
| Posted: Tue Dec 27, 2005 3:48 pm Post subject: |
|
|
Status
Its looking like 1.1B is winding up to be a great release, as work on the refactoring tasks is moving along as planned. For those involved, I will post details on any API changing updates here, as a few more are planned for the upcomming milestone (sorry Lars).
I got permission to re-license the COFF loader code, so I will proceed with updating those soon. This is good news as I have a considerable amount of refactoring left to do within those files.
Coalescing the DynamicLibrary stuff is going to take a little more engineering than I thought. The problem, is that there's enough variance between how the loaders handle module storage that I'm going to have to really push things around. In the end, it'll be worth it just to set the stage for future optimizations.
DynamicModule will likely follow suit with DynamicModule by coalescing into something more homogenous. Per the previous discussion here with Kris, I may add a getAttributes() method to DynamicModule, so per-module metadata will be possible as well. That way single-module libs (like .obj) can directly expose compiled-in metadata should anyone want to use it.
| Code: | //just spitballing here
module my.module.example;
const char[][char[]] _DDLAttribs = [
"std.version" : "1.0",
"notes" : "an example of in-code metadata"]; |
I'm 95? sure that any .obj that DMD produces can be interrogated for const data w/o linking against the runtime, which makes this very possible.
The bless tool could be modified to pull these in automatically, or to have that operation manually suppressed. Line-command attribs and properties-files would override the internal values.
The only drawback I can think of here, is that .lib style binaries are going to have no such facility as there's no way to tell which internal .obj should speak for the whole library. In that case, wrapping as a .ddl is the only way to go for big files like that.
I've also toyed with the idea of putting more proper reflection hooks at both the library and module levels. Things like getCtor and getObject are very doable, even if they require some rather complex template code. I'm also contemplating the need for runtime equivalents for these, as you can only do so much at compile-time; querying D symbols irrespective of its host module namespace is the #1 problem here.
_________________
-- !Eric.t.Anderton at gmail |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
