FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Status Log
Goto page Previous  1, 2, 3  Next
 
Post new topic   Reply to topic     Forum Index -> DSP
View previous topic :: View next topic  
Author Message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Jun 03, 2005 10:16 pm    Post subject: Re: Status Log Reply with quote

DSP

In my efforts to push the last pieces of the puzzle into place, I added in the named-servlet facility to the tag library and the runtime. This means that the request.servletName() property is matched to an applicable handler in the servlet body, and that any given dll can have multiple named servlets (as defined by dsp:servlet).

There are some codegen bugs releated to this, as the examples don't work quite right yet. The worst of which is that the named servlet in index.dsp somehow is generated with the *entire* contents of the .dsp script, producing a mirror copy of the default servlet. Wierd.

The code generator did get some improvments this round. The runtime __writer reference is now extinct, as it was extra baggage and causing problems of its own. "response.getWriter" is now used directly for output since the code generator is smart enough to concat things together for few calls to getWriter(). Next step: getting the code generator to stick raw text pieces together to reduce the number of runtime string concats.

6/04/05
- added named servlet capabililtities
- modified dsp:servlet
- modified runtime behavior for dsp:tag
- modified runtime behavior for custom tags
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Jun 16, 2005 9:51 am    Post subject: Re: Status Log Reply with quote

Took a small break from DSP to scratch another itch. I should be back to DSP coding by week's end.

The itch? Ever tried to maintain a website via FTP, w/o using a local server for testing?

You're not alone: that method is a huge PITA for a whole cavalcade of reasons. FTP clients are infuratingly braindead when it comes to moving files: they don't allow you to synchronize file trees for squat. Plus, a client won't upload modified files automatically so you find yourself in a very wide development loop:

    1. save files in text editor
    2. launch/switch-to FTP client
    3. find the files you touched
    4. find the target folder for each and queue/upload
    5. test in browser
    6. go back to text editor
I want to eliminate (summary execution style) steps 2,3 and 4. Those alone cost me real time each iteration and to be frank: the computer is there to do work for me, not the other way around. Wink

Now some products out there solve this problem by demanding your hard-earned-cash. FTPDrive is one solution for Windows that fits the bill, FTPSync is another.

What would be ideal is the combination of the two for free: an open-source rsync-like client that will sync-on-demand when the local files are touched. No mapped-drives, just a deamon or app that watches and moves files all on its own.

So, I'm writing one. Progress has been rapid and extremely encouraging.

As it turns out, Mango makes it obscenely easy to talk FTP with a server, including uploading files. The Win32 SDK also has methods that allow you to check if anything has been modified in a given file tree; i've tested this and its definately doable. (Does anyone know how to pull this off in Linux/Unix? I'm assuming that there's some kind of "FTPFS" or somesuch that you'd use instead of a custom app?)

So now its a matter of meshing the filesystem notifications with the FTP client. The result should be a single-threaded 'daemon' of sorts that just watches the notifications and translates them into FTP commands. Sprinkle some command-line options on top and off we go.

When this is done, I'll be able to maintain websites as if the files were stored locally or over a file-share. Twisted Evil

Now all I need is a name for it. Sad
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Fri Jun 17, 2005 11:30 am    Post subject: Re: Status Log Reply with quote

?
Still scratching.

I ran into this nifty function in the Win32 SDK.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/readdirectorychangesw.asp

ReadDirectoryChangesW() does pretty much what it sounds like. It gives you a list of file system events that ocurred between calls for a given file tree.

There is a humongous catch to this when trying to harvest events accurately, so one can reproduce a file tree. The events do not specify if they are a directory or a file. One can always query the file system and ask "hey, is this a directory or not?" But what if your file/directory has been deleted? That's right, you're screwed.

You can't work around it by calling the funciton twice per iteration to get the file-events and then the directory-events separately. This can't be done because the API simply does not give you any event ordinal or timestamp. So your order-of-operations between directory and file productions has now been destroyed, as you now have two queues that relate to an original queue by some unknown ordering.

My current solution involves nothing less than having your code maintain a virtual directory tree in memory, and refer to it when resolving deletion events. I suppose this is better than trying to reproduce all of the behavior in ReadDirectoryChanges from scratch. So ultimately, ReadDirectoryChanges becomes more of a shortcut, which is fine by me.

Also, I'd like to expand this thing to behave much like a rsync-style tool which would require it to syncronize the entire tree from the very start. So maintaining an in-memory directory tree could only accelerate things.

DSP

I may consider rolling this research into DSP at a later date. It might be nice to have the server rebuild servlet libs on its own as opposed to on-demand like they are now.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Jun 21, 2005 8:37 pm    Post subject: Reply with quote

Watcher

Itch scratched.

I've dubbed this little subproject "Watcher" and I've put up a request to have it added to dsource as a separate project. For now you can download the working, if buggy version from here:

http://svn.dsource.org/projects/dsp/downloads/watcher.zip

Using it is fairly simple. Just like with rsync, use a uri to specify your username, host, port and destination directory.

Code:
watcher c:\dev\web ftp://pragma@ftp.myserver.com/www


The above will map the local directory "c:\dev\web" to "/www" on the remote server. That's really all there is to it. Directory and file modifications will be redirected to the configured FTP server. The application will run silently, so you may want to rebuild with -debug to get all the diagnostic and FTP console output.

Exclamation One word of warning: you will be prompted for your password which will echo to the screen. I'm currently asking around how to rectify this.

Happy Hacking! Very Happy
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Jun 22, 2005 6:19 pm    Post subject: Reply with quote

Watcher

Watcher has been added to the forum. I'll be maintaining it separately from DSP from here on out.

http://www.dsource.org/forums/viewforum.php?f=64

DSP

DSP is presently on hold, as Mango needs to be brought up to DMD 0.127 before I can do much more.

On the drawing board, I've taken some time to familiarize myself with AJAX development techniques. I'm actually quite fond of the idea, and think it has some untapped potential. As for DSP, I hope to provide enough functionality with the runtime to support AJAX-style applications. This will probably come in the form of both .d and .js code libraries.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Jul 06, 2005 11:24 am    Post subject: Reply with quote

I took advantage of the D-coding offseason by rebuilding the home office. The spare bedroom has now been fully converted into a library and two-computer lab. I'm also making the switch from cable to DSL, so hopefully I'll save a few bucks in the process too.

Mango has been patched in the SVN repos, so I'll be back at work bugfixing and improving DSP. Also, don't forget Watcher, which has been moved to it's own project space: http://www.dsource.org/projects/watcher
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Jul 10, 2005 9:34 pm    Post subject: Reply with quote

DSP

Today I went looking for bugs, and squashed a few in the DSP codebase.

- finished fromString() runtime string conversion lib
- fixed subtle in/out mistake in OutCode.d
+ (bugfix) Lib cache 'ballooning' feature doesn't behave correctly. Causes aggregate .dlls to be permanently loaded.
+ (bugfix) <?dsp:output?> and <dsp:output> both require a closing ';' to function

The following bugs are still outstanding:

- Issues with paths on custom tags.. requires "./" to get local directory
- Codegen bug with named servlet generation.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Sun Jul 17, 2005 1:27 pm    Post subject: Reply with quote

DSP

Wound up tightening up the implementation by eliminating bugs and streamlining <dsp:tag>, which had the most issues. Named servlets now are generated correctly and can be refereced via custom tags externally. Also, support for '#line' statements in the generated code has been added, which greatly enhances debugging.

I'm really glad I reduced the size of the platform tag set, as the engine itself is really the star for this project. Its worked out to be less bugtracking and fixing than the last model (Alpha2). There are some quirks to this particular design that are going to be tough to fix.

As it turns out, I never completely side-stepped the problem of intra-dll type management. This isn't any more apparent than in the dsp_argument.dsp example which yields the following error message:

Code:
Could not unbox from type char[] to char[].


The problem is that std.boxer is performing a direct comparison between Typeinfo instances, much like cast() and the rest of the D language does things. So 'char[]' from dll A is not the same as 'char[]' from dll B.

The solution? Something that can handle type conversions in this manner akin to the Exportable Class support I wrote some time ago. Its either this, or force all custom tag arguments to be strings and leave it at that. We'll see how this unfolds in the weeks to come.

Don't forget, that to run the dspconf suite you need to explicitly type in dspconf/index.dsp. There are no .htaccess like configurations for the server yet, which is also on the drawing board to come.

Also, after hacking around with JRun and ColdFusion at work last week, I've come to realization that there is a serious disconnect between the developer's environment and the administrator's in so-called "Enterprise" platforms; Java's not the only one to blame here.

There should be an API (optional for security reasons) that gives the coder access to changing the server's configuration, behavior and state during runtime. It doesn't make much sense to have to leave the runtime environment, which knows the most about what is happening, just to tweak this behavior or that.

Let's face it: not all of us have shell access to a development machine or can afford to 'bounce' the server after its already up. So why not just give that power back to the developer via a friendly API?

Beta 2 TODO:
- Better Variant support than std.boxer that is resilient across dll boundaries.

Beta 3 TODO:
- .htaccess-like app config ("web.xml" to sit alongside "config.xml"?)
- Review server design for 'resetless' configuration changes on the fly.
- Pipes to support builds, so temp files aren't needed.
- Build.exe support
- Tag grammar enforcment and validation support

Bugs:
- DMD keeps littering .map files in the application root.

Changes:

7/11/05
- changed default web.xml for dspconf to /dspconf from /foo/bar

7/16/05
- added line number support for code generator
+ (bugfix) Issues with paths on custom tags.. requires "./" to get local directory
- eliminated dependency attribute from <dsp:tag>
- added expression support for filename attribute on <dsp:tag>
- named servlet codgen fixed
- dll cache locking implemented
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Tue Jul 19, 2005 8:47 am    Post subject: Thoughts Reply with quote

D

I've given the notion of passing attribute values between servlets (across DLL boundaries) a great deal of thought over the course of this project. No matter what, the issue keeps rearing its ugly head: how do you reconcile objects that rely on code that is volatile in nature.

Its not beyond D to resolve, and in my esitmation would require nothing less than a custom GC and some special library hooks. Such a GC (under Win32 anyway) would need the following qualities:

- will not run a destructor if it cannot be 'reached' w/o causing a GPF (provides a failsafe and should probably be in D right now anyway)
- is aware of dll memory spaces and treats the entire library as collectable so un-used libraries are unloaded via the GC cycle
- will manage entire dll lifespan (ties into above) and possibly classbinding (see below)
- will copy loaded dlls to a temporary location before loading to allow said dll to be modified on-the-fly (also helps with resource lock-up in the event of a crash)

Also, the runtime will need to do no less than:
- provide hooks to access, manipulate and track all the new GC behavior
- provide an advanced type system that allows multiple revisions of types to operate side-by-side (may require GUID-based look-aside for type comparisons)
- provide "soft casting" or a unified type system so that similar types in separate dlls can be matched up

The last bullet is a tricky one. I've made some headway here by writing a custom cast operator that compares class names rather than Typeinfo addresses. This works reasonably well, but the casted objects remain 'brittle' beyond the original cast (they still have references to foreign Typeinfo).

Now by "unified type system" I'm picturing something more along the lines of how Java dynamically binds Classinfo references (by name) in a given .class file to the current runtime. D could do the same thing, by publishing D-specific .dll files that have a Typeinfo fixup hook or map of some kind. The dll could be provided with the addresses of Typeinfo instances (and vtables) for everything it needs. The result would be a *smaller* library that can expose class instances that don't fail on basic casts and work seamlessly with the existing runtime.

In short, think of it like what the Linker does with .obj files, only at runtime instead of at compile time. At the cost of additional lookup data, and loading time, we get a more flexible and resilient system. And of course, it's 100? optional.

DSP

In light of all this, I've decided to not allow (and discourage) the following behaviors in DSP.

- Custom tag arguments cannot be objects of any kind.
- Depending on the Variant-type's degree of support, only certain array types will be allowed
- Any Session or Application scope will also only contain scalars and arrays of scalars.

In the short run, this will be less to engineer around which means that the overall product will benefit. In the long run, it will be a limitation to be overcome provided a solution can be found free from too many "gotchas" (we have too many of those already).

Also, its looking more and more like the testbed server is taking on qualities that would make it apt for either inclusion into Mango, or ito be made into a Mango-based HTTP/Servlet server akin to JRun or Tomcat.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Jul 27, 2005 8:33 am    Post subject: Reply with quote

DSP

My efforts are going to be effectively stalled this week, due to a whole slew of personal obligations. This isn't entirely a bad thing, as I am still ruminating over how to compose a dll-capable variant type; it's trickier than it sounds.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Mon Aug 08, 2005 11:27 am    Post subject: Reply with quote

DSP

I began work on a tried-and-true Variant type the other week, as a replacement for std.box.

The problem with the standard box implementation is that it is not safe to use between "code domains", that is between a dll and its client. The implementation simply reduces the domain specific typeinfo into some flags that can be more universally digested. The process is reversed when the variant is used to fetch the stored type.

The only issue is that working with the TypeInfo subclasses, as provided in object.d isn't implemented correctly in DMD/Phobos. Depending on how you get your TypeInfo instance (via typeid() or by varargs), casting to a TypeInfo subclass may or may not work. In the meantime, I'm considering adopting the workaround that Burton Radons has coded into std.box. While not pretty, it works pretty well.
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Wed Aug 10, 2005 12:48 pm    Post subject: Reply with quote

D

What a detour!

It looks like I may be rounding off the current version of DSP to Beta 2 so that I may place it on the back-burner for just a bit longer.

It has come to my attention that the current issues that I've had with TypeInfo in D and dlls are limited to windows only. ELF binaries/objects in linux resolve external references by interrogating the host executable.

To that end, there is http://edll.sourceforge.net , which covers most of the bases already under win32. All that its missing is an OMF backend. To wit, OpenWatcom's "Public License" is non-commercial, so that leaves me starting from scratch. Wish me luck.

So when all things are said and done, we may have a dlopen() dlclose() interface under windows that will load raw .obj files just as we'd expect. As a bonus, this will make porting to linux *easier* than before.

I can't say if it'll work this way, but we may even get GC management of such resources for free as well.

For now I'll be working this as a D port of EDLL until further notice. I may eventually spin it off as D Dynamic Libraries (.ddl) instead, simply becuase I like the ring of it. Besides, D .obj files are going to contain a good number of D specific imports that typical .obj files won't have.

DSP
Anyway, the implications of this kind of technology for DSP (as well as virtually any other project) are obvious.

- No more TypeInfo problems
- Seamless class interop between dynamic libraries
- *Tiny* dynamic library sizes (no more compiled-in runtimes)
- No more faults when exceptions are thrown from dll code
- Potentially fix GC problems with dll based code including premature collections
- No more "dll handle locking" issues common with bad crashes.

Java, eat your heart out. Twisted Evil
_________________
-- !Eric.t.Anderton at gmail


Last edited by pragma on Wed Aug 10, 2005 1:35 pm; edited 1 time in total
Back to top
View user's profile Send private message Yahoo Messenger
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Wed Aug 10, 2005 1:34 pm    Post subject: Reply with quote

pragma wrote:
So when all things are said and done, we may have a dlopen() dlclose() interface under windows that will load raw .obj files just as we'd expect. As a bonus, this will make porting to linux *easier* than before.

I can't say if it'll work this way, but we may even get GC management of such resources for free as well.

For now I'll be working this as a D port of EDLL until further notice. I may eventually spin it off as D Dynamic Libraries (.ddl) instead, simply becuase I like the ring of it. Besides, D .obj files are going to contain a good number of D specific imports that typical .obj files won't have.

Frickin' awesome!

Last year, I wrote Walter suggesting something similar: a runtime-based relocating linker/loader ~ it would truly resolve all the dynamic loading issues, and would clearly differentiate D from other static compilation languages (combining the best of both worlds). Good on ya, Eric! I wish you the best of luck with this one; particularly so since it's how I would like to see D module-loading operate.

BTW: I completely dropped all usage of DLL's since Walter released his notion of 'support' earlier in the year. Instead, I took the route of the executable monitoring its own disk-image for updates, and then swapping itself on the fly as necessary. It works, but can become tricky to share things like sockets during the hand-off. What you're suggesting is infinately more refined.
Back to top
View user's profile Send private message
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Wed Aug 10, 2005 10:19 pm    Post subject: Reply with quote

This does look exciting, Eric! Keep up the good work. You've been holding on longer than most! Wink

-JJR
Back to top
View user's profile Send private message
pragma



Joined: 28 May 2004
Posts: 607
Location: Washington, DC

PostPosted: Thu Aug 11, 2005 7:11 am    Post subject: Reply with quote

kris wrote:

Frickin' awesome!

Last year, I wrote Walter suggesting something similar: a runtime-based relocating linker/loader ~ it would truly resolve all the dynamic loading issues, and would clearly differentiate D from other static compilation languages (combining the best of both worlds). Good on ya, Eric! I wish you the best of luck with this one; particularly so since it's how I would like to see D module-loading operate.


Exactly. I kept seeing how Java does things and asked myself: why not D? Then I learn just this week that ELF binaries have done what I've wanted for god only knows how long. DLL's should have worked like this in the first place.

Quote:

BTW: I completely dropped all usage of DLL's since Walter released his notion of 'support' earlier in the year. Instead, I took the route of the executable monitoring its own disk-image for updates, and then swapping itself on the fly as necessary. It works, but can become tricky to share things like sockets during the hand-off.


Nice. A completely self-healing system.

Just a thought: simply write the HANDLE values to a flatfile to be investigated on load and you should be all set. AFAIK, windows shouldn't care what process is using the handle, just as long as the value is correct.

Taking another nod from Java and .NET, I now understand that DLL's should be considered "legacy support" when all this is said and done. I don't blame you for resorting to such exotic workarounds: I've investigated plenty of odd soultions myself.

Quote:
What you're suggesting is infinately more refined.


I look at it as a step in the logical progression of things. Someone was going to have to do this eventually. Smile
_________________
-- !Eric.t.Anderton at gmail
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> DSP All times are GMT - 6 Hours
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group