FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Developer tasks: Descent / Mmrnmhrm integration

 
Post new topic   Reply to topic     Forum Index -> Descent
View previous topic :: View next topic  
Author Message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Thu Aug 16, 2007 9:15 am    Post subject: Developer tasks: Descent / Mmrnmhrm integration Reply with quote

Having finished a basic code completion feature in Mmrnmhrm (available at SVN), and nearing a 0.1 release, I'm now thinking about what next tasks to take on.
If we want to integrate Mmrnmrhm features into Descent, then, like I said in the other thread, the key to that is the dtool component. dtool is where the parser, ASTs, reference resolver (both for find ref and code completion) are. DTool code is both Mmrnhmrm and Eclipse independent, altough it doesn't work by itself.
But before we even think about integrating semantic features, I would sugest first integrating the parser, as that's pretty much the first line of functionality. I've just finished updating dtool's parser to the latest descent version, and it would be quite nice if you could integrate them. That would mean a few changes (which have already implemented in my branch of the parser):
Adding some visitor classes, and the accept0 methods for DMD Nodes, and a minor hierarchy change (should not bring conflicts).
And if we are to share the same code base, we need to decide how to do it. One way is to split the parser component of descent.core into a separate plugin. Another way is to simply have dtool have descent.core has a dependency. I'm still not sure what would be preferable, so do comment if you have any preference, altough that last option is likely the the best, for starting. In any case, another requirement of integration is that now we would have to be mindful of changes not breaking each other's plugin code (not likely, but could happen).

I'll detail this more on MSN when I have the chance.
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Thu Aug 16, 2007 1:19 pm    Post subject: Reply with quote

I'm really looking forward to this
Back to top
View user's profile Send private message
fraserofthenight



Joined: 08 Apr 2007
Posts: 33
Location: Seattle, WA

PostPosted: Fri Aug 17, 2007 4:08 pm    Post subject: Reply with quote

That sounds like a really good idea. Ary has the final say, but if it doesn't break any existing functionality, I'm cool with your making any changes you want. I doubt accept0 will break much of anything, so go for it. What hierarchy change were you thinking?

Right now, Ary is working on resolving a couple OutOfMemory issues, so if there's no objections, I can work on dtool integration. I think making dtool dependent on descent.core would be easier for us and would maintain the overall structure of the JDT, however if Mmrnhmrm will continue to live in independence, it may be better to split off the parser.

I think we should also work on some sort of indexing of references. The auto-import feature of the JDT is my favorite part of it by far (I don't think I've written a Java import statement in years...) so I'd like to see that feature implemented as soon as possible. It'd be nice to have markings for unresolved symbols in general, which can be done without complete semantic analysis as long as every reference is bound, though I haven't looked through dtool, so I'm not sure if this is possible with the current implementation (template/mixin expansion especially might make this difficult).

Write me an e-mail - fraserofthenight@gmail.com - and we'll see what we can work out.
Back to top
View user's profile Send private message Send e-mail AIM Address MSN Messenger
asterite



Joined: 01 Jun 2006
Posts: 235
Location: Buenos Aires, Argentina

PostPosted: Fri Aug 17, 2007 8:38 pm    Post subject: Reply with quote

In fact, today phoenix integrated he's part of the parser with Descent... since I started to change some things to use char[] instead of String, I still need to fix some problems to commit the final, integrated version.

After that, you could try to get some of dtool functionality, that would be awsome. (we can still continue porting the semantic, since maybe that will get at to the same point as dtool, but with more advantages... I don't know).

I know I've started the project, but I'm realy clueless about some stuff, like how people generaly works with "compiles to native" languages, since I almost always work with Java and C# (that's why integration with a compiler is taking so long). So you guys can make lots of decisions and suggestions, and I'll surely take them if I'm sure I know less than you (which should happen more than 90% of the time Razz).
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Sat Aug 18, 2007 5:09 am    Post subject: Reply with quote

fraserofthenight wrote:
That sounds like a really good idea. Ary has the final say, but if it doesn't break any existing functionality, I'm cool with your making any changes you want. I doubt accept0 will break much of anything, so go for it. What hierarchy change were you thinking?

Right now, Ary is working on resolving a couple OutOfMemory issues, so if there's no objections, I can work on dtool integration. I think making dtool dependent on descent.core would be easier for us and would maintain the overall structure of the JDT, however if Mmrnhmrm will continue to live in independence, it may be better to split off the parser.


Like Ary said, I've already committed my changes to descent.core. The hierarchy change is having:
ASTDmdNode -> ASTNode -> TreeNode & others
The TreeNode class is just to reuse some generic tree behavior, but there shouldn't be any issues nor any differences in using that.
The ASTDmdNode -> ASTNode split is because the nodes from the dtool AST (ASTNeoNode) also inherit from that ASTNode. This is made that way as to allow to convert only some nodes and not all (to improve performance), thus having a "hybrid" AST, but having the visitor and all other mechanism still work correctly. This is not currently done by dtool (all nodes are converted), but there are some nodes (like Expression nodes) which are pratically the same as their dmd conterparts, and with some minor non-structural changes to the original dmd nodes, they wouldn't need to be converted at all. (Unlike the nodes related do definitions and references, whose structure is heavily modified and sanitized) Stuff like this should very important to performance, because having two entire ASTs created is a great performance waste.

I've also converted dtool to be dependent on descent.core, altough I havent commited those changes yet. The plugin binary dependency is not a problem for Mmrnmrhm, only source issues matter. For example, its not critical but it would be nice, from an organization and software engineering perspective, that the parser component of descent.core would be contained in it's own plugin (er even just a source folder) and not have any dependencies on the rest of descent.core. (Currently it depends on DDocComent, AST's api level, PublicScanner and one or other minor classes)

fraserofthenight wrote:

I think we should also work on some sort of indexing of references. The auto-import feature of the JDT is my favorite part of it by far (I don't think I've written a Java import statement in years...) so I'd like to see that feature implemented as soon as possible. It'd be nice to have markings for unresolved symbols in general, which can be done without complete semantic analysis as long as every reference is bound, though I haven't looked through dtool, so I'm not sure if this is possible with the current implementation (template/mixin expansion especially might make this difficult).

Write me an e-mail - fraserofthenight@gmail.com - and we'll see what we can work out.

Indexing would be quite nice, altough I was thinking of using for other feature: Open Type. I find very, *very* useful, especially for large projects (like by workspace which has Descent+Mmrnmrm+JDT+Eclipse Platform:P). But it would take some time to implement: unlike other features of JDT which I have examined a bit (like model LRU caching) I haven't looked on how this one works. I only know they search for definitions and then store than info in an index (both memory and disk based I think), but don't know any details.
Also the next feature I'd like to go into, is adding support for definition lookup to expressions nodes, and have a simple builder (invoking bud or something). I also have to start working on my thesis doc.
But you could take a stab at it of course. Smile

BTW, I've just noticed I had the wrong MSN address in my forum profile, I've corrected it, but anyway I've also added you to MSN.
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
stack_overflow



Joined: 18 Aug 2007
Posts: 3
Location: Brasil

PostPosted: Sat Aug 18, 2007 10:41 pm    Post subject: Reply with quote

Did you have some clue where code completion (intellisense) will be functional in descent?
The project is great! with intellisense it will be the best D IDE ever.

Thanks.
Back to top
View user's profile Send private message MSN Messenger
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Sun Aug 19, 2007 5:04 pm    Post subject: DTLK Reply with quote

OMG! I'm so excited with what I've found thid weekend that I'm nearing an orgasm! Twisted Evil Did anyone of you knew about the DLTK project:
http://www.eclipse.org/dltk/ ? I just looked into it in more detail this weekend and found it was just what I was looking for! What is DLTK? Basicly, if you check the creation slides, it is, among other things, a framework designed to structure and refactor JDT's code to allow it to be used by other IDEs! It creates an API that client IDEs can extend so as to have:
Quote:
1 Structured source model, with delta management.
2 Indexed source code. (By default all declarations are indexed). Fast search engine.
3 Many free UI, like ScriptExplorer, Outline, Search and so on.

This is fraking awesome! And although the name seems to indicate DLTK is made for dynamic languages, for what I've seen, it should be equally useful for a static, compiled language like D. Check this video which demos the Ruby IDE implementation based on DLTK:
http://us.xored.com/~haiodo/EclipseCon07/dltk-ruby.mov
It's pretty awesome for such a recent language as Ruby, and furthermore, I think we could have that same set of features with D, in not too long from now! I've already started porting Mmrnmhrm to DLTK! Check it out, DLTK changes our whole development situation!

A little background on how I came to DLTK:
I was personally never too fond of the ideia of blindly porting JDT over, without external help and without fully understanding it's internal workings (begin the control freak I am Rolling Eyes ). That was part of the reason I developed Mmrnmhrm seperatly, from scratch, except for the parser(altough also copying some of the few code of JDT I took the time to fully understand, which was mostly UI stuff).
However I gradually (actually recently) realized that it would take a lot of work to do from scratch all that boilerplate UI and core code that JDT (and consequently Descent as well) already had. Especially the core is major feature of Descent, that some users may not be aware. I mean, Descent has many strong points, such as the debbuger integration, the formatter, all the UI boilerplate code supporting the model, the editor, preferences etc., but the core's language element features, such as resource delta processing, model delta generation (for IDE clients), and especially the automatic handling and caching of language elements are all powerful features, and very hard to implement. The last aspect in particular (the caching) should be very important for IDE performance in large projects.
Because of this I favored having dtool's features integrated into Descent, but I did wish there was a more structured (and evolution friendly) way to support all of those JDT common language features. In Mmrnmhrm itself I did already write the code separating D specific parts from language-generic parts (not that I had enough code worth reusing though).
While integrating the parser I dabbled on this, and then I recalled the DLTK project, which I already had heard of several weeks ago, but didn't give it much tought since I thought it was for dynamic languages and did not even see what their framework provided. But this weekend I did stumble upon it, and having looked into it in more detail, I was amazed. The words "Generalizes JDT code and follows its architecture" blew me away. That project makes all the sense for Eclipse. After checking that it could be used for static languages as well (there isn't really any reason why it shouldn't), I was more than sold. I only wish I had found it sooner, as it makes several things in Mmrnmhrm obsolote (and many more in Descent)
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
phoenix



Joined: 06 Mar 2005
Posts: 68

PostPosted: Sat Aug 25, 2007 6:12 am    Post subject: Reply with quote

A summary of my last talk with Ary:
I've converted Mmrnmhrm to DLTK (available at SVN), preserving all functionality it had before convertion, and also gaining:
* A java-like language model with caching, several kinds of buildpath entries, full UI boilerplate code, (Mmrnmhrm's previous native implementation was very rudimentary in all these aspects.)
* Miscelleanous UI boilerplate code, such a better editor (code folding, quick outline, bracket matching), preference pages for coloring, code templates & compilers.
* Indexing of model type elements (class, structs, etc.) and a working Open Type feature based on it. Smile

This is enough to show that DLTK is viable and worthwhile. Plus the fact that DLTK has just been accepted as an Eclipse Foundation project means it should have reliable support in the future, should any issues arise.
I'm looking next to integrating Type Hierarchy and Search features, (but right now I'm going to have to suspend that and work on my thesis document which is late Sad )
Note: one does not have to use DLTK AST hierarchy to integrate with DLTK (except using ASTNode as a base class), it just means one will have to do more work to integrate the other semantic features of DLTK if you don't use their hierarchy.
_________________
Bruno Medeiros
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Descent All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group