Development of Descent

The parts of Descent

Descent is built on top of Eclipse, as a plugin. It consists of many parts, but mainly:

1. A lexer, parser, and code for semantic analysis.
2. A global model, so you can query these things:
   • What D projects are in the workspace?
   • Which packages are in each of them?
   • Which modules are in the packages?
   • Which top level declarations each module contains? (functions, classes, structs, interfaces, templates, etc.).

The first point is implemented as a port of DMD's lexer, parser and semantic analysis. The decision of doing it this way instead of using, for example, ANTLR or another parser generator, is because:

• A port will always work similarly to DMD (if no bugs were introduced in the porting, of course).
• Updates to the D specification are acquired by Descent re-porting the diff of the versions.
• Possibility of reusing the semantic code for UI purposes, like:
  • Autocompletion.
  • Go to definition.
  • Bindings: http://www.eclipse.org/articles/article.php?file=Article-JavaCodeManipulation_AST/index.html
  • Compile-time evaluation.
  • Templates and CTFE debugging.
  • Showing compile errors as soon as you stop typing, without the need to recompile the file.

The semantic code is still a work in progress and below I'll intend to explain which modifications were needed/are being done for Descent.

The second point is a copy of the JDT project ( http://www.eclipse.org/jdt/ ), adapted for D. The decision of not making the project from scratch is because:

• The Java Model ( http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.jdt.doc.isv/guide/jdt_int_model.htm ) is very similar to the D Model. The only difference is that D has structs, unions, templates, and conditional declarations.
• Once the Java Model is replaced with the D model and some Java-specific code is replaced with D-specific code, some features should magically start working. Indeed, this happened with the outline view, the Project Explorer extensions, folding, go to definition, autocompletion, source on hover, syntax coloring, semantic coloring, the ddoc view, open type, TODO tasks, etc. That is a lot of bolierplate code that was better if avoided.
• Plugins written for JDT can be retrofitted to work with Descent.

Issues with he DMD port

While porting the DMD lexer, parser and semantic code we encountered some problems:

• Since the parser is not meant to be used by UI code, it merely stores line count information. Source range information (i.e. start position and end position of nodes of the grammar) were added.
• It uses a lot of global variables. It's ok, it's an executable meant to compile some files to generate object files or executables. But Descent will use this code many times and for different sets of files, and it may be doing this in different threads at the same time, so global variables were moved to a SemanticContext? class. This class is passed in each step of the semantic analysis. It stores information about the project configuration (version and debug levels, include paths, etc.) and some other stuff. References to global variables were replaced by references to members of the SemanticContext?.
• It is not meant to reuse files parsed in old executions. Each time you execute DMD, it RE-parses the imported files. This is impossible to do in Descent for performance reasons.

The third problem is critical. It means modifying the semantic code in a way very different to the one done for SemanticContext?.

Before starting with this modification, a lot of tests were written for the semantic code. Dstress was also used, and currently about 100 tests fail out of 4700.

The semantic analysis modification

The first thing modified in the semantic analysis was the loading of imported modules. DMD does this by parsing the file and generating an instance of a class called Module. Instead of parsing the file each time, we use the cached D Model, which is already built (each time a file is saved, the model for that file is updated).

In order to impact as little as possible in DMD's semantic code, the Module class was abstracted by an IModule interface. Some public members were changed to methods (getters/setters), and some methods found in Module were added to the IModule interface.

This led to a big refactor where many top-level declarations, like classes, methods, templates, etc., were hidden by interfaces. The implementation for some of this was trivial. For example, Module had a public variable called "members", which contained the top-level members of the module. This variable was changed by a method called "members()", and was implemented by IModule by using the members in the D model. Others were trickier, and some, like templates, and template instances, remain partially unimplemented.

Man, porting all that code and making those modifications is sure boring

Just a little. In the meantime, other, more fun things, were implemented. Some are features of the last released version, like the formatter, the ddoc view, the debugger, and open type dialog. Most was written in parallel with the semantic code, like autocompletion, go to definition, show source on hover, show compile-time evaluation of variable or function call on hover, show inactive code as grey (code that is not reachable due to version and debug levels, or static ifs that evaluate to false). JUnit-like functionality is also being added for D's built-in unit test declarations.

What's missing for the next big release?

The semantic analysis must be finished. It currently happens that some editors won't open because of exceptions thrown because some stuff is still unimplemented, or incorrectly implemented. Once this is finished, autocompletion needs to be tuned, some UI must be created for the user to configure the project (version and debug levels, include paths, etc.), and maybe bindings could be finished (but this affects more a programmer that is intending to build plugins on top of Descent). Also, Unicode support is having issues (it may be a UTF-8 vs. UTF-16 thing), so anyone with expertise in that area, please help out!

Once this is done, we think the project will give D a big shove, because a good IDE is essential for many users.

In near-term future versions, we are considering adding:

• A builder
• Ability to generate Ddoc with interlinks
• Refactoring
• Inline expansion of templates/mixins
• A "compiler view" of the source, so the user can see what their source looks like after all compile-time expansions; helpful for template debugging

Can I help?

Sure! There are currently two active developers in the project: Ary Borenszweing and Robert Fraser. You can imagine writing what's missing by two guys, in their spare time, could take months, even years!

If you read all of the above, have some knowledge about Java and the Eclipse platform, you just need to know where everything is located to start helping us. We write a lot of tests to verify that the code works, so chances of introducing bugs are decreased.

Some important packages and classes

• descent.internal.compiler.parser.*: the DMD port.
• descent.internal.compiler.lookup.*: the abstracted interfaces in the semantic analysis, like IModule.
• descent.core.dom.*: the internal AST generated by the DMD front-end is transformed to a friendlier one, like the one provided with JDT. This is where it is found. Also bidnings implementation is found here.
• descent.internal.codeassist.*: here you can found the CompletionEngine? (used for autocompletion), the SelectionEngine? (used to tell which element in the model is referred in some source range of a file), and the EvaluationEngine? (used to get hold of compile-time evaluations produced by DMD).
• descent.internal.compiler.SourceElementParser?: used to update the D module for a single file.
• descent.core.*: interfaces for the D model.
• descent.internal.core.*: implementation of the interfaces found in descent.core.*.

Java scares me; what else can I help with?

While we'd love help on the coding, we realize not everybody is willing to commit to such a large project. There are three other major areas we could use help in:

Documentation

Descent has a lot of "secret" features, such as the ability to compare revisions, a trace.log viewer, a formatter and more. Strong documentation, preferably with screenshots, could help quite a bit. Both wiki documentation and internal documentation (to ship with the plugin itself), could be very helpful. It doesn't need to be in English, either!

Testing

Descent is a very ambitious project, with a lot of facets. As we add new functionality, we'd love for as many users as possible to try out the functionality and report back. Usability analysis would also be helpful -- that is, what's difficult or non-intuitive? How "discoverable" is Descent (can you figure out feature X without documentation? Is the UI easy to navigate for someone who hasn't used Eclipse before?). For this, you can just grab the latest trunk version, load up your project and report back on what issues you see.

Translation

DMD's error messages are in one language. Descent can report those same error messages in many. Thanks to Eclipse's NLS system, Descent is all ready to be translated. If you're a (preferably native) speaker of another language, from Ancient Mayan to 13375p34k, we'd love your help in bringing Descent to a new audience.