| View previous topic :: View next topic |
| Author |
Message |
OOPMan
Joined: 06 Feb 2007
Posts: 6
|
 Posted: Tue Feb 06, 2007 11:56 am Post subject: Derelict Makefiles |
 |
|
Hey all, my first post in these parts 
Nice work with the Derelict bindings by the way, very cool beans.
My only gripe lies in that I really do not like Bud or the buildme method of compiling stuff like this.
In particular, buildme takes bleeding ages to compile...
My personal preference is for GNU Make makefiles (Which are also fairly portable thanks to things like MSys and the like...)
As a result I've been working on a makefile for Derelict which I expanded into something a little more general. The result is a template makefile for D libraries and another for D applications, along with a Derelict-customised version of the library template file.
I think the makefiles I've put together are ready for release and testing (Beyond test runs compiling Derelict that is...) so I've uploaded on my development blog, which can be found here.
Well, anyway, I'd welcome comments and confirmation that it actually compiles Derelict and anything else properly...
Usage is easy enough, as always:
"make" builds
while
"make clean" cleans the build
The makefiles generate objects, output, DDocs and .DI files in one go
So, yeah, comments would be welcome...
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI |
|
| Back to top |
|
 |
aldacron
Joined: 05 May 2004
Posts: 1322
Location: Seoul, South Korea
|
| Posted: Tue Feb 06, 2007 8:53 pm Post subject: Re: Derelict Makefiles |
|
|
| OOPMan wrote: | | Nice work with the Derelict bindings by the way, very cool beans. |
Thanks!
| Quote: |
My only gripe lies in that I really do not like Bud or the buildme method of compiling stuff like this. |
Interestingly enough, Derelict started out with makefiles for Digital Mars Make. The reason I switched to buildme.d and Bud is because DM Make is not 100% compatible with GNU Make. Plus, I needed separate makefiles for Windows and Linux. Even if I were to support GNU make for Windows, not everyone uses MSys so I would need a separate version for the console. Yuck. The buildme for bud seemed an excellent solution, and I still think it is.
| Quote: | | In particular, buildme takes bleeding ages to compile... |
Hmm. The recent changes I made, which allow buildme to compile with DMD or GDC directly, resulted in long build times when going that route, true. But when compiling with Bud, it's quite fast. DerelictGL hangs for just a bit, but it's still light years faster in comparison to not using Bud. The script will now use DMD or GDC by default, so to compile with Bud make sure you pass it on the command line, i.e.:
dmd -run buildme.d bud
OR
gdmd -run buildme.d bud
Without the bud option, yes it takes ages (especially GDC), but with Bud it's quite fast. Did you run it like that?
| Quote: |
My personal preference is for GNU Make makefiles (Which are also fairly portable thanks to things like MSys and the like...)
|
That's actually what I use when I program C. Since using D, I've gotten used to not using IDEs and fallen in love with Crimson Editor. So even with C now I just edit with CE, and debug with gdb & compile with GNU Make via MSys.
| Quote: |
As a result I've been working on a makefile for Derelict which I expanded into something a little more general. The result is a template makefile for D libraries and another for D applications, along with a Derelict-customised version of the library template file.
|
Sounds interesting. Did you see Gregor's Rebuild project yet?
| Quote: |
Well, anyway, I'd welcome comments and confirmation that it actually compiles Derelict and anything else properly...
Usage is easy enough, as always:
"make" builds
while
"make clean" cleans the build
The makefiles generate objects, output, DDocs and .DI files in one go
So, yeah, comments would be welcome... |
While I don't mind using Make for C app, I'm not too keen on them for D, since Bud and Rebuild are available and, IMO, are better options. But when I have time I'll see about downloading this and giving you some feedback.
_________________
The One With D | The One With Aldacron | D Bits |
|
| Back to top |
|
|
OOPMan
Joined: 06 Feb 2007
Posts: 6
|
| Posted: Wed Feb 07, 2007 2:29 am Post subject: Re: Derelict Makefiles |
|
|
| aldacron wrote: | | Interestingly enough, Derelict started out with makefiles for Digital Mars Make. The reason I switched to buildme.d and Bud is because DM Make is not 100% compatible with GNU Make. Plus, I needed separate makefiles for Windows and Linux. Even if I were to support GNU make for Windows, not everyone uses MSys so I would need a separate version for the console. Yuck. The buildme for bud seemed an excellent solution, and I still think it is. |
Hmmmm, the joy of make comptability troubles, eh?
Well, my makefiles do specifically state that they're GNU Make compatible and that windows users should use MSys, so I'm not to worried
| Quote: | Hmm. The recent changes I made, which allow buildme to compile with DMD or GDC directly, resulted in long build times when going that route, true. But when compiling with Bud, it's quite fast. DerelictGL hangs for just a bit, but it's still light years faster in comparison to not using Bud. The script will now use DMD or GDC by default, so to compile with Bud make sure you pass it on the command line, i.e.:
dmd -run buildme.d bud
OR
gdmd -run buildme.d bud
Without the bud option, yes it takes ages (especially GDC), but with Bud it's quite fast. Did you run it like that? |
Yeah, the bud version compiles much faster, for sure, but it's a pain that the plain buildme takes so long. I'm guessing it's slow slow because:
1: Library collection is performed on each .O file
2: File deletion is performed on each .O file
Is there a reason for this? It seems to be the source of slowdown in the GL package. At the very least file deletion could be handled with a recursive "rm -rf *.o" rather than on a per-file basis.
As for the library collection, is there a reason why it's handle the way it is? It seems to result in a horribly bloated output library file as well...
Speed-wise my makefiles seem to be faster than buildme or bud, although bud does provide some competition
| Quote: | Sounds interesting. Did you see Gregor's Rebuild project yet?
|
Nope, thanks for the link, I'll give it a test
| Quote: | | While I don't mind using Make for C app, I'm not too keen on them for D, since Bud and Rebuild are available and, IMO, are better options. But when I have time I'll see about downloading this and giving you some feedback. |
Hmmmm, any particular reasons? Personally I prefer my makefiles because I like to be sure I know *exactly* what is and isn't being compiled, generated and cleaned, rather than leaving those decisions up to a rather "black box-ish" tool like bud
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI |
|
| Back to top |
|
|
Crispy
Joined: 26 Nov 2005
Posts: 67
|
| Posted: Wed Feb 07, 2007 3:14 am Post subject: Re: Derelict Makefiles |
|
|
| OOPMan wrote: | | Personally I prefer my makefiles because I like to be sure I know *exactly* what is and isn't being compiled, generated and cleaned, rather than leaving those decisions up to a rather "black box-ish" tool like bud |
To me, that sounds suspiciously similar to the arguments that assembly language programmers made when railing against C. "You've got no control over what the compiler's doing!" 
I like things to just work. I don't like having to spend the time setting up (and subsequently babysitting) a makefile. I've never had a problem with bud that wasn't my fault, and never a major problem - you just point it vaguely in the direction of your code and it does the right thing. Nothing could be simpler or easier. |
|
| Back to top |
|
|
aldacron
Joined: 05 May 2004
Posts: 1322
Location: Seoul, South Korea
|
| Posted: Wed Feb 07, 2007 3:26 am Post subject: Re: Derelict Makefiles |
|
|
| OOPMan wrote: |
Yeah, the bud version compiles much faster, for sure, but it's a pain that the plain buildme takes so long. I'm guessing it's slow slow because:
1: Library collection is performed on each .O file
2: File deletion is performed on each .O file |
Actually, not anymore. Over the course of several days, I went with a few different approaches. In the final revision from last week, the object files are passed to lib/ar in groups according to directory and deleted as groups as well. That means that for each package except DerelictGL, all of the object files are manipulated in one go. With DerelictGL, there are a handful of subdirectories and an equal number of object groups.
| Quote: | | Is there a reason for this? It seems to be the source of slowdown in the GL package. At the very least file deletion could be handled with a recursive "rm -rf *.o" rather than on a per-file basis. |
Yeah, the per-file thing was painfully slow. It's better now that I'm grouping them. DerelictGL is still slow when compiling, though. The problem with the DerelictGL objects is that when compiling with DMD on Windows, the lib tool kept complaining when I passed it all of the object files at once. Even when passing them all individually I had to keep increasing the page size, which resulted in warnings every time the library was linked.
What I need to do is peek at the source for Bud and see how Derek got things to go so fast.
| Quote: | | It seems to result in a horribly bloated output library file as well... |
You know, I didn't even notice this. I hadn't bothered to check the size of the library files. I just compared two builds, one using Bud and one without, and I see what you mean. This is largely from my ignorance of the lib tool. I'll need to figure out a way around this.
| Quote: |
| Quote: | | While I don't mind using Make for C app, I'm not too keen on them for D, since Bud and Rebuild are available and, IMO, are better options. But when I have time I'll see about downloading this and giving you some feedback. |
Hmmmm, any particular reasons? Personally I prefer my makefiles because I like to be sure I know *exactly* what is and isn't being compiled, generated and cleaned, rather than leaving those decisions up to a rather "black box-ish" tool like bud |
Heh, well, because I don't have to tell it what to compile, generate and clean. Makefiles are just a PITA to maintain. Bud's brf's are quite simple by comparison. Actually, I haven't encountered any cases where I've needed more control over the process. When I want to exclude source modules from being compiled, such as when I'm linking with the libraries those modules make up, I just use -X to tell Bud not to compile them. Everything else I actually import gets parsed, compiled, and linked. It doesn't get any easier than that.
_________________
The One With D | The One With Aldacron | D Bits |
|
| Back to top |
|
|
OOPMan
Joined: 06 Feb 2007
Posts: 6
|
| Posted: Wed Feb 07, 2007 5:40 am Post subject: Re: Derelict Makefiles |
|
|
| aldacron wrote: | | Actually, not anymore. Over the course of several days, I went with a few different approaches. In the final revision from last week, the object files are passed to lib/ar in groups according to directory and deleted as groups as well. That means that for each package except DerelictGL, all of the object files are manipulated in one go. With DerelictGL, there are a handful of subdirectories and an equal number of object groups. |
Hah, it would figure that I had a 2 week old trunk release It's been updated now
| aldacron wrote: | Yeah, the per-file thing was painfully slow. It's better now that I'm grouping them. DerelictGL is still slow when compiling, though. The problem with the DerelictGL objects is that when compiling with DMD on Windows, the lib tool kept complaining when I passed it all of the object files at once. Even when passing them all individually I had to keep increasing the page size, which resulted in warnings every time the library was linked.
What I need to do is peek at the source for Bud and see how Derek got things to go so fast. |
Weird, so Windows has issues with the GL thing? Ah well, it's a pain keeping either Windows or Linux satisfied, let alone both...
As for bud, I figure it probably passes as many input source files to the D compiler in one go as possible in order to minimise calls to the compiler...
| aldacron wrote: | | You know, I didn't even notice this. I hadn't bothered to check the size of the library files. I just compared two builds, one using Bud and one without, and I see what you mean. This is largely from my ignorance of the lib tool. I'll need to figure out a way around this. |
I was a bit shocked myself. Once again, the problem is in the GL package I suspect. With the per-file version it was creating a .a for every source file, which ended up being very wasteful...
| aldacron wrote: | | Heh, well, because I don't have to tell it what to compile, generate and clean. Makefiles are just a PITA to maintain. Bud's brf's are quite simple by comparison. Actually, I haven't encountered any cases where I've needed more control over the process. When I want to exclude source modules from being compiled, such as when I'm linking with the libraries those modules make up, I just use -X to tell Bud not to compile them. Everything else I actually import gets parsed, compiled, and linked. It doesn't get any easier than that. |
Hehehe, actually, the template makefiles that I put together for D don't need to be told *exactly*. You just plonk a list of source directories into a variable and it uses a funky foreach trick to generate a list of source files from everything relevant in the directories listed Take a look at the makefile to see how it works. I got the trick from someone else and in retrospect I should credit them. Time to dig details methinks...
The only downside is illustrated by the GL binding for derelict. In order for things to compile one needs to open wingdi.d in DerelictGL\derelict\opengl and enclose the entire file in a version(Windows) {} to stop the compiler from throwing up on Linux
Otherwise, it's actually a very nice makefile that basically automates almost everything for you, kinda bud-ish really. Just plonk in source directories, import directories and set it running
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI |
|
| Back to top |
|
|
OOPMan
Joined: 06 Feb 2007
Posts: 6
|
| Posted: Wed Feb 07, 2007 9:16 am Post subject: |
|
|
Some big and somewhat nasty news. A couple of news items actually.
I've tracked down what is causing the huge slowdown in compilcation while also bloating the GL bindings to such huge extents.
Just a note for anyone else reading. I was communicating with aldacron via PM regarding a problem compiling the examples, which is what led me to discover this problem.
Right, okay, so, first problem.
Use of the -inline parameter by both buildme.d and my own makefiles siginificantly slows complication down under Linux (Possibly windows as well) and has the effect of horribly bloating the GL bindings (Most of the GL .o files end up at least 500kb in size, if not larger...) However, this does not seem to affect bud in any way, and it compiles everything fine.
I discovered this after I added the -inline parm to my makefiles and compilication time and output library sizes shot through the roof. I haven't finished testing this yet on Windows, but am doing so now in the background...
EDIT: The problem does not seem to manifest on windows. Possibly it's a problem with the linux version of DMD 1.004. I'll test with gdmd to see if that has similar problems...
This is very very weird, since unless I'm mistaken the Bud scripts also cause -inline to be used...is this correct?
Okay, the second problem is weirder and more troublesome. Some how my makefiles and the vanilla buildme.d script as well are producing non-functional library files under Linux. I haven't determined whether this is happening on Windows, but I'll check in a bit. Bud is not producing corrupt files, but I'm not sure why this is...
I discovered this while trying compile the SDL example under Linux. First, I built the derelict libs with Bud as follows:
dmd -run buildme.d bud
I then copied the .a libs to my /usr/local/lib and tried compiling the SDL demo with:
dmd sdl_ex1.d -I../DerelictSDL -I../DerelictUtil -L-ldl -L-lDerelictSDL -L-lDerelictUtil
The demo compiled fine and worked perfectly...
I then compiled the relevant packages (DerelictSDL and DerelictUtil) with the normal buildme.d script and, after cleaning executables, etc tried compiling the demo again.
The linking process failed, complaining of undefined symbol references to various name-mangled SDL and util symbols.
Finally, I compiled a libderelict.a using my own makefiles (Both without the crazy -inline problem switch and with it as well, just for testings sake) and tried compiling the demo again.
I got the very same undefined symbol ref errors!
This was freaking me out a little so I went and opened the Bud produced libDerelictSDL.a file and checked file sizes. I then did the same with the various other .a files. I was kinda shocked to find that for some reason the Bud produced .o files contained within the .a libraries were a bit larger than the .o files produced by buildme.d or the makefiles.
For example, active.o was 2.3kb in the Bud produced version and only 1.0kb in the libraries produced by buildme.d and the makefiles.
This leads me to believe that Bud is doing something that my makefiles and the buildme.d script are not, something that is actually essential. I'm pretty sure that Bud is also stripping the symbols, so I don't think it's this, but I'll check again just to be totally sure...
At the moment I'm wondering if the format of the .o files being produced is somewhat different with bud. Maybe the source will reveal more...
aldacron, do you have any ideas what could be causing these problems?
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI |
|
| Back to top |
|
|
OOPMan
Joined: 06 Feb 2007
Posts: 6
|
| Posted: Wed Feb 07, 2007 10:36 am Post subject: |
|
|
Woop, found the solution to problem 2.
Bud was not stripping symbols, while the makefiles and buildme.d were.
As you can guess, when I removed the strip command my library started working again
Now all I need to do is test the -inline parm with gdc...
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI |
|
| Back to top |
|
|
aldacron
Joined: 05 May 2004
Posts: 1322
Location: Seoul, South Korea
|
| Posted: Wed Feb 07, 2007 11:42 am Post subject: |
|
|
| OOPMan wrote: | Bud was not stripping symbols, while the makefiles and buildme.d were.
As you can guess, when I removed the strip command my library started working again
|
Wow. I thought stripping was supposed to be a good thing. Should I remove that from the buildme?
_________________
The One With D | The One With Aldacron | D Bits |
|
| Back to top |
|
|
JJR
Joined: 22 Feb 2004
Posts: 1104
|
| Posted: Wed Feb 07, 2007 3:53 pm Post subject: Re: Derelict Makefiles |
|
|
| OOPMan wrote: | | Quote: | | While I don't mind using Make for C app, I'm not too keen on them for D, since Bud and Rebuild are available and, IMO, are better options. But when I have time I'll see about downloading this and giving you some feedback. |
Hmmmm, any particular reasons? Personally I prefer my makefiles because I like to be sure I know *exactly* what is and isn't being compiled, generated and cleaned, rather than leaving those decisions up to a rather "black box-ish" tool like bud |
I just want to point out that bud and rebuild both allow you to see exactly what is going into the final library or executable via -v switch. Therefore, I don't think your argument is strong in this regard. If they didn't do that, I might be a little hesitant to use such tools as well.
Makefiles have a formidable learning curve and can be frustrating to maintain because of compatibility issues: not everyone uses GNU make so most large projects end up supporting multiple make types in order to build cross-platform projects.
The fact that they are available as an option is good, however, for those developers that feel a need for them. Certain situations might even favour their use. Tango, for example, still uses makefiles for building the low level library components. But it would be admittedly harder for new users to start using Tango, if they had to use makefiles everytime they wanted to test or use Tango features in their own projects. Bud and Rebuild are huge time savers in this regard.
Bud and Rebuild are very good solutions for a quick turnaround cyle in editing/compiling/testing. They allow you to take your mind off the nitty gritty details and concentrate on the project itself. |
|
| Back to top |
|
|
Derek Parnell
Joined: 22 Apr 2004
Posts: 408
Location: Melbourne, Australia
|
| Posted: Wed Feb 07, 2007 4:18 pm Post subject: Re: Derelict Makefiles |
|
|
| OOPMan wrote: | | I prefer my makefiles because I like to be sure I know *exactly* what is and isn't being compiled, generated and cleaned, rather than leaving those decisions up to a rather "black box-ish" tool like bud |
I suspect you misunderstand what Bud can do and can't do.
Bud examines the source files to work out what needs to be built, then it creates two sets of command lines - one for the compiler and one for the linker. It then submits these command lines to the operating system.
You can use the -V and/or -test switch to see what is in those command lines. The -test builds the command lines but does not submit them to be run. The command lines are created in <target>.rsp and <target>.ksp files so you can look at them to see EXACTLY what will be submitted. There is no 'black-box' because I wanted to see what Bud was doing too.
_________________
--
Derek
skype name: derek.j.parnell |
|
| Back to top |
|
|
OOPMan
Joined: 06 Feb 2007
Posts: 6
|
| Posted: Thu Feb 08, 2007 2:14 am Post subject: |
|
|
Eek, my bad, sorry if it seemed like I was performing a character assassination of Bud
I do understand what Bud does and doesn't do, I just don't like it too much
As for the make vs. make compatibility issues, I get what you're all saying, but as I said before, these makefiles state specifically in the headers that they are GNU Make makefiles and hence it's not my problem if someone ignores the spec and tries to use them with some other version of make. That may seem a little callous, but hey, you have to draw the line somewhere
As for the issues about makefiles being tedious to maintain, not really. That only applies if your makefile is written using the basic make stuff. As the makefiles I put up illustrate, source/object/etc file list population can be entirely automated without too many problems, so one's makefile need not contain 5 zillion entries, one for each file or so
On a side note, I should point out that the trick used in my makefiles was originally illustrated via the makefiles included with devkitPro. I just jiggered it a bit to suit my needs
But anyway, I'm not really smacking down Bud or Rebuild. You're welcome to use them, but I still prefer make for this kind of thing... (I'm not one for re-inventing the wheel
EDIT: Updated the makefiles download over at the blog link I posted. So now the makefiles actually create a usable library
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
