FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Linux Library Linking Issues Explained

 
Post new topic   Reply to topic     Forum Index -> Build
View previous topic :: View next topic  
Author Message
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Sat Oct 08, 2005 1:28 am    Post subject: Linux Library Linking Issues Explained Reply with quote

I've finally tracked down a "minor" issue with build. Well, it not so much a problem with build, I guess; but build propogates the error so to speak when I use linux.

It goes like this:

- Compile a D library with build...
- Compile your project with build and include your D library on the command line with the -L-l command (assuming your D libraries are on the system library path).

If you do the above, the project link phase will die on missing symbol errors mostly relating to missing phobos symbols. libphobos.a is automatically included in all d programs, so these errors at first don't make sense. But to appease the linker we add a -L-lphobos back on to the command line (second time). This clears up all previous errors. Program links. But executable crashes on run.

Solution:

Instead of using the linker passthrough switch for libraries (-L-l), avoid it like the plague when you are linking your own D libraries in with your program (the method otherwise works fine with C system libraries, though)! Use a direct link to your d library with the full path instead:

dmd project.d /my/library/path/libmylib.a [other options]

At least I tried the above with dmd directly (using -L-l with dmd causes the same problem as that with build). I assume it should work the same with build. At least build should submit the above in the same form to dmd or else trouble continues. If the library is submitted this way, the resulting executable runs correctly with no errors. Furthermore there is no need to try to fix the problem with a tacked on -L-lphobos for symbol resolution.

What happens is that dmd seems to get the required symbols immediately submitted to the linker when using the above format, whereas the -L-l switch tacks them on late, which somehow causes the linker to not see phobos symbols for your -L-l included D library. Trying to fix the missing symbols with a redundant -L-lphobos causes major conflicts that crash the program.

I'm not sure if the pragma(link...) command causes the same problem, but I'm pretty sure it does as long as it passes on the command in a -L-l form to the linker. Once again this is true of D libraries. C libraries don't seem to have this issue.

So perhaps this is more of a warning than a request for a fix. Although it would be appropriate if the pragma(link...) could be fixed to submit the library to the command line in the appropriate format, otherwise D libraries won't link in as expected and the resulting program won't run. I'm not sure what action is necessary, since in actually fact using an -L-l should work in the first place!

-JJR
Back to top
View user's profile Send private message
Derek Parnell



Joined: 22 Apr 2004
Posts: 408
Location: Melbourne, Australia

PostPosted: Sat Oct 08, 2005 2:56 am    Post subject: Re: Linux Library Linking Issues Explained Reply with quote

JJR wrote:
I've finally tracked down a "minor" issue with build. Well, it not so much a problem with build, I guess; but build propogates the error so to speak when I use linux.
-JJR


A very timely post. I'm currently rewritting the linker invocation logic in Build. Last night I completed the Windows section and I'm going to tackle the Linux section today (and tomorrow I guess).

The major change is that I no longer have DMD invoke the linker. Build now calls it directly, after all the compilations are completed. This will allow me to be very precise about how the linker is started and what switches it gets, etc....

I've done this for a number reasons: To support the idea of embedding compiler options in source files for specific compliation requirements, and to allow alternative library locations without having to supply these via sc.ini file, finally to allow building multiple projects in one call.

I'll examine your findings and I will definitely need help in testing and debugging the next Linux version of Build.
_________________
--
Derek
skype name: derek.j.parnell
Back to top
View user's profile Send private message
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Sat Oct 08, 2005 4:35 am    Post subject: Reply with quote

Perfect!

Then count me in for the testing phase.

-JJR
Back to top
View user's profile Send private message
teqdruid



Joined: 11 May 2004
Posts: 390
Location: UMD

PostPosted: Sat Oct 08, 2005 10:30 am    Post subject: Re: Linux Library Linking Issues Explained Reply with quote

JJR wrote:
So perhaps this is more of a warning than a request for a fix. Although it would be appropriate if the pragma(link...) could be fixed to submit the library to the command line in the appropriate format, otherwise D libraries won't link in as expected and the resulting program won't run. I'm not sure what action is necessary, since in actually fact using an -L-l should work in the first place!


The correct behavior for pragma(link) is using -L-l (for dmd) or just -l if you're going right to gcc. Doing anything else is just plain wrong. Of course this is assuming that the link pragma is for linking in libraries.

As for your linking issues, I've never had any problems whatsoever with -L-l. Can you post a list of steps to reproduce the issue?

~John Demme
Back to top
View user's profile Send private message Send e-mail AIM Address
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Sat Oct 08, 2005 4:04 pm    Post subject: Reply with quote

John,

I agree that using -L-l is generally the proper approach for submitting a library to the linker. That's why, if it worked correctly all the time, it would be the preferred way to link in a D library. When passing a D library directly to DMD, however, I've found that it only works if you include the library directly on the command line (in this case DMD manages submitting the library to the linker). This way is no less incorrect where D specific libraries are concerned. It's just an inappropriate option where the pragma(link, ...) directive is used.

For some strange reason, as I describe previously -L-l does not work properly in some cases. In the next few days, I'll try to submit an example for you to try that reproduces the problem (been busy lately).

Also, I qualified that -L-l should work in the first place. Thus, I agree that pragma(link, ...) should be using this method. But since it appeared to be failing, I was open to an alternate solution to solve my problem.

I think it odd that, in my case, dmd seems to submit libraries to the linker in very different orders depending on whether an -L-l switch is used or the library is supplied directly on the command line.

I'll try to submit an example that reveals the problem I'm experiencing.

-JJR
Back to top
View user's profile Send private message
teqdruid



Joined: 11 May 2004
Posts: 390
Location: UMD

PostPosted: Sat Oct 08, 2005 7:05 pm    Post subject: Reply with quote

We've had this argument before with build. It's possible that dmd is changing the order of the -l's and that's doing something, but since Derek is re-writing it to use gcc directly, that shouldn't be an issue anymore.

If there's an issue with the -l switch, we need to get that issue fixed rather that attempt a rather difficult workaround. The only other way for build to do it is to search through the library path in the same manner as gcc, which seems unnecessary to me.

Considering how hard it was to get it right last time, I'd just rather not go through it again. Sorry if my response seemed a little hard-lined.

~John Demme
Back to top
View user's profile Send private message Send e-mail AIM Address
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Sat Oct 08, 2005 8:46 pm    Post subject: Reply with quote

My gaff must have been that I didn't realize the discussion occurred before (or maybe I just forgot Sad ). I'm afraid I wasn't arguing for any particular method of fixing the problem. I was just providing the only solution I knew myself.

Since Derek seems to be working on a overall solution, perhaps there's nothing more for me to really say on the matter now. I'm very greatful for any solution, though. Smile

Given that you've wrangled about these linking issues before, I can certainly understand your minor impatience. Smile

-JJR
Back to top
View user's profile Send private message
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Sun Oct 09, 2005 4:27 pm    Post subject: Reply with quote

As fate would have it, the problem is no longer reproducible in its entirety. I'm not sure why since the original problem used to happen all the time for me.

Yet, adding another -L-lphobos is still necessary to avoid missing symbol errors in the linker. This should not be the case as phobos is already submitted to the linker at an earlier stage (as revealed by the build output).

Nonetheless the resulting executable now runs without error contrary to what I indicated before. That's good news. So if Derek can streamline how build organizes the linking process so that -L-l works without adding another -L-lphobos then all should be just about perfect.

-JJR
Back to top
View user's profile Send private message
Carlos



Joined: 19 Mar 2004
Posts: 396
Location: Canyon, TX

PostPosted: Sun Oct 09, 2005 8:37 pm    Post subject: Reply with quote

I'm not sure of what you're doing, John, and I don't remember the details exactly, but I think something like this happens: when you do "dmd foo.d -L-lmylib", DMD passes to gcc "gcc -o foo foo.o -lphobos -lm -lpthread -lmylib", or something like that, so if mylib depends on phobos, ld fails. The workaround is to just compile with DMD and then do "gcc -o foo foo.o -lmylib -lphobos -lm -lpthread".
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Derek Parnell



Joined: 22 Apr 2004
Posts: 408
Location: Melbourne, Australia

PostPosted: Sun Oct 09, 2005 10:47 pm    Post subject: Reply with quote

Carlos wrote:
I'm not sure of what you're doing, John, and I don't remember the details exactly, but I think something like this happens: when you do "dmd foo.d -L-lmylib", DMD passes to gcc "gcc -o foo foo.o -lphobos -lm -lpthread -lmylib", or something like that, so if mylib depends on phobos, ld fails. The workaround is to just compile with DMD and then do "gcc -o foo foo.o -lmylib -lphobos -lm -lpthread".


The linker does one pass through each library and so if a module in one library references a module in an earlier library there will be problems. That is why sometimes one needs to have a library specified multiple times so that multiple passes can be made of it.
_________________
--
Derek
skype name: derek.j.parnell
Back to top
View user's profile Send private message
JJR



Joined: 22 Feb 2004
Posts: 1104

PostPosted: Tue Oct 11, 2005 5:41 pm    Post subject: Reply with quote

Carlos,

You're correct. I'm afraid I came to the same conclusion after some trial and error. I wish I had played with this a little more before posting so much redundant mumbo jumbo.

It makes sense now. Although it didn't seem to work correctly before (as in several months ago). Thanks, guys, for putting up with me on this one.

-JJR
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Build All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group