FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Library naming convention implementation
Goto page Previous  1, 2
 
Post new topic   Reply to topic     Forum Index -> Build
View previous topic :: View next topic  
Author Message
Derek Parnell



Joined: 22 Apr 2004
Posts: 408
Location: Melbourne, Australia

PostPosted: Wed Sep 20, 2006 5:59 pm    Post subject: Reply with quote

There are some fundamental assumptions that we are both making that do not agree with each other.

Gregor wrote:
Through this entire discussion, you've given no thought to shared object files. This is not good, nor is it promising.

This, for example. What is a 'shared object file'? There is no such thing in the Windows environment and that's where I work. I do not work in Linux. In Windows there are shared libraries (DLL) but you know that. I believe that the equivalent in Linux are the .so files. But these are libraries and not object files. They contain object files but are not object files themselves, as far as I'm aware.

However, if you really meant 'shared libraries', then you are wrong. I have given thought to these. And that is where I think we have a huge misunderstanding.

In Windows, a DLL is 'loaded' at runtime by the application. The name of the library and its location are not necessarily known at compile time, so the file I/O is resolved at run time. Any build process, be it my Build or make, cannot help us at run time.

Gregor wrote:
D should be capable of using .so and .dll files.


It already is. Why do you think that D can't use .so/.dll files?

Gregor wrote:

Derek Parnell wrote:
Gregor wrote:
That's mostly what I'm saying, with one exception: The libraries don't form a directory hierarchy like you're suggesting. This can make linking against shared object files at runtime a nightmare.


Huh? Hang on. Linking at runtime is NOT the responsibility of Build. The application controls this and Build doesn't know anything about it. Build only deals with static linkage. It cannot deal with runtime linkage because the location and library names are determined at run time and not compile time.


This assertion is valid for linking against static libraries, but falls flat with shared libraries. The runtime linker expects shared libraries to be in a standard location, so the compile-time linker also needs to expect them to be in a standard location.


I don't know about Linux, but the Windows world doesn't work like that. DLL files don't have to be in the standard (default) location, nor is there such a concept as a standard place for compile-time libraries. In Windows, the shared library files can be anywhere, as can the static libraries. This is why Build needs to tell the linker where to look for libraries and why Build can't help with run-time linking.

Gregor wrote:
Derek Parnell wrote:
Gregor wrote:
The naming convention is that the package name is encoded into the library file name. So, with
Code:
import a.b.c.d.e.f.g;


it will look through every library file name, from most specific to most general, for one that exists. If it exists, and is following the naming convention, it contains all the symbols being imported.

But where will it look for these library files? I mean, what is the directory, or list of directories, that it should scan through looking for libraries?


The standard system library path.


There is no such thing in Windows and I have no idea what it is in Linux. And it can only apply to static linking anyway because the runtime name of a library might not be known until the application actually executes.


Gregor wrote:

Derek Parnell wrote:
Gregor wrote:
The encoding of the package name into the library name is simple, the library is named (under UNIX):
libD.<package name>.<extension>

But why do they have to be named so? Is it so the humans can look at the name and make a reasonable guess as to what modules it should contain? The linker doesn't give shit about the name so any library found in the expected places could be used.


The runtime linker does care when they're shared object files, that is, .so and .dll files.

Why? I don't get it. The linker needs to be told what name to load into RAM but it surely doesn't care about the format or content of the name!?

Gregor wrote:

Derek Parnell wrote:
Gregor wrote:

So, for a.b.c.d.e.f.g, it would look at:
libD.a.b.c.d.e.f.<valid extensions>
libD.a.b.c.d.e.<valid extensions>
etc.


But in which directory is "libD.a.b.c.d.e.f.<valid extensions>" expected to be found in?


The standard system library path.


Windows has no concept of this. And what do I use for Linux? I mean, how does Build find out what the 'The standard system library path' is?

Gregor wrote:

Derek Parnell wrote:
Gregor wrote:
The other side of this is that build ought to be able to produce these libraries as well, and this production should be painless. Just add some flag, and it'll turn every package into a library (or further subdivide if subdivisions are specified by the upstream author).

Hmmmm... got to think that idea through a bit more, I suspect.


You have to or I have to? It's a workable, logical plan as far as I'm concerned.

I have to think it though. It isn't that simple in the Windows world. There are some special things required for a DLL to operate correctly and these must be supplied in source code somewhere as each DLL can do things differently. Again, I don't know about Linux .so files.
_________________
--
Derek
skype name: derek.j.parnell
Back to top
View user's profile Send private message
Gregor



Joined: 05 May 2006
Posts: 72
Location: Portland, OR

PostPosted: Thu Sep 21, 2006 1:38 am    Post subject: Reply with quote

First, I'm just going to try to respond to everything.

aldacron wrote:
It seems to me you are suggesting that DMD should be able to accept shared libraries on the command line at compile time. If that's what you are getting at, I can see where the confusion is coming from.

First, that's a definite Unixism. You are talking about standard library names, runtime linkers and standard library paths. That's all in the Linux world. Derek comes from the Windows world (so do I, which is why this conversation is causing my head to spin). I think your intent is getting lost in translation. The only compiler I know of on Windows that can accept a DLL on the command line is MingW -- and most people don't even know that since I often find myself bringing it up in different forums.

On the Unices, you can pass a shared object file to GCC at compile time and the runtime linker will pull it in at runtime. Windows doesn't function that way. On Windows, you link to a static library which contains the DLL exports and the DLL is loaded at runtime. The term "runtime linker" doesn't exist in the Windows world AFAIK.


...ow.

aldacron wrote:
Second, there's no naming convention on Windows for library files or DLLs


And? As I said, this can be layered on top of a non-naming-convention as well as UNIX's (or any other) convention.

aldacron wrote:
nor is there a standard library path.


The compiler doesn't maintain such a path?!

aldacron wrote:
You can set up a LIBRARY environment variable that some compilers will use, but there is certainly no standard path. Windows has a standard search pattern for DLLs at runtime, starting with the working directory, so most apps just drop any non-system DLLs there. But there certainly isn't any naming convention, nor does there need to be.


I'm not suggesting a general naming convention, but a naming convention for D libraries in particular.

If I understand, apps in Windows don't generally share .dll files (blech), hence the confusion over wanting a naming convention at all.

aldacron wrote:
Finally, I don't think it is clear at all what you are proposing that Build should do. Are you suggesting that Build should pass any shared library files the app requires on to the compiler as it does already for static libraries?


I'd like for library files (whether shared or static, though apparently maybe only static on Windows given what you've been telling me) to be a transparent alternative you including the entire source to the library you're linking against with -I

aldacron wrote:
This is a confusing conversation. If you could be a little more clear about exactly what you want Build to do with shared object files it would be helpful.


I'm going to make another post after this one (perhaps several hours after, I'm busy Razz), trying to explain everything in a more Windows-compatible way.


aldacron wrote:
Gregor wrote:
My purpose is to make library files an alternative to building everything from source. The two components of that are:

1) Building library files from a source directory.
2) Linking against said library files while building another program.


Build can already build libraries from a source directory and link against them when building another program. It only builds an executable when it encounters a main or WinMain method.


I'm sorry, I didn't mean to imply that it couldn't, what I meant was that it would build libraries as requested with some specific standard of naming (unless explicitly overridden), such that it could then link against those libraries without your needing to explicitly -l them in. That is, it would link against libraries where the current convention is to compile the imported prerequisites.

aldacron wrote:
Gregor wrote:
My ONLY PURPOSE at present is to make linking against libraries a transparent alternative to building entirely from source.


What does this mean?


If you have some tool, foobar, that depends upon mango.io, you should have these alternatives to get it to link:

1) -I against mango's directory, compile mango into it from source
2) Copy mango in
(the above 2 are already in place)
3) -I against a directory with .di files for mango, and build will find the appropriate .a, .lib or .so files.

aldacron wrote:
Gregor wrote:

This involves slight modifications in the compiling and heavy modifications in the linking phases you describe.

As per what "problem" I'm solving, I'm trying to prevent a problem from happening, not solve one. Eventually, people will start making library files with D. I've had enough experience with bizarre conventions, so I want to make sure that there's a simple, sane convention in place before (shared) libraries become prevalent.

It seems to me like the most logical place for these changes to be made is the tool used to build software - and the de facto standard for that is build.


Wouldn't the better place for any changes needed be in the compiler and the linker? What does patching Build have to do with anything?[/quote]

The compiler and linker are perfectly capable of linking against all sorts of libraries I'm sure. The issue is that some tools are intended to be compiled to libraries and can basically only be linked against that way, others are intended to be copied in (see methods 1 and 2 above), and others don't even care or know. It think that these should be transparent alternatives, not huge, difficult, source-altering alternatives.


aldacron wrote:
Gregor wrote:
In short: build will still do what it does. But it will also support building libraries and building with libraries.


Build already does support that. And if the compiler supports building shared libraries, build supports that, too. If the compiler accepts shared libraries on the command line, Build supports that, too.


My point is: You shouldn't have to supply them on the command-line. Build should be able to find them as an alternative to building the entire source.



Derek Parnell wrote:
There are some fundamental assumptions that we are both making that do not agree with each other.


I'm realizing that XD

Derek Parnell wrote:
Gregor wrote:
Through this entire discussion, you've given no thought to shared object files. This is not good, nor is it promising.

This, for example. What is a 'shared object file'? There is no such thing in the Windows environment and that's where I work. I do not work in Linux. In Windows there are shared libraries (DLL) but you know that. I believe that the equivalent in Linux are the .so files. But these are libraries and not object files. They contain object files but are not object files themselves, as far as I'm aware.


.so = shared object file = shared library

Derek Parnell wrote:
However, if you really meant 'shared libraries', then you are wrong. I have given thought to these. And that is where I think we have a huge misunderstanding.


What I mean is for any sort of library to be an alternative for -I'ing in source.

Derek Parnell wrote:
In Windows, a DLL is 'loaded' at runtime by the application. The name of the library and its location are not necessarily known at compile time, so the file I/O is resolved at run time. Any build process, be it my Build or make, cannot help us at run time.


The build tool links in the library, somewhere in the library the name is specified (I guess for Windows you link in a static library which specifies the name of the shared library). Therefore, the build tool does help if it automatically selects the right library.

Derek Parnell wrote:
Gregor wrote:
D should be capable of using .so and .dll files.


It already is. Why do you think that D can't use .so/.dll files?


I need to add the words "as a transparent alternative to -I'ing in source."

Derek Parnell wrote:
Gregor wrote:
This assertion is valid for linking against static libraries, but falls flat with shared libraries. The runtime linker expects shared libraries to be in a standard location, so the compile-time linker also needs to expect them to be in a standard location.


I don't know about Linux, but the Windows world doesn't work like that. DLL files don't have to be in the standard (default) location, nor is there such a concept as a standard place for compile-time libraries. In Windows, the shared library files can be anywhere, as can the static libraries. This is why Build needs to tell the linker where to look for libraries and why Build can't help with run-time linking.


Again: build should find libraries, whether static or shared (apparently static for Windows), as a transparent alternative to -I'ing in source files.


I need to make a more monolithic post towards Windows to explain everything again...

I'll make such a post in a few hours.
Back to top
View user's profile Send private message AIM Address
Gregor



Joined: 05 May 2006
Posts: 72
Location: Portland, OR

PostPosted: Thu Sep 21, 2006 4:22 am    Post subject: Reply with quote

OK, here's the monolithic post. I hope I can explain precisely what I'm trying to do here.

But first, some nomenclature ... I'll try to be consistent and use OS-agnostic names for things, but in case I forget:

.so = shared object file = shared library
.a = archive = static library
standard library directory = wherever the linker looks for libraries


The current methodology that build uses to compile software is to either:
1) Include the entire source of all prerequisite libraries in the source being compiled
or
2) Include the prefix to the source of prerequisites with the -I flag to build.

This method works quite well, but has a number of problems:
1) It requires the rebuild of all prerequisite libraries every time they're used.
2) It's incapable of using shared libraries, so the library used can't be upgraded without upgrading the tool itself (apparently this is the norm on Windows, but this is ridiculous on UNIX).

The one huge advantage that it has is that there's only one representation for code: source. Because of this, the naming is all very simple: imports can be directly mapped to source files.

What I'm proposing is an alternative, using libraries (whether shared or static). The most important thing about this alternative is that it should be totally transparent. That is, the user should be able to build with either complete source to all of the prerequisites or with libraries, without needing to worry about which is being used.

The problem with libraries in many languages is that there's no inherant mapping of import names to library names. So, the two names must be maintained separately, in source and in the Makefile (or configure, or whatever else). build is capable of building libraries using this no-rules convention, but it makes building tools a hassle for the builder. On Windows, the builder is usually whoever wrote the source, so it's not so much of a problem, but on UNIX the builder can be anyone, so dealing with all these library files is a huge hassle.

Solution: unify the namespaces. If library names correspond to import names, then any import can be mapped to either a source file, or both a .di file and a library file.

To do this, build would have to be able to:
1) Build libraries using a standardized naming convention, and install them to some path that either the compiler or build itself implicitly recognizes.
2) When building a tool with dependencies, map imports to library names, and compile in those dependencies automatically.

The importance of the standardized naming convention is simply that build be able to find them again without the user needing to specify anything. Hence, transparently.

On Windows, there would (apparently) generally be only static libraries compiled, so the only advantage is the amount of time taken to compile. Even that is significant, however: imagine porting Firefox and all of its dependencies to D. (Is there any way to automatically generate .dll files and the .lib files needed to link against them, without crazy __declspec etc?)

On UNIX, there would be support for both static and shared libraries (which are far more interchangeable at compile time in UNIX than they are on Windows), so the naming convention comes into play again: When you link against a shared library, the runtime linker/loader will look for an appropriately-named library at runtime. So, the name would have to remain consistent there.

The summary: In D, imports and files share a unified namespace: one .d file is one module, one directory is one package, etc, etc, etc. The only difference is that imports use .'s where files use whatever the directory delimiter is. My proposal is to make D library files (where appropriate) also share this namespace. The libraries would be named after the package they correspond to. The advantage is simple: the builder does not need to specify what libraries to link against, and in fact can either link against libraries or source with virtually no change to their build command (except to -I a directory with .d source files instead of .di interface files).
Back to top
View user's profile Send private message AIM Address
Gregor



Joined: 05 May 2006
Posts: 72
Location: Portland, OR

PostPosted: Thu Sep 21, 2006 8:11 pm    Post subject: Reply with quote

I think this could be even further explained with an example.

Let's say you have mango, and you want to make a library of it, and then use that library. For this example, imagine that I have a program called Foobar which simply prints the text "Foobar", and inexplicably requires mango.

With the current convention:

(No build explicitly of mango)

Code:

~$ cd foobar
~/foobar$ build foobar -I~/mango # Or copy the source in
~/foobar$ ./foobar
Foobar
~/foobar$ # foobar depends on no libraries other than the standard system ones



With libraries:

Code:

~$ cd mango
~/mango$ build --some-flag-to-tell-build-to-make-libraries
~/mango$ ls
mango/
libD.mango.so
libD.mango.a
~/mango$ build --some-flag-to-tell-build-to-make-and-install-d-interface-files-and-libraries
~/mango$ cd ..
~$ cd foobar
~/foobar$ build foobar
~/foobar$ ./foobar
Foobar
~/foobar$ # foobar depends on libD.mango.so
Back to top
View user's profile Send private message AIM Address
aldacron



Joined: 05 May 2004
Posts: 1322
Location: Seoul, South Korea

PostPosted: Thu Sep 21, 2006 11:49 pm    Post subject: Reply with quote

Okay, so you want to be able to compile an application without specifying libraries on the command line. What's wrong with creating a Build Response File and specifying the libraries there? You only need to do it once (ex. Foobar.brf):

Code:

-Imangopath
-Xmango
mango.a
Foobar.d


build @Foobar

If you decide you want to use mango's source in the build instead, you just remove the -Xmango and mango.a (or, I suppose it would be -lmango if using GDC?) from the BRF, then the mango source is used instead. The result is the same. It's transparent when you execute the build command.
_________________
The One With D | The One With Aldacron | D Bits
Back to top
View user's profile Send private message Send e-mail
Gregor



Joined: 05 May 2006
Posts: 72
Location: Portland, OR

PostPosted: Fri Sep 22, 2006 12:01 am    Post subject: Reply with quote

Because:

1) That limits the decision to the source maintainer exclusively, and is not flexible to any option.
2) That option is (apparently) compiler-specific.
3) It doesn't even attempt to solve the creation of libraries.
4) It still leaves these two namespaces (imports and libraries) separate, even though they are intuitively quite similar.
5) It relies on source maintainers or builders to establish their own convention for naming, since there's no good default provided to them. Establishing standards is a good thing. Have you ever used autoconf? Establishing standards is a very good thing.
Back to top
View user's profile Send private message AIM Address
Derek Parnell



Joined: 22 Apr 2004
Posts: 408
Location: Melbourne, Australia

PostPosted: Fri Sep 22, 2006 12:50 am    Post subject: Reply with quote

Gregor wrote:
OK, here's the monolithic post. I hope I can explain precisely what I'm trying to do here.

But first, some nomenclature ... I'll try to be consistent and use OS-agnostic names for things, but in case I forget:

.so = shared object file = shared library
.a = archive = static library
standard library directory = wherever the linker looks for libraries

The current methodology that build uses to compile software is to either:
1) Include the entire source of all prerequisite libraries in the source being compiled
or
2) Include the prefix to the source of prerequisites with the -I flag to build.

This method works quite well, but has a number of problems:
1) It requires the rebuild of all prerequisite libraries every time they're used.


Not really. It would only recompile source files that have been modifed. It only builds an object file if either the object file doesn't exist or is older than the source that builds it.
Gregor wrote:

2) It's incapable of using shared libraries, so the library used can't be upgraded without upgrading the tool itself (apparently this is the norm on Windows, but this is ridiculous on UNIX).


Not at all! I don't where you are getting your Windows info from Smile

The reason for a DLL is so that one can update the DLL without having to recompile the application that uses the functions contained in it.

I write Windows apps all the time and I never have to rebuild a DLL each time I recompile my application. In fact, compiling an application and compiling a DLL are totally independant of each other and neither is aware of the other during the recompilation process. There are only 'connected' up at run time and that is only by explicit coding on the application developer's part. I take it that this is not how its done in the Linux world.


There is already a facility within Build to associate a library with an application source file. Look up the pragma 'link' in the Build docs.

Using your 'mango' example ...

Assuming that the mango library has already been built and the '.di' files are distributed with the library. Our application file would look like...

Code:

import mango.whatever;  // assume contains the display function 
version (build) pragma(link, "mango"); // tells build to insert -Lmango on the linker command line
void main()
{
   display("Foobar");
}

Then to compile it using the mango library...
Code:

~/foobar$ build foobar
~/foobar$ ./foobar
Foobar


This is functionality already in Build.

And to build the mango library, all the mango team needs is a file containing a list of the modules that are supposed to go into the library.
For example ...
Code:

module all;
version(build)
{
   pragma(include, "mango.io");
   pragma(include, "mango.xml");
   pragma(include, "mango.tcp");
   . . .
   pragma(nolink);  // tells Build to not run the linker step
   pragma(target, "mango.a");
}


Then to build the library ...
Code:

build all -clean -full -H


So this allows the mango team to control what goes into a mango library, and what sort of library to create (shared and/or static), and allows the application developers to use which evey mango library they have (shared and/or static). All without messing about with unnecessary switches or odd names.


Now what I could do is improve the 'all.d' module by allowing wildcards in the include pragma, for example ...
Code:

pragma(include , "mango.*");

_________________
--
Derek
skype name: derek.j.parnell
Back to top
View user's profile Send private message
Gregor



Joined: 05 May 2006
Posts: 72
Location: Portland, OR

PostPosted: Fri Sep 22, 2006 9:01 am    Post subject: Reply with quote

It's clear that this is quickly regressing towards flamewar status. I don't understand what your core contention with my concept is. If I can simply figure that out, then I can properly argue, but as-is, I'm flustering to make a proper argument because I have no idea what you want me to prove.

From your last post, it seems like you think using build with my LNC would be complicated. This couldn't be further from the truth! My ultimate, happy conclusion would be basically exactly the flags you showed above, with only one caveat: There is a guaranteed link between module names and libraries. That's it.

In an attempt to figure out your contention, I have the following questions:

1) Do you agree that D packages are fundamentally similar to libraries?
2) Do you agree that this similarity could be exploited by guaranteeing a connection between the two?
3) Do you agree that such a guarantee could make the build process more automatic?
3) Do you agree that such a guarantee would make the build process easier?
4) Do you believe that my proposed changes would be very complex?
5) Do you believe that the current methodology, requiring modifying .d files or .brf libraries to use libraries rather than against-source builds, is superior?
6) Do you believe that the flexibility of the current methodology and the methodology you've been showing in the last posts trumps the reliable guarantee provided by my proposed solution?
7) Do you believe that my additions would break all current methodologies, or that that is implicitly a bad thing with a still-pre-1.0 language?
8 ) Is it your opinion that requiring a change to source to burn-in library names et al is more flexible than being able to simply connect imports to libraries?
9) What do you believe is my core argument?

FYI, my core argument is this: D packages and libraries are fundamentally similar, and this similarity can be exploited to make the entire building process easier and more reliable. The current methodology requires that these namespaces are maintained separately (if you use libraries), making the entire process more complicated, which is part of why everybody just -I's source files rather than using libraries.

EDIT: Had one more question
Back to top
View user's profile Send private message AIM Address
Gregor



Joined: 05 May 2006
Posts: 72
Location: Portland, OR

PostPosted: Fri Sep 22, 2006 9:35 am    Post subject: Reply with quote

Thank you, I think I understand our difference of opinion.

Unfortunately, I think this comes down to a core philosophy on which we disagree, and the three things you should never argue online are philosophy, religion and programming. We have two of them already Razz ... so about that Jesus fella' ...

Derek Parnell wrote:
Gregor wrote:

From your last post, it seems like you think using build with my LNC would be complicated. This couldn't be further from the truth! My ultimate, happy conclusion would be basically exactly the flags you showed above, with only one caveat: There is a guaranteed link between module names and libraries. That's it.

I don't know if its complicated or not. It sounds an awful lot like something that the tool can already do. Thus I'm confused. And as for the link between module names and library names - who cares? Obviously you do but I have never felt a desire to force a link between the two. It has never given me any grief so I don't understand your huge concern.


I spend my life compiling software - I do it for work, I do it at home, some of my primary open source projects are geared directly at the task of compiling software. The fact is, things are in general a mess. And the reason? With no standard to easily conform to, people do things however they want, so then compiling something involves reeling in all the possible garbage people may have felt inclined to do.

The solution? Well, it's been used successfully by Java (and probably by .NET but I'm not really familiar with it). By establishing and yes, even imposing, a standard before the language proliferates to a point-of-no-return, processes become easier for the future generations who then have to deal with and use the code written by others. It will never be perfect, it's impossible to solve every problem, but if keeping in line is easy and non-detrimental, people tend to do so. If keeping things in line also makes processes easier, why not?

Derek Parnell wrote:
Gregor wrote:

In an attempt to figure out your contention, I have the following questions:

1) Do you agree that D packages are fundamentally similar to libraries?

No. I've never considered this to be so. In my mind, a package is a collection of modules, and a library is a collection of packages.


But a package is (or can be) also a collection of packages.

Derek Parnell wrote:
I cannot see why a library cannot contain modules from various packages, or even partial packages for that matter.


Because then, the process of tracking down exactly which library corresponds to the imports you're using gets exponentially more complex. Is it in libmango, or libmango_io_primitives? Is the bar package in libfoo, just because it's distributed with libfoo? What if the maintainer arbitrarily feels like changing that one day? What if there's a secondary implementation (a fork or similar)? Who's correct?

Furthermore, libraries containing several packages are not advantageous in any way to just having several libraries (IMHO).

Derek Parnell wrote:
Gregor wrote:

3) Do you agree that such a guarantee could make the build process more automatic?

If we assume that a library is a package, then yes. But at the cost of imposing an arbitrary restriction on the coders.


There are thousands of "arbitrary" restrictions imposed onto coders. Why is it that Windows coders always feel obliged to make things output PE .exe files, instead of writing an ELF loader and writing ELF binaries with full .so support? Oh yes, because it's easier and not particularly disadvantageous. Arbitrary restrictions are a good thing if they make life easier on everyone.

EDIT: And this is hardly arbitrary.

Derek Parnell wrote:
Gregor wrote:

4) Do you believe that my proposed changes would be very complex?

If we assume that a library is a package, then maybe. There would be a small performance hit during the build process as it scanned for libraries.


The performance hit of looking at files is basically negligible, and at compile time it's definitely unimportant.

In my opinion, the changes would be fairly noncomplex.

Derek Parnell wrote:
Gregor wrote:

6) Do you believe that the flexibility of the current methodology and the methodology you've been showing in the last posts trumps the reliable guarantee provided by my proposed solution?

Yep.


This is, in my opinion, where we'll never agree. In my opinion, the reliable guarantee provided by a standard library naming convention is by-far more important than flexibility in creating libraries.

Derek Parnell wrote:
Gregor wrote:

7) Do you believe that my additions would break all current methodologies, or that that is implicitly a bad thing with a still-pre-1.0 language?

Not all, but certainly most. Imposing a naming convention on programmers is like trying to herd cats.


And how do you herd cats? With a fence. If you have thirty cats in a chain-link fence, and only three of them squeeze through or around it, that's a lot better than having thirty cats in an open field and trying to catch them all.

Furthermore, I have no intention of making current methodologies impossible, of course coders should be able to name things whatever they damn-well please. Conventions are not hard-and-fast rules, they're defaults. Everything would have to be overrideable to be sane and flexible. My only proposed change is to the default behavior.

The fact is that, when a default is presented which solves the problem and works, there's no compelling reason to do otherwise.

Derek Parnell wrote:
Gregor wrote:

9) What do you believe is my core argument?

That by imposing the naming convention, magic will happen.


Not magic, just ease-of-use for D programmers in general. The entire point of imposing a convention on libraries is to unify these potentially-different namespaces (imports and libraries) into one, so you only have to go digging about for one piece of information to use 3rd-party code.

Derek Parnell wrote:
Gregor wrote:

FYI, my core argument is this: D packages and libraries are fundamentally similar, and this similarity can be exploited to make the entire building process easier and more reliable. The current methodology requires that these namespaces are maintained separately (if you use libraries), making the entire process more complicated, which is part of why everybody just -I's source files rather than using libraries.


I think we are thinking very differently. I maintain that Build can already achieve what you want to ultimately do but not via the method you wish to do it.


Yes, it can be made to achieve similar functionality. But if there's no convention for naming as at least a default, who would ever use it? This is where your herding-cats analogy comes in. If there's nothing actively encouraging programmers to conform to a logical standard, they inevitably will not.

Derek Parnell wrote:
Namely, it is already possible to Build an application that uses libraries by just running
Code:

build myapp


In your thinking the application source code would contain ...
Code:

import some.pack.age.mod;

and in my thinking it would contain...
Code:

import some.pack.age.mod;
version (build) pragma(link, "thelibrary");


Your code forces the library to have a specific name and mine gives the freedom to not follow any specific convention.


Again, our primary difference of opinions. I believe that this freedom is not only not valuable, but harmful.

Derek Parnell wrote:
However, I can offer this...

I could add a new pragma(autolink, "on"); and then use all the following imports until end-of-file or a matching pragma(autolink, "off"); to scan for libraries, and if found add them to the linker step. Thus given import some.pack.age.mod; I would look for ...

    libD.some.*.(a|so)
    libD.some.pack.*.(a|so)
    libD.some.pack.age.(a|so)

and upon finding one or more of these file specs, add found ones to the linker command line.

A new switch for Build to use, or avoid using, the new pragma might be useful too.


Anything non-default will never be used, so it defeats the entire purpose of trying to create a convention. While this is (half of) precisely what I'm looking for, it would never work because:

1) Without any reason or easy way to, nobody would use the convention during creation of libraries.
2) If this is not the default, nobody will ever use it - this sort of convention is virtually all-or-nothing. A few people not conforming to it won't hurt the community as a whole, but if only a few people are conforming to it it may as well not be there.


EDIT: Well, we've gone into a timewarp again. For those trying to read this, imagine that this post is after the next one.
Back to top
View user's profile Send private message AIM Address
Derek Parnell



Joined: 22 Apr 2004
Posts: 408
Location: Melbourne, Australia

PostPosted: Fri Sep 22, 2006 9:53 am    Post subject: Reply with quote

Gregor wrote:
It's clear that this is quickly regressing towards flamewar status.

I'm sorry you feel that way. I'm not trying to argue any point nor score points. I'm trying to understand what it is that you want but my background knowledge is not up to the task, thus I ask seemingly stupid questions and say some dumb things.

Gregor wrote:
I don't understand what your core contention with my concept is.

I don't understand it. It seems pointless. I know it can't be so, thus I suspect I really do not grasp what it is you are asking for.

Gregor wrote:
If I can simply figure that out, then I can properly argue, but as-is, I'm flustering to make a proper argument because I have no idea what you want me to prove.

I'm not asking nor requiring you to prove anything. I'm trying to find out what you are asking for. I simply just don't get it.

It seems that you are saying that every package must also be a shared library, and that is just wrong in my mind. Why must that be so?

Gregor wrote:

From your last post, it seems like you think using build with my LNC would be complicated. This couldn't be further from the truth! My ultimate, happy conclusion would be basically exactly the flags you showed above, with only one caveat: There is a guaranteed link between module names and libraries. That's it.

I don't know if its complicated or not. It sounds an awful lot like something that the tool can already do. Thus I'm confused. And as for the link between module names and library names - who cares? Obviously you do but I have never felt a desire to force a link between the two. It has never given me any grief so I don't understand your huge concern.

Gregor wrote:

In an attempt to figure out your contention, I have the following questions:

1) Do you agree that D packages are fundamentally similar to libraries?

No. I've never considered this to be so. In my mind, a package is a collection of modules, and a library is a collection of packages.

I cannot see why a library cannot contain modules from various packages, or even partial packages for that matter.

Gregor wrote:

2) Do you agree that this similarity could be exploited by guaranteeing a connection between the two?

If we assume that a library is a package, then yes.
Gregor wrote:

3) Do you agree that such a guarantee could make the build process more automatic?

If we assume that a library is a package, then yes. But at the cost of imposing an arbitrary restriction on the coders.
Gregor wrote:

3) Do you agree that such a guarantee would make the build process easier?

If we assume that a library is a package, then yes. But more restrictive.
Gregor wrote:


4) Do you believe that my proposed changes would be very complex?

If we assume that a library is a package, then maybe. There would be a small performance hit during the build process as it scanned ofr libraries.
Gregor wrote:


5) Do you believe that the current methodology, requiring modifying .d files or .brf libraries to use libraries rather than against-source builds, is superior?

Apparently Wink
Gregor wrote:


6) Do you believe that the flexibility of the current methodology and the methodology you've been showing in the last posts trumps the reliable guarantee provided by my proposed solution?

Yep.
Gregor wrote:

7) Do you believe that my additions would break all current methodologies, or that that is implicitly a bad thing with a still-pre-1.0 language?

Not all, but certainly most. Imposing a naming convention on programmers is like trying to herd cats.
Gregor wrote:

8 ) Is it your opinion that requiring a change to source to burn-in library names et al is more flexible than being able to simply connect imports to libraries?

No. Because in the end they amount to the same thing - source code entries.
Gregor wrote:


9) What do you believe is my core argument?

That by imposing the naming convention, magic will happen.
Gregor wrote:


FYI, my core argument is this: D packages and libraries are fundamentally similar, and this similarity can be exploited to make the entire building process easier and more reliable. The current methodology requires that these namespaces are maintained separately (if you use libraries), making the entire process more complicated, which is part of why everybody just -I's source files rather than using libraries.


I think we are thinking very differently. I maintain that Build can already achieve what you want to ultimately do but not via the method you wish to do it.

Namely, it is already possible to Build an application that uses libraries by just running
Code:

build myapp


In your thinking the application source code would contain ...
Code:

import some.pack.age.mod;

and in my thinking it would contain...
Code:

import some.pack.age.mod;
version (build) pragma(link, "thelibrary");


Your code forces the library to have a specific name and mine gives the freedom to not follow any specific convention.

However, I can offer this...

I could add a new pragma(autolink, "on"); and then use all the following imports until end-of-file or a matching pragma(autolink, "off"); to scan for libraries, and if found add them to the linker step. Thus given import some.pack.age.mod; I would look for ...

    libD.some.*.(a|so)
    libD.some.pack.*.(a|so)
    libD.some.pack.age.(a|so)

and upon finding one or more of these file specs, add found ones to the linker command line.

A new switch for Build to use, or avoid using, the new pragma might be useful too.
_________________
--
Derek
skype name: derek.j.parnell
Back to top
View user's profile Send private message
Gregor



Joined: 05 May 2006
Posts: 72
Location: Portland, OR

PostPosted: Fri Sep 22, 2006 10:07 am    Post subject: Reply with quote

Derek Parnell wrote:
I could add a new pragma(autolink, "on"); and then use all the following imports until end-of-file or a matching pragma(autolink, "off"); to scan for libraries ...


Just in case it wasn't clear in the first place, I want to make it perfectly clear that I'm willing to write all of the code to support this, unless you'd prefer to. Considering that I'm the one trying so hard to push this convention to the unwilling masses, implementing it is the least I could do Rolling Eyes
Back to top
View user's profile Send private message AIM Address
Gregor



Joined: 05 May 2006
Posts: 72
Location: Portland, OR

PostPosted: Fri Sep 22, 2006 4:18 pm    Post subject: Reply with quote

A friend of mine suggested posting a clear, real-world example to make my purpose clear.

I am, as we've well established, only versed in UNIX nomenclature, so I'll preface this with some notes for translation:

1) The include path (that is, where header files for C/C++ or D import files go) is generally /usr/include, or probably something special for DMD. With GDC it's detectable through a simple call and could of course be customized in the .cfg file. GDC can also use /usr/include/d, which is nice for keeping C and C++ separate from D. Of course, this could all be set up by build too.
2) The lib path (that is, where the compiler expects to find library files) is generally /usr/lib. DMD, to my recollection, has its own lib path on Windows and uses the standard lib path on GNU/Linux, so /usr/lib should work fine there too.
3) I'm still incredibly confused about how .DLL files work, but FYI, shared object files on UNIX are totally interchangeable with static libraries. No change to code is required to use one vs the other, or even to the compile line. Most compilers will prefer the shared one unless you tell them to do otherwise. The real difference is that, if you use a shared library, at runtime (before your code has been started, in fact), all of the shared libraries you specify will be loaded. I'm not going into the .so versioning, as it's a bit beyond the scope.

In this example, I'm going to use mango from the program foobar (I wish I could think up a program that uses mango from the top of my head, but I can't).

Code:

~$ cd mango
~/mango$ # I am now in the mango directory, ready to compile its source into libraries
~/mango$ build -libs
# build now compiles all of the packages into libraries, perhaps consulting some file specifying how the author wished for the libraries to be divided.  In this example, we'll imagine that the author specified that there should be one lib for most of mango, but that mango.io should be separated.  This step also creates the .di files which will be necessary for using these libraries.
~/mango$ ls
...
libD.mango.a
libD.mango.so
libD.mango.io.a
libD.mango.io.so
...
~/mango$ build -lib-install [-prefix=<where you want to install the libs if you don't want the default>]
# This would install the libraries you just created into the lib path, and install the .di files into the include path.  Realistically, this would require root privileges, but that's beyond the point of this example.
~/mango$ ls /usr/lib/libD.*
/usr/lib/libD.mango.a
/usr/lib/libD.mango.so
/usr/lib/libD.mango.io.a
/usr/lib/libD.mango.io.so
~/mango$ ls /usr/include/d/mango
/usr/include/d/mango/cache
/usr/include/d/mango/cipher
...
~/mango$ # note that none of this process required that the source specify that mango can only be used as a library.  Since the source doesn't specify any imports being required to use this, it could be -I'd or copied in with no source changes, if that's what somebody preferred to do
~/mango$ cd ~
~$ cd foobar
~/foobar$ # I am now in the directory for foobar, my example program
~/foobar$ build foobar
# foobar has an 'import mango.sys.OS' in it, so build goes in search of a matching library.  Going from most specific to most general, it first searches for libD.mango.sys.{a,so}, then for libD.mango.{a,so}.  It finds libD.mango.so, and links that in.  If it didn't find one, it would of course tell the user.
# because the name is standardized, I can copy foobar over to any system that has mango installed, and have no fear that it will associate itself with the proper library.
~/foobar$ build foobarextreme
# foobarextreme is a version which includes io, and has 'import mango.io.Stdout',  In a similar fashion, build looks through and finds libD.mango.io.so.  Because mango.io depends on other parts of mango, some import would eventually make it find libD.mango.so
~/foobar$ # now, let's say I decided I wanted to include mango with foobar.  Why?  Who knows, but I want to so I will.  It's as easy as copying it in:
~/foobar$ cp -a ../mango/mango .
~/foobar$ # and compiling again
~/foobar$ build -clean
~/foobar$ build foobar
# now, since it found the source, it'll use that instead.  NO change was needed to the source to tell it it's no longer being used as a library.


I don't know whether that explained anything at all, but I hope it may have helped a bit.

A final note: I feel like I cannot possibly stress this enough:

What's great about build is it consolidates the D language concept of imports and the more basic concept of including files in a build into one concept. I feel that using a standard naming in libraries brings them into that same space: the humble 'import' can then represent anything it could theoretically require, without touching the code. Because of this, to me this idea seems like a simple extension of build's current vision:

Quote:

[build's] main purpose is to reduce the need for developers to keep track of which files are needed to be compiled and/or linked to form either an execuable file or library file.


By suggesting that developers should add the pragma(lib) construct, you're requiring them to keep track of one more file that needs to be linked - it seems to me like that violates the very principal that I like about build in the first place.
Back to top
View user's profile Send private message AIM Address
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Build All times are GMT - 6 Hours
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group