dmds new release: runtime interface + 64 bit

Well the first thing is avoiding passing -m32 to the C compiler in bob. I used a script in the meantime. I just began making the first changes, but it is still quite a bit to do and to check. One should be sure not to miss changes to phobos/dmd that should be transferred to tango, some changes began quite some time ago, and were not done to tango because nobody tracks dmd releases yet. I attach my actual changes, if someone wants to use them as base to build a complete patch.

If you remove -m32 from the C compiler what happens then when someone with a 64bit G compiler tries to build Tango with a 32bit DMD. It won't work. Maybe there should be a switch to build 32bit Tango on a 64bit system (would that be cross compiling?). BTW what's the reason for "alias size_t wordtype;"?

the binary bob version doesn't seem to pass -m32 to the compiler. I just used -o"-m64"

I am not that confident in my patch, someone might check over it

more work on 64bit dmd

Replying to Marenz:

I am not that confident in my patch, someone might check over it

I think you should use "ref" instead of "inout".

Uh. I guess the copied code had inout, because I never use it. Will check over it on Monday.

Main problem right now is the fact, that the druntime code uses object.talign which tango doesn't have. Should we implement it or can the druntime code be rewritten to not use it?

replaced inout with ref

the talign changes are not compiler specific they can be easily ported one should "correct" the typeinfo of the various compilers, and their object.d basically typeinfo has a field more with the aligment info

at compile time the aligment is known, so one can easily put size_t.sizeof as default and then in specifice typeinfo override it with the alignof of that type I haven't looked at the details but it shoudl be rather straightforward to do for all compilers, so that one has a single object.di without version(xxx) in it

added talign to TypeInfo? and in the compiler dependent files

synced with newest rev

all but a few things patched

I fixed all but two errors. Anyone an idea how to fix those?

dmd -c -I./tango/core -I. -I./tango/core/vendor -m64 -version=WithDateTime? -oftango-core-rt-compiler-dmd-rt-lifetime-m64--version-WithDateTime?.o ./tango/core/rt/compiler/dmd/rt/lifetime.d ./tango/core/rt/compiler/dmd/rt/lifetime.d(339): Error: e2ir: cannot cast foo(ti,pdim,ndims) of type void[] to type ulong ./tango/core/rt/compiler/dmd/rt/lifetime.d(398): Error: e2ir: cannot cast foo(ti,pdim,ndims) of type void[] to type ulong

also need fix the _d_arrayliteralTX issue .

I build bob catch this:

/home/opt/dmd1/bin/../lib/libtango.a(tango-sys-Process-release.o): In function `_D5tango3sys7Process7Process4argsMFAaAAaXAAa':
../tango/sys/Process.d:(.text._D5tango3sys7Process7Process4argsMFAaAAaXAAa+0x1b): undefined reference to `_d_arrayliteralTX'
/home/opt/dmd1/bin/../lib/libtango.a(tango-sys-Process-release.o): In function `_D5tango3sys7Process7Process7executeMFAaAAaXv':
../tango/sys/Process.d:(.text._D5tango3sys7Process7Process7executeMFAaAAaXv+0x1b): undefined reference to `_d_arrayliteralTX'
/home/opt/dmd1/bin/../lib/libtango.a(tango-time-chrono-Gregorian-release.o): In function `_D5tango4time6chrono9Gregorian9Gregorian4erasMFZAk':
../tango/time/chrono/Gregorian.d:(.text._D5tango4time6chrono9Gregorian9Gregorian4erasMFZAk+0x10): undefined reference to `_d_arrayliteralTX'
/home/opt/dmd1/bin/../lib/libtango.a(tango-time-chrono-Hebrew-release.o): In function `_D5tango4time6chrono6Hebrew6Hebrew4erasMFZAk':
../tango/time/chrono/Hebrew.d:(.text._D5tango4time6chrono6Hebrew6Hebrew4erasMFZAk+0x12): undefined reference to `_d_arrayliteralTX'
/home/opt/dmd1/bin/../lib/libtango.a(tango-time-chrono-Hijri-release.o): In function `_D5tango4time6chrono5Hijri5Hijri4erasMFZAk':
../tango/time/chrono/Hijri.d:(.text._D5tango4time6chrono5Hijri5Hijri4erasMFZAk+0x12): undefined reference to `_d_arrayliteralTX'
collect2: ld returned 1 exit status
--- errorlevel 1

Fixed compilation errors in deh.d

newest patch contains some hacks for unittest and bob to build 64 bit. I will do that properly once everything works.

@sleets: it's way more than just _d_arrayliteralTX. In detail, what is missing is:

_aaDelX
_aaGetRvalueX
_aaGetX
_aaInX
_d_arrayappendcTX
_d_arrayliteralTX
_d_assocarrayliteralTX
_memsetDouble
_memsetFloat

I'll see what they did in phobos for that and copy it over.

removed whitespace diffs, implemented the _aa* functions (not tested yet)

successfully compiles and links now, unittest does no longer segfault, but simply outputs nothing and exits with code 1. A simple test program does segfault.

forgot a change

further changes. can't recall what.

and more changes.

Normal Cout("test"); works, throwing and catching an exception works also.

The big unittesting of tangos fails though. I don't know why, because it just outputs nothing but also doesn't segfault.

What does segfault is using Stdout. I try now to locate problematic modules and investigate why they fail.

some debugging output, and forgot a complation error

The following is a small example program that causes a crash. I think it happens while calling rt_finalize

module main;

import tango.io.Console;

void main ( )
{
    char[] test = new char[2000000];
}

going to check the GC changes

arraycatnT had some more version(X86) else which I missed because he didn't use X86_64 but just "else". Took me 2 days to find this one. and I fixed a strange bug in the GC. I don't understand how 64bit worked with other compilers if my if is really correct. check it out.

Okay, current problem is something with vararg. I made a simple test program that fails:

module main;

//import tango.io.Console;
//import tango.io.Stdout;

import tango.stdc.stdio : printf;
import tango.core.Vararg;

void vararg ( ... )
{
    printf(" ############# vararg: %llx\n", _argptr);
    printf(" ############# *vararg: %llx\n", *cast(int*)_argptr);
    
}

void normalarg( int t)
{
    printf(" ############# normalarg: %llx\n", &t);
    printf(" ############# *normalarg: %llx\n", t);

}

void main ( )
{
    int test = 524;
    printf(" ############# test: %llx\n", &test);
    printf(" ############# *test: %llx\n", test);

    vararg(test);
    normalarg(test);
}

Output is

 ############# test: 7fff0615f118
 ############# *test: 20c
 ############# vararg: 7fff0615f0d0
 ############# *vararg: 8
 ############# normalarg: 7fff0615f0f8
 ############# *normalarg: 20c

added argTypes where it was missing, only for dmd so far

The d_arrayliteralTX is still missing in win32 .

(Posting again as first time went wrong).

I'm just going to note that these runtime functions have changed in GDC:

In gdc/rt/aaA.d
Old:
BB* _d_assocarrayliteralTp(TypeInfo_AssociativeArray ti, size_t length, void* keys, void* values)
New:
BB* _d_assocarrayliteralTp(TypeInfo_AssociativeArray ti, void[] keys, void[] values)

In gdc/rt/lifetime.d
Old:
byte[] _d_arrayappendcTp(TypeInfo ti, ref byte[] x, void *argp)
New:
byte[] _d_arrayappendcTp(TypeInfo ti, inout byte[] x, size_t n)


And newly added runtime function:
void* _d_arrayliteralTp(TypeInfo ti, size_t length)

Implementations same as DMD.

Regards

Replying to ibuclaw:

(Posting again as first time went wrong). I'm just going to note that these runtime functions have changed in GDC: {{{ In gdc/rt/aaA.d Old: BB* _d_assocarrayliteralTp(TypeInfo?_AssociativeArray ti, size_t length, void* keys, void* values) New: BB* _d_assocarrayliteralTp(TypeInfo?_AssociativeArray ti, void[] keys, void[] values) In gdc/rt/lifetime.d Old: byte[] _d_arrayappendcTp(TypeInfo? ti, ref byte[] x, void *argp) New: byte[] _d_arrayappendcTp(TypeInfo? ti, inout byte[] x, size_t n) And newly added runtime function: void* _d_arrayliteralTp(TypeInfo? ti, size_t length) }}} Implementations same as DMD. Regards

Are you sure that You got all? What about all the *X functions that were added? Maybe they are only for the 64 bit version...

found some more missing version()s. Fixed tango layout class. unittests run up to a certain point.

The unittest runs now till it segfaults.. and there is a problem when displaying the stacktrace for each failed unittest..

http://paste.dprogramming.com/dpf2y4pa

I will first fix all the segfaults, then all the asserts. If some one can fix the stacktrace problem, I would appreciate it.

fixed all segfaults that happened when running the unittests

Current status: I think I got most things working. The failing unittests are still to fix (and sometimes I neither see a stacktrace nor the assert that triggered, so it's not always easy to fix).

One big mess is basically everything that uses varargs. The 64bit api is different from the 32bit api. You can't use _argptr directly anymore but are forced to use the va_* functions. This is particular ugly for all the layoutfunctions because you have to init va_start with a special variable _va_argsave which is only avaliable in the function that had the "..." (But only in 64 bit!). So you have to va_start and va_end in that function. As layout converts everything to a void*[], you have to have several versions for various compilers and architectures.

I have added many functions to the other compiler-runtimes, too, but I couldn't test them, so if someone with ldc/gdb/dmd and windows/linux (64/32bit) feels inclined to contribute: TEST STUFF!

I know that some are still missing for the other compilers than dmd and will take care of those.

Fawzi suggests to use variadic templates instead of varargs here http://dsource.org/projects/tango/ticket/1042 . The call would be the same, the usage a little different. You would define/mixin a template in front of your variable function and your function could simply get an array of Variant[] or another fixed/defined way. That way you could also just forward the Variant-Varargs to other Variant-vararg functions

Oh yes, variants toVarargs function is also broken on x86_64 now. And I don't even know how to fix it, if it is possible at all.

I start liking fawzis suggestion more. Each time I work with varargs, a little more.

I'd love to get some more input from you guys.

Some examples of how the template usage could look like:

void func(...) 
{
    mixin Vararg;   
 /* now available:  
varArg.arguments  -> void*[] and   
vararg.types -> TypeInfo[] 
*/ 
}
// or the other suggestion:

 mixin Vararg!(&original_function);   
void original_function(uint something, VarArgs args, VarTypes types) {.. }

The first one is less easy to implement, but looks more elegant to the user and it still uses varargs, just transforms them to something that can be used the same way everywhere. Any compiler dependent stuff would be hidden.

The second one would just remove varargs and replaces them with a template that gets mixed in.

The failing testcase of Layout.d is actually dmd/phobos fault, see http://d.puremagic.com/issues/show_bug.cgi?id=5694

The still failing testcase of Layout.d (after turning the previously triggering asserts off) is also dmd/phobos fault: http://d.puremagic.com/issues/show_bug.cgi?id=5698

Fixed unittest for Layout.d

The unittest for layout now runs successfully (after turning of the asserts concerning the before-mentioned tickets).

The backtrace is always one line because the c-backtrace function that is being used only manages to extract one frame. My guess is that dmd fucks something up here and I bet it's the same as it is for the gdb errors. There is already a ticket for that, but I've seen no progress there so far

The first bug in dmd & phobos is fixed, which means that I reenabled the according asserts in Layout.d. See new patch. Glad this is going forward.

assimilated patch to phobos, reenabled asserts/unittests for layout.d

Fixed Process unittest .. but no idea how.

Fixed the failing unittest for sys.HomeFolder? and an internal one of the runtime. The last three failing tests are actually just two because one is dependent on the other. The first is in math.Probability and I have no idea about that stuff, so someone who does might want to check that. The the other one is in regex, I didn't look to closely at that one yet.

I have looked into the unittest failure in IEEE.d. Here is a small test case:

import tango.math.IEEE;
void main() {
assert(feqrel(real.min/8,real.min/17)==3);
}

The assert is hit using "dmd -m64 test.d && ./test", but vanishes when used with -O and/or -release. -m32 makes it works as well. Phobos for D2 has the same unittest and feqrel implementation. I have tested dmd.2.052 and it has the same problem with -m64 vs -m32.

There might be a dmd ticket already. So far it's a dmd issue. A hotfix would be to add a dummy function call before "if (pd[F.EXPPOS_SHORT] == 0)".

http://d.puremagic.com/issues/show_bug.cgi?id=5809

I couldn't reproduce the failure in tango.math.Probability, but it may be the same bug. No idea about the regex issue.

Funny enough, ldc has the exact same 3 unittest failures. To me, it looks like the patch is ready to commit, as the remaining bugs are likely dmd (frontend) bugs.

@mwarning: I am pretty sure it is the same bug in Probability. Thanks for having a look at that :)

took care of the last remaining things. should be ready to commit now

ah no, I forgot to add the interface changes for the other compilers. Gotta do that this evening

gdc is more work than I thought. My vararg changes for the new dmd interface broke gdc at all those parts. I created a new branch dmd-1.067 (okay, unlucky name) and applied all my changes there.

as just mentioned, gdc breaks at several parts. Compile it with

PATH=/home/mathias/gdc-bin/usr/local/bin/:$PATH build/bin/linux32/bob -r=gdc -l=libtango-gdc -c=gdc -o="-fversion=WithDateTime? -m64" .

and see yourself (path being where your gdc is located at). I couldn't test ldc so far as I lack the proper llvm version. But I'll get to that.

unittest regressions in branches/dmd-1.067 r5651:
DC=ldc PLATFORM=linux ARCH=64 build/script/unittest.sh

• tango.core.Variant
• tango.text.convert.Layout

Is there any reason why the following code shouldn't build? It breaks functionality in mango which compiled and worked with tango in trunk. I'm new to D so I'm not really familiar with inout yet.

import tango.net.http.HttpConst;

class HttpTest {
	this() {
		
	}
	
	void sendError (inout HttpStatus status) {
	}
}

void main(char[][] args) {
	auto test = new HttpTest();
	test.sendError(HttpResponses.NotFound);
}

src/main.d(14): Error: cannot modify final variable 'NotFound?'

There's another error in my application I can't explain. If the source as above gets modified like this...

import tango.net.http.HttpConst;

private import	tango.util.container.HashMap;

class HttpTest {
	private static const HashMap!(char[], void*) handlers;

	static this() {
		handlers = new HashMap!(char[], void*)();
	}
	
	void makeError () {
		sendError(HttpResponses.NotFound);
	}
	
	void sendError (inout HttpStatus status) {
	}
}

void main(char[][] args) {
	auto test = new HttpTest();
}

I suddenly get 2 compilation errors:

src/main.d(13): Error: cannot modify final variable 'NotFound'
/opt/dmd/bin/../import/tango/util/container/HashMap.d(155): Error: function tango.util.container.HashMap.HashMap!(char[],void*).HashMap.Iterator.opApply (int delegate(ref char[] key, ref void* value)) does not match parameter types (_error_ delegate(ref char[] k, ref void* v))
/opt/dmd/bin/../import/tango/util/container/HashMap.d(155): Error: cannot implicitly convert expression (__dgliteral1) of type _error_ delegate(ref char[] k, ref void* v) to int delegate(ref char[] key, ref void* value)

Now it also complains about the HashMap?. If I comment out the sendError() call inside makeError() _both_ errors are gone. How could they be related?

I would guess the compiler has become more strict. I would guess it will work if you remove "inout". "inout" is the old keyword for a reference parameter ("ref" is the one you want to use), it behaves just like a reference parameter in C++.

If you declare a parameter as "ref" then it will be possible to change the parameter and it will be visible for the caller. Example:

void foo (ref int i)
{
  i = 4;
}

int a = 3;

foo(a);
assert(a == 4);

The two fields in HttpStatus are declared as final, I don't know if that is still valid.

They are final in HttpStatus?, that's correct. Before I switched from trunk to the 1.067 tango branch that piece of code (which in this form is used in mango) compiled and worked.

To my understanding inout can be used when you have a method which takes a parameter in various mutations(immutable, const, ...) and returns a value of the same kind.

Instead of having to write those two (in C++):

const char* someStringMethod(const char* text); char* someStringMethod(char* text);

you can write in D:

inout(char)* (inout(char)* text);

Maybe that's just a basic use case though... I will have to modify the mango code then. It's not really compatible with tango at the moment.

Thanks, Rainer

Replying to ArKay:

To my understanding inout can be used when you have a method which takes a parameter in various mutations(immutable, const, ...) and returns a value of the same kind.

Yes, that's how it works in D2.

Another DMD update, another incompatibility? Just switched from Linux to Windows so I installed the shiny 1.068 which came out 2 days ago.

C:\D\dmd\windows\bin\..\import\tango\core\sync\Atomic.d(480): opcode expected, not cmpxch8b

Probably related to this little change?

- Fix spelling of cmpxchgb8

Of course he also misspelled it in the Changelog, it should be "cmpxchg8b" :)

Both Atomic.ds have to be fixed.

I've moved that spelling bug into another ticket, see #2052.

The dmd-1.067 branch was merged in [5652]. I think we can close this one now.