Ticket #1724 (reopened defect)

Opened 15 years ago

Last modified 14 years ago

Zlibstream in_chunk allocation memory problem

Reported by:	lars_kirchhoff	Assigned to:	DRK
Priority:	minor	Milestone:	1.0
Component:	Tango	Version:	0.99.8 Sean
Keywords:		Cc:

Description

Zlibstream consumes lots of memory, when used for decompressing a high number of files. One reason seems the allocation of the in_chunk within the constructor with

ubyte[] in_chunk = new ubyte[CHUNKSIZE];

I suggest using

ubyte[CHUNKSIZE] in_chunk

right away.

Attachments

zlibstream.txt (0.7 kB) - added by lars_kirchhoff on 08/14/09 13:41:24.
zlibstream_reset.txt (1.9 kB) - added by lars_kirchhoff on 09/10/09 17:35:42.

Change History

08/14/09 13:41:24 changed by lars_kirchhoff

attachment zlibstream.txt added.

(in reply to: ↑ description ) 08/14/09 16:50:32 changed by DRK

status changed from new to assigned.
owner changed from kris to DRK.

That's what the old behaviour was before it was changed; the old style was crashing GDC.

Besides which, this wouldn't reduce pressure on the GC: it STILL needs to allocate that memory, except with that change it would do so as part of the class instead of separately.

A better solution would be to allow ZlibStream? objects to be reused. Perhaps adding

void reset(InputStream);
void reset(InputStream, Encoding, int windowBits);

which then re-initialises the object would be best.

08/16/09 13:55:54 changed by DRK

status changed from assigned to closed.
resolution set to fixed.

(In [4858]) Added reset interface to the Zlib & Bzip2 compression streams so the objects can be re-used, closes #1724. Also changed ZlibInput?'s and BzipInput?'s flush methods to be overrides of close instead, since overriding flush made absolutely no sense at all.

09/09/09 17:21:52 changed by tomni

status changed from closed to reopened.
resolution deleted.

Sorry but I've to disagree on that solution. While using the latest DMD 1.047 and tango trunk the new patch produces a segmentation fault after reusing decode. The problem is somehow related to this line:

line 280: auto len = source.read(in_chunk);

Somehow in_chunk can't be correctly re-used a second time. I'd really like to post the gdb output but with the latest trunk it just produces

Dwarf Error: Cannot find DIE at 0x349f9 referenced from DIE at 0x31b87 
[in module /root/workspace/eclipse/HelloWorld/bin/main]

Anyway, changing the CHUNKSIZE in line 43 to 1024 resolves the issue somehow. I'm stuck.

09/09/09 21:45:23 changed by larsivi

DMD 1.047 appears to have several issues, so I'm not sure this is proven to be a Tango issue. We'd rather drop support for a broken version of DMD than putting in workarounds.

09/10/09 17:34:55 changed by lars_kirchhoff

The problem with the new reset is, that it does not set the stream object correctly. The stream object was passed to the init function, but never used. I removed it from the stream as a parameter from the init. Additionally I set the new stream to either "source" or "sink".

I've added a new patch above for the current trunk.

09/10/09 17:35:42 changed by lars_kirchhoff

attachment zlibstream_reset.txt added.

09/10/09 17:48:45 changed by lars_kirchhoff

and actually I'm using DMD 1.043 :-) so it's not an issue related to DMD 1.047

02/07/10 21:37:28 changed by larsivi

milestone changed from 0.99.9 to 1.0.

Ticket Navigation