Download Reference Manual
The Developer's Library for D
About Wiki Forums Source Search Contact

Table of Contents

  1. Conduits
  2. Buffers

Conduits and Buffers

The Conduit and Buffer form the basis for tango IO.

Conduits

Tango IO is stream oriented: each data end-point is described by a Tango construct called a conduit, which plays host to both an InputStream and an OutputStream. For example, there are conduits for communicating with files, sockets, the console, and so on. Some conduits require explicit connections to the end-point, whereas others connect implicitly – socket conduits are of the explicit variety, while file and console conduits are usually implicit. There are conduit extensions to enable asynchronous read and write operations, supported via various OS-specific facilities (see tango.io.selector).

Conduits are bi-directional, supporting contiguous reads and writes. Reading is facilitated via an associated InputStream, and writing via an OutputStream, accessed via the conduit input() and output() methods respectively. In general, an application interacts with conduit streams via read/write methods, each accepting a void array and returning an integer representing the quantity populated or consumed respectively. Note that the return value may, in some cases, be less than the array-length provided. In particular, the reserved constant IConduit.Eof will be returned for end-of-flow conditions:

InputStream input ();
OutputStream output ();

InputStream

void clear ();
IConduit conduit ();
uint read (void[] dst);

OutputStream

void flush ();
IConduit conduit ();
uint write (void[] src);
OutputStream copy (InputStream src);

OutputStream methods are provided to fully flush an array and to copy the content of another conduit. These two methods may stall whilst completing, depending upon the conduit end-point.

Conduits support the notion of stream-filters: one or more filters may be attached to the conduit and each may intercept and mutate data as it flows through them. Filters are connected in reverse order via these conduit methods, which return the previous filter attached:

InputStream attach (InputStream filter);
OuputStream attach (OutputStream filter);

Conduit filters are expected to be chained together where, for example, a read() in one filter invokes the read() of its ancestor. However, filters may redirect flow as they see fit. The filter base-classes are implemented in the tango.io.Conduit module. Note that these filters are effectively contained within the conduit itself, rather than the filter being exposed as the primary gateway for IO activity. This allows for conduit-specifics to remain exposed (such as socket attributes, or file seeks) rather than being hidden behind a less rich facade.

Once a conduit is no longer in use it should be explicitly closed by the application, in order to perform a conduit-specific set of cleanup actions and to release any attached filters.

Conduit Exceptions

As a rule, conduits do not throw exceptions because some error conditions are often considered valid. For example, IConduit.Eof is generally returned to the application intact. However, exceptions will be raised where the underlying OS detects some type of fault.

Buffers

Buffering a conduit applies the equivalent of a "sliding window" to the content therein, and simplifies many common operations such as token-parsing or output concatenation. Buffer can also act as a throughput enhancer for certain IO applications. For example: buffering enables an underlying conduit to read and write large chunks of data instead of discrete (small) data elements, it provides a temporary space for certain algorithms to mutate streaming content efficiently, and can support efficient mapping of data-record content into user-memory. Buffers are full duplex, and maintain distinct read and write locations – hence it is possible to be writing near the end of a buffer whilst reading near the beginning.

Buffer provides the necessary function to handle many types of buffering activity, from simple append operations to the more prosaic producer/consumer balancing between conduits of differing bandwidths. Buffer also supports multiple clients. For example, more than one reader and/or writer may be attached and the buffer will maintain a common state across them. This can be handy where, for example, an application must deal with a mixed or interleaved protocol – such clients should operate contiguously, as opposed to concurrently. When used without a protocol client, buffer itself is data-type agnostic. It operates as a smart array, flushing and/or refilling itself via a conduit as necessary.

Whilst a buffer is often attached to a conduit, it can happily be used standalone instead as a memory-based accumulator. In both cases, buffer clients are added in an identical manner, and operate in the same manner with one distinction: without an attached conduit the buffer cannot be automatically flushed when filled to capacity, so an exception will be raised where this occurs. A growing buffer can be used to handle this scenario where appropriate.

Common usage of a Buffer includes the following methods:

IBuffer append(void[] src);
uint read(void[] dst);
void[] slice();
IBuffer fill();
IBuffer flush();
IBuffer clear();
IConduit conduit();

These methods are typically used less often:

bool skip();
bool truncate(uint extent);
bool next(uint delegate(void[]) tokenizer);
void[] getContent();
IBuffer setContent(void[] content);
IBuffer copy(IConduit src);
IConduit compress();

Another buffer variety wraps OS facilities to expose memory-mapped files; buffer memory is mapped directly onto a (usually) large file, which can then be treated as just another buffer or indeed as a void[].

Buffer Exceptions

When not attached to a conduit, a buffer underflow and overflow condition may occur causing an exception to be thrown. Where appropriate, use a growing buffer to correctly manage an overflow condition.