Question

I’m new to boost and its iostreams package and finding the documentation a bit thin. Hopefully someone will set me straight. I’m trying to convert a small piece of C# streams code I wrote a while back for reading in a compressed stream.

byte[] data = new byte[length - 1];
file.Read(data, 0, data.Length);

Stream ret = new ZlibStream(new MemoryStream(data), CompressionMode.Decompress, true);
return ret;

Data from part of a file is read into a memory buffer which feeds the zlib decompressor. The consumer of the stream will pick away at it over time, and when it is finished will call Close() which combined with the garbage collector will clean up all of the resources. Note: an important distinction is I am not trying to decompress an entire file, just a small part of one. The file has already been seeked to some interior location and length is small relative to the full size of the file.

I’m trying to come up with the best equivalent to this in C++ code with Boost. So far I have this moral equivalent to the above (untested):

char * data = new char[length - 1];
_file.read(data, length - 1);

io::stream_buffer<io::basic_array_source<char> > buffer(data, length - 1);

io::filtering_stream<io::input> * in = new io::filtering_stream<io::input>;         
in->push(io::zlib_decompressor());
in->push(buffer);

return in;

I assume I could return the filtering_stream wrapped in a shared_ptr which would save the consumer from having to worry about deleting the stream, but I also have that data buffer new’d up in there. Ideally I would like the consumer to just call close() on the stream and some mechanism (e.g. callback) would clean up the underlying resources in the filter. Requiring the consumer to pass the stream to an explicit release function is also acceptable, but I’m still not entirely sure how to get the underlying data buffer back in the first place.

Cleaner alternative solutions are also welcome.

Update 1

I’ve tried loosely seizing on the comment by Cat Plus Plus about a std::vector-backed driver. That’s not quite what I’ve done, but this is what I have come up with so far. In the following code, I have a boost::shared_array-backed driver, based on the boost driver examples.

namespace io = boost::iostreams;

class shared_array_source
{
public:

    typedef char            char_type;
    typedef io::source_tag  category;

    shared_array_source (boost::shared_array<char> s, std::streamsize n)
        : _data(s), _pos(0), _len(n)
    { }

    std::streamsize read (char * s, std::streamsize n)
    {
        std::streamsize amt = _len - _pos;
        std::streamsize result = (std::min)(amt, n);

        if (result != 0) {
            std::copy(_data.get() + _pos, _data.get() + _pos + result, s);
            return result;
        }
        else {
            return -1;
        }
    }

private:
    boost::shared_array<char> _data;
    std::streamsize _pos;
    std::streamsize _len;
};

Then I have my function that returns a stream

io::filtering_istream * GetInputStream (...)
{
    // ... manipulations on _file, etc.

    boost::shared_array<char> data(new char[length - 1]);
    _file.read(data.get(), length - 1);

    shared_array_source src(data, length - 1);
    io::stream<shared_array_source> buffer(src);

    io::filtering_istream * in = new io::filtering_istream;
    in->push(io::zlib_decompressor());
    in->push(buffer);

    // Exhibit A
    // uint32_t ui;
    // rstr->read((char *)&ui, 4);

    return in;
}

And in my main function of a test program:

int main () {
    boost::iostreams::filtering_istream * istr = GetInputStream();

    // Exhibit B
    uint32_t ui;
    rstr->read((char *)&ui, 4);

    return 0;
}

Ignore the fact that I’m returning a pointer that will never be freed — I’m keeping this as simple as possible. What happens when I run this? If I uncomment the code at Exhibit A, I get a proper readout in ui. But when I hit Exhibit B, I crash deep, deep, deep down in Boost (sometimes). Well crap, I went out of scope and things broke, some local must be deconstructing and messing everything up. data is in a shared_array, in is on the heap, and the compressor is constructed according to docs.

Are one of the boost constructors or functions grabbing a reference to an object on the stack (namely io::stream or filtering_stream’s push)? If that’s the case, I’m sort of back to square one with unmananaged objects on the heap.

Answer 1

Ultimately I had to give up on trying to get Boost iostreams to clean up after themselves. While creating a custom array device that internally used a boost::shared_array solved the problem of my data buffer being left on the heap, it turns out that boost::iostreams::filtering_stream / streambuf take a reference to an object pushed into the chain, which means it was capturing a reference to my device on the stack (as much as I could infer from the source and behavior, anyway). I could new up the device on the heap, but that just puts me back to square one.

My following solution moves the creation of the stream into a thin std::istream wrapper, which can then be returned inside a boost::shared_ptr, allowing all resources to be cleaned up when the last reference is destroyed.

template <class Compressor>
class CompressedIStream : public std::basic_istream<char, std::char_traits<char> >
{
public:
    CompressedIStream (std::istream& source, std::streamsize count)
        : _data(new char[count]), _buffer(_data, count), std::basic_istream<char, std::char_traits<char> >(&_filter)
    {
        source.read(_data, count);

        _filter.push(Compressor());
        _filter.push(_buffer);
    }

    virtual ~CompressedIStream ()
    {
        delete[] _data;
    }

private:
    char * _data;
    io::stream_buffer<io::basic_array_source<char> > _buffer;
    io::filtering_istreambuf _filter;
};

boost::shared_ptr<std::istream> GetInputStream (...)
{
    // ... manipulations on _file, etc.

    typedef CompressedIStream<io::zlib_decompressor> StreamType;
    return boost::shared_ptr<StreamType>(new StreamType(_file, length - 1));
}