Question

I have read that boost iostreams supposedly supports 64 bit access to large files semi-portable way. Their FAQ mentions 64 bit offset functions, but there is no examples on how to use them. Has anyone used this library for handling large files? A simple example of opening two files, seeking to their middles, and copying one to the other would be very helpful.

Thanks.

Answer 1

Short answer

Just include

#include <boost/iostreams/seek.hpp>

and use the seek function as in

boost::iostreams::seek(device, offset, whence);

where

• device is a file, stream, streambuf or any object convertible to seekable;
• offset is a 64-bit offset of type stream_offset;
• whence is BOOST_IOS::beg, BOOST_IOS::cur or BOOST_IOS::end.

The return value of seek is of type std::streampos, and it can be converted to a stream_offset using the position_to_offset function.

Example

Here is an long, tedious and repetitive example, which shows how to open two files, seek to offstets >4GB, and copying data between them.

WARNING: This code will create very large files (several GB). Try this example on an OS/file system which supports sparse files. Linux is ok; I did not test it on other systems, such as Windows.

/*
 * WARNING: This creates very large files (several GB)
 * unless your OS/file system supports sparse files.
 */
#include <boost/iostreams/device/file.hpp>
#include <boost/iostreams/positioning.hpp>
#include <cstring>
#include <iostream>

using boost::iostreams::file_sink;
using boost::iostreams::file_source;
using boost::iostreams::position_to_offset;
using boost::iostreams::seek;
using boost::iostreams::stream_offset;

static const stream_offset GB = 1000*1000*1000;

void setup()
{
    file_sink out("file1", BOOST_IOS::binary);
    const char *greetings[] = {"Hello", "Boost", "World"};
    for (int i = 0; i < 3; i++) {
        out.write(greetings[i], 5);
        seek(out, 7*GB, BOOST_IOS::cur);
    }
}

void copy_file1_to_file2()
{
    file_source in("file1", BOOST_IOS::binary);
    file_sink out("file2", BOOST_IOS::binary);
    stream_offset off;

    off = position_to_offset(seek(in, -5, BOOST_IOS::end));
    std::cout << "in: seek " << off << std::endl;

    for (int i = 0; i < 3; i++) {
        char buf[6];
        std::memset(buf, '\0', sizeof buf);

        std::streamsize nr = in.read(buf, 5);
        std::streamsize nw = out.write(buf, 5);
        std::cout << "read: \"" << buf << "\"(" << nr << "), "
                  << "written: (" << nw << ")" << std::endl;

        off = position_to_offset(seek(in, -(7*GB + 10), BOOST_IOS::cur));
        std::cout << "in: seek " << off << std::endl;
        off = position_to_offset(seek(out, 7*GB, BOOST_IOS::cur));
        std::cout << "out: seek " << off << std::endl;
    }
}

int main()
{
    setup();
    copy_file1_to_file2();
}