Question

I want to save raw data chunks to a file, And later on read those chunks one by one. This is no big deal except the following doubt:

What exact bytes to use as a delimiter, i.e to identify end of one chunk and beginning of next ? Given that chunk data might also contain such a sequence of bytes by random chance.

Notes: chunks are of variable size and contain random data. They are jpeg images actually.

Answer 1

You could first write the length of the chunk to the file as a fixed-size value, e.g. a 4 bytes integer, followed by the data itself:

public void appendChunk(byte[] data, File file) throws IOException {
    DataOutputStream stream = null;
    try {
        stream = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(file, true)));
        stream.writeInt(data.length);
        stream.write(data);
    } finally {
        if (stream != null) {
            try {
                stream.close();
            } catch (IOException e) {
                // ignore
            }
        }
    }
}

If you later have to read the chunks back from that file, you start by reading the length of the first chunk. You now can decide whether to read the chunk data, or whether to skip it and continue with the next chunk.

public void processChunks(File file) throws IOException {
    DataInputStream stream = null;
    try {
        stream = new DataInputStream(new BufferedInputStream(new FileInputStream(file)));
        while (true) {
            try {
                int length = stream.readInt();
                byte[] data = new byte[length];
                stream.readFully(data);
                // todo: do something with the data
            } catch (EOFException e) {
                // end of file reached
                break;
            }
        }
    } finally {
        if (stream != null) {
            try {
                stream.close();
            } catch (IOException e) {
                // ignore
            }
        }
    }
}

You can also add other meta-data about the chunks, like writing the original name of the file with stream.writeUTF(…). You only have to make sure that you write and read the same data in the same order.