Question

I have a binary protocol with some unknown amount of initial header data (unknown length until header is fully decoded) followed by a stream of data that should be written to disk for later processing.

I have an implementation that decodes the header and then writes the file data to disk as it comes in from one or more frames in ChannelBuffers (so my handler subclasses FrameDecoder, and builds the message step by step, not waiting for one ChannelBuffer to contain the entire message, and writing the file data to disk with each frame). My concern is whether this is enough, or if using something like ChunkedWriteHandler does more than this and is necessary to handle large uploads.

Is there a more optimal way of handling file data than just writing it directly to disk from the ChannelBuffer with each frame?

Answer 1

It should be enough as long as the throughput is good enough. Otherwise, you might want to buffer the received data so that you don’t make system calls too often (say, 32KiB bounded buffer) when the amount of received data is not large enough.

Netty could be even faster if it exposed the transferTo/From operation to a user, but such a feature is not available yet.