Question

I have a situation where I need to process some data from a stream line by line. The problem is that the encoding of the data is not known in advance; it might be UTF-8 or any legacy single-byte encoding (e.g. Latin1, ISO-8859-5, etc). It will not be UTF16 or exotics like EBCDIC, so I can reasonably expect \n to be unambiguous, so in theory I can split it into lines. At some point, when I encounter an empty line, I will need to feed the rest of the stream somewhere else (without splitting it into lines, but still without any reencoding); think in terms of HTTP-style headers followed by an opaque body.

Here is what I got:

function processStream(stream) {
    var buffer = '';

    function splitLines(data) {
        buffer += data;
        var lf = buffer.indexOf('\n');
        while (lf >= 0) {
            var line = buffer.substr(0, lf - 1);
            buffer = buffer.substr(lf + 1);
            this.emit('line', line);
            lf = buffer.indexOf('\n');
        }
    }

    function processHeader(line) {
        if (line.length) {
            // do something with the line
        } else {
            // end of headers, stop splitting lines and start processing the body
            this
            .removeListener('data', splitLines)
            .removeAllListeners('line')
            .on('data', processBody);
            if (buffer.length) {
                // process leftover buffer as part of the body
                processBody(buffer);
                buffer = '';
            }
        }
    }

    function processBody(data) {
        // do something with the body chunks
    }

    stream.setEncoding('binary');
    stream
    .on('data', splitLines)
    .on('line', processHeader);
}

It does the job, but the problem is that the binary encoding is deprecated and will probably disappear in the future, leaving me without that option. All other Buffer encodings will either mangle the data or fail to decode it altogether if (most likely, when) it does not match the encoding. Working with Uint8Array instead will mean slow and inconvenient Javascript loops over the data just to find a newline.

Any suggestions on how to split a stream into lines on the fly, while remaining encoding-agnostic without using binary encoding?

Answer 1

Disclaimer: I’m not a Javascript developer.

At some point, when I encounter an empty line, I will need to feed the rest of the stream somewhere else (without splitting it into lines, but still without any reencoding)

Right. In that case, it sounds like you really don’t want to think about the data as text at all. Treat it as you would any binary data, and split it on byte 0x0A. (Note that if it came from Windows to start with, you might want to also remove any trailing 0x0D value.)

I know it’s text really, but without any encoding information, it’s dangerous to impose any sort of interpretation on the data.

So you should keep two pieces of state:

• A list of byte arrays
• The current buffer

When you receive data, you logically want to create a new array with the current buffer prepending the new data. (For efficiency you may not want to actually create such an array, but I would do so to start with, until you’ve got it working.) Look for any 0x0A bytes, and split the array accordingly (create a new byte array as a “slice” of the existing array, and add the slice to the list). The new “current buffer” will be whatever data you have left after the final 0x0A.

If you see two 0x0A values in a row, then you’d go into your second mode of just copying the data.

This is all assuming that the Javascript / Node combination allows you to manipulate binary data as binary data, but I’d be shocked if it didn’t. The important point is not to interpret it as text at any point.