Question

The problem is that sending large serialized JSON (over 16,000 characters) over a net socket gets split into chunks. Each chunk fires the data event on the receiving end. So simply running JSON.parse() on the incoming data may fail with SyntaxError: Unexpected end of input.

The work around I’ve managed to come up with so far is to append a null character ('\u0000') to the end of the serialized JSON, and check for that on the receiving end. Here is an example:

var partialData = '';
client.on( 'data', function( data ) {
    data = data.toString();
    if ( data.charCodeAt( data.length - 1 ) !== 0 ) {
        partialData += data;
        // if data is incomplete then no need to proceed
        return;
    } else {
        // append all but the null character to the existing partial data
        partialData += data.substr( 0, data.length - 1 );
    }
    // pass parsed data to some function for processing
    workWithData( JSON.parse( partialData ));
    // reset partialData for next data transfer
    partialData = '';
});

One of the failures of this model is if the receiver is connected to multiple sockets, and each socket is sending large JSON files.

The reason I’m doing this is because I need to pass data between two processes running on the same box, and I prefer not to use a port. Hence using a net socket. So there would be two questions: First, is there a better way to quickly pass large JSON data between two Node.js processes? Second, if this is the best way then how can I better handle the case where the serialized JSON is being split into chunks when sent?

Answer 1

1. You can use try…catch every time to see if it is a valid json. Not very good performance though.
2. You can calculate size of your json on sending side and send it before JSON.

3. You can append a boundary string that’s unlikely be in JSON. Your \u0000 – yes, it seems to be a legit way. But most popular choice is newline.

4. You can use external libraries like dnode which should already do something I mentioned before. I’d recommend trying that. Really.

One of the failures of this model is if the receiver is connected to multiple sockets, and each socket is sending large JSON files.

Use different buffers for every socket. No problem here.