Question

I’m trying to utilise TCP sockets in Node.js to communicate with a Lua program. There are two issues I’m trying to get my head around. Firstly buffers.

Buffers As I understand it, when data is provided to your code from the socket, it will be whatever data has been received so far (a stream rather than packets). So when you read the data received, it may be broken from what was sent.

i.e. data received
{ schools : [“Long

vs data sent
{ schools : [{“Longwood”, “Hillbrow”}] }

The way to get around this is to put your data into a ‘buffer’ and split it by whatever method you use to show the end of that piece of information. Commonly appears to be new line.

My questions here:

1. Can you end up with more 2 or more complete sections of data in the buffer, how would you handle this? For loop?
2. The same issue appears to be present with data leaving the socket. I have noticed however in other code examples, people use the Node.js Buffer before writing to the socket. Why is this not used for incoming data?
3. If more data is written than can be handled to leave, does Node handle this for you or do you need to come up with a method.

Finally I seem to be misunderstanding the data side. Does all data sent and received need to be converted to binary and back? I wish to send JSON data back and forth only. I think there in lies my confusion. For example:

var myQuestion = "Is this acceptable and will I encounter any issues?
socket.write(myQuestion);

Many thanks for your time.

Answer 1

1. Yes, you could definitely end up with several sections of data. It entirely depends on the code you write on either end.
2. I don’t quite follow what you mean by “use the buffer before writing”. Buffers are just pure binary data, they are used for both incoming and outgoing data. Node also generally accepts strings for sending, and assumes they are UTF8 if no encoding is given. You can also get strings when receiving if you call setEncoding.
3. The OS has buffers for incoming data on the TCP socket, and when the buffer is full (being emptied too slowly by node), it will stop accepting packets until there is space.

First off, Buffer is binary. A buffer is simply an array of bytes, and nothing more. Buffers can be created from strings, and node will do this automatically in some cases. In your case, I would recommend that you call socket.setEncoding('utf8'). That will automatically convert incoming data into a string to simplify your parsing.

As far as processing and splitting the data, it is up to you. TCP only provides a stream of bytes arriving in the same order they were send. As you said, you can gather up the bytes and when a newline is received, you take everything before that and parse it as JSON. This should work fine. You can use any character that wouldn’t pop up inside JSON. As long as the programs doing the JSON serialization do not add newlines, then you are set. As ‘data’ is emitted, you can check the string for newline, and if not, then add it to any previously received data, and if you find it, then split it up, add the existing data.