I need to decode binary messages coming at a high rate(>1000 msgs/sec) and store them in a DB(not decided on which one) using JAVA. There would be multiple TCP connections coming into this server each with its own stream of binary data that needs to be processed.
The messages are not separated by any “flags”.
The starting of the message has a 4 byte length field. It is followed by a fixed header.
The payload of the message would be in turn multiple messages, each having a fixed header and followed by bit masks(32 bits) that determine what other fields are present. Each bit mask field is 32 bits and the bits 32-30(MSB -32 / Big endian) specify the length of each of the optional fields. All the other bits(29-1), if “ON” means that the field is present in the message.
For example, if bits 32-30 are 100, and bit 1 is “1”, then field XXX follows the bit mask field and it is 4 bytes long. If bit 2 is “0”, the field YYY is not present in the message and so on.
There would be multiple bit mask fields(optional) present but limited by a max number.
I’m new to java(c/C++ background) so questions maybe …
1) I’m thinking to design the app in the regular way that the “main” thread receives the connections and creates a “worker thread A” to process messages on that socket. I’m thinking of letting the config file drive whether the “workerThread A” creates a threadPool for handling each message or does it on his own. I’ll implement the former and check out the performance and see whether it needs to be improved.
My question is, are netty or Apache Mina good options to consider? Since it is a POC effort, i need to crank it up quickly.
2) I thought of using nio – SocketChannel and ByteBuffer. But it seems like that I cannot read a specified number of bytes from the socket? I’m thinking it would be easier to “readInt()” to get the length and then read “length” number of bytes from the socket to get a complete one message an then parse it. Is DataInputStream a better one to use? Any performance impacts will be there from using oio vs nio?
3) Should i be looking at any frameworks to decode the message? I looked at Google Protocol buffers a little bit but it does not appear as if it would support decoding bit mask fields.
I would create one worker thread per connection.
I would use DataInputStream, unless you determine this is not fast enough. There is a small performance impact, but its unlikely to matter at 1000 msg/sec.
I would just decode the message as it arrives using the JDK. I haven’t found third party libraries to make things simpler in this situation.