I’ve been using WebServices at moving data across the wire and that has served me pretty well. It excels at sending small pieces of data. As soon as you have to move deep object trees with lots of properties, the resulting XML soup takes 100k of data and turns it into a 1MB.
So I’ve tried IIS Compression, but it left me underwhelmed. It compressed data well, but the trade off was in compression/decompression. Then I’ve serialized the objects via BinaryFormatter and sent that across. This was better, however, speed of encode/decode still remains.
Anyway, I am hearing that I am stuck in the 00s and now there are better ways to send data across the wire such as ProtocolBuffers, MessagePack, etc…
Can someone tell me whether these new protocols will be better suited for sending large pieces of data and whether I am missing some other efficient ways to do this?
By efficient, I mean amount of bandwidth, speed of encode/decode, speed of implementation, etc…
It depends on what’s making up the bulk of your data. If you’ve just got lots of objects with a few fields, and it’s really the cruft which is “expanding” them, then other formats like Protocol Buffers can make a huge difference. I haven’t used MessagePack or Thrift, but I would expect they could have broadly similar size gains.
In terms of speed of encoding and decoding, I believe that both Marc Gravell’s implementation of Protocol Buffers and my own will outperform any of the built-in serialization schemes.