Question

I have to send a large amount of xml data through sockets.
Example of xml file (server side generates such a file):

<message>
 <data>
  <param1>12345</param1>
  <param2>234</param2>
 </data>
...
 <data>
  <param1>321</param1>
  <param2>34234</param2>
 </data>
</message>

And then client receive such data.
Client have to parse this file and insert parameters to db.

Client receives this data by parts (size of byte):

byte[] receivedData = new byte[1024];
int receivedBytesLen = streamFromServer.read(receivedData);

How can I process this data?

1. For example, if I read all the file data to the variable on the client side and then process. I think it’s not good – may be problems (large amount of data).
2. Write all data to the temp file and then to process? Create file, delete…
3. Read and process in the same time. But the xml is not full.
4. Something else?

Answer 1

A few years ago, I developed a solution for a broadcaster who was using a sports wire service that worked EXACTLY this way.

A few tips from them:

1. Make sure every message has a unique identifier. It can be something as simple as a guid attribute in the document element.
2. Transmit the messages multiple times (unless this is a point-to-point and you have an acknowledgement built into your solution). The client should maintain a hashtable of successfully processed messages’ unique identifiers, and disregard duplicates it has already handled.
3. Keep your messages small. Do you really need all the “data” nodes to be in one message? Can you not send them as individual messages?
4. Separate your messages with a non-viable XML character. CR’s and LF’s won’t work. This service used {0x02} bytes to separate messages. Use that on the client side to find the separation points between XML documents.