Question

I need to be able to delimit a stream of binary data. I was thinking of using something like the ASCII EOT (End of Transmission) character to do this.

However I’m a bit concerned — how can I know for sure that the particular binary sequence used for this (0b00000100) won’t appear in my own binary sequences, thus giving a false positive on delimitation?

In other words, how is binary delimiting best handled?

EDIT: …Without using a length header. Sorry guys, should have mentioned this before.

Answer 1

You’ve got five options:

• Use a delimiter character that is unlikely to occur. This runs the risk of you guessing incorrectly. I don’t recommend this approach.
• Use a delimiter character and an escape sequence to include the delimiter. You may need to double the escape character, depending upon what makes for easier parsing. (Think of the C \0 to include an ASCII NUL in some content.)
• Use a delimiter phrase that you can determine does not occur. (Think of the mime message boundaries.)
• Prepend a length field of some sort, so you know to read the following N bytes as data. This has the downside of requiring you to know this length before writing the data, which is sometimes difficult or impossible.
• Use something far more complicated, like ASN.1, to completely describe all your content for you. (I don’t know if I’d actually recommend this unless you can make good use of it — ASN.1 is awkward to use in the best of circumstances, but it does allow completely unambiguous binary data interpretation.)