Question

I am trying to use struct.unpack() to take apart a data record that ends with an ASCII string.

The record (it happens to be a TomTom ov2 record) has this format (stored little-endian):

• 1 byte
• 4 byte int for total record size (including this field)
• 4 byte int
• 4 byte int
• variable-length string, null-terminated

unpack() requires that the string’s length be included in the format you pass it. I can use the second field and the known size of the rest of the record — 13 bytes — to get the string length:

str_len = struct.unpack("<xi", record[:5])[0] - 13
fmt = "<biii{0}s".format(str_len)

then proceed with the full unpacking, but since the string is null-terminated, I really wish unpack() would do it for me. It’d also be nice to have this should I run across a struct that doesn’t include its own size.

How can I make that happen?

Answer 1

The size-less record is fairly easy to handle, actually, since struct.calcsize() will tell you the length it expects. You can use that and the actual length of the data to construct a new format string for unpack() that includes the correct string length.

This function is just a wrapper for unpack(), allowing a new format character in the last position that will drop the terminal NUL:

import struct
def unpack_with_final_asciiz(fmt, dat):
    """
    Unpack binary data, handling a null-terminated string at the end 
    (and only at the end) automatically.

    The first argument, fmt, is a struct.unpack() format string with the 
    following modfications:
    If fmt's last character is 'z', the returned string will drop the NUL.
    If it is 's' with no length, the string including NUL will be returned.
    If it is 's' with a length, behavior is identical to normal unpack().
    """
    # Just pass on if no special behavior is required
    if fmt[-1] not in ('z', 's') or (fmt[-1] == 's' and fmt[-2].isdigit()):
        return struct.unpack(fmt, dat)

    # Use format string to get size of contained string and rest of record
    non_str_len = struct.calcsize(fmt[:-1])
    str_len = len(dat) - non_str_len

    # Set up new format string
    # If passed 'z', treat terminating NUL as a "pad byte"
    if fmt[-1] == 'z':
        str_fmt = "{0}sx".format(str_len - 1)
    else:
        str_fmt = "{0}s".format(str_len)
    new_fmt = fmt[:-1] + str_fmt

    return struct.unpack(new_fmt, dat)

---

>>> dat = b'\x02\x1e\x00\x00\x00z\x8eJ\x00\xb1\x7f\x03\x00Down by the river\x00'
>>> unpack_with_final_asciiz("<biiiz", dat)
(2, 30, 4886138, 229297, b'Down by the river')