Byte oriented data types in python

[John Machin]

On Jan 26, 10:53 am, "Martin v. Löwis" <mar... at v.loewis.de> wrote:
> > It deals with variable sized fields just fine:
>
> > dtype = 18
> > dlength = 32
> > format = "!BB%ds" % dlength
>
> > rawdata = struct.pack(format, (dtype,dlength,data))
>
> I wouldn't call this "just fine", though - it involves
> a % operator to even compute the format string. IMO,
> it is *much* better not to use the struct module for this
> kind of problem, and instead rely on regular string
> concatenation.
>

IMO, it would be a good idea if struct.[un]pack supported a variable *
length operator that could appear anywhere that an integer constant
could appear, as in C's printf etc and Python's % formatting:

dlen = len(data)
rawdata = struct.pack("!BB*s", dtype, dlen, dlen, data)
# and on the other end of the wire:
dtype, dlen = struct.unpack("!BB", rawdata[:2])
data = struct.unpack("!*s", rawdata[2:], dlen)
# more than 1 count arg could be used if necessary
# *s would return a string
# *B, *H, *I, etc would return a tuple of ints in (3.X-speak)

I've worked with variable-length data that looked like
   len1, len2, len3, data1, data2, data3
and the * gadget would have been very handy:
len1, len2, len3 = unpack('!BBB', raw[:3])
data1, data2, data3 = unpack('!*H*i*d', raw[3:], len1, len2, len3)

Note the semantics of '!*H*i*d' would be different from '!8H2i7d'
because otherwise you'd need to do:
bundle = unpack('!*H*i*d', raw[3:], len1, len2, len3)
data1 = bundle[:len1]
data2 = bundle[len1:len1+len2]
data3 = bundle[len1+len2:]