struct: type registration?

[Giovanni Bajo]

Giovanni Bajo wrote:

> You need struct.unpack() to parse these datas, and you need custom
> packer/unpacker to avoid post-processing the output of unpack() just
> because it just knows of basic Python types. In binary structs, there
> happen to be *types* which do not map 1:1 to Python types, nor they
> are just basic C types (like the ones struct supports). Using custom
> formatter is a way to better represent these types (instead of
> mapping them to the "most similar" type, and then post-process it).
>
> In my example, "S" is a basic-type which is a "A 0-terminated 20-byte
> string", and expressing it in the struct format with the single
> letter "S" is more meaningful in my code than using "20s" and then
> post-processing the resulting string each and every time this happens.


Another compelling example is the SSH protocol:
http://www.openssh.com/txt/draft-ietf-secsh-architecture-12.txt
Go to section 4, "Data Type Representations Used in the SSH Protocols", and it
describes the data types used by the SSH protocol. In a perfect world, I would
write some custom packers/unpackers for those types which struct does not
handle already (like the "mpint" format), so that I could use struct to parse
and compose SSH messages. What I ended up doing was writing a new module
sshstruct.py from scratch, which duplicates struct's work, just because I
couldn't extend struct. Some examples:

client.py:        cookie, server_algorithms, guess, reserverd =
sshstruct.unpack("16b10LBu", data[1:])
client.py:            prompts = sshstruct.unpack("sssu" + "sB"*num_prompts,
pkt[1:])
connection.py:        pkt = sshstruct.pack("busB", SSH_MSG_CHANNEL_REQUEST,
self.recipient_number, type, reply) + custom
kex.py:        self.P, self.G = sshstruct.unpack("mm",pkt[1:])

Notice for instance how "s" is a SSH string and unpacks directly to a Python
string, and "m" is a SSH mpint (infinite precision integer) but unpacks
directly into a Python long. Using struct.unpack() this would have been
impossible and would have required much post-processing.

Actually, another thing that struct should support to cover the SSH protocol
(and many other binary protocols) is the ability to parse strings whose size is
not known at import-time (variable-length data types). For instance, type
"string" in the SSH protocol is a string prepended with its size as uint32. So
it's actual size depends on each instance. For this reason, my sshstruct did
not have the equivalent of struct.calcsize(). I guess that if there's a way to
extend struct, it would comprehend variable-size data types (and calcsize()
would return -1 or raise an exception).
-- 
Giovanni Bajo