[Python-Dev] Identifying magic prefix on Python files?

[Tim Peters]

[Ka-Ping Yee, with more magical ideas]

This is contentious every time it comes up because of "backward
compatibility".  The contentious part is that no two people come into it
with the same idea of what "backward compatible" means, exactly, and it
usually drags on for days until people realize that.  In the meantime,
everyone thinks everyone else is an idiot <wink>.

So far as the docs go, imp.get_magic() returns "a string", and that's all it
says.  By that defn, it would be darned hard to think of any scheme that
isn't backward compatible.  OTOH, PyImport_GetMagicNumber() returns "a
long", so there's good reason to preserve that version-checking must not
rely on more than 4 bytes of info.

Then you have /F's post, which purports to give a "fully backward
compatible" scheme, despite changing what probably appears <wink> to be
almost everyting.

It takes a long time to reverse-engineer what the crucial invariants are for
each person, based on what they propose and what they complain about in
other proposals.

I don't have time for that now, so will leave it to someone else.  It would
help if people could spell out directly which invariants they do and don't
care about (e.g., you can *infer* that /F cares about

    exactly 4 bytes magic number (but doesn't care about content)
then
    exactly 4 bytes file timestamp
then
    a blob that marshal believes is a single object
then
    that's it

but doesn't care that, e.g., checking the 4-byte magic number alone is
sufficent to catch binary files opened in text mode (but somebody else will
care about that!)).

Since virtually none of this has been formalized via an API, virtually all
code outside the distribution that deals with this stuff is cheating.  Small
wonder it's contentious ...