[Python-ideas] A new .pyc file format

[Facundo Batista]

2008/4/25, Gabriel Genellina <gagsl-py2 at yahoo.com.ar>:

>  The problem is that this format is *too* simple. It can't be changed, nor
>  can accomodate other fields if desired. I propose using a more flexible

But how do you think that these extended pyc's will be used? I mean,
are there use cases for this more complex pyc? Or they just will be
more complex, but with the same information than before, for years,
because nobody needs this flexibility?


>  Anyway the change is "safe", in the sense that any old code expecting the
>  MAGIC number in the first 4 bytes will reject the new format as invalid
>  and not process it.

Maybe what we can do here is that, for some Python versions (say, 3.0,
and maybe 3.1), the importer will try to import in the new form, and
if recognizes it as invalid *and* finds the some MAGIC numbers in the
first 4 bytes, just import it in the old fashion way..


>  - making the size of all sections multiple of 4 may be a good idea, so
>  marshaled code should be padded with up to 3 NUL bytes at the end.

Why? Has this something to do with memory alignment? I don't see the
benefit of this extra rule.


>  - 32 bits should be enough for all sizes (and 640k should be enough for
>  all people...)

Regarding this, and the 44 bytes of overhead you said, I checked which
is the average size of the .pyc in the Linux system I had at hand:

$ locate pyc | xargs ls -l | awk 'BEGIN {a=0; c=0} {a += $5; c+=1} END
{print c, a/c}'
4960 9069.12

I think that both overhead and 32b for size are ok.


>  Implementation:
>
>  I don't have a complete implementation yet, but if this format is approved
>  (as is or with any changes) I could submit a patch. I've made a small but
>  incompatible modification in the currently used .pyc format in order to
>  detect all places where this change would impact, and they're not so many
>  actually.

Yes, you should start writing a PEP (any help you need here, we can
talk about it in the next Python Argentina meeting, ;).

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/