[Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Terry Reedy]

On 6/13/2012 1:19 PM, Brett Cannon wrote:

> On Tue, Jun 12, 2012 at 2:16 PM, Terry Reedy <tjreedy at udel.edu
> <mailto:tjreedy at udel.edu>> wrote:
>
>     http://bugs.python.org/__issue12982 <http://bugs.python.org/issue12982>
>
>     Currently, cpython requires the -O flag to *read* .pyo files as well
>     as the write them. This is a nuisance to people who receive them
>     from others, without the source. The originator of the issue quotes
>     the following from the doc (without giving the location).
>
>     "It is possible to have a file called spam.pyc (or spam.pyo when -O
>     is used) without a file spam.py for the same module. This can be
>     used to distribute a library of Python code in a form that is
>     moderately hard to reverse engineer."
>
>     There is no warning that .pyo files are viral, in a sense. The user
>     has to use -O, which is a) a nuisance to remember if he has multiple
>     scripts and some need it and some not, and b) makes his own .py
>     files used with .pyo imports cached as .pyo, without docstrings,
>     like it or not.
>
>     Currently, the easiest workaround is to rename .pyo to .pyc and all
>     seems to work fine, even with a mixture of true .pyc and renamed
>     .pyo files. (The same is true with the -O flag and no renaming.)
>     This suggests that there is no current reason for the restriction in
>     that the *execution* of bytecode is not affected by the -O flag.
>     (Another workaround might be a custom importer -- but this is not
>     trivial, apparently.)
>
>
> In Python 3.3 it's actually trivial.

For you. Anyway, I am sure Michael of #12982 is using an earlier version.

>     So is the import restriction either an accident or obsolete holdover?
>
>
> Neither. .pyo files are actually different from .pyc files in terms of
> what bytecode they may emit. Currently -O causes all asserts to be left
> out of the bytecode and -OO leaves out all docstrings on top of what -O
> does. This makes a difference if you are trying to introspect at the
> interpreter prompt or are testing things in development and want those
> asserts to be triggered if needed.

I suggested to Michael that he should request an all-.pyc library for 
that reason.

>     If so, can removing it be treated as a bugfix and put into current
>     releases, or should it be treated as an enhancement only for a
>     future release?
>
>
> The behaviour shouldn't change. There has been talk of doing even more
> aggressive optimizing under -O, which once again would cause an even
> larger deviation between a .pyo file and a .pyc file (e.g. allowing
> Python code to hook into the peephole optimizer or an entirely new AST
> optimizer).

Would such a change mean that *reading* a .pyo file as if it were a .pyc 
file would start failing? (It now works, by renaming the file.)
If so, would not it be better to rely on having a different magic number 
*in* the file rather than on its mutable external name?

You just said above that evading import restriction by name is now 
trivial. So what is the point of keeping it. If, in the future, there 
*are* separate execution pathways*, and we want the .pyo pathway closed 
unless -O is passed, then it seems that that could only be enforced by a 
magic number in the file.

*Unladen Swallow would have *optionally* produced cache files completely 
different from current bytecode, with a different extension. A couple of 
people have suggested using wordcode instead of bytecode. If this were 
also introduced as an option, its cache files would also need a 
different extension and magic number.

>     Or is the restriction an intentional reservation of the possibility
>     of making *execution* depend on the flag? Which would mean that the
>     restriction should be kept and only the doc changed?
>
> The docs should get updated to be more clear.

-- 
Terry Jan Reedy