[Python-Dev] Inconsistent behaviour in import/zipimport hooks

[Ulrich Berning]

Phillip J. Eby schrieb:

>At 04:33 PM 11/9/2005 -0800, Guido van Rossum wrote:
>  
>
>>On 11/9/05, Phillip J. Eby <pje at telecommunity.com> wrote:
>>    
>>
>>>By the way, while we're on this subject, can we make the optimization
>>>options be part of the compile() interface?  Right now the distutils has to
>>>actually exec another Python process whenever you want to compile
>>>code with
>>>a different optimization level than what's currently in effect, whereas if
>>>it could pass the desired level to compile(), this wouldn't be necessary.
>>>      
>>>
>>Makes sense to me; we need a patch of course.
>>    
>>
>
>But before we can do that, it's not clear to me if it should be part of the 
>existing "flags" argument, or whether it should be separate.  Similarly, 
>whether it's just going to be a level or an optimization bitmask in its own 
>right might be relevant too.
>
>For the current use case, obviously, a level argument suffices, with 'None' 
>meaning "whatever the command-line level was" for backward 
>compatibility.  And I guess we could go with that for now easily enough, 
>I'd just like to know whether any of the AST or optimization mavens had 
>anything they were planning in the immediate future that might affect how 
>the API addition should be structured.
>
>  
>
I'm using a totally different approach for the above problem. I have 
implemented two functions in the sys module, that make the startup flags 
accessible at runtime. This also solves some other problems I had, as 
you will see in the examples below:


The first function makes most of the flags readable (I have ommited the 
flags, that are documented as deprecated in the code):

sys.getrunflag(name) -> integer

Return one of the interpreter run flags. Possible names are 'Optimize', 
'Verbose', 'Interactive', 'IgnoreEnvironment', 'Debug', 
'DivisionWarning', 'NoSite', 'NoZipImport', 'UseClassExceptions', 
'Unicode', 'Frozen', 'Tabcheck'. getrunflag('Optimize') for example 
returns the current value of Py_OptimizeFlag.


The second function makes a few flags writable:

sys.setrunflag(name, value) -> integer

Set an interpreter run flag. The only flags that can be changed at 
runtime are Py_VerboseFlag ('Verbose') and Py_OptimizeFlag ('Optimize'). 
Returns the previous value of the flag.


As you can see, I have also introduced the new flag Py_NoZipImport that 
can be activated with -Z at startup. This bypasses the activation of 
zipimport and is very handy, if you edit modules stored in the 
filesystem, that are normally imported from a zip archive and you want 
to test your modifications. With this flag, there is no need to delete, 
rename or update the zip archive or to modify sys.path to ensure that 
your changed modules are imported from the filesystem and not from the 
zip archive.


And here are a few usable examples for the new functions:

1.)  You have an application, that does a huge amount of imports and 
some of them are mysterious, so you want to track them in verbose mode. 
You could start python with -v or -vv, but then you get hundreds or 
thousands of lines of output. Instead, you can do the following:

import sys
import ...
import ...
oldval = sys.setrunflag('Verbose', 1) # -v, use 2 for -vv
import ...
import ...
sys.setrunflag('Verbose', oldval)
import ...
import ...

Now, you get only verbose messages for the imports that you want to track.

2.) You need to generate optimized byte code (without assertions and 
docstrings) from a source code, no matter how the interpreter was started:

import sys
...
source = ...
oldval = sys.setrunflag('Optimize', 2) # -OO, use 1 for -O
bytecode = compile(source, ...)
sys.setrunflag('Optimize', oldval)
...

3.) You have to build a command line for the running application (e.g. 
for registration in the registry) and need to check, if you are running 
a script or a frozen executable (this assumes, that your freeze tool 
sets the Py_FrozenFlag):

import sys
...
if sys.getrunflag('Frozen'):
    commandline = sys.executable
else:
    commandline = '%s %s' % (sys.executable, sys.argv[0])
...

NOTE: My own freeze tool sib.py, which is part of the VendorID package 
(www.riverbankcomputing.co.uk/vendorid) doesn't set the Py_FrozenFlag 
yet. I will provide an update soon.

----

And now back to the original subject:

I have done nearly the same changes, that Osvaldo provided with his 
patch and I would highly appreciate if this patch goes into the next 
release. The main reason why I changed the import behavior was 
pythonservice.exe from the win32 extensions. pythonservice.exe imports 
the module that contains the service class, but because 
pythonservice.exe doesn't run in optimized mode, it will only import a 
.py or a .pyc file, not a .pyo file. Because we always generate bytecode 
with -OO at distribution time, we either had to change the behavior of 
pythonservice.exe or change the import behavior of Python.
It is essential for us to remove assertions and docstrings in our 
commercial Python applications at distribution time, because assertions 
are meaningful only at development time and docstrings may contain 
implementation details, that our customers should never see (this makes 
reverse engineering a little bit harder, but not impossible).
Another important reason is, that the number of files in the standard 
library is reduced dramatically. I can have .py and .pyc files in 
lib/pythonX.Y containing assertions and docstrings and put optimized 
code in lib/pythonXY.zip. Then, if I need docstrings, I just bypass 
zipimport as described above with -Z. On the customer site, we only need 
to install the zip archive (our customers are not developers, just end 
users; they may not even recognize that we use Python for application 
development).

Guido, if it was intentional to separate slightly different generated 
bytecode into different files and if you have good reasons for doing 
this, why have I never seen a .pyoo file :-)

For instance, nobody would give the output of a C compiler a different 
extension when different compiler flags are used.

I would appreciate to see the generation of .pyo files completely 
removed in the next release. Just create them as .pyc files, no matter 
if -O or -OO is used or not. At runtime, Python should just ignore (jump 
over?) assert statements when started with -O or ignore assert 
statements and docstrings when started with -OO if they are in the .pyc 
file.

----

There are two reasons, why I haven't started a discussion about those 
issues earlier on this list:

1.) I have done those changes (and a lot of other minor and major 
changes) in Python 2.3.x at a time when Python 2.4.x came up and we 
still use Python 2.3.5. I just wanted to wait until I have the next 
release of our Python runtime environment (RTE) ready that will contain 
the most recent Python version.

2.) In the last months, my time was limited, because I first had to 
stabilize the current RTE (developers and customers were waiting on it).


A few notes about this Python runtime environment:

The RTE that I maintain contains everything (mainly, but not only 
Python) that is needed to run our applications on the customer site. The 
applications are distributed as frozen binaries either containing the 
whole application or only the frozen main script together with an 
additional zip archive holding the application specific modules and 
packages. Tools like py2exe or cx_Freeze follow a different approach: 
they always package a kind of runtime environment together with the 
application.  There is nothing bad with this approach, but if you 
provide more than one application, you waste resources and it may be 
harder to maintain (bugfixes, updates, etc.).
Our RTE currently runs on AIX, HP-UX, IRIX, Windows and Linux, 
SunOS/Solaris will follow in the near future. It gets installed together 
with the application(s) on the customer site if it is not already there 
in the appropriate version. It is vendor specific (more in the sense of 
a provider, not necessarily in the sense of a seller) and shouldn't 
interfere with any other software already installed on the target site.

This RTE is the result of years of experience of commercial application 
development with Python (we started with Python-1.3.x a long long time 
ago). My guideline is: Application developers and/or customers should 
not worry about the requirements needed to get the applications running 
on the target site. Developers should spend all their time for 
application development and customers can expect a complete installation 
with only a very small set of system requirements/dependencies.

In general, I would like to discuss those things that increase Python's 
usability especially in a commercial environment and things that make 
Python more consistent across platforms (e.g. unifying the filesystem 
layout on Windows and UNIX/Linux), but I don't know if this is the right 
mailing list.

Ulli