[Python-Dev] Should Python's library modules be written to help the freeze tools?

[Bob Ippolito]

On Jan 30, 2005, at 5:50 PM, Martin v. Löwis wrote:


> Tony Meyer wrote:
>> The main question (to steal Thomas's words) is whether the library  
>> modules
>> should be written to help the freeze tools - if the answer is 'yes',  
>> then
>> I'll submit the above as a patch for 2.5.
>
> The answer to this question certainly is "yes, if possible". In this
> specific case, I wonder whether the backwards compatibility is still
> required in the first place. According to PEP 291, Greg Smith and
> Barry Warsaw decide on this, so I think they would need to comment
> first because any patch can be integrated. If they comment that 2.1
> compatibility is still desirable, your patch would be fine (I guess);
> if they say that the compatibility requirement can be dropped for 2.5,
> I suggest that the entire exec statement is removed, along with the
> conditional clause.

py2app <http://undefined.org/python/#py2app> handles this situation by  
using a much richer way to analyze module dependencies, in that it can  
use hooks (called "recipes") to trigger arbitrary behavior when the  
responsible recipe sees that a certain module is in the dependency  
graph.  This is actually necessary to do the right thing in the case of  
extensions and modules that are not friendly with bytecode analysis.   
Though there are not many of these in the standard library, a few  
common packages such as PIL have a real need for this.  Also, since  
modulegraph uses a graph data structure it is much better suited to  
pruning the dependency graph.  For example, pydoc imports Tkinter to  
support an obscure feature, but this is almost never desired in the  
context of an application freeze tool.  py2app ships with a recipe that  
automatically breaks the edge between pydoc and Tkinter  
<http://svn.red-bean.com/bob/py2app/trunk/src/py2app/recipes/ 
docutils.py>, so if Tkinter is not explicitly included or used by  
anything else in the dependency graph, it is correctly excluded from  
the resultant application bundle.

In order to correctly cover the Python API, I needed to ALWAYS include:  
unicodedata, warnings, encodings, and weakref because they can be used  
by the implementation of Python itself without any "import" hints  
(which, if py2exe also did this, would've probably solved Tony's issue  
with bsddb).  Also, I did an analysis of the Python standard library  
and I discovered that the following (hopefully rather complete) list of  
implicit dependencies (from  
<http://svn.red-bean.com/bob/py2app/trunk/src/modulegraph/ 
find_modules.py>):

     {
         # imports done from builtin modules in C code (untrackable by  
modulegraph)
         "time":         ["_strptime"],
         "datetime":     ["time"],
         "MacOS":        ["macresource"],
         "cPickle":      ["copy_reg", "cStringIO"],
         "parser":       ["copy_reg"],
         "codecs":       ["encodings"],
         "cStringIO":    ["copy_reg"],
         "_sre":         ["copy", "string", "sre"],
         "zipimport":    ["zlib"],
         # mactoolboxglue can do a bunch more of these
         # that are far harder to predict, these should be tracked
         # manually for now.

         # this isn't C, but it uses __import__
         "anydbm":       ["dbhash", "gdbm", "dbm", "dumbdbm", "whichdb"],
     }

I would like to write a PEP for modulegraph as a replacement for  
modulefinder at some point, but I can't budget the time for it right  
now.  The current implementation is also largely untested on other  
platforms.  I hear it has been used by a Twisted developer to create  
Windows NT services using py2exe (augmenting py2exe's rather simple  
dependency resolution mechanism), however I'm not sure if that is in  
public svn or not.  If the authors of the other freeze tools are  
interested, they can feel free to use modulegraph from py2app -- it is  
cross-platform code under MIT license, but I can dual-license if  
necessary (however I think it should be compatible with cx_freeze,  
py2exe, and Python itself).  The API is purposefully different than  
modulefinder, but it is close enough such that most of the work  
involved is just removing unnecessary kludges.

-bob