[Distutils] formencode as .egg in Debian ??

[Phillip J. Eby]

At 08:12 PM 11/23/2005 +0100, Matthias Urlichs wrote:
>Hi,
>
>Phillip J. Eby:
> > I'm thinking that perhaps I should add an option like
> > '--single-version-externally-managed' to the install command so that you
> > can indicate that you are installing for the sake of an external package
> > manager that will manage conflicts and uninstallation needs.  This would
> > then allow installation using the .egg-info form and no .pth files.
> >
>You might shorten that option a bit. ;-)  I agree that this would be a
>good option to have.

I try to use very long names for options that can have damaging effects if 
used indiscriminately.  A project that's installed the "old-fashioned way" 
(which is what this does, apart from adding .egg-info) is hard to uninstall 
and may overwrite other projects' files.  So, it is only safe to use if the 
files are being managed by some external package manager, and it further 
only works for a single installed version at a time.  So the name is 
intended to advertise these facts, and to discourage people who are just 
reading the option list from trying it out to see what it does.  :)


> > >People will often inspect sys.path to understand where Python
> > >is looking for their code.
> >
> > As I pointed out, eggs give you much better information on this.
>
>The .egg metadata does. That, as you say, is distinct from the idea of
>packaging the .egg as a zip file. Most likely, one that includes .pyc
>files which were byte-compiled with different file paths; That causes no
>problems whatsoever ... until you get obscure ideas like trying to step
>through the code with pdb, or opening it in your editor to insert an
>assertion or a printf, trying to figure out why your code breaks.  :-/

This is actually what the .egg-info mode was designed for.  That is, doing 
development of the project.  A setuptools-based project can run "setup.py 
develop" to add the project's source directory to sys.path, after 
generating an .egg-info directory in the project source if necessary.  This 
allows you to do all your development right in your source checkout, and of 
course all the file paths are just fine, and the egg metadata is available 
at runtime.  You can then deploy the project as an .egg file or directory.

(Also, for the .egg directory format, note that easy_install recompiles the 
.pyc/.pyo files so their paths *do* point to the .egg contents instead of 
the original build paths.  The issues with zipfiles and precompiled .pyc 
files are orthogonal to anything about setuptools, eggs, etc.; they will 
bite you in today's Python no matter what's in the zipfile or who 
precompiled the .pyc files.  I do have some ideas for fixing both of these 
problems in future versions of Python, but they're rather off-topic for all 
the lists we are currently talking on.)


>That's not exactly negotiable. Debian has a packaging format which
>resolves generic installation dependencies on its own. Therefore it
>cannot depend on Python-specific .egg metadata. Therefore we need a way
>to translate .egg metadata to Debian metadata.

Yes, that's precisely what I was suggesting would be helpful.  As Vincenzo 
already mentioned, the egg metadata is a good starting point for defining 
the Debian metadata.  I'm obviously not proposing changing Debian's 
metadata system.  Well, maybe it wasn't *obvious* that I wasn't proposing 
that, but in any case I'm not.  :)


> > I remain concerned about how such packages will work with namespace
> > packages, since namespace packages mean that two different distributions
> > may be supplying the same __init__.py files, and some package managers may
> > not be able to deal with two system packages (e.g. Debian packages, RPMs,
> > etc.) supplying the same file, even if it has identical contents in each
> > system package.
> >
>Debian packaging has a method to explicitly rename a different package's
>file if it conflicts with yours ("dpkg-divert"; it does _not_ depend on
>which package gets installed first). IMHO that's actually superior
>randomly executing only one of these files, since you are aware that
>there is a conflict (the second package simply doesn't install if you
>don't fix it), and thus can handle it intelligently.

The two kinds of possible conflicts are namespace packages, and 
project-level resources.

A namespace package is more like a Java package than a traditional Python 
package.  A Java package can be split across multiple directories or jar 
files; it doesn't have to be all in one place.  Thus you can have lots of 
jars with org.apache.* classes in them.

Python, however, requires packages to have an __init__.py file, and by 
default the entire package is assumed to be in the directory containing the 
__init__.py file.  However, as of Python 2.3, the 'pkgutil' module was 
introduced in the Python standard library which allowed you to create a 
Java-style "namespace package", automatically combining package directories 
found on different parts of sys.path.  So, if in one sys.path directory you 
had a 'zope.interface' package, and in another you had a 'zope.publisher' 
package, these would be combined, instead of the first one being treated as 
if it were all of 'zope.*', and the second being completely 
ignored.  However, *each* of the subpackages needs its own zope/__init__.py 
file for this to work.

So, the issue here is that if you install two projects that contain zope.* 
packages into the *same* directory (e.g. site-packages), then there will be 
two different zope/__init__.py files installed at the same location, even 
though they will have the same content (a short snippet of code to activate 
the namespace mechanism via the pkgutil module or via setuptools' 
pkg_resources module).

To date, there are only a small number of these namespace packages in 
existence, but over time they will represent a fairly large number of 
*projects*.  As I go through the breakup of the PEAK meta-project into 
separate components, I expect to have a dozen or so projects contributing 
to the peak.* and peak.util.* namespace packages.  Ian Bicking's Paste 
meta-project has a paste.* namespace package spread out in two or three 
subprojects so far.  There has been some off-and-on discussion about 
whether Zope 3 will move to eggs instead of their own zpkg tool (which has 
issues on Windows and Mac OS that eggs do not), and in that case they will 
likely have a couple dozen components in zope.* and zope.app.*.

So, for the long-term solution of wrapping Python projects in Debian 
packages, the namespace issue needs to be addressed, because renaming each 
project's zope/__init__.py or whatever isn't going to work very 
well.  There has to be one __init__.py file, or else such projects need to 
be installed in their own .egg directories or zipfiles to avoid collisions.

The second collision issue with --single-version-externally-managed is 
top-level resource collisions.  Some existing projects that are not 
egg-based manipulate their install_data operation in such a way that they 
create files or directories in site-packages directly, rather than inside 
their own package data structures.  Setuptools neither encourages nor 
discourages this, because it doesn't cause any problems for any egg layout 
except the .egg-info one -- and the .egg-info one was originally designed 
to support development, not deployment.  In the development scenario, any 
such files are isolated to the source tree, and for deployment the .egg 
file or directory keeps each projects' contents completely isolated.

So, what I'm saying is that putting all projects in the same directory (as 
all "traditional" Python installations do) has some inherent limitations 
with respect to namespace packages and top-level resources, and these 
limitations are orthogonal to the question of egg metadata.  The .egg 
formats were created to solve these problems (including clean upgrades, 
multi-version support, and uninstallation in scenarios where a package 
manager isn't usable), and so the other features that they enable will be 
increasingly popular as well.

In other words, as people make more use of PyPI (because they now really 
*can*), more people will put things on PyPI, and the probability of package 
name conflicts will increase more rapidly.  The natural response will be a 
desire to claim uber-project or organizational names (like paste.*, peak.*, 
zope.*, etc.) putting individual projects under sub-package names.  (For 
example, someone has already argued that I should move RuleDispatch's 
'dispatch' package to 'peak.dispatch' rather than keeping the top-level 
'dispatch' name all to myself.)

So, I'm just saying that using the --single-version-externally-managed 
approach requires that a package manager like Debian grow a way to handle 
these namespace packages safely and sanely.  One possibility is to create 
dummy packages that contain only the __init__.py file for that namespace, 
and then have the real packages all depend on the dummy package, while 
omitting the __init__.py.  So, perhaps each project containing a 
peak.util.* subpackage would depend on a 'python2.4-peak.util-namespace' 
package, which in turn would depend on a 'python2.4-peak-namespace' 
package.  It's rather ugly, to say the least, but it would work as long as 
upstream developers never put anything in namespace __init__.py files 
except for the pkg_resources.declare_namespace() call.

(By the way, since part of an egg's metadata lists what namespace packages 
the project contains code or data for, the generation of these dependencies 
can be automated as part of the egg-to-deb conversion process.)

Or, of course, the .egg directory approach can also be used to bypass all 
collision issues, but this brings sys.path and .pth files back into the 
discussion.  On the other hand, it can possibly be assumed that anything in 
a namespace package can be used only after a require() (either implicit or 
explicit), so maybe the .pth can be dropped for projects with namespace 
packages.  These are possibilities worth considering, since they avoid the 
ugliness of creating dummy packages just to hold namespace __init__.py files.