[Python-Dev] Re: [Distutils] Questions about distutils strategy

[Guido van Rossum]

> [Guido]
> > > It's not Mark's fault, it's Microsoft's fault.  If you don't do
> > > things the way MS wants you to, experienced Windows users will
> > > gripe, misunderstand what you do, etc.
> [Tim] 
> > Something just occurred to me:  MS's guidelines aren't arbitrary,
> > they actually have very good reasons.  In the case of putting all
> > an app's crucial info in the Registry, it's the only way to allow
> > a site administrator to set policy and site options remotely (an
> > admin can fiddle other machines' registries remotely).  This
> > works very well indeed when there's only "one copy" of an app on
> > a machine (or at most one copy "per user").
[Gordon]
> And actually, the business about separate subtrees for the 
> machine's configuration and the user's configuration is pretty 
> clever. MS doesn't explain it well, and it gets misused, but 
> when done right, it's a lot simpler than the maze of .xxxrc files 
> you sometimes find in other OSes.

I agree.  And I am guilty of not even try to find MS' explanation -- I
just looked in the registry at what other apps did and tried to mimic
that (plus what Mark had already done), without really knowing what I
was doing.  I now know a little better -- see the end of this message.

> In my Linux version, I went to the heart of the matter - 
> getpath.c. It occurs to me that getpath.c might do better to 
> follow a normal bootstrap process - ie,  create the absolute 
> minimal sys.path required to go to the next step. Then the 
> rest of what goes on in getpath.c could be written in Python. 
> Maybe that Python code needs to get frozen in (to prevent 
> bozos from destroying an installation by stepping on 
> getpath.py), but it would make it a lot easier to create 
> independent installations, and also reduce the variations 
> between platforms at the C level. (Then again, I've never heard 
> of anyone stepping on exceptions.py.)

Yes, this is exactly what was proposed in the thread on the Big Import
Rewrite.

> If some registry manipulation primitives were exposed (say, 
> through ntpath) that would mean that Windows developers 
> could (if they wanted) play by the MS rules with at least the 
> option of not stepping on each other.

That's a good idea.  These functions are already available through
Mark's win32api extension -- much of which will eventually (I hope
before 1.6 is out!) become part of the core distribution.

In the mean time, I've been thinking a bit more about how Python
should be using the Windows registry.  (It's clear to me that Python
should use the registry -- those who disagree can go build their own
Python distribution.)

The basic ideas of Python's current registry usage are sound: there's
a resource built into the DLL which is part of the key into the
registry used for all information.

The problem lies in which key is used.  All versions of Python 1.5.x
(1.5, 1.5.1, 1.5.2) use the same key!  This is a main cause of
trouble, because it means that different versions cannot peacefully
live together even if the user installs them into different
directories -- they will all use the registry keys of the last version
installed.  This, in turn, means that someone who writes a Python
application that has a dependency on a particular Python version (and
which application worth distributing doesn't :-) cannot trust that if
a Python installation is present, it is the right one.  But they also
cannot simply bundle the standard installer for the correct Python
version with their program, because its installation would overwrite
an existing Python application, thus breaking some *other* Python apps
that the user might already have installed.

(There's a solution for app builders who are willing to do a lot of
work -- you can change the registry key resource in the DLL.  For
example, Alice comes with its own version of Python 1.5.1 and it uses
"1.5.1-alice" as its registry key.  The Alice installer installs
Python in a subdirectory of the Alice installation directory and
points the 1.5.1-alice registry entries there.  The problem is that
this is a lot of work for the average app builder.)

I thought a bit about how VB solves this.  I think that when you wrap
up a VB app in, all the support code (mostly a big DLL) is wrapped
with it.  When the user runs the installer, the DLL is installed
(probably in the WINDOWS directory).  If a user installs several VB
apps built with the same VB version, they all attempt to install the
exact same DLL; of course the installers notice this and optimize it
away, keeping a reference count.  (Ignoring for now the fact that
those reference counts don't always work!)  If an app builty with a
different VB version is installed, it has a DLL with a different name,
and that is installed separately.  Other support files, I presume, are
dealt with in much the same way.  Voila, there's the theory.

How can we do something similar for Python?

A app written in Python should need to install only three or four
files:

- a driver EXE to start the app
- a copy of the Python DLL
- the Python library in an archive
- the app code in an archive

The latter two could be combined into a single archive, but I propose
that we use two archives so that the DLL and the Python library
archive can be shared between installations of independent Python apps
as long as they use the exact same Python version and don't need
additional 3rd party packages.  (I believe that Jim A's proposal
combines the archives with the EXE and the DLL, reducing the number of
files to two.  That's fine too.)

Is there a use for the registry here at all?  Maybe not.  (I notice
that VB seems to have a single registry entry, pointing to a DLL; all
other VB files also seem to live there.)

Complications:

- Some apps may need a custom extension module, which has to be
  installed as a PYD file.  So it seems that there needs to be a
  directory per app, and perhaps per version of the app (if the app
  distributor cares).

- Some apps need other, non-pyc files (e.g. data tables or help
  files); it would be handy if these could be stored in the archives as
  well.

- Some standard extension modules are in their own PYD files; these
  also need to be installed.  They aren't typically marked with a
  version, so perhaps a path directory per version of Python (if not per
  installed app) is wise.

- How to distribute an app that needs 3rd party stuff, e.g. Tcl/Tk, or
  PIL, or NumPy?  Their Python code can easily be wrapped up in another
  archive with a standard name incorporating a version number; but the
  required PYD and DLL files are a separate story.  (E.g. for Tkinter,
  you need _tkinter.pyd which links against tcl80.dll.)  Basically the
  same solution as for standard PYD files can work; the needed DLL files
  can be installed either systemwide (if they have a reliable version
  number in their name, like tcl80.dll) or in the per-app or per-package
  directory (like NumPy).

- Presumably, the archives will contain PYC files only.  This means
  that tracebacks will not show source code, only line numbers.  For Jim
  A, this is probably exactly what he wants (if the user gets a
  traceback, his "robust app" has miserably failed, and he takes it in
  pride that this doesn't happen).  But for some others, access to the
  sources could be essential.

  For example, I might want to distribute IDLE using this mechanism;
  users of IDLE who are curious about the standard library (or about
  IDLE itself) should be able to open the source for an arbitrary module
  (and maybe even edit it, although that's not a priority and perhaps
  should even be discouraged).  Library source access is an important
  feature of the IDLE debugger as well.

  A way out for IDLE is to install a classic distribution of the Python
  library sources, into the filesystem at an IDLE specific location.
  Other apps, with only the need for source code in tracebacks, might
  choose to to have the PY files in the archives sitting next to the PYC
  files, and somehow the traceback mechanism should be accessing the
  archive to get a hold of the source.

And yes, I realize that Jim A's latest offering solves most of these
problems to a large extent -- well done.  (Jim, would you care to
comment on the issues that you don't address?  Will you address them
in a future version?)

Final notes:

There are two different problems here.  One is how to distribute
Python apps robustly to end users who don't particular care about
Python.  This is Jim A's problem (and he has a solution that works for
him).  In general the solutions here try to isolate the installed app
from other Python installations.  I'm proposing that at least the DLL
and the Python library archive can probably be shared between apps
without reducing robustness if we keep track more carefully of version
numbers.

The other problem is how to distribute packages of Python and
extension modules for use by Python users.  These typically need to
drop into some existing Python installation.  This is Paul Dubois'
problem with NumPy (amongst others) and is the current focus of the
distutil SIG.

However I believe that there could be a lot of common infrastructure
that would help us create better solutions for both problems.  For
package distribution, common infrastructure (a.k.a. standards) is
essential.  For app distribution, common infrastructure isn't so
important (since the solutions strive for total isolation, there's no
problem if different apps use solutions).  However, this changes when
app creators want to distribute robust self-sufficient apps that use
3rd party packages -- then the 3rd party packages must allow being
packaged up using the app distribution creator of choice.

Solving this compound problem (creating package distributions that can
be redistributed easily as part of robust Python app distributions)
should be an important goal for the infrastructure we're building
here.  The Big Import Rewrite ought to add this to its list of
objectives if it isn't already on it.  My guess is that the solution
for this compound problem will increase the dependency of app
distribution tools on the package distribution infrastructure; which
to me seems like a Good Thing because it would lead to more code
sharing.

--Guido van Rossum (home page: http://www.python.org/~guido/)