[Python-Dev] Internal documentation for egg formats now available

[M.-A. Lemburg]

Phillip J. Eby wrote:
> At 06:47 PM 4/27/2006 +0200, M.-A. Lemburg wrote:
>> Just read that you are hijacking site.py for setuptools'
>> "just works" purposes.
> 
> "hijacking" isn't the word I'd use; "wrapping" is what it actually
> does.  The standard site.py is executed, there is just some pre- and
> post-processing of sys.path.

Whatever you call it: you're fiddling with the standard
Python distribution again, without letting your users
know about it.

Note that sitecustomize.py is intended for site local
changes that admin wants to implement.

>> Please be aware that by allowing .pth files in all PYTHONPATH
>> directories you are opening up a security hole - anyone with
>> write-permission to one of these .pth files could manipulate
>> other user's use of Python.
> 
> FUD.  Write access to a PYTHONPATH-listed directory already implies
> complete control over the user's Python environment.  This doesn't
> introduce any issues that aren't implicit in the very existence of
> PYTHONPATH.

Really ? How often do you check the contents of a .pth
file ? How often do you check sys.path of a running
interpreter to see whether some .pth file has added extra
entries that you weren't aware of ?

Note that I was talking about the .pth file being
writable, not the directory. Even if they are not
writable by non-admins, the .pth files can point
to directories which are.

Combine that with the -m option of the Python interpreter
and you have the basis for a nice exploit.

>> That's the reason why only site-packages .pth files are
>> taken into account, since normally only root has write
>> access to this directory.
> 
> False.  On OS X, Python processes any .pth files that are found in the
> ~/Library/Python2.x/site-packages directory. 

Hmm, that was new to me. OTOH, it's a very special case
and falls under the "normally only root" clause :-) -
Mac users are often enough their own admins; I suppose
that's why Jack added this:

            if sys.platform == 'darwin':
                # for framework builds *only* we add the standard Apple
                # locations. Currently only per-user, but /Library and
                # /Network/Library could be added too

> (Which means, by the way,
> that OS X users don't need most of these hacks; they just point their
> install directory to their personal site-packages, and it already Just
> Works.  Setuptools only introduces PYTHONPATH hacks to make this work on
> *other* platforms.)
>
>> The added startup time for scanning PYTHONPATH for .pth
>> files and processing them is also not mentioned in the
>> documentation. Every Python invocation would have to pay
>> for this - regardless of whether eggs are used or not.
> 
> FUD again.  This happens if and only if:
> 
> 1. You used easy_install to install a package in such a way that the
> .pth file is required (i.e., you don't choose multi-version mode or
> single-version externally-managed mode)
> 
> 2. You include the affected directory in PYTHONPATH
> 
> So the idea that "every Python invocation would have to pay for this" is
> false.  People who care about the performance have plenty of options for
> controlling this.
>
> Is there a nice HOWTO that explains what to do if you care more about
> performance than convenience?  No.  Feel free to contribute one.

Here's a HOWTO to optimize startup time, without loosing
convenience:

* build Python with all builtin modules statically linked
  into the interpreter

* setup PYTHONPATH to just include the directories you
  really care about

* avoid putting any ZIP archives on PYTHONPATH, except
  for python24.zip (there are better alternatives; see
  below)

* avoid using .pth files

If you run a lot of scripts, you'll also want to do keep
this in mind:

* start Python with option -s (avoids importing site.py
  and among other things, parsing .pth files), if you're
  running a script

More startup time can be saved by using an approach like
the one described in mxCGIPython which adds the complete
Python stdlib to the interpreter binary, effectively
turning the Python installation into a single file:

http://www.egenix.com/files/python/mxCGIPython.html

>> I really wish that we could agree on an installation format
>> for package (meta) data that does *not* rely on ZIP files.
> 
> There is one already, and it's used if you select single-version
> externally-managed mode explicitly, or if you install using --root.

No, I'm talking about a format which has the same if not
more benefits as what you're trying to achieve with the
.egg file approach, but without all the magic and hacks.

It's not like this wouldn't be possible to achieve.

>> All the unnecessary magic that you have in your design would
>> just go away - together with most of the issues people on this
>> list have with it.
> 
> Would you settle for a way to make a one-time ~/.pydistutils.cfg entry
> that will make setuptools act the way you want it to?  That is, a way to
> make setuptools-based packages default to
> --single-version-externally-managed mode for installation on a given
> machine or machine(s)?  That way, you'll never have to wonder whether a
> package uses setuptools or not, you can just "setup.py install" and be
> happy.

Not really.

I would like Python to remain a place that's free
of unnecessary magic in the first place, not by having to
configure setuptools to disable it's magic.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 27 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::