[Distutils] [buildout] path usage and proposal

[Jim Fulton]

This proposal is limited to buildout and how buildout handles paths.

Jim

On Jul 6, 2009, at 7:57 PM, David Lyon wrote:

>
> Good Evening Jim,
>
> On Mon, 6 Jul 2009 11:48:25 -0400, Jim Fulton <jim at zope.com> wrote:
>
>> ..
>> Paths are also used by running scripts to search for packages.  When
>> buildout generates a script, it generates a search path based on the
>> working set.  Note that it adds the working set locations to the base
>> path that the script starts with.
>
> True
>
>> For existing behavior, we can work smarter by searching the base path
>> + the current install paths for installed distribution and passing an
>> empty search path the index used to look for uninstalled  
>> distributions.
>
> That's already how it works.
>
>> Note that not every distribution found on the base path is
>> importable.  In particular, an egg may be found on the base path, but
>> the egg itself must be added to the Python path to be importable.
>> The base path is used for 2 things:
>>
>> - A place to look for installed distributions
>> - A place to import things from at run time.
>
> Yes, and the importing is done via .PTH files...
>
>> It would be cleaner to separate these uses, although, for usability,
>> it might be simpler to keep them together.  There's probably little
>> harm in including a path in sys.path that *only* includes eggs.
>
> The only way to enforce that is to change the interpretor itself..
>
> Within site.py... otherwise.. enforce it with some sort of packaging
> tool.... lol
>
> Are you suggesting something like a directory "SITE-EGGS" ?
>
>> So hear's a proposal:
>>
>> 1. Add an include-site-packages option, defaulting to true.  If  
>> false:
>>   set the base path to those paths not added by running site.py.
>>   We'll get this by evaling the output of:
>>
>>     python -Sc "import sys; print (repr(sys.path))"
>>
>>   The base path will be used for searching for distrbutions *and* for
>>   computing the run-time path.  The script preamble will become
>>   someting like::
>>
>>      sys.path[:] = ... # set sys.path to the base path
>>      for path in ...: # iterate over working set using expression we
>> use now
>>          if path not in sys.path:
>>              sys.path.insert(0, path)
>>
>>   Note:
>>   a. We completely replace sys.path
>>   b. If a distro's location was already in the base path, we don't
>>      insert it.  This will help avoid accidently putting a base path
>>      ahead of egg-supplied paths.
>
> It seems clunky....
>
>> 2. Add a search-path option.  By default, it will be the base path.
>>   It can be manipulated in 2 ways:
>>
>>   - It can be set.  If this is done, the base path will be ignored.
>>
>>   - It can be incremented, for example, to cherry-pick the foo  
>> package
>>     on ubuntu:
>>
>>       search-path += /usr/share/pycentral/foo/site-packages
>>
> ..
>
>> Thoughts? Questions?
>
> All the functionality you are asking for already exists within python
> already... afiak there's already provision to add extra paths and do  
> all
> these extra things.
>
> Under linux, there is this great place to put packages ..
> /usr/local/pythonx.x/.. and under windows there's also a spot in
> \Documents and Settings\User\My Documents.
>
> EasyInstall will let us install there if we ask it to....
>
> The big problem is that we/users aren't doing any of this all that
> well at the moment because we/they forget the command lines.
>
> Maybe I don't know buildout too well... so I don't know the issue..
>
> Maybe you just need a better tool for installing packages under
> buildout.. maybe I could modify my python package manager in some
> way to support buildout better....
>
> http://sourceforge.net/projects/pythonpkgmgr/
>
> In a future version of the python package manager, I will
> probably add capabilities to move packages from one place
> to another... haha... generate buildout files..
>
> None of these things really require anything extra from
> distutils than is already there....
>
> Regards
>
> David
>
>
>
>
>

--
Jim Fulton
Zope Corporation