[Distutils] Importable wheels using distlib/distil

[Nick Coghlan]

On Sat, Mar 30, 2013 at 8:52 AM, PJ Eby <pje at telecommunity.com> wrote:
> On Fri, Mar 29, 2013 at 4:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> You don't lose the place where you want the inserts to happen. Without
>> the marker, you end up having to come up with a heuristic for "make
>> insertions here" and that gets messy as you modify the path
>> (particularly since other code may also modify the path without your
>> involvement).
>
> But at least you can tell exactly what the order is by inspecting sys.path.

Agreed that introspection support for metapath importers and path
hooks is currently lacking.

>> Using a path hook to say "process these additional path
>> entries here" cleans all that up and lets you precisely control the
>> relative precedence of the additions as a group, without needing to
>> care about their contents, or the other contents of sys.path.
>
> Then can't you just bracket the entries with "<start-versioned>" and
> "<end-versioned>"  then?  ;-)
>
> (That would actually work right now without a path hook, since strings
> that refer to non-existent paths are optimized away even in Python
> 2.5+.)

Sure, you *could*, but then you're effectively embedding a list inside
another list and it would be a lot cleaner to just say "at this point,
go consult that other dynamically modified list over there".

> Also, btw, pkg_resources's sys.path munging is far more aggressive
> than anything with wheels needs to be, because pkg_resources is
> *always* working with individual eggs, not plain install directories.
> If you are only using standalone wheels to handle alternate versions,
> then you *definitely* want those standalone wheels to override other
> things, so a strategy of always placing them immediately before their
> containing directory is actually safe.  (In effect, treating the
> containing directory as the insertion marker.)

Yes, that was the other notion I had - insert the extra directory
immediately before the directory where the metadata was located.

> So, if the standalone wheel is in site-packages, then activating it
> would place it just ahead of site-packages -- i.e., the same place
> you're saying it should go.
>
> And as I believe I mentioned before, a single marker for insertion
> points doesn't address the user site-packages, app directory or app
> plugin directories, etc.  AFAICT your proposal only addresses the
> needs of system packages, and punts on everything else.

No, it doesn't - all it does is provide a clear demarcation between
the default system path provided by the interpreter and the ad hoc
runtime modifications used to gain access to additional packages that
aren't available by default. At the moment, if you print out sys.path,
you have *no idea* what it originally looked like before the
application started modifying it (process global shared state is
always fun that way).

*What* directories can be added is then entirely up to the manipulation API.

I guess we could easily enough snapshot the path during the
interpreter initialisation to help diagnose issues with post-startup
modifications.

> At the least, may I suggest that instead of using a marker, if you
> must use a path hook, simply install the path hook as a wrapper for a
> specific directory (e.g. site-packages), and let it process insertions
> for *that directory only*, rather than having a single global notion
> of "all the versioned packages".

It's not really "all the versioned packages", it's "all the packages
found through this particular path manipulation API".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia