[Distutils] PEP 517: Build system API

[Nick Coghlan]

On 29 November 2016 at 21:41, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Thu, Nov 24, 2016, at 11:10 PM, Paul Moore wrote:
> But even without that, they're not ideal. I routinely have development
> installs of quite a lot of different packages. If each one of those is
> an entry on sys.path from a .pth file, there's a performance penalty, as
> every 'import qux' has to stat() in dozens of directories which don't
> contain qux at all:
>
> stat('.../foo/qux.py')
> stat('.../foo/qux/__init__.py')
> stat('.../foo/qux.so')
> stat('.../bar/qux.py')
> stat('.../bar/qux/__init__.py')
> ...
> stat('.../qux/qux.py')   # At last!

The Python 3 import system (and the importlib2 backport) cache
directory listings rather than making repeated stat calls, so the
performance penalty of longer paths is dramatically reduced relative
to the old "n stat calls per directory" approach. (For NFS based
imports with network round trips involved, we're talking multiple
orders of magnitude reductions in application start-up times)

> If you put tooling scripts/modules in the root of your project
> directory, it also increases the chance of name clashes.
>
> Maybe we should look into something more like symlinks for doing
> editable installs, so a file called '.../site-packages/qux.pylink' would
> contain the path to the qux module/package. This would only be read on
> 'import qux', so it wouldn't affect other imports at all.

The main challenge is to do this kind of thing without breaking PEP
376 (the directory based installation metadata layout), and without
coupling the adoption period to CPython adoption cycles (where 2.7
hasn't even completely supplanted 2.6, let alone the 3.x series taking
over from 2.x).

And executable .pth file *could* be used to bootstrap a new system,
but then the interaction with setuptools may get even more complicated
due to relative import orders (as far as I know the order of
processing for pth files is entirely arbitrary).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia