[Python-3000] Proposal: No more standard library additions

[Phillip J. Eby]

At 06:58 PM 10/15/2006 +0200, Martin v. Löwis wrote:
>Phillip J. Eby schrieb:
> > Last, but not least, the distutils suffer from an excess of policy
> > flexibility in areas where flexibility is undesirable from a community
> > perspective.  There should be "one obvious way" to do a lot of things
> > that the distutils provide a ridiculous number of ways to do (e.g.
> > insane project directory layouts including package-by-package directory
> > mapping).  This is not something that affects the distutils themselves
> > much, but it essentially prevents the distutils from improving in
> > certain ways because there are so few assumptions one can make about its
> > use.
>
>While each of these points might be true, I also believe they can
>be fixed. For example, it would be possible to deprecate-then-remove
>a non-standard package mapping. I'm not sure how many packages actually
>use such a mapping,

They are not common in published packages (although PIL is a notable 
exception).  But they often are in use in unpublished packages, judging by 
distutils-sig threads inquiring about the use of more esoteric directory 
layouts with setuptools.  Setuptools tries hard to support arbitrary 
package mappings, but there are some things that just won't work, like 
making the in-development source code available on sys.path.


> >> Actually, it is possible to add a new compiler to some Python
> >> installation, so that users can pass it to the --compiler option,
> >> and distutils packages don't have to be modified for that.
> >> You just have to extend the dictionary
> >> distutils.ccompiler.compiler_class.
> >
> > Yes, and how do you accomplish *that*?  You have to either modify some
> > Python code (e.g. sitecustomize) or use a big -c to python when running
> > the setup script.
>
>There is a third option: write a .pth file, and put that on the path.

Which means you are now importing the distutils whenever Python is started, 
just as in the sitecustomize case.


> > With entry point extension, it is only necessary to have an extension
> > installed on sys.path; it is not necessary to run some code to do the
> > registration.
>
>One thing that bothers me about setuptools is the strange, non-intuitive
>terminology. What is "entry point extension"? If you'd call it "ini
>files listing command names", I could understand much better.

When I name a new thing, I give it a new name.  If I called it something 
you merely *thought* you understood, I would have a much harder time 
correcting any misunderstanding.  It is intended that you ask "what is 
that?", because then there is some chance you will listen to my answer, or 
that you will investigate to find the answer yourself.  :)

Terminology is also a group/social engineering tool, as well as an 
educational engineering tool.  In the same way that such documents as the 
Zen of Python create a community vocabulary of concepts such as EIBTI, so 
too do setuptools terms create a community vocabulary by which its users 
can recognize each other's level of understanding.  You can observe this in 
blog discussions, hallway conversations at PyCon, etc., where the idea that 
"entry points are cool" is like a little secret handshake among the 
informed, and the opportunity to *explain* them to outsiders provides an 
opportunity for social reward.

A lot of people leave these kinds of things to chance; I consider them as 
much a part of the design of a total system, as anything in the actual 
code.  (See also WSGI "middleware" for a similar social phenomenon, and 
Kathy Sierra's "Creating Passionate Users" blog for general discussion of 
the concepts involved in creating a passionate user community.)

Certainly it is true that there are people who do not care about these 
things or see them as counterproductive.  However, if I am doing "community 
engineering", I do not allow this to get in the way, as those people are 
simply not a "compatible platform" for the socialware I'm creating.  There 
are enough people who *will* run socialware that there is always a viable 
and even vital community for it.


> > Adding setuptools-style plugin ability to the distutils is very simple,
> > and doesn't require a wholesale reorganization.  After all, setuptools
> > manages to do it!
>
>Ok, if that's the most desired property, I think it should be added
>(provided somebody is willing to contribute, of course); one doesn't
>have to wait for 3.0 to add it.

Of course, and I've already said this could be done in the 2.x series.  It 
doesn't even need the whole of setuptools to be incorporated into the 
stdlib; the pkg_resources module suffices.  (Of course, there would need to 
be additions to support building and installing the entry point data.)


> > However, there are areas of build functionality that
> > the distutils does not cover or provides incomplete coverage for, such
> > as configuration and build of more complex C libraries and programs,
> > shared libraries, documentation production, and so on, that the existing
> > architecture of the distutils doesn't really encompass.
>
>I agree it doesn't cover it. I don't see why it isn't "easy" to add
>it (where "easy" really means "as easy doing it stand-alone").

If you look at how much distutils infrastructure ends up being reinvented 
in specific distutils extension, because the original wasn't customizable 
in a useful way, you would see that there is in many respects very little 
difference between adding something to the distutils, and "doing it 
standalone".  Distutils' lower layers encode a bit too many assumptions 
from its upper layers, and it's rarely customizable in the places you need 
it to be.  But even if you have the ability to change the distutils itself, 
you may be stuck because changing these assumptions means breaking the 
existing distutils, but not having any decent way to *discover* that you 
have done so via tests.

Having done massive extensions to the distutils, I think I'm qualified to 
say that, if I had the choice, I would've built something that was merely 
API-compatible with the distutils, borrowing code and ideas from the 
distutils, rather than layering on top of it.

Of course, I didn't have that choice, because one of my design goals was to 
"embrace and extend" by supporting automated installation of a large number 
of existing packages.  However, with setuptools' current marketshare, an 
API-compatible version of setuptools that wasn't based on the current 
distutils architecture could likely be quite successful.


> > Meanwhile, the higher-level layer is based on the idea of dependency
> > between sequentially invoked commands, rather than dependency between
> > things being built.  This makes it difficult to deal with concepts like
> > having a bunch of reST release note docs combined with a LaTeX
> > documentation build.
>
>This I don't understand. Couldn't there be a subcommand of "build" that
>creates the documentation? Of course, that command would have to do
>the time-stamp checking, and use whatever build machinery is appropriate
>(make, ant, scons, ...).

What I'm saying is that the reason there is no build_docs command is 
because documentation build processes aren't sufficiently uniform to fit 
within the distutils' simple command-oriented framework.  Most use of 
distutils extensions "in the wild" are trying to solve issues related to 
this.  For example, Twisted tries to have C extensions that are optionally 
built, based on whether a compiler is available and what libraries are 
available.  This is another scenario that doesn't really work in the 
distutils' concept framework.



> > So, distutils' extensibility *could* be improved dramatically in the 2.x
> > line by adding setuptools-style plugin entry points.  However, adding
> > any *new* build functionality to the distutils is probably going to be
> > better off being based on something that offers an extensible
> > object-dependency layer, ala zc.buildout.
>
>I don't mind adding a time-stamp-or-content-based dependency
>infrastructure to the standard library, either, and it might be useful
>to reimplement the existing build commands on top of that
>(for example, it bothers me that the standard setup.py doesn't take
>  header file dependencies into account, for the modules of the standard
>  library).
>That's still no reason to drop distutils.

I'm not sure what you mean.  Do you mean that there should be a set of 
modules *called* distutils?  I agree with that, and that it should offer 
something closely resembling the existing API, so that a large subset of 
existing packages could continue to work.

But do you also mean that those "distutils" modules must be created only by 
modifying the existing ones?  In that case I disagree.

I personally don't believe that the distutils code can be rehabilitated 
without clean-rooming it for testability.  The lifecycle for effective 
testing against the *enormous* variations in how it is used, means that to 
make any substantial change you must rule out backward compatibility except 
for a strictly defined feature set.  And if you do that, there is nothing 
stopping you from simply implementing a testable and less-fragile 
architecture from the beginning, and then putting a limited 
backward-compatibility layer on top of that.  The result would be far less 
fragile than building new floors on a shaky foundation, as setuptools has 
tried to do.


> > (Which reminds me.  Does anybody have any idea why the distutils compile
> > modules in build/?  There seems to be no point to this, since any
> > installation of the modules is going to need to recompile them.  Even
> > bdist operations end up recompiling them in a different location!)
>
>It was added to build_py.py with this checkin:
>
>r17820 | gward | 2000-10-02 04:19:04 +0200 (Mon, 02 Oct 2000) | 8 lines
>
>Added the ability to do byte-compilation at build time, currently off
>by default (since compiling at install time works just fine).  Details:
>   - added 'compile' and 'optimize' options
>   - added 'byte_compile()' method
>   - changed 'get_outputs()' so it includes bytecode files
>A lot of the code added is very similar to code in install_lib.py;
>would be nice to factor it out further.
>
>Notice that it is meant to be off by default, and, indeed, for me,
>it is off. Not sure why you are getting .pyc files in your
>build directory.

That's not what I said.  I asked why it even has this capability at 
all.  The checkin comment seems to agree that it is useless, "since 
compiling at install time works just fine".  :)