Thoughts on some stdlib modules

[Robert Kern]

Fredrik Lundh wrote:
> Robert Kern wrote:
> 
> 
>>>(the Linux distributors know how to do this: look for good stuff that's
>>>either actively maintained or simple and solid enough to live for a while,
>>>make sure the licenses are good enough, bundle the latest and greatest
>>>version, ship tested versions at regular intervals, update when necessary,
>>>and pass bugs and patches upstream.  why not use the same approach
>>>for Python's standard distribution?)
>>
>>The reason they don't is because making a distribution like this is a royal pain in the ass. It 
>>makes no sense to make the Python standard distribution like this. Only insane people do this. I'm 
>>one of them.
> 
> 
> with enough volunteers, and a distributed development environment, nobody
> needs to be insane.  if you try to channel everything through a small number of
> CPython developers, everyone will loose their sanity, and the users will suffer.

Agreed, which is why I think that proper package management and a 
CPyAN-type system is the way to go, not trying to stuff everything into 
a single distribution.

>>"Updating when necessary" is problematic and really requires proper package management. Python is 
>>not an OS and cannot control all of the little factors that make package management feasible for 
>>OS distributions to do.
> 
> 
> "updating when necessary" means "adding new components to the standard library
> when there are new stable releases available, and someone has time to run the test
> suite and contact the upstream provider if necessary".

Yes, that's the "royal pain in the ass" that I was referring to. Sounds 
easy, but it soaks up far more time than you (I) possibly thought it 
would. I think that pumping out a new release of the whole standard 
library every time one package gets updated is rather absurd and 
aggravates rather than ameliorates the problem.

Allow me to take an example that I'm sure you are familiar with (and can 
correct me if I get things wrong). You were once asked why you didn't 
use Numeric arrays as the memory store for PIL image objects. Your 
(good) reason was that Numeric was not part of the standard library and 
you didn't want to add a dependency on another 3rd party package. Now 
let's suppose that Numeric was included in this standard library and PIL 
wasn't. If you wanted to use Numeric arrays, you could not simply depend 
on the standard library; you would have to depend on specific releases 
of the standard library because Numeric is changing what with numarray 
and the impending Numeric/numarray unification (actually, we ought to 
talk about that[1]). What's more, the version numbers of the standard 
library that you would have to depend on would have no particular 
correlation with the versioning of the underlying component on which you 
are really depending. And then someone is going to want to use a package 
that requires stdlib < 2.5.1-1004 *and* a package that requires stdlib > 
2.5.1-1010.

CPyAN-type systems will, if they make good on their promises, avoid this 
hassle.

[1] As a side note, with the Numeric/numarray unification, the Numeric 
people are developing an array interface that will allow seamless and 
copyless interfacing between Numeric and other packages that either 
produce something array-like (e.g. PIL) or can consume something 
array-like (e.g. gui toolkits). With this protocol, PIL can support 
Numeric arrays (or anything else array-ish like whatever image classes 
in are in a gui toolkit) without needed to depend on the modules themselves.

   http://numeric.scipy.org/

Since PIL is one of the compelling reasons for such an interface, we'd 
like your input on it. If you have comments or questions, please let us 
know on the Numpy mailing list.

   http://lists.sourceforge.net/lists/listinfo/numpy-discussion

>>What's more, once you *do* have proper package management on the platforms that you care about, it 
>>doesn't matter what is or isn't in the standard distribution.
> 
> 
> which means that at the moment, the best way to get a Python module into the
> hands of users is to 1) provide windows installers that are compatible with the
> "official" distribution, and 2) make it easy for downstream providers to package
> your stuff, and leave it to them to take care of the rest.

At the moment, yes, more or less. However, neither the standard Windows 
and Mac installers are what I consider to be "proper package management."

> but if the library in the standard distribution doesn't really matter, it should be
> stripped down to minimum, to avoid overloading the CPython maintainers.

I agree, but the CPyAN systems have to come out of the vapour first.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter