[SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation

[David Cournapeau]

On Wed, Dec 30, 2009 at 3:36 AM, René Dudfield <renesd at gmail.com> wrote:
> On Tue, Dec 29, 2009 at 2:34 PM, David Cournapeau <cournape at gmail.com> wrote:
>> On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield <renesd at gmail.com> wrote:
>>
>>> Buildout is what a lot of the python community are using now.
>>
>> I would like to note that buildout is a solution to a problem that I
>> don't care to solve. This issue is particularly difficult to explain
>> to people accustomed with buildout in my experience - I have not found
>> a way to explain it very well yet.
>
> Hello,
>
> The main problem buildout solves is getting developers up to speed
> very quickly on a project.  They should be able to call one command
> and get dozens of packages, and everything else needed ready to go,
> completely isolated from the rest of the system.
>
> If a project does not want to upgrade to the latest versions of
> packages, they do not have to.  This reduces the dependency problem a
> lot.  As one package does not have to block on waiting for 20 other
> packages.  It makes iterating packages daily, or even hourly to not be
> a problem - even with dozens of different packages used.  This is not
> theoretical, many projects iterate this quickly, and do not have
> problems.
>
> Backwards compatibility is of course a great thing to keep up... but
> harder to do with dozens of packages, some of which are third party
> ones.  For example, some people are running pygame applications
> written 8 years ago that are still running today on the latest
> versions of pygame.  I don't think people in the python world
> understand API, and ABI compatibility as much as those in the C world.
>
> However buildout is a solution to their problem, and allows them to
> iterate quickly with many participants, on many different projects.
> Many of these people work on maybe 20-100 different projects at once,
> and some machines may be running that many applications at once too.
> So using the system pythons packages is completely out of the question
> for them.

This is all great, but I don't care about solving this issue, this is
a *developer* issue. I don't mean this is not an important issue, it
is just totally out of scope.

The developer issues I care about are much more fine-grained (corrent
dependency handling between target, toolchain customization, etc...).
Note however that hopefully, by simplifying the packaging tools, the
problems you see with numpy on 2.6 would be less common. The whole
distutils/setuptools/distribute stack is hopelessly intractable, given
how messy the code is.

>
> It is very easy to include a dozen packages in a buildout, so that you
> have all the packages required.

I think there is a confusion - I mostly care about *end users*. People
who may not have compilers, who want to be able to easily upgrade one
package, etc...

David