[Distutils] Other ideas from today's packaging meetup at EuroPython

[Nick Coghlan]

On 26 July 2014 23:49, Steve Dower <Steve.Dower at microsoft.com> wrote:
> "Will that also allow me to put ‘macports’ or ‘homebrew’ in there you can
> create an Cython-0.20.1-cp27-none-lmacosx_x86_64-macports_10.10.whl
> distribution?"
>
> Just how many wheels are people going to have to publish? Who has that many
> dev machines? Without a build farm, I can't see this being more helpful than
> frustrating (and the build farm is going to need a way to automatically get
> non-Python dependencies, which is not our business, so that's a long way off
> IMO).

Yeah, I actually suspect feeding into a PyPI PPA (Ubuntu) and COPR
(Fedora/RHEL/CentOS) may be a more sensible model for the Linux
communities, with conda/binstar as a way to become *completely*
independent of the distro package managers (i.e. handling the delivery
of the Python runtime and other external dependencies as well, not
just the Python modules). That way we'd be offloading the task of
providing the appropriate external dependencies to Launchpad/COPR for
the integrated libraries, and to binstar for the distro independent
ones.

Potentially enabling that kind of approach is one of the reasons I've
emphasised a redistributor friendly design for metadata 2.0 - not
everybody wants to do their own system integration the way SaaS
developers do, and one of the advantages of having both pip and conda
available as tools is that we can stake out two different positions on
the "redistributor friendliness" scale at the same time:

- pip can focus on managing packages *within* an existing Python
installation. It's applicable regardless of how you obtained that
Python installation in the first place, and hence plays nice with
redistributors (who each have their own way of providing Python and
external dependencies)
- conda can focus on being a compelling "cross platform platform" for
delivery direct to end users, bypassing the other redistributor
channels

So pip becomes a tool for cooperating with redistributors, while conda
becomes a tool for competing with them. I actually had some serious
discussions with the conda folks at SciPy about facilitating closer
collaboration between them and PyPA, and came to an explicit
conclusion that it made sense for conda to remain a distinct community
that works-with-but-is-not-part-of PyPA - not only is conda itself a
complex beast with multiple subcommunities, but it's also in far more
direct competition with the redistributor channels than pip is :)

(Hmm, I don't think I passed along the details of my SciPy keynote to
the list. For anyone interested, the talk is at
http://pyvideo.org/video/2785/python-beyond-cpython-adventures-in-software-dis
and notes are at
https://bitbucket.org/ncoghlan/misc/src/default/talks/2014-07-scipy/talk.md?at=default)

> An organisation building their own wheels in-house and configuring their
> machines to use an OS specific index/wheelhouse sounds like a much more
> feasible idea that works now and could do with more traction and
> encouragement. (Someone mentioned earlier that they do this - apologies for
> forgetting who.)

Yeah, this is the model we were intending to encourage when we blocked
wheel uploads to PyPI for platforms other than Windows and Mac OS X.
It avoids a lot of the gnarly dependency management problems.

Unfortunately, there's no generally accepted technique for automating
it at this point (Although there's an issue open suggesting the
addition of a feature along these lines to devpi:
https://bitbucket.org/hpk42/devpi/issue/110/build-put-wheel-for-pypi-released-package)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia