[Distutils] User Viewpoint: Packaging History, Confusion and User Advice

Wed Apr 24 16:55:31 CEST 2013

Hi everybody,

because there is no central place to look at and probably not all decisions
can be found in written form (like the abandoning of the fellow ship list,
when and why Tarek Ziadé decided to work on other things then
distribute/distutils2 and so on) I thought about writing a short summary of
the current state of packaging efforts as I understand it. I hope it has
value for people who need to make decisions in packaging their own projects
as well as help you packaging developers to see it a little bit from your
user's point of view.

All of it might not be correct or it might be incomplete. It simply
reflects my current state of investigation about that topic. I also hope
nobody gets angry, if he sees things differently or considers his effort
not embraced enough.

Short history about my reasons for this thread
---------------------------------------------------------------

End of last year I started a Python project and wanted to make sure users
can install my project without much effort. The main documentation of
Python 2.7 confused me deeply and it only talked about distutils and
setuptools. I simply wanted to add the required other packages, that they
can be automatically installed with my project, without the user manually
installing anything (which already let to problems in my project pre-alpha).

For some reason I ran into the pip installer and added, thanks to the
distutils-sig list [1] , the `install_requires` parameter to the setup()
function. In conclusion my problem was solved, although I still didn't
understand anything!

Concluding that being confused might mean that other people are also
confused, I started an effort to change the distutils part of the core
documentation [2][3]. I hoped for some guidance, but the basic problem for
that initiative was that I simply didn't know enough (and probably still
don't know enough) and nothing could help but investigating. For now I am
finished with that investigation and will think about continuing about a
documentation update in the short future.

I am writing this mail, because I hope I can give others and my future self
an overview of what all the hours of reading half finished documents and
mailing list threads got me until today. If it makes sense and there is a
central place to put such information I'd like to improve this text with
your assistance and put it there.

Documentation Advice For Users
----------------------------------------------

>From all guides to this complicated topic the best one seems to be the
inofficial guide by Scott Torborg: "How To Package Your Python Code" [4].
It refuses to discuss all the different projects and their interaction, but
presents a clear, simple path to follow that gets most things done, at
least much more then I probably want from packaging in the short future.

A great introduction about the complexities of this topic are in the
chapter "Python Packaging" in the open book "The Architecture of Open
Source Applications" [XX]. It's great, because it states different view
points on the topic, the different requirements and contains diagrams. Oh
how we people need diagrams to understand complex relationships!

Other resources aren't that useful in their current states: The core
documentation [5] is confusing, the often cited "Hitchhikers Guide to
Packaging" [6] seems to be a work in progress from 2009(?), and many
documentations from packaging tools themselves are not really complete,
either.

As a side note I like many sections in the distlib documentation [7]. It's
not complete and has some rough edges, but you can understand a lot about
what it should do from a first read, without too much frustration. That's
huge for any tool/framework in it's alpa state! I am a believer that good
human readable text (a.k.a. a good plan) results in good software, so I
have high hopes for this development.

Implementation Advice For Users
----------------------------------------------

In most cases setuptools is the way to go. distutils simply doesn't have
some fundamental features like `install_requires`. Taking other tools
doesn't seem to be necessary anymore, because distribute should be merged
into setuptools [8], so under the hood it's features will be used in the
short future anyway, Distutils2 is not an option due to nobody working on
it anymore (see the mailing list for example [9]), and distlib is currently
still in alpha and will likely be added under the hood into setuptools
anyway, when it's time.

Packaging efforts/project
-----------------------------------

(This only consideres projects close to the mainline development. There are
other tools like bento, or zc.buildout, which might be good or not, but
don't influence the mainline topics very much, so they are omitted in this
overview)

distutils (classic) - the old school, deprecated, pretty much unmaintained
packaging tooling; still the official standard [4] (Note: reasons [13])

setuptools - fork of distutils, which wasn't maintained for a long time but
will now be merged with it's successor [8]. Contains clear improvements
over distutils like requirements handling. Not all design decisions are
still appreciated, while some of them are based on the distutils mistake
and can't be fixed directly. (finding sources for these arguments will be
tough, because nobody cites any previous sources, when making such
arguments, but probably the following 2 threads contain these arguments at
least once: "Status in packaging 3.3" [10], "packaging location?" [11])

distribute - fork of setuptools [12], which is supported to the current
day. I don't know if it has any feature improvements over setuptools but I
guess it should be at least less buggy. Is currently merged back into
setuptools, which is a work in progress. (Note: Merge notification [8])

distutils2/packaging - a complete rewrite(?) [13] considering new
requirements, features and bug fixes. Was too big a task and couldn't be
finished until today. Was planned to be added to Python 3.3, which wasn't
possible, due to lack of work force for the size of the project. Currently
abandoned(?)

distlib - following the fail of distutils2 a fundamental set of futures
shall now be build into a framework that packaging tools like setuptools
can use to build stable, interoperable packaging mechanisms.

distil - a packaging tool like setuptools purely built on top of the
current state of distlib (?)

easy_install - an install script that shipped with setuptools. maintainance
state probably connected to setuptool's (?)

pip - an installer that seems to be developed in parallel with
distribute(?). From my feeling I'd say there is no much talk about this
online, because it simply works as expected.

Sources
--------------------

[1] http://mail.python.org/pipermail/distutils-sig/2013-February/019743.html
[2] http://mail.python.org/pipermail/distutils-sig/2013-February/019799.html
[3] http://mail.python.org/pipermail/docs/2013-March/013550.html
[4] http://www.scotttorborg.com/python-packaging/minimal.html
[5] http://docs.python.org/2.7/distutils/index.html (text is the same in
all following docs' versions as far as I could see)
[6] http://guide.python-distribute.org/index.html
[7] http://distlib.readthedocs.org/en/latest/index.html
[8] http://mail.python.org/pipermail/distutils-sig/2013-March/020126.html
[9]
https://groups.google.com/forum/?fromgroups#!forum/the-fellowship-of-the-packaging
[10]
http://mail.python.org/pipermail/python-dev/2012-June/thread.html#120430
[11]
http://mail.python.org/pipermail/python-dev/2012-September/thread.html#121689
[12] http://ziade.org/category/packaging,%20python.html
[13]
http://ziade.org/2010/03/03/the-fate-of-distutils-pycon-summit-packaging-sprint-detailed-report/
[XX] http://www.aosabook.org/en/packaging.html (added afterwards as
honourable mention)

Feedback welcome!

Cheers,
Erik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/8ec90927/attachment-0001.html>