[Distutils] How to implement ‘setup.py’ functionality that itself needs third-party distributions (was: Module from install breaks subsequent install of different distribution)

[Marius Gedminas]

On Tue, Jan 20, 2015 at 12:50:14PM +1100, Ben Finney wrote:
> Tres Seaver <tseaver at palladion.com> writes:
> > setuptools itself is extensible by means of entry points. Both entry
> > points in your example (as cited by Marius) demonstrate that: one adds
> > support for a new keyword argument to 'setup()'[1], and the other
> > defines a new "writer" for 'egg-info'[2]. By design, both of those are
> > supposed to be loaded / available for any invocation of 'setup()' in a
> > Python where the are installed (not merely for packages which
> > "mention" them).
> 
> What recourse do I have, then?
> 
> I'm using entry points because it seems to be the only way I can declare
> functionality that resides in a module alongside the ‘setup.py’ which
> itself needs third-party packages.

Are you aware that those entry points will be used for every single
package the user tries to install after installing python-daemon?

> * During the distribution build stage (‘./setup.py build’ or earlier),
>   I want to parse a reST document and derive some of the distribution
>   metadata from that.

And presumably you don't want to use regexps for that. ;)

> * The ‘setup.py’ is the wrong place for this; it's complex and deserves
>   its own module which can be imported for unit testing.
> 
> * This also is totally unrelated to the functionality this distribution
>   installs, so it doesn't belong in any of the packages to distribute
>   for installation.
> 
> * So I place it in a separate top-level module, ‘version’, only for use
>   within ‘setup.py’.

These three items sound very reasonable and shouldn't be causing
problems.

> * That module itself needs a third-party distribution (‘docutils’) from
>   PyPI. So I need that distribution to be installed *before* the
>   ‘version’ module can be used. Apparently the ‘setup_requires’ option
>   is the only way to do that.

This is the point where my experience with setup_requires would tempt me
to go "hey, regexes aren't _that_ bad". ;)

> * Then, in order to declare how that functionality is to be used for
>   commands like ‘./setup.py egg_info’, I have no other way than
>   declaring a Setuptools entry point.

I think there's also the option of providing custom command classes to
the setup() function.

For completeness' sake there's also the option of monkey-patching
distutils/setuptools internals (not recommended).

> * Declaring a Setuptools entry point makes Setuptools think the
>   distribution-specific module must be imported by every other
>   distribution in the same run.

*And in every subsequent run.*  This is important.

Entry points live in the installed package metadata.

You shouldn't ever define an entry point that points to a package or
module that won't be installed.

>   Of course it's not available there so those distributions fail to install.
> 
> * It even thwarts the installation of ‘docutils’, the very distribution
>   that is needed for ending this circular dependency.
> 
> What am I missing? How can I implement complex functionality specific to
> packaging this distribution, without making an utter mess?

I'll attempt some suggestions:

  1. Simplify metadata extraction so it doesn't rely on docutils.

  2. Implement metadata extraction using custom command classes[*] and
     setup_requires.

     (Sorry about the lack of specifics: I've never done this.  I'm not
     even 100% certain the things you want can be done this way.)

     [*] https://docs.python.org/2/distutils/extending.html#integrating-new-commands,
         the bits about setup(..., cmdclass=...); not the bits about
         command_packages.

  3. Duplicate the metadata in your setup.py; have a unit test that
     extracts it from your ReST and from your setup.py and compares the
     two.  Since your release process should involve a "run all tests"
     step, this ought to prevent you from making releases with
     mismatched metadata.

     (Not a big fan of this one, but it's better than some alternatives,
     like having duplicated metadata that must be checked for
     consistency by hand.)

  4. Move your entry points into an installed module that checks the
     name of the package it's working on and does nothing if it's not
     "python-daemon".

     (Not a fan of this one, it adds a moving part that can break any
     subsequent Python package installation.)

  5. Use some replacement of setup_requires as suggested elsewhere in
     this thread, so you could import docutils at the top level of a
     setup.py.

     (I've never tried this approach and have no idea how reliable it
     is.)

HTH,
Marius Gedminas
-- 
   Since this protocol deals with Firewalls there are no real security
   considerations.
                -- RFC 3093
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150120/68d925e0/attachment-0001.sig>