[Distutils] Twisted packaging in Debian (was Re: Pre-pre-PEP: Requirements for the Python BUILDS Specification)

[Jean-Paul Calderone]

On Tue, 07 Oct 2008 17:11:19 +0200, Josselin Mouette <joss at debian.org> wrote:
>Le mardi 07 octobre 2008 à 10:02 -0400, Jean-Paul Calderone a écrit :
>> The expectations of the Nevow developers was that a file included in Nevow,
>> nevow_widget.py, would be installed to
>>
>>   /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py
>
>This expectation is wrong. You’re shipping it as a Python module, the
>only thing you can expect is to be able to import it.
>

Leaving aside whether or not I think that is a reasonable restriction to
impose, I don't see the relevance.  The only reason the module can be
imported on Debian now is because of the special measures included (by
upstream developers, not by Debian) in twisted/plugins/__init__.py.  I
can't understand how this could be part of Debian's Python packaging
policy, so perhaps I'm missing something very fundamental.

> [snip - "specific, private, plugins directory" which I may not understand,
> but doubt is relevant]

>> The ideal fix, from my perspective as a Twisted developer, is to install
>> the nevow_widget.py file into
>>
>>   /usr/lib/python2.5/site-packages/twisted/plugins/
>
>As explained on the debian-python list (see
>http://lists.debian.org/debian-python/2008/05/msg00032.html), there are
>strong reasons for not putting managed modules in the same directory as
>modules included in the Python distribution. So you can’t expect modules
>to be in a specific directory (but this shouldn’t be a problem since
>these are not modules, hmmm?).

They're modules.  They get imported.  That makes them modules, right?

    exarkun at charm:~$ python
    Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) 
    [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import twisted.plugins.nevow_widget
    >>> 

Do we have the same definition of "module"?

I chose the phrase "The ideal fix, from my perspective as a Twisted
developer" very intentionally.  If the file ends up somewhere else
but it's still an importable module, fine - if, from a Debian packager's
perspective that's better, fine.  I don't believe that the current
situation is better from a Debian packager's perspective, though, since
it involves carrying a patch that reverts an upstream bugfix.

> [snip - .pyc files in /var - not directly relevant]
> [snip - use of .pth file - not directly relevant]
>
>However this has nothing to do with the inability of Twisted to cope
>with multiple module paths. Let’s take another example that is not
>Debian specific: what if I want to install a plugin to /usr/local? The
>fact that you are abusing the python modules directory forces me to
                                                        ^^^^^^^^^^^^
>install it to /usr/local/python2.5/site-packages/twisted/plugins instead
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>of e.g. /usr/local/share/twisted/plugins, where a normal application or
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>library would be looking for its plugins. And there, I have to deal with
>namespaces and other python module specificities that should be
>irrelevant for plugins.

No, this is wrong.  Install it to /usr/local/share/ if you want.  It will
work just fine.  Why do you think it won't?  I'd rather you _didn't_, at
least not without a couple other changes, and I don't understand why you
would _want_ to, but it's possible.

>
>In the beginning, Debian only expected the Python modules directory to
>contain, well, Python modules. It seems even that is too much to expect,
>since module developers are taking advantage of a set of features
>(module file location) that were initially meant for introspection.

I assume you're talking about the "Python modules directory" in the install
target (not, say, in an upstream VCS).

Twisted plugins are modules, so Debian's expectations (be they valid or
otherwise, I make no judgement) aren't violated by them (oops, except for
dropin.cache files, but we haven't really talked about those yet and I
don't think they're actually contentious).

>> Have I succeeded in explaining why the Twisted plugin system isn't making
>> any unusual requirements of the installation system nor using the Python
>> module system in an unusual way?
>
>Not really. You have succeeded in explaining what is the exact nature of
>the unusual requirements you make, though. Thanks for that, as it is
>much clearer than the original explanations I received.

I'm glad this was clearer.  I'm not sure I've yet succeeded in entirely
explaining the system.  I hope this follow up goes a little bit further
toward that goal.

> [snip - I can't expect to see any improvements yet]

Jean-Paul