[Distutils] Platform specific destinations in wheel files?

[Nick Coghlan]

On 21 December 2013 21:40, Paul Moore <p.f.moore at gmail.com> wrote:
> On 21 December 2013 10:57, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> And these are the generic directories that aren't necessarily Python specific:
>>
>>     scripts: directory for script files.
>>     data: directory for data files.
>
> It's worth noting that "data", although in essence a platform neutral
> term, is in practice not very well defined. Projects as a result use
> it inconsistently, and usually based on their expectations from the
> platforms they are used to. One thing I have seen data being used for
> a lot is documentation - and not consistently - some projects use
> <data>/man, others use <data>/share/man, and others
> <data>/<projectname>.
>
> I'd suggest the following changes in this area:
>
> 1. Add a "docs" directory to hold documentation. We may need to have a
> debate on how this should be structured, and how to handle backward
> compatibility, but this is a genuine need that the current scheme does
> not cover.

Evolution of sysconfig as we go forward is indeed something for us to
think about. It may be a case where we need to actually decouple the
wheel spec from the implicit dependency on those definitions. The
stdlib related paths don't make sense to use in a wheel file, so what
probably makes sense is for us to make just purelib, platlib, include
and platinclude depend on sysconfig (since they're installing to
Python-specific locations and need the assistance from the standard
library), while other paths like scripts and docs would be defined
independently of sysconfig.

> 2. Add some more explicit wording on how "data" should (and should
> not) be used. For instance, having FHS-style structures like
> share/man/man1 under "data" doesn't really help with cross-platform
> compatibility that much - it just provides somewhere to dump the data
> on Windows while still interoperating with standard tools on a POSIX
> system.

I'd be interesting in reading such guidance, because I really have no
idea... (I'd never even noticed the option until Daniel pointed it out
to me recently)

>> This is still only a half-baked idea at this point, but I'm currently
>> leaning towards keeping the "<name>.data" sysconfig subdirectories in
>> the wheel format cross platform (and somewhat Python specific), and
>> adding a new "<name>.app" subdirectory in parallel.
>>
>> A wheel that had content in "app" would be inherently platform
>> specific - you wouldn't be permitted to use the new directory in a
>> cross-platform wheel file. The defined subdirectories of app would
>> also be platform specific.
>
> It might be worth having the possibility of optional parts of "app".
> To use the docs example again, it would be easy to see projects
> putting Unix manpages in app, and so unintentinally making the
> distribution platform specific, when in practice it would be perfectly
> OK to simply omit the manpages on Windows. Other app files (for
> example config files in etc) *will* be essential for proper operation,
> of course.

I think a separate docs path would be a better solution for that.
There may be a couple of others we decide to support directly (like an
admin_scripts path that maps to sbin under FHS and to the same place
that scripts maps to on Windows).

>> All POSIX systems would at least support the "fhs" subdirectory. For a
>> system install, this would map to "/", for a virtualenv install it
>> would map to the root of the virtualenv and for a user install it
>> would map to "~/.local".
>>
>> I'm not sure what other subdirectories would be appropriate for
>> Windows or Mac OS X, although I expect being able to install into
>> Program Files and Application Data would be interesting for Windows
>> apps, and into an application folder for Mac OS X.
>
> I'm not sure Program Files is that useful - very few apps would
> install into *both* Program Files and an application area
> (site-packages). But Application Data might be useful. Another
> possibility would be the start menu - but that's not precisely a
> filesystem location, you'd need to put metadata into the "Start Menu"
> area of the wheel, and use the Windows API to create the start menu
> items at install time.
>
>> It's really the potential for FHS support that drives my interest in
>> the idea, but if we're going to do something like this at all, it
>> shouldn't be POSIX specific.
>
> Agreed in principle. But in practice, do you expect the wheel spec to
> define exactly how each scheme under "app" is mapped? And do you
> expect every tool to implement those features? That implies a lot of
> work getting consensus and a change to the wheel spec every time
> someone thinks of a new platform-specific area that's worth
> supporting.
>
> Maybe this is something that should be handled by the post-install
> hook support. The "app" area could be an area that is reserved in the
> wheel for platform-specific data as you say, but there is no specified
> behaviour for tools to implement. Rather:
>
> 1. Tools must make the "app" area accessible in post-install hooks
> (how is yet to be defined, simplest answer just put it in a temporary
> location on disk, and tell the hooks the location of that location)
> 2. Tools should provide an extensibility mechanism for installing
> system-wide processors for app data (so that we don't need every wheel
> to ship the FHS-handling post-install hooks)
> 3. Standard processing for particular app areas (for example, app/fhs)
> could be defined independently of the wheel spec.
>
> One of the motivations I have here is to keep the install behaviour
> from getting too complex. We need to remember that pip is not the only
> wheel installer (there's distlib and wheel itself). So I don't want to
> mandate too much complexity into the install phase (for example,
> script wrappers aren't handled in wheel install, and I'm not sure what
> distlib does - so there's implementation differences creeping in right
> now).

Oh, I like that idea. Let me get the next draft of the metadata 2.0
PEPs cleaned up and published, and then we can discuss it further (I
think it ties into nicely with some of the changes I've made in
response to PJE's feedback on the last draft).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia