[Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

#1 is pretty straightforward. An entry-point format Python
pre/post/etc. script may do.

I have some ideas for the FHS, though I fear it's full of bikesheds:

1. Allow all GNU directory variables as .data/* subdirectories
(https://www.gnu.org/prep/standards/html_node/Directory-Variables.html).
The distutils names will continue to be allowed.

packagename-1.0.data/mandir/...

2. Make data_files useful again. Interpolate path variables into
distutils data_files using $template syntax. (Only allow at
beginning?)

data_files=[('$mandir/xyz', ['manfile', 'other_man_file'])

In addition to $bindir, $mandir, etc. it will be important to allow
the package name and version to be interpolated into the install
directories.

Inside the wheel archive, you will get
packagename-1.0.data/mandir/manfile and
packagename-1.0.data/mandir/other_man_file

3. Write the install paths (the mapping from $bindir, $mandir, $prefix
etc. to the actual paths used) to one or more of a .py, .json, or
.dist-info/* based on new metadata in WHEEL:

install-paths-to: wheel/_paths.py

It is critical that this be allowed to work without requiring the end
user to look for it with pkg_resources or its pals. It's also good to
only write it if the installed package actually needs to locate its
file categories after it has been installed.

This will also be written inside the wheel itself with relative paths
to the .data/ directory.

4. Allow configurable & custom paths. The GNU paths could be
configured relative to the distutils paths as a default. We might let
the user add additional paths with a configuration dict.

    paths = {
             "foo" : "$bar/${quux}",
             "bar" : "${baz}/more/stuff",
             "baz" : "${quux}/again",
             "quux": "larry"
             }

5. On Windows, no one will really care where most of these files go,
but they probably won't mind if they are installed into separate
directories. Come up with sensible locations for the most important
categories.


On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft <donald at stufft.io> wrote:
>
> On Apr 13, 2015, at 10:39 AM, David Cournapeau <cournape at gmail.com> wrote:
>
> Hi there,
>
> During pycon, Nick mentioned there was interest in updating the wheel format
> to support downstream distributions. Nick mentioned Linux distributions, but
> I would like to express interest for other kind of downstream distributors
> like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work
> for Enthought).
>
> Right now, wheels have the following limitations for us:
>
> 1. lack of post/pre install/removing
> 2. more fine-grained installation scheme
> 3. lack of clarify on which tags vendors should use for custom wheels: some
> packages we provide would not be installable on "normal" python, and it
> would be nice to have a scheme to avoid confusion there as well.
>
> At least 1. and 2. are of interest not just for us.
>
> Regarding 2., it looks like anything in the <wheel_name>.data/data directory
> will be placed as is in sys.prefix by pip. This is how distutils scheme is
> defined ATM, but I am not sure whether that's by design or accident ?
>
> I would suggest to use something close to autotools, with some tweaks to
> work well on windows. I implemented something like this in my project bento
> (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
> but we could of course tweak that.
>
> For 1., I believe it was a conscious decision not to include them in wheel
> 1.0 ? Would it make sense to start a discussion to add it to wheel ?
>
> I will be at the pycon sprints until wednesday evening, so that we can flesh
> some concrete proposal first, if there is enough interest.
>
> As a background: at Enthought, we have been using eggs to distribute
> binaries of python packages and other packages (e.g. C libraries, compiled
> binaries, etc...) for a very long time. We had our own extensions to the egg
> format to support this, but I want to get out of eggs so as to make our own
> software more compatible with where the community is going. I would also
> like to avoid making ad-hoc extensions to wheels for our own purposes.
>
>
> To my knowledge, (1) was purposely punted until a later revision of Wheel
> just to make it easier to land the “basic” wheel.
>
> I think (2) is a reasonable thing as long as we can map it sanely on all
> platforms.
>
> I’m not sure what (3) means exactly. What is a “normal” Python, do you
> modify Python in a way that breaks the ABI but which isn’t reflected in the
> standard ABI tag?
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>