[Distutils] Add additional file categories for distutils, setuptools, wheel

[Paul Moore]

On 19 April 2015 at 04:42, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 18 April 2015 at 21:31, Daniel Holth <dholth at gmail.com> wrote:
>> Remember when distutils-sig didn't believe that Windows users actually
>> lacked compilers?

:-) I remember the days before we had package_data, when projects had
to abuse the data file support in setuptools to create $prefix/share,
or $prefix/docs. Projects like moin and cx_Oracle did this, for
example. It looked a mess on Windows installations :-)

>> graphite-web and carbon are two examples of Python applications with
>> lots of data files, that want to install config files into an obvious
>> place so you can edit them and get the applications up and running.
>> Because Python packaging has such terrible support for applications
>> that are not libraries, it's unnecessarily hard to get these packages
>> up and running -- especially if you want to put them in a virtualenv.

Thanks. If I get the time, I'll have a look at those packages and see
how their requirements might translate to Windows.

> It's a fairly standard practice in Fedora/RHEL/CentOS to use setup.py
> to define the build process in the RPM spec file, even if the package
> itself is never distributed via the upstream package index (e.g.
> beaker-project.org is built that way, as is pulpproject.org).
>
> Fedora's packaging policy for redistribution of upstream Python
> projects also switched last year to favour using "pip install" in the
> RPM build process over invoking "setup.py install" directly.
>
> Historically, all the "extra bits" needed to comply with FHS have
> lived in the spec file, independently of the upstream packaging
> system, requiring changes in two places for certain kinds of packaging
> modifications, and frequently rendering the projects undeployable in a
> virtual environment for no good reason.
>
> The benefit of Daniel's proposal is that it should make it feasible to
> modify many of these projects to be virtualenv friendly, and then
> *automate* the process of generating FHS policy compliant downstream
> packages. That will be a big step towards "package for PyPI, get your
> conda/Nix/Debian/Fedora packaging for free", so it feeds directly into
> my own interests in streamlining the redistribution pipeline in
> Fedora.

OK, thanks. That clarifies why this is needed, but it does read to me
as being something that projects should use sparingly, and only when
they are facing the sort of Linux packaging issues that it's designed
to solve. That's fine - my worry is that it'll be easy for projects to
see it as the "normal" way to structure their data, and as a result
we'll see a move away from package_data (which was introduced
precisely to solve the sorts of problems that the Unix model of
dedicated directories introduced in the context of Windows, zipimport
and virtualenv).

> From a Windows perspective, I believe this change mostly has the
> potential to make services that were previously Linux-only solely for
> packaging related reasons available on Windows as well. However, there
> may also be an opportunity to better automate the process of
> generating wix-based installers from PyPI packages (see
> http://wixtoolset.org/documentation/manual/v3/howtos/files_and_registry/add_a_file.html)
> rather than generating Windows installers directly (if I understand
> the tooling correctly, introducing wix into that process should offer
> the same kind of potential for better platform integration that
> integrating with distro package managers offers on Linux).

The biggest problem I see from a Windows POV is that installing files
outside of the package data structure is actively hostile to tools
like py2exe, and cx_Freeze, which are widely used for bundling
standalone Windows applications written in Python. Such tools won't be
able to find data installed via the proposed file category mechanism.

As a possible compromise, how about an approach where on Linux system
installs (or more accurately "those install schemes that are relevant
to the distribution packaging issue you described") the file
categories are installed into dedicated directories, as described. But
on other installs (virtualenvs, Windows, maybe OSX) the file
categories map to locations within package_data, so that they can be
accessed via normal mechanisms like loader.get_data. Application code
would need some support code to locate and read the files, but that's
true whether we go for this proposal or an "outside of site-packages"
scheme. Also, some things may even be better designated as "don't
install" in certain schemes (man files, for example, are pretty much
useless on Windows).

Beyond this issue, I do have some concerns over the specific locations
proposed, but they are better addressed as part of the PEP review
process, once we have a specific proposal to review.

Paul