[Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[M.-A. Lemburg]

On 10.02.2016 14:00, Oscar Benjamin wrote:
> On 10 February 2016 at 12:21, M.-A. Lemburg <mal at egenix.com> wrote:
>>> So "easy to achieve" still needs someone to take the time to deal with
>>> these sorts of issue. It's the usual process of the people willing to
>>> put in the effort get to choose the direction (which is also why I
>>> just provide feedback, and don't tend to offer my own proposals,
>>> because I'm not able to commit that sort of time).
>>
>> Wait. You are missing the point that the setup.py interface
>> already does work, so no extra effort is needed. All that's
>> needed is some documentation of what's currently being used,
>> so that other tools can support the interface going forward.
> 
> You can see an example of a minimal setup.py file here:
> 
> https://github.com/oscarbenjamin/setuppytest/blob/master/setuppytest/setup.py
> 
> I wrote that some time ago and don't know if it still works (that's
> the problem with just having a de facto standard).

Agreed, and something that we should address in a PEP.

>> At the moment, pip this interface is only defined by
>> "what pip uses" and that's a moving target.
> 
> The setup.py interface is a terrible interface for tools like pip to
> use and for tools like flit to emulate. 

I'm not saying that it's a great interface, but it's one that
by far most sdists out there support.

> Currently what pip does is to
> invoke
> 
>     $ python setup.py egg_info --egg-base $TEMPDIR
> 
> to get the metadata. It is not possible to get the metadata without
> executing the setup.py which is problematic for many applications.
> Providing a static pypa.json file is much better: tools can read a
> static file to get the metadata.

Depends on which kind of meta data you're after. sdist packages
do include the static PKG-INFO file which has the version 1.0
meta data. This doesn't include dependencies or namespace
details, but it does have important data such as version,
package name, description, etc.

> To install a distribution pip runs:
> 
>     $ python setup.py install --record $RECORD_FILE         \
>                         --single-version-externally-managed
> 
> So the setup.py is entirely responsible not just for building but also
> for installing everything. This makes it very difficult to develop a
> system where different installer tools and different build tools can
> cooperate to allow end users to specify installation options. It also
> means that the installer has no direct control over where any of the
> files are installed.

Why is that ? The install command is very flexible in allowing
you to define where the various parts are installed.

When defining a minimal set of supported options, the
various --install-* options should be part of this.

It would also be possible to separate the build and install
steps, since distutils is well capable to do this.

However, I'm not sure where this aspect fits in relation to the
proposed PEP, since it is targeting the operation of building
the package and wrapping it into a wheel file, so the
bdist_wheel command would have to be used instead.

pip wheel pkg runs this command:

python setup.py bdist_wheel -d targetdir

> If you were designing this from scratch then there are some obvious
> things that you would want to do differently here. The setup.py
> interface also has so much legacy usage that it's difficult for
> setuptools and pip to evolve. The idea with this proposal is to
> decouple things by introducing a new interface with well defined and
> sensible behaviour.

In the end, you'll just be defining a different standard
to express the same thing in different ways.

The setup.py interface was never designed with integration in mind
(only with the idea to provide an extensible interface; and I'm not
going get into the merits of setuptools additions such as
--single-version-externally-managed :-)) but it's still
quite usable for the intended purpose.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 10 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________
2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/