[Distutils] draft PEP: manylinux1

Fri Jan 22 05:48:33 EST 2016

On 22.01.2016 11:03, Nathaniel Smith wrote:
> On Fri, Jan 22, 2016 at 1:33 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 21.01.2016 20:05, Matthew Brett wrote:
>>> Hi,
>>>
>>> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>>> By using the version based approach, we'd not run into this
>>>>>> problem and gain a lot more.
>>>>>
>>>>> I think it's better to start with a small core that we *know* works,
>>>>> then expand later, rather than trying to make the first iteration too
>>>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>>>> than newer ones.
>>>>
>>>> My argument is that the file based approach taken by the PEP
>>>> is too limiting to actually make things work for a large
>>>> set of Python packages.
>>>>
>>>> It will basically only work for packages that do not interface
>>>> to other external libraries (except for the few cases listed in
>>>> the PEP, e.g. X11, GL, which aren't always installed or
>>>> available either).
>>>>
>>>> IMO, testing the versions of a set of libraries is a safer
>>>> approach. It's perfectly fine to have a few dependencies
>>>> not work in a module because an optional system package is not
>>>> installed, e.g. say a package comes with UIs written in
>>>> Qt and one in GTK.
>>>
>>> Please forgive my slowness, but I don't understand exactly what you
>>> mean.  Can you give a specific example?
>>>
>>> Say my package depends on libpng.
>>>
>>> Call the machine I'm installing on the client machine.
>>>
>>> Are you saying that, when I build a wheel, I should specify to the
>>> wheel what versions of libpng I can tolerate on the the client
>>> machine, and if if the client does have a compatible version, then pip
>>> should raise an error, perhaps with a useful message about how to get
>>> libpng?
>>>
>>> If you do mean that, how do you want the PEP changed?
>>
>> I already posted a change proposal earlier on in the thread.
>> I'll repeat it here (with a minor enhancements):
> 
> Okay, I think I get it now. I'll try to repeat back to summarize and
> see if I have understood your proposal correctly:
> 
> In the PEP 513 "manylinux1" approach, when users do 'pip install foo',
> then one of three things happens:
> 1) they get a working foo and are immediately good-to-go, or
> 2) pip says "I'm sorry, there's no compatible wheel", or
> 3) something else happens, in which case this is a bug, and the spec
> provides some framework to help us determine whether this is a bug in
> the wheel, a bug in pip, or a bug in the spec.
> 
> In your approach, users do 'pip install foo', and then pip installs
> the wheel, and then when they try to use the wheel they get an error
> message from the dynamic linker about missing libraries, and then the
> user has to read the docs or squint at these error messages in order
> to figure out what set of apt-get / yum / pacman / ... commands they
> need to run in order to make foo work. (And possibly there is no such
> combination of commands that will actually work, because e.g. the
> wheel was linked against Debian's version of libbar.so.7 and Fedora's
> version of libbar.so.7 turns out to have an incompatible ABI, or
> Fedora simply doesn't provide a libbar.so.7 package at all.)

pip could be made to check the wheel for missing library
dependencies in order to provide help with cases where
additional packages are needed, but overall, yes, that's
the way it should work, IMO.

It's better to have wheels than not to have them, since
installing an additional system package is by far easier
than trying to compile packages from source (this will
usually also require additional -dev packages to be
installed).

>>  * no lock-out of package authors who would like to push
>>    wheel files for their packages to PyPI, but happen to
>>    use libraries not in the predefined list of the original
>>    draft PEP
>
> https://mail.python.org/pipermail/distutils-sig/2016-January/028050.html

Embedding additional libraries in the wheels files to overcome
deficiencies in the PEP design simply doesn't feel right
to me.

People who rely on Linux distributions want to continue
to do so and get regular updates for system packages from
their system vendor. Having wheel files override these
system packages by including libs directly in the wheel
silently breaks this expectation, potentially opening
up the installations for security holes, difficult to
track bugs and possible version conflicts with already
loaded versions of the shared libs.

IMO, that's much worse than having to install additional
system packages to make a Python wheel work.

The embedding approach also creates licensing problems,
since those libs may be under different licenses than the
package itself. And of course, it increases the size of the
wheel files, causing more bandwidth to be necessary,
more disk space to be used for wheel caches, etc.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 22 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/