[Distutils] draft PEP: manylinux1

[Nick Coghlan]

On 22 January 2016 at 02:53, M.-A. Lemburg <mal at egenix.com> wrote:
> On 21.01.2016 17:13, Nathaniel Smith wrote:
>> On Jan 21, 2016 2:07 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>>>
>>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>> By using the version based approach, we'd not run into this
>>>>> problem and gain a lot more.
>>>>
>>>> I think it's better to start with a small core that we *know* works,
>>>> then expand later, rather than trying to make the first iteration too
>>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>>> than newer ones.
>>>
>>> My argument is that the file based approach taken by the PEP
>>> is too limiting to actually make things work for a large
>>> set of Python packages.
>>>
>>> It will basically only work for packages that do not interface
>>> to other external libraries (except for the few cases listed in
>>> the PEP, e.g. X11, GL, which aren't always installed or
>>> available either).
>>
>> The list of allowed libraries is exactly the same list of libraries as are
>> required by the Anaconda python distribution, so we *know* that it works
>> for about a hundred different python packages, including lots of tricky
>> ones (the whole scientific stack), and had been tested by tens or hundreds
>> of thousands of users. (I posted a link above to some actual working,
>> compliant pyside and numpy packages, which we picked for testing because
>> they're particular interesting/difficult packages that need to interface to
>> external libraries.) Yes, various extra tricks are needed to get things
>> working on top of this base, including strategies for shipping libraries
>> that are not in the baseline set, but these are problems that can be solved
>> on a project-by-project basis, and don't need a PEP.
>
> And that's the problem: The set is limited to the needs
> of the scientific community and there to the users of
> one or two distributions only.
>
> It doesn't address needs of others that e.g. use Qt or GTK as
> basis for GUIs, people using OpenSSL for networking, people
> using ImageMagick for processing images, or type libs for
> type setting, or sound libs for doing sound processing,
> codec libs for video processing, etc.

This is fine - at the moment *everyone* is locked out from publishing
Linux wheels to PyPI, so I'm entirely OK with biting off incremental
chunks that meet the needs of different sections of the community,
rather than trying to maintain an ever-expanding one-size-fits-all
platform definition.

However, it does suggest a possible alternative approach to naming
these compatibility subsets: what if the name of this particular
platform compatibility tag was something like "linux-sciabi1", rather
than "manylinux1"?

That way, if someone later wanted to propose "linux-guiabi1" or
"linux-audioabi1" or "linux-videoabi1", that could be done.

The auditwheel utility is already designed to support this "multiple
compatibility policy" approach, and the idea of using Docker-based
build environments also lends itself well to that model.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia