From oscar.j.benjamin at gmail.com  Mon Jan  4 09:27:01 2016
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 4 Jan 2016 14:27:01 +0000
Subject: [Distutils] Trouble using setuptools to build separate c++
 extension
In-Reply-To: <56761D38.5010409@gmail.com>
References: <56761D38.5010409@gmail.com>
Message-ID: <CAHVvXxT5NpYhmUMmO2Rzv0OyQM9Qo4pmLnjvSBuNnUEJGZ8kiw@mail.gmail.com>

On 20 December 2015 at 03:15, Thomas Nyberg <tomnyberg at gmail.com> wrote:
> Hello I'm having trouble understanding the right way to build a c++ module
> using setuptools. I've been reading the docs, but I'm confused where I
> should be putting my build options. Everything builds fine on its own. I
> have my sources in src/ and my headers in include/.
>
> My first problem is that I'm having trouble figuring out where to put my
> build flags. Here is the Makefile I'm currently using:
>
> --------
> srcs=$(wildcard *.cpp)
> srcs+=$(wildcard src/*.cpp)
> objs=$(patsubst %.cpp,%.o,$(srcs))
>
> cc=g++
> ccflags=-std=c++11 -g -O3 -fPIC
> includes=-I. -I./include/ -I/usr/include/python2.7/ -I/usr/include/boost
> libflags=-L. -L/usr/lib/x86_64-linux-gnu
> ldflags= -shared -Wl,--export-dynamic
>
> patent_computer_cpp.so: $(objs)
>         $(cc) $(libflags) $(ldflags) $(objs) -o patent_computer_cpp.so
> -lboost_python -lpython2.7
>
> %.o:%.cpp
>         $(cc) $(ccflags) $(includes) -c -o $@ >lt;
> --------
>
> Unfortunately I can't post the sources, but they compile fine to produce the
> `patent_computer_cpp.so` file which can be imported as a module. Maybe I
> should also point out that I'm using boost-python (I don't think this is the
> issue though).
>
> I just can't figure out how to get setuptools.Extension to use these build
> flags. I've seen recommendations online saying that I should set CFLAGS as
> an environment variable and set OPT='' as an environment variable as well,
> but this just feels wrong given the simplicity of my setup. (Besides the
> shared object doesn't seem to compile correctly in this case.) I've tried
> using the extra_compile_args option in setup.py, but that fails.
>
> Is there a way to avoid setting environment variables like this or is this
> the accepted way to build this kind of software? Am I missing some obvious
> docs somewhere? Thanks for any help.

Hi Thomas,

Whether or not what you're currently doing is acceptable really
depends. Who needs to install this? Do you know which OS or compiler
they will use etc? Are you hoping that this will be installed by pip?

The setup.py is supposed to allow the end user a bit of freedom to use
different OS/compiler etc. and free up the module author from needing
to know exactly where the Python header files and link libraries will
be on each platform. OTOH if you are just writing for exactly one
platform then what you have may be more convenient and flexible.

I would approach this step-by-step to convert that to using setup.py.
So the first part is to tell extension about all of the cpp files and
get it to use g++ to compile it. The next step is to tell about the
additional include locations. The next step is the additional
libraries that need linking. There are arguments to Extension and
setup for each of these things. The last part is to get the exact
right compiler flags.

--
Oscar

From arstechnica at contractor.net  Fri Jan  8 04:18:14 2016
From: arstechnica at contractor.net (ars technica)
Date: Fri, 8 Jan 2016 10:18:14 +0100
Subject: [Distutils] Problems using pip on Ubuntu when 2.7,
 3.4 and 3.5 are installed
Message-ID: <trinity-37077f80-8a79-4daf-94b7-b3f856a2e291-1452244694467@3capp-mailcom-lxa05>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160108/96bf685e/attachment.html>

From leorochael at gmail.com  Fri Jan  8 09:43:10 2016
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Fri, 8 Jan 2016 12:43:10 -0200
Subject: [Distutils] Problems using pip on Ubuntu when 2.7,
 3.4 and 3.5 are installed
In-Reply-To: <trinity-37077f80-8a79-4daf-94b7-b3f856a2e291-1452244694467@3capp-mailcom-lxa05>
References: <trinity-37077f80-8a79-4daf-94b7-b3f856a2e291-1452244694467@3capp-mailcom-lxa05>
Message-ID: <CAMxX_oPxChGqwmkTMmukGce2hfDNBEZCsxZ66MO_pt0V+UmmgQ@mail.gmail.com>

Hi Mr(s) Versions,

Your "locate" output shows that you have a manual installation of Python
3.4 on /usr/local, and it's possible that "/usr/local/bin" is ahead of
"/usr/bin" in your PATH, which is why pip3 installs into Python 3.4.

>From your error message with Python 3.5, it seems like your Python 3.5 is a
"System Python", i.e., a Python that was installed with the package manager
for your system, but that your Python development packages wasn't installed
yet.

If yours is a Debian style system, you might need to install the
"python-dev" or perhaps a "python3.5-dev" package. E.g.:

   - apt-get install python-dev

If your system is Redhat based (Fedora, RHEL (do we already have python3.5
in any RHEL release?)), then you're looking for a "python-devel" or
"python3.5-devel" package. Something like:

   - yum install python-dev

Hope it helps.

Best regards,

Leo


On 8 January 2016 at 07:18, ars technica <arstechnica at contractor.net> wrote:

> Hi,
>
> thank you for the tutorial here
> https://docs.python.org/3.5/installing/index.html
>
> I have a problem using pip to install packages to 3.5. I now have three
> versions installed and pip only addresses the two old ones.
>
> The command pip installs to 2.7
> pip3 installs to 3.4 and that leaves 3.5 all alone.
>
> I tried to install numpy with the command
> python3.5 -m pip install numpy
>
> but got the error that Python.h could not be located. When asked bash
> *locate* Python.h
>    it returned
> /usr/local/include/python3.4m/Python.h
>
>
> My question therefore is. How do I setup my environment to be able to
> install packages to all the Python versions? Should I do a
> virtualenvironment for each version or can I change some configuration file
> to let PIP know that it should check the Py3.5 folder when desired?
>
> Best regards,
>
> lost in versions.
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160108/7fa58d5d/attachment.html>

From ncoghlan at gmail.com  Sat Jan  9 04:26:32 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Jan 2016 19:26:32 +1000
Subject: [Distutils] Problems using pip on Ubuntu when 2.7,
 3.4 and 3.5 are installed
In-Reply-To: <trinity-37077f80-8a79-4daf-94b7-b3f856a2e291-1452244694467@3capp-mailcom-lxa05>
References: <trinity-37077f80-8a79-4daf-94b7-b3f856a2e291-1452244694467@3capp-mailcom-lxa05>
Message-ID: <CADiSq7f0-F2fB9unNBon48BTu2Q6ymDG0Juc0LhkQGj88vEvQw@mail.gmail.com>

On 8 January 2016 at 19:18, ars technica <arstechnica at contractor.net> wrote:
> Hi,
>
> thank you for the tutorial here
> https://docs.python.org/3.5/installing/index.html
>
> I have a problem using pip to install packages to 3.5. I now have three
> versions installed and pip only addresses the two old ones.
>
> The command pip installs to 2.7
> pip3 installs to 3.4 and that leaves 3.5 all alone.
>
> I tried to install numpy with the command
> python3.5 -m pip install numpy

In the absence of a virtual environment, using "-m" as you have done
here is the right way to invoke pip while ensuring it's using the
intended Python installation.

> but got the error that Python.h could not be located. When asked bash
> locate Python.h
>    it returned
> /usr/local/include/python3.4m/Python.h

This is where things can get a bit messy, since Linux tends to assume
system-wide builds by default.

> My question therefore is. How do I setup my environment to be able to
> install packages to all the Python versions? Should I do a
> virtualenvironment for each version or can I change some configuration file
> to let PIP know that it should check the Py3.5 folder when desired?

If you're fine with building NumPy from source yourself, then creating
a virtual environment for each version should work. However, I believe
there may still be some potential to end up running into missing
header files for some of NumPy's external dependencies when building
for a Python version that wasn't provided by your distro.

If you'd prefer to avoid any configuration and debugging of
C/C++/FORTRAN build processes, then you may prefer to grab miniconda
and download distro-independent pre-built binaries:
http://conda.pydata.org/miniconda.html

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mailtosijojose at gmail.com  Wed Jan 13 09:30:51 2016
From: mailtosijojose at gmail.com (Sijo Jose)
Date: Wed, 13 Jan 2016 20:00:51 +0530
Subject: [Distutils] Unable to build language-check
Message-ID: <CAG5wUb_iPZRTJuEiyEGeCWEM1PvjtU+JAARi0aW=RpB8t6182A@mail.gmail.com>

The language-check (https://pypi.python.org/pypi/language-check)library has
the dependency of '*3to2*' in *python 2.7*,
In a virtualenv it works fine with pip,that is

*pip install  3to2*
*pip install language-check*


But its failing with easy_install in venv

*easy_install 3to2*
*easy_install language-check* (It fails)

Due to this I'm not able to build my project using *setuptools*.

-- 
*Regards*
*Sijo Jose*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160113/58d01b9b/attachment.html>

From erik.m.bray at gmail.com  Wed Jan 13 15:54:06 2016
From: erik.m.bray at gmail.com (Erik Bray)
Date: Wed, 13 Jan 2016 15:54:06 -0500
Subject: [Distutils] Unable to build language-check
In-Reply-To: <CAG5wUb_iPZRTJuEiyEGeCWEM1PvjtU+JAARi0aW=RpB8t6182A@mail.gmail.com>
References: <CAG5wUb_iPZRTJuEiyEGeCWEM1PvjtU+JAARi0aW=RpB8t6182A@mail.gmail.com>
Message-ID: <CAOTD34a2=MSYoUkjnv6C0DtONizrN7GKocZHpKB+L4L8uqF4dQ@mail.gmail.com>

On Wed, Jan 13, 2016 at 9:30 AM, Sijo Jose <mailtosijojose at gmail.com> wrote:
> The language-check (https://pypi.python.org/pypi/language-check)library has
> the dependency of '3to2' in python 2.7,
> In a virtualenv it works fine with pip,that is
>
> pip install  3to2
> pip install language-check
>
>
> But its failing with easy_install in venv
>
> easy_install 3to2
> easy_install language-check (It fails)
>
> Due to this I'm not able to build my project using setuptools.

Why do you need to install it with easy_install?  pip is the
recommended installer for Python packages.

Erik

From erik.m.bray at gmail.com  Thu Jan 14 00:46:58 2016
From: erik.m.bray at gmail.com (Erik Bray)
Date: Thu, 14 Jan 2016 00:46:58 -0500
Subject: [Distutils] Unable to build language-check
In-Reply-To: <CAG5wUb_iPZRTJuEiyEGeCWEM1PvjtU+JAARi0aW=RpB8t6182A@mail.gmail.com>
References: <CAG5wUb_iPZRTJuEiyEGeCWEM1PvjtU+JAARi0aW=RpB8t6182A@mail.gmail.com>
Message-ID: <CAOTD34bFNvpX43vBN2hLvctBngNKN_XH_HXgrpLERuaJbTwE3g@mail.gmail.com>

On Wed, Jan 13, 2016 at 9:30 AM, Sijo Jose <mailtosijojose at gmail.com> wrote:
> The language-check (https://pypi.python.org/pypi/language-check)library has
> the dependency of '3to2' in python 2.7,
> In a virtualenv it works fine with pip,that is
>
> pip install  3to2
> pip install language-check
>
>
> But its failing with easy_install in venv
>
> easy_install 3to2
> easy_install language-check (It fails)
>
> Due to this I'm not able to build my project using setuptools.

After trying it myself and looking at the resulting traceback, I see
what's going on here.

language-check performs some extensive shenanigans to run 3to2 on its
code, and somewhere along the line it assumes that 3to2 is installed
in a directory.  However when you install it with easy_install, it is
installed as an egg by default.  And since it is marked zip-safe it's
installed as a zipped egg file instead of a directory.

My recommendation for installing your package is to run `pip install
.`, rather than `./setup.py install`, which there is a movement to get
away from.  Only installing with pip will get you out of a lot of
trouble you might run into with easy_install.

If you *must* use easy_install run it with -Z, at least when
installing 3to2, to make sure it's installed unzipped.  You might also
raise the issue with the developers of language-check since this is a
bug in their setup.py.

Best,
Erik

From njs at pobox.com  Thu Jan 14 00:55:16 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 13 Jan 2016 21:55:16 -0800
Subject: [Distutils] heads-up on a plot to bring linux wheels to pypi
Message-ID: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>

Hi all,

Just wanted give distutils-sig a heads-up that there seems to be some
momentum gathering around a plot to bring linux wheels to pypi:

  https://github.com/manylinux/manylinux

Basically the idea is to define a standard baseline linux environment
(available libraries + version numbers + ABI of these), give it a name
(for now, "manylinux"), and then provide tools to build wheels against
this environment, teach pip how to recognize that the system it's on
conforms to the environment, and then convince pypi to start accepting
wheels with this as a platform tag :-). And if we carefully define the
baseline environment based on the experiences of the popular
scientific python distros (Continuum's Anaconda + Enthought's Canopy)
we can be highly confident that essentially all desktop + server linux
distros will be compatible with these wheels.

This strategy is orthogonal to the more ambitious efforts to define an
interface between wheels and the platform package manager; they can
proceed in parallel and potentially complement each other. The goal
here is just to get linux up to feature parity with windows and osx.

Status:
- we have a draft policy
- there's a cool tool for scanning wheels and checking whether they
conform to the policy: https://github.com/manylinux/auditwheel
- there's a draft docker image to make it easy to build such wheels
(https://github.com/manylinux/manylinux/pull/2)

To do:
- bikeshed the name ("manylinux" was picked after about 2 minutes of
discussion so as to get started. maybe "genericlinux" would be better?
I dunno.)
- build some test wheels
- write a proper PEP
- convince pip and pypi maintainers that this is a good idea ;-)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org

From ncoghlan at gmail.com  Thu Jan 14 05:12:59 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Jan 2016 20:12:59 +1000
Subject: [Distutils] heads-up on a plot to bring linux wheels to pypi
In-Reply-To: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
References: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
Message-ID: <CADiSq7enoYVDy_fU_XmvWHOSBTKAxB0coHehf5kt_kb=yiZ4_w@mail.gmail.com>

On 14 January 2016 at 15:55, Nathaniel Smith <njs at pobox.com> wrote:
> Hi all,
>
> Just wanted give distutils-sig a heads-up that there seems to be some
> momentum gathering around a plot to bring linux wheels to pypi:
>
>   https://github.com/manylinux/manylinux
>
> Basically the idea is to define a standard baseline linux environment
> (available libraries + version numbers + ABI of these), give it a name
> (for now, "manylinux"), and then provide tools to build wheels against
> this environment, teach pip how to recognize that the system it's on
> conforms to the environment, and then convince pypi to start accepting
> wheels with this as a platform tag :-). And if we carefully define the
> baseline environment based on the experiences of the popular
> scientific python distros (Continuum's Anaconda + Enthought's Canopy)
> we can be highly confident that essentially all desktop + server linux
> distros will be compatible with these wheels.

Very nice!

> This strategy is orthogonal to the more ambitious efforts to define an
> interface between wheels and the platform package manager; they can
> proceed in parallel and potentially complement each other. The goal
> here is just to get linux up to feature parity with windows and osx.

That sounds like a good strategy to me - improved automation of
upstream -> downstream package conversions is a nice feature to have,
but it's a redistributor oriented one, moreso than an end user focused
one.

For end users, a pragmatic approach to defining a baseline "manylinux"
ABI based on the experience of cross-distro redistributors will
provide more benefit, sooner.

> Status:
> - we have a draft policy
> - there's a cool tool for scanning wheels and checking whether they
> conform to the policy: https://github.com/manylinux/auditwheel
> - there's a draft docker image to make it easy to build such wheels
> (https://github.com/manylinux/manylinux/pull/2)
>
> To do:
> - bikeshed the name ("manylinux" was picked after about 2 minutes of
> discussion so as to get started. maybe "genericlinux" would be better?
> I dunno.)

Since you've already set up the GitHub group etc as "manylinux", I
think it makes sense to just call it "good enough". "manylinux" is an
accurate description (as *many* Linux distributions provide the ABI
subset you're targeting, but not all of them), and "genericlinux" has
potential to cause confusion with "linux-generic" kernel packages.

> - build some test wheels
> - write a proper PEP
> - convince pip and pypi maintainers that this is a good idea ;-)

While I've historically advocated against the idea of defining our own
"Linux platform ABI" subset, the fact that Enthought and Continuum are
successfully distributing pre-built binaries through the simple "use
CentOS 5.11" approach seems promising.

In terms of non-scientific packages, the main group I'd suggest
getting in touch with is pycryptography, as we'll probably want to
baseline a more recent version of OpenSSL than the one in CentOS 5.11.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Jan 14 05:19:28 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Jan 2016 20:19:28 +1000
Subject: [Distutils] heads-up on a plot to bring linux wheels to pypi
In-Reply-To: <CADiSq7enoYVDy_fU_XmvWHOSBTKAxB0coHehf5kt_kb=yiZ4_w@mail.gmail.com>
References: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
 <CADiSq7enoYVDy_fU_XmvWHOSBTKAxB0coHehf5kt_kb=yiZ4_w@mail.gmail.com>
Message-ID: <CADiSq7dGZ_5katH25VeZoCuOsyZQECXv3h91wMROEroefO3Vyw@mail.gmail.com>

On 14 January 2016 at 20:12, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 14 January 2016 at 15:55, Nathaniel Smith <njs at pobox.com> wrote:
>> - build some test wheels
>> - write a proper PEP
>> - convince pip and pypi maintainers that this is a good idea ;-)
>
> While I've historically advocated against the idea of defining our own
> "Linux platform ABI" subset, the fact that Enthought and Continuum are
> successfully distributing pre-built binaries through the simple "use
> CentOS 5.11" approach seems promising.
>
> In terms of non-scientific packages, the main group I'd suggest
> getting in touch with is pycryptography, as we'll probably want to
> baseline a more recent version of OpenSSL than the one in CentOS 5.11.

Ah, looking at https://github.com/manylinux/auditwheel, I see anything
linking to OpenSSL would fail the platform audit, at least for the
current draft policy. That also seems like a potentially reasonable
approach (although it could lead to complaints about "Why doesn't
project <X> offer a wheel file?")

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From njs at pobox.com  Thu Jan 14 06:13:09 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 14 Jan 2016 03:13:09 -0800
Subject: [Distutils] heads-up on a plot to bring linux wheels to pypi
In-Reply-To: <CADiSq7dGZ_5katH25VeZoCuOsyZQECXv3h91wMROEroefO3Vyw@mail.gmail.com>
References: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
 <CADiSq7enoYVDy_fU_XmvWHOSBTKAxB0coHehf5kt_kb=yiZ4_w@mail.gmail.com>
 <CADiSq7dGZ_5katH25VeZoCuOsyZQECXv3h91wMROEroefO3Vyw@mail.gmail.com>
Message-ID: <CAPJVwB==vV9=RXSZUYNPApXUVVkKFvGc8oRMLKep_rgEo3jQNw@mail.gmail.com>

On Thu, Jan 14, 2016 at 2:19 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 14 January 2016 at 20:12, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 14 January 2016 at 15:55, Nathaniel Smith <njs at pobox.com> wrote:
>>> - build some test wheels
>>> - write a proper PEP
>>> - convince pip and pypi maintainers that this is a good idea ;-)
>>
>> While I've historically advocated against the idea of defining our own
>> "Linux platform ABI" subset, the fact that Enthought and Continuum are
>> successfully distributing pre-built binaries through the simple "use
>> CentOS 5.11" approach seems promising.
>>
>> In terms of non-scientific packages, the main group I'd suggest
>> getting in touch with is pycryptography, as we'll probably want to
>> baseline a more recent version of OpenSSL than the one in CentOS 5.11.
>
> Ah, looking at https://github.com/manylinux/auditwheel, I see anything
> linking to OpenSSL would fail the platform audit, at least for the
> current draft policy. That also seems like a potentially reasonable
> approach (although it could lead to complaints about "Why doesn't
> project <X> offer a wheel file?")

Yeah, it's very unfortunate, that's exactly the library that you'd
like to be able to depend on, so we checked specifically. But it turns
out that openssl has broken ABI over the relevant time-period, so
there's no choice, you can't rely on the system version and have to
statically link :-(.

Though I guess this is no worse than than if you want to distribute a
wheel that needs openssl on Windows.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org

From tritium-list at sdamon.com  Thu Jan 14 06:26:31 2016
From: tritium-list at sdamon.com (Alexander Walters)
Date: Thu, 14 Jan 2016 06:26:31 -0500
Subject: [Distutils] heads-up on a plot to bring linux wheels to pypi
In-Reply-To: <CAPJVwB==vV9=RXSZUYNPApXUVVkKFvGc8oRMLKep_rgEo3jQNw@mail.gmail.com>
References: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
 <CADiSq7enoYVDy_fU_XmvWHOSBTKAxB0coHehf5kt_kb=yiZ4_w@mail.gmail.com>
 <CADiSq7dGZ_5katH25VeZoCuOsyZQECXv3h91wMROEroefO3Vyw@mail.gmail.com>
 <CAPJVwB==vV9=RXSZUYNPApXUVVkKFvGc8oRMLKep_rgEo3jQNw@mail.gmail.com>
Message-ID: <569785E7.5030004@sdamon.com>



On 1/14/2016 06:13, Nathaniel Smith wrote:
> On Thu, Jan 14, 2016 at 2:19 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 14 January 2016 at 20:12, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> On 14 January 2016 at 15:55, Nathaniel Smith <njs at pobox.com> wrote:
>>>> - build some test wheels
>>>> - write a proper PEP
>>>> - convince pip and pypi maintainers that this is a good idea ;-)
>>> While I've historically advocated against the idea of defining our own
>>> "Linux platform ABI" subset, the fact that Enthought and Continuum are
>>> successfully distributing pre-built binaries through the simple "use
>>> CentOS 5.11" approach seems promising.
>>>
>>> In terms of non-scientific packages, the main group I'd suggest
>>> getting in touch with is pycryptography, as we'll probably want to
>>> baseline a more recent version of OpenSSL than the one in CentOS 5.11.
>> Ah, looking at https://github.com/manylinux/auditwheel, I see anything
>> linking to OpenSSL would fail the platform audit, at least for the
>> current draft policy. That also seems like a potentially reasonable
>> approach (although it could lead to complaints about "Why doesn't
>> project <X> offer a wheel file?")
> Yeah, it's very unfortunate, that's exactly the library that you'd
> like to be able to depend on, so we checked specifically. But it turns
> out that openssl has broken ABI over the relevant time-period, so
> there's no choice, you can't rely on the system version and have to
> statically link :-(.
>
> Though I guess this is no worse than than if you want to distribute a
> wheel that needs openssl on Windows.
>
> -n
>
Theoretically you can statically link openssl on windows, or otherwise 
include it.  An odd benefit (in this case, only) of an operating system 
that includes nothing is a culture of vendoring everything.

From glyph at twistedmatrix.com  Thu Jan 14 06:57:38 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Thu, 14 Jan 2016 03:57:38 -0800
Subject: [Distutils] heads-up on a plot to bring linux wheels to pypi
In-Reply-To: <CADiSq7enoYVDy_fU_XmvWHOSBTKAxB0coHehf5kt_kb=yiZ4_w@mail.gmail.com>
References: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
 <CADiSq7enoYVDy_fU_XmvWHOSBTKAxB0coHehf5kt_kb=yiZ4_w@mail.gmail.com>
Message-ID: <53925647-42D8-413E-85D6-2DFFCF77F416@twistedmatrix.com>


> On Jan 14, 2016, at 2:12 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> In terms of non-scientific packages, the main group I'd suggest
> getting in touch with is pycryptography, as we'll probably want to
> baseline a more recent version of OpenSSL than the one in CentOS 5.11.
> 

1. It's "cryptography" (or "PyCA's Cryptography", or "cryptography.io"), not "pycryptography".  This is an important distinction because "PyCrypto" is the crappy, old thing you should not use, and "cryptography" is the new hotness.

2. On every other platform where they distribute wheels, the Cryptography developers have statically linked both OpenSSL and libffi; I was tangentially involved in the effort to do this on OS X, and in the process of debugging that, I learned that the Linux toolchain is fairly similar.  I would imagine that they'd want to statically link OpenSSL the same way, for the same reasons, on Linux.  Cryptography does regular releases to bundle in newer OpenSSLs, generally more often than the underlying platforms do.  (Since Cryptography does not directly export OpenSSL's API as such, it's easier to do multi-verison compatibility with Python than with C.)

In fact I am going to go out on a limb and say that I think Cryptography could be ready to go with this in a few weeks if PyPI just started allowing Linux wheels.  We've discussed using ancient-CentOS containers for building static binaries the same way PyPy does.  The potentially tricky part is just building the static new versions of OpenSSL from scratch on old systems, I think...

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160114/bf9829f6/attachment.html>

From njs at pobox.com  Thu Jan 14 07:01:08 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 14 Jan 2016 04:01:08 -0800
Subject: [Distutils] heads-up on a plot to bring linux wheels to pypi
In-Reply-To: <53925647-42D8-413E-85D6-2DFFCF77F416@twistedmatrix.com>
References: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
 <CADiSq7enoYVDy_fU_XmvWHOSBTKAxB0coHehf5kt_kb=yiZ4_w@mail.gmail.com>
 <53925647-42D8-413E-85D6-2DFFCF77F416@twistedmatrix.com>
Message-ID: <CAPJVwBngmfgpg5LX3LewtcAEBBYPtCWLgiJpAvfAHXTq=F_wsA@mail.gmail.com>

On Thu, Jan 14, 2016 at 3:57 AM, Glyph Lefkowitz
<glyph at twistedmatrix.com> wrote:
[...]
> In fact I am going to go out on a limb and say that I think Cryptography
> could be ready to go with this in a few weeks if PyPI just started allowing
> Linux wheels.  We've discussed using ancient-CentOS containers for building
> static binaries the same way PyPy does.  The potentially tricky part is just
> building the static new versions of OpenSSL from scratch on old systems, I
> think...

https://phusion.github.io/holy-build-box/
https://github.com/phusion/holy-build-box/blob/master/image/build.sh#L308

-n

-- 
Nathaniel J. Smith -- http://vorpus.org

From mailtosijojose at gmail.com  Thu Jan 14 00:29:28 2016
From: mailtosijojose at gmail.com (Sijo Jose)
Date: Thu, 14 Jan 2016 10:59:28 +0530
Subject: [Distutils] Unable to build language-check
In-Reply-To: <CAOTD34a2=MSYoUkjnv6C0DtONizrN7GKocZHpKB+L4L8uqF4dQ@mail.gmail.com>
References: <CAG5wUb_iPZRTJuEiyEGeCWEM1PvjtU+JAARi0aW=RpB8t6182A@mail.gmail.com>
 <CAOTD34a2=MSYoUkjnv6C0DtONizrN7GKocZHpKB+L4L8uqF4dQ@mail.gmail.com>
Message-ID: <CAG5wUb8ZgEN4xDrEAwUQyRB5dKgHyi5G6yhgxkZUnQ+LYCKYLA@mail.gmail.com>

Hi Erik,

I'm not able to build my project using setuptools, it fails to install
language-check library,

see my setup.py below.
------------------
setup(
    name='name',
    version='0.0.1',
    description=' TA',
    long_description=DESCRIPTION,
    license='Proprietary License',
    author='',
    author_email='',
    packages=find_packages(),
    install_requires=[
               "language-check==0.7.5",
               "3to2==1.1.1",
    ],
While executing *python setup.py install*, the language-check dependency is
not getting resolved.
Is there any workaround for this...?

Thanks
Sijo Jose

On Thu, Jan 14, 2016 at 2:24 AM, Erik Bray <erik.m.bray at gmail.com> wrote:

> On Wed, Jan 13, 2016 at 9:30 AM, Sijo Jose <mailtosijojose at gmail.com>
> wrote:
> > The language-check (https://pypi.python.org/pypi/language-check)library
> has
> > the dependency of '3to2' in python 2.7,
> > In a virtualenv it works fine with pip,that is
> >
> > pip install  3to2
> > pip install language-check
> >
> >
> > But its failing with easy_install in venv
> >
> > easy_install 3to2
> > easy_install language-check (It fails)
> >
> > Due to this I'm not able to build my project using setuptools.
>
> Why do you need to install it with easy_install?  pip is the
> recommended installer for Python packages.
>
> Erik
>



-- 
*Regards*
*Sijo Jose*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160114/23ac8269/attachment.html>

From stuaxo2 at yahoo.com  Thu Jan 14 14:47:40 2016
From: stuaxo2 at yahoo.com (Stuart Axon)
Date: Thu, 14 Jan 2016 19:47:40 +0000 (UTC)
Subject: [Distutils] data_files ?
References: <143447675.4210655.1452800860667.JavaMail.yahoo.ref@mail.yahoo.com>
Message-ID: <143447675.4210655.1452800860667.JavaMail.yahoo@mail.yahoo.com>

Hi,? ?Is data_files deprecated ? ? I use it to install files to a known location. ?I had a look at some of the tickets and am still not 100% sure.
package_data won't really work, as the location isn't know.
(I have the package 'vext' which can find the files installed by vext.gi vext.pygtk etc)?S++
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160114/c8821fd9/attachment.html>

From nate at bx.psu.edu  Thu Jan 14 16:47:42 2016
From: nate at bx.psu.edu (Nate Coraor)
Date: Thu, 14 Jan 2016 16:47:42 -0500
Subject: [Distutils] heads-up on a plot to bring linux wheels to pypi
In-Reply-To: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
References: <CAPJVwBkcMRM35pNWzwC879wG2osZ88Oz6=+TBDimOnQPheqkTQ@mail.gmail.com>
Message-ID: <CALT861F+9QLgwOU+ui0dd0sQNB=7ydqjO+VXKOTsj1eCAZm1_g@mail.gmail.com>

On Thu, Jan 14, 2016 at 12:55 AM, Nathaniel Smith <njs at pobox.com> wrote:

> Hi all,
>
> Just wanted give distutils-sig a heads-up that there seems to be some
> momentum gathering around a plot to bring linux wheels to pypi:
>
>   https://github.com/manylinux/manylinux
>
> Basically the idea is to define a standard baseline linux environment
> (available libraries + version numbers + ABI of these), give it a name
> (for now, "manylinux"), and then provide tools to build wheels against
> this environment, teach pip how to recognize that the system it's on
> conforms to the environment, and then convince pypi to start accepting
> wheels with this as a platform tag :-). And if we carefully define the
> baseline environment based on the experiences of the popular
> scientific python distros (Continuum's Anaconda + Enthought's Canopy)
> we can be highly confident that essentially all desktop + server linux
> distros will be compatible with these wheels.
>

Hi Nathaniel,

This is exciting news. As you probably know, I've been working in this
space as well. In fact, the January 2016 release of Galaxy[1] will be the
first to use wheels to distribute or framework dependencies, and they're
installed via pip. We also have a Docker-based build system that automates
the process of building these wheels[2].

Unfortunately, this necessitates our own modified versions of pip[3] and
wheel[4] for messing with the platform tags. My previous discussion here on
distutils-sig on the subject stalled on the discussion of these
environments, but it sounds like you're making progress where that stopped.

If there is momentum to manylinux and it will gain the favor of pip/PyPI
maintainers, I would be happy to transfer my effort there. I've written a
linux distro detection library that may be of some use if it's not possible
to base all environments off Centos 5. I, and the Galaxy project as a
whole, are very motivated to make official Linux wheels a reality.


> This strategy is orthogonal to the more ambitious efforts to define an
> interface between wheels and the platform package manager; they can
> proceed in parallel and potentially complement each other. The goal
> here is just to get linux up to feature parity with windows and osx.
>
> Status:
> - we have a draft policy
> - there's a cool tool for scanning wheels and checking whether they
> conform to the policy: https://github.com/manylinux/auditwheel
> - there's a draft docker image to make it easy to build such wheels
> (https://github.com/manylinux/manylinux/pull/2)
>
> To do:
> - bikeshed the name ("manylinux" was picked after about 2 minutes of
> discussion so as to get started. maybe "genericlinux" would be better?
> I dunno.)
>

FWIW, as an illumos proponent, this same work as is being done for Linux
will transfer over to other "distro" OSes like illumos.


> - build some test wheels
> - write a proper PEP
> - convince pip and pypi maintainers that this is a good idea ;-)
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>


Not directly related, but Galaxy is moving toward Conda as its tool
dependency distribution system, so we are likely to converge efforts in
other places as well.

--nate

[1] https://github.com/galaxyproject/galaxy/
[2] https://github.com/galaxyproject/starforge/
[3] https://github.com/natefoo/pip/tree/linux-wheels
[4] https://bitbucket.org/natefoo/wheel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160114/74a2cbb9/attachment.html>

From p.f.moore at gmail.com  Fri Jan 15 09:12:03 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 15 Jan 2016 14:12:03 +0000
Subject: [Distutils] PEP 440 ad PEP 508 requirement formats
Message-ID: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>

Pip refers to PEP 440 when defining the format of a requirement (see
https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers).
But PEP 508 *also* defines a requirement - the title implies that it's
for dependency specification, but the content of the PEP says "the
language defined is a compact line based format which is already in
widespread use in pip requirements files".

So what's the relationship between PEP 440 and PEP 508? Which one is
the definitive definition of what is a "requirement"? The "packaging"
library implements specifiers which conform (according to the docs) to
PEP 440.

IMO, we're in danger of switching from having too few standards to
having too many...

Can someone clarify?
Thanks,
Paul

From donald at stufft.io  Fri Jan 15 09:14:29 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 15 Jan 2016 09:14:29 -0500
Subject: [Distutils] PEP 440 ad PEP 508 requirement formats
In-Reply-To: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>
References: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>
Message-ID: <427EA1E6-2384-49A5-AE0B-64E7E0BA072D@stufft.io>


> On Jan 15, 2016, at 9:12 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> Pip refers to PEP 440 when defining the format of a requirement (see
> https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers).
> But PEP 508 *also* defines a requirement - the title implies that it's
> for dependency specification, but the content of the PEP says "the
> language defined is a compact line based format which is already in
> widespread use in pip requirements files".
> 
> So what's the relationship between PEP 440 and PEP 508? Which one is
> the definitive definition of what is a "requirement"? The "packaging"
> library implements specifiers which conform (according to the docs) to
> PEP 440.
> 
> IMO, we're in danger of switching from having too few standards to
> having too many...
> 
> Can someone clarify?
> Thanks,
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


A PEP 508 requirement contains a PEP 440 specifier. E.g. something like `requests >= 1.0` is a PEP 508 requirement format, which contains a distribution name `requests` and a PEP 440 version specifier `>= 1.0`.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160115/fa312cd6/attachment-0001.sig>

From p.f.moore at gmail.com  Fri Jan 15 09:23:10 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 15 Jan 2016 14:23:10 +0000
Subject: [Distutils] PEP 440 ad PEP 508 requirement formats
In-Reply-To: <427EA1E6-2384-49A5-AE0B-64E7E0BA072D@stufft.io>
References: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>
 <427EA1E6-2384-49A5-AE0B-64E7E0BA072D@stufft.io>
Message-ID: <CACac1F9X-+iQKSA=Oqr5aFJvK3eDKazUo1D+21S00r0=aTgCnQ@mail.gmail.com>

On 15 January 2016 at 14:14, Donald Stufft <donald at stufft.io> wrote:
> A PEP 508 requirement contains a PEP 440 specifier. E.g. something like `requests >= 1.0` is a PEP 508 requirement format, which contains a distribution name `requests` and a PEP 440 version specifier `>= 1.0`.

Cool. So should the documentation for pip install be updated to say
that a requirement conforms to PEP 508 (minus the url_req part, which
the PEP notes is not implemented by pip)? I'm working on a small doc
update at the moment, so if that's right, I'll make that change too.

Paul

From donald at stufft.io  Fri Jan 15 09:24:17 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 15 Jan 2016 09:24:17 -0500
Subject: [Distutils] PEP 440 ad PEP 508 requirement formats
In-Reply-To: <CACac1F9X-+iQKSA=Oqr5aFJvK3eDKazUo1D+21S00r0=aTgCnQ@mail.gmail.com>
References: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>
 <427EA1E6-2384-49A5-AE0B-64E7E0BA072D@stufft.io>
 <CACac1F9X-+iQKSA=Oqr5aFJvK3eDKazUo1D+21S00r0=aTgCnQ@mail.gmail.com>
Message-ID: <299CCE46-3FB4-4011-A174-C21A58CD8538@stufft.io>


> On Jan 15, 2016, at 9:23 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 15 January 2016 at 14:14, Donald Stufft <donald at stufft.io> wrote:
>> A PEP 508 requirement contains a PEP 440 specifier. E.g. something like `requests >= 1.0` is a PEP 508 requirement format, which contains a distribution name `requests` and a PEP 440 version specifier `>= 1.0`.
> 
> Cool. So should the documentation for pip install be updated to say
> that a requirement conforms to PEP 508 (minus the url_req part, which
> the PEP notes is not implemented by pip)? I'm working on a small doc
> update at the moment, so if that's right, I'll make that change too.
> 
> Paul


Yes, that?s correct.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160115/b49c59b1/attachment.sig>

From ncoghlan at gmail.com  Sat Jan 16 10:46:59 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Jan 2016 01:46:59 +1000
Subject: [Distutils] PEP 440 ad PEP 508 requirement formats
In-Reply-To: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>
References: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>
Message-ID: <CADiSq7cYob8Mf8cnhrEqukJyJ+PH6wsmutYex-oxFobRpWtEGQ@mail.gmail.com>

On 16 January 2016 at 00:12, Paul Moore <p.f.moore at gmail.com> wrote:
> Pip refers to PEP 440 when defining the format of a requirement (see
> https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers).
> But PEP 508 *also* defines a requirement - the title implies that it's
> for dependency specification, but the content of the PEP says "the
> language defined is a compact line based format which is already in
> widespread use in pip requirements files".
>
> So what's the relationship between PEP 440 and PEP 508? Which one is
> the definitive definition of what is a "requirement"? The "packaging"
> library implements specifiers which conform (according to the docs) to
> PEP 440.

As Donald already noted, 508 is the higher level specification, that
includes 440 by reference to cover the "version specifier" section of
a full dependency specifier

The key parts that 508 adds above and beyond 440 are:

- package names (the current rules were previously only given in the 426 draft)
- extras
- environment markers

The main benefit of keeping the two levels separate is that there's a
*lot* of arcana in PEP 440 around version ordering and how that
relates to version comparison operators that you don't really need to
think about when working at the level of processing a list of
dependency specifiers.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From robertc at robertcollins.net  Sat Jan 16 14:13:35 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Sun, 17 Jan 2016 08:13:35 +1300
Subject: [Distutils] PEP 440 ad PEP 508 requirement formats
In-Reply-To: <CADiSq7cYob8Mf8cnhrEqukJyJ+PH6wsmutYex-oxFobRpWtEGQ@mail.gmail.com>
References: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>
 <CADiSq7cYob8Mf8cnhrEqukJyJ+PH6wsmutYex-oxFobRpWtEGQ@mail.gmail.com>
Message-ID: <CAJ3HoZ0NdiaXC7xBYz3WiCx4-LSnFE=KSaa9ci=HYBvwS4ScOg@mail.gmail.com>

Oh that reminds me, I noticed the other day that PEP 426 references
James' draft markers PEP that actually ended up in 508. Whats the
process for fixing that? Just do it in hg?

On 17 January 2016 at 04:46, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 16 January 2016 at 00:12, Paul Moore <p.f.moore at gmail.com> wrote:
>> Pip refers to PEP 440 when defining the format of a requirement (see
>> https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers).
>> But PEP 508 *also* defines a requirement - the title implies that it's
>> for dependency specification, but the content of the PEP says "the
>> language defined is a compact line based format which is already in
>> widespread use in pip requirements files".
>>
>> So what's the relationship between PEP 440 and PEP 508? Which one is
>> the definitive definition of what is a "requirement"? The "packaging"
>> library implements specifiers which conform (according to the docs) to
>> PEP 440.
>
> As Donald already noted, 508 is the higher level specification, that
> includes 440 by reference to cover the "version specifier" section of
> a full dependency specifier
>
> The key parts that 508 adds above and beyond 440 are:
>
> - package names (the current rules were previously only given in the 426 draft)
> - extras
> - environment markers
>
> The main benefit of keeping the two levels separate is that there's a
> *lot* of arcana in PEP 440 around version ordering and how that
> relates to version comparison operators that you don't really need to
> think about when working at the level of processing a list of
> dependency specifiers.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud

From ncoghlan at gmail.com  Sun Jan 17 01:16:39 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Jan 2016 16:16:39 +1000
Subject: [Distutils] PEP 440 ad PEP 508 requirement formats
In-Reply-To: <CAJ3HoZ0NdiaXC7xBYz3WiCx4-LSnFE=KSaa9ci=HYBvwS4ScOg@mail.gmail.com>
References: <CACac1F-BovdcDRPXG0jLA9JXd3cQPycT=Sn7-vScuws-6du+GA@mail.gmail.com>
 <CADiSq7cYob8Mf8cnhrEqukJyJ+PH6wsmutYex-oxFobRpWtEGQ@mail.gmail.com>
 <CAJ3HoZ0NdiaXC7xBYz3WiCx4-LSnFE=KSaa9ci=HYBvwS4ScOg@mail.gmail.com>
Message-ID: <CADiSq7d65Ms4sDAe9irUkqM_AGBRZyPkGD73KbC8MuB=qqLP2w@mail.gmail.com>

On 17 January 2016 at 05:13, Robert Collins <robertc at robertcollins.net> wrote:
> Oh that reminds me, I noticed the other day that PEP 426 references
> James' draft markers PEP that actually ended up in 508. Whats the
> process for fixing that? Just do it in hg?

It's probably better to fix
https://github.com/pypa/interoperability-peps first, and then copy it
over to hg.

With 508 now accepted, it could be useful to strip the now redundant
sections from 426 (I think it's only "extras" that currently goes
beyond what 508 defines).

In the vein of "PEP 426 will be done when we've run out of things to
remove, rather than things to add", I've also been considering the
semantic dependencies idea, and thinking it would be beneficial to
move that out to a proposed extension, while reverting the baseline
metadata to the well established "setup_requires" and
"install_requires".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From njs at pobox.com  Wed Jan 20 22:55:12 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 20 Jan 2016 19:55:12 -0800
Subject: [Distutils] draft PEP: manylinux1
Message-ID: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>

Hi all,

Here's a first draft of a PEP for the manylinux1 platform tag
mentioned earlier, posted for feedback. Really Robert McGibbon should
get the main credit for this, since he wrote it, and also the docker
image and the amazing auditwheel tool linked below, but he asked me to
do the honors of posting it :-).

BTW, if anyone wants to try this out, there are some test
"manylinux1-compatible" wheels at
  https://vorpus.org/~njs/tmp/manylinux-test-wheels/repaired
for PySide (i.e. Qt) and numpy (using openblas). They should be
installable on any ordinary linux system with:
  pip install --no-index -f
https://vorpus.org/~njs/tmp/manylinux-test-wheels/repaired $PKG
(Note that this may require a reasonably up-to-date pip -- e.g. the
one in Debian is too old, which confused me for a bit.)

(How they were created: docker run -it quay.io/manylinux/manylinux
bash; install conda because to get builds of Qt and OpenBLAS because I
was too lazy to do it myself; pip wheel PySide / pip wheel numpy;
auditwheel repair <the resulting wheels>, which copies in all the
dependencies to make the wheels self-contained. Just proof-of-concept
for now, but they seem to work.)

----

PEP: XXXX
Title: A Platform Tag for Portable Linux Built Distributions
Version: $Revision$
Last-Modified: $Date$
Author: Robert T. McGibbon <rmcgibbo at gmail.com>, Nathaniel J. Smith
<njs at pobox.com>
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 19-Jan-2016
Post-History: 19-Jan-2016


Abstract
========

This PEP proposes the creation of a new platform tag for Python package built
distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
external dependencies limited restricted to a standardized subset of
the Linux kernel and core userspace ABI. It proposes that PyPI support
uploading and distributing Wheels with this platform tag, and that ``pip``
support downloading and installing these packages on compatible platforms.


Rationale
=========

Currently, distribution of binary Python extensions for Windows and OS X is
straightforward. Developers and packagers build wheels, which are assigned
platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload these
wheels to PyPI. Users can download and install these wheels using tools such
as ``pip``.

For Linux, the situation is much more delicate. In general, compiled Python
extension modules built on one Linux distribution will not work on other Linux
distributions, or even on the same Linux distribution with different system
libraries installed.

Build tools using PEP 425 platform tags [1]_ do not track information about the
particular Linux distribution or installed system libraries, and instead assign
all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
this ambiguity, there is no expectation that ``linux``-tagged built
distributions compiled on one machine will work properly on another, and for
this reason, PyPI has not permitted the uploading of wheels for Linux.

It would be ideal if wheel packages could be compiled that would work on *any*
linux system. But, because of the incredible diversity of Linux systems -- from
PCs to Android to embedded systems with custom libcs -- this cannot
be guaranteed in general.

Instead, we define a standard subset of the kernel+core userspace ABI that,
in practice, is compatible enough that packages conforming to this standard
will work on *many* linux systems, including essentially all of the desktop
and server distributions in common use. We know this because there are
companies who have been distributing such widely-portable pre-compiled Python
extension modules for Linux -- e.g. Enthought with Canopy [2]_ and Continuum
Analytics with Anaconda [3]_.

Building on the compability lessons learned from these companies, we thus
define a baseline ``manylinux1`` platform tag for use by binary Python
wheels, and introduce the implementation of preliminary tools to aid in the
construction of these ``manylinux1`` wheels.


Key Causes of Inter-Linux Binary Incompatibility
================================================

To properly define a standard that will guarantee that wheel packages meeting
this specification will operate on *many* linux platforms, it is necessary to
understand the root causes which often prevent portability of pre-compiled
binaries on Linux. The two key causes are dependencies on shared libraries
which are not present on users' systems, and dependencies on particular
versions of certain core libraries like ``glibc``.


External Shared Libraries
-------------------------

Most desktop and server linux distributions come with a system package manager
(examples include ``APT`` on Debian-based systems, ``yum`` on
``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
responsibilities, the installation of shared libraries installed to system
directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
on one or more of these shared libraries, and thus function properly only on
systems where the user has the proper libraries (and the proper
versions thereof), either installed using their package manager, or installed
manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
to notify the runtime linker of the location of the depended-upon shared
libraries.


Versioning of Core Shared Libraries
-----------------------------------

Even if author or maintainers of a Python extension module with to use no
external shared libraries, the modules will generally have a dynamic runtime
dependency on the GNU C library, ``glibc``. While it is possible, statically
linking ``glibc`` is usually a bad idea because of bloat, and because certain
important C functions like ``dlopen()`` cannot be called from code that
statically links ``glibc``. A runtime shared library dependency on a
system-provided ``glibc`` is unavoidable in practice.

The maintainers of the GNU C library follow a strict symbol versioning scheme
for backward compatibility. This ensures that binaries compiled against an older
version of ``glibc`` can run on systems that have a newer ``glibc``. The
opposite is generally not true -- binaries compiled on newer Linux
distributions tend to rely upon versioned functions in glibc that are not
available on older systems.

This generally prevents built distributions compiled on the latest Linux
distributions from being portable.


The ``manylinux1`` policy
=========================

For these reasons, to achieve broad portability, Python wheels

 * should depend only on an extremely limited set of external shared
   libraries; and
 * should depend only on ``old`` symbol versions in those external shared
   libraries.

The ``manylinux1`` policy thus encompasses a standard for what the
permitted external shared libraries a wheel may depend on, and the maximum
depended-upon symbol versions therein.

The permitted external shared libraries are: ::

    libpanelw.so.5
    libncursesw.so.5
    libgcc_s.so.1
    libstdc++.so.6
    libm.so.6
    libdl.so.2
    librt.so.1
    libcrypt.so.1
    libc.so.6
    libnsl.so.1
    libutil.so.1
    libpthread.so.0
    libX11.so.6
    libXext.so.6
    libXrender.so.1
    libICE.so.6
    libSM.so.6
    libGL.so.1
    libgobject-2.0.so.0
    libgthread-2.0.so.0
    libglib-2.0.so.0

On Debian-based systems, these libraries are provided by the packages ::

    libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
    libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0

On RPM-based systems, these libraries are provided by the packages ::

    ncurses libgcc libstdc++ glibc libXext libXrender
    libICE libSM mesa-libGL glib2

This list was compiled by checking the external shared library dependencies of
the Canopy [1]_ and Anaconda [2]_ distributions, which both include a wide array
of the most popular Python modules and have been confirmed in practice to work
across a wide swath of Linux systems in the wild.

For dependencies on externally-provided versioned symbols in the above shared
libraries, the following symbol versions are permitted: ::

    GLIBC <= 2.5
    CXXABI <= 3.4.8
    GLIBCXX <= 3.4.9
    GCC <= 4.2.0

These symbol versions were determined by inspecting the latest symbol version
provided in the libraries distributed with CentOS 5, a Linux distribution
released in April 2007. In practice, this means that Python wheels which conform
to this policy should function on almost any linux distribution released after
this date.


Compilation and Tooling
=======================

To support the compilation of wheels meeting the ``manylinux1`` standard, we
provide initial drafts of two tools.

The first is a Docker image based on CentOS 5.11, which is recommended as an
easy to use self-contained build box for compiling ``manylinux1`` wheels [4]_.
Compiling on a more recently-released linux distribution will generally
introduce dependencies on too-new versioned symbols. The image comes with a
full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
well as the latest releases of Python and pip.

The second tool is a command line executable called ``auditwheel`` [5]_. First,
it inspects all of the ELF files inside a wheel to check for dependencies on
versioned symbols or external shared libraries, and verifies conformance with
the ``manylinux1`` policy. This includes the ability to add the new platform
tag to conforming wheels.

In addition, ``auditwheel`` has the ability to automatically modify wheels that
depend on external shared libraries by copying those shared libraries from
the system into the wheel itself, and modifying the appropriate RPATH entries
such that these libraries will be picked up at runtime. This accomplishes a
similar result as if the libraries had been statically linked without requiring
changes to the build system.

Neither of these tools are necessary to build wheels which conform with the
``manylinux1`` policy. Similar results can usually be achieved by statically
linking external dependencies and/or using certain inline assembly constructs
to instruct the linker to prefer older symbol versions, however these tricks
can be quite esoteric.


Platform Detection for Installers
=================================

Because the ``manylinux1`` profile is already known to work for the many
thousands of users of popular commercial Python distributions, we suggest that
installation tools like ``pip`` should error on the side of assuming that a
system *is* compatible, unless there is specific reason to think otherwise.

We know of three main sources of potential incompatibility that are likely to
arise in practice:

* A linux distribution that is too old (e.g. RHEL 4)
* A linux distribution that does not use glibc (e.g. Alpine Linux, which is
  based on musl libc, or Android)
* Eventually, in the future, there may exist distributions that break
  compatibility with this profile

To handle the first two cases, we propose the following simple and reliable
check: ::

    def have_glibc_version(major, minimum_minor):
        import ctypes

        process_namespace = ctypes.CDLL(None)
        try:
            gnu_get_libc_version = process_namespace.gnu_get_libc_version
        except AttributeError:
            # We are not linked to glibc.
            return False

        gnu_get_libc_version.restype = ctypes.c_char_p
        version_str = gnu_get_libc_version()
        # py2 / py3 compatibility:
        if not isinstance(version_str, str):
            version_str = version_str.decode("ascii")

        version = [int(piece) for piece in version_str.split(".")]
        assert len(version) == 2
        if major != version[0]:
            return False
        if minimum_minor > version[1]:
            return False
        return True

    # CentOS 5 uses glibc 2.5.
    is_manylinux1_compatible = have_glibc_version(2, 5)

To handle the third case, we propose the creation of a file
``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
contents: ::

   [manylinux1]
   compatible = true

where the supported values for the ``manylinux1.compatible`` entry are the
same as those supported by the ConfigParser ``getboolean`` method.

The proposed logic for ``pip`` or related tools, then, is:

0) If ``distutils.util.get_platform()`` does not start with the string
   ``"linux"``, then assume the current system is not ``manylinux1``
   compatible.
1) If ``/etc/python/compatibility.conf`` exists and contains a ``manylinux1``
   key, then trust that.
2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
   current system can handle ``manylinux1`` wheels.
3) Otherwise, assume that the current system cannot handle ``manylinux1``
   wheels.


Security Implications
=====================

One of the advantages of dependencies on centralized libraries in Linux is
that bugfixes and security updates can be deployed system-wide, and
applications which depend on on these libraries will automatically feel the
effects of these patches when the underlying libraries are updated. This can
be particularly important for security updates in packages communication
across the network or cryptography.

``manylinux1`` wheels distributed through PyPI that bundle security-critical
libraries like OpenSSL will thus assume responsibility for prompt updates in
response disclosed vulnerabilities and patches. This closely parallels the
security implications of the distribution of binary wheels on Windows that,
because the platform lacks a system package manager, generally bundle their
dependencies. In particular, because its lacks a stable ABI, OpenSSL cannot be
included in the ``manylinux1`` profile.


Rejected Alternatives
=====================

One alternative would be to provide separate platform tags for each Linux
distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
``debian_jessie``, etc. Nothing in this proposal rules out the possibility of
adding such platform tags in the future, or of further extensions to wheel
metadata that would allow wheels to declare dependencies on external
system-installed packages. However, such extensions would require substantially
more work than this proposal, and still might not be appreciated by package
developers who would prefer not to have to maintain multiple build environments
and build multiple wheels in order to cover all the common Linux distributions.
Therefore we consider such proposals to be out-of-scope for this PEP.


References
==========

.. [1] PEP 425 -- Compatibility Tags for Built Distributions
   (https://www.python.org/dev/peps/pep-0425/)
.. [2] Enthought Canopy Python Distribution
   (https://store.enthought.com/downloads/)
.. [3] Continuum Analytics Anaconda Python Distribution
   (https://www.continuum.io/downloads)
.. [4] manylinux1 docker image
   (https://quay.io/repository/manylinux/manylinux)
.. [5] auditwheel
   (https://pypi.python.org/pypi/auditwheel)

Copyright
=========

This document has been placed into the public domain.

..

   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:


-- 
Nathaniel J. Smith -- https://vorpus.org

From randy at thesyrings.us  Wed Jan 20 23:02:55 2016
From: randy at thesyrings.us (Randy Syring)
Date: Wed, 20 Jan 2016 23:02:55 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <56A0586F.9000206@thesyrings.us>

FWIW, really excited to be seeing progress on this!

*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/

On 01/20/2016 10:55 PM, Nathaniel Smith wrote:
> Hi all,
>
> Here's a first draft of a PEP for the manylinux1 platform tag
> mentioned earlier, posted for feedback. Really Robert McGibbon should
> get the main credit for this, since he wrote it, and also the docker
> image and the amazing auditwheel tool linked below, but he asked me to
> do the honors of posting it :-).
>
> BTW, if anyone wants to try this out, there are some test
> "manylinux1-compatible" wheels at
>    https://vorpus.org/~njs/tmp/manylinux-test-wheels/repaired
> for PySide (i.e. Qt) and numpy (using openblas). They should be
> installable on any ordinary linux system with:
>    pip install --no-index -f
> https://vorpus.org/~njs/tmp/manylinux-test-wheels/repaired $PKG
> (Note that this may require a reasonably up-to-date pip -- e.g. the
> one in Debian is too old, which confused me for a bit.)
>
> (How they were created: docker run -it quay.io/manylinux/manylinux
> bash; install conda because to get builds of Qt and OpenBLAS because I
> was too lazy to do it myself; pip wheel PySide / pip wheel numpy;
> auditwheel repair <the resulting wheels>, which copies in all the
> dependencies to make the wheels self-contained. Just proof-of-concept
> for now, but they seem to work.)
>
> ----
>
> PEP: XXXX
> Title: A Platform Tag for Portable Linux Built Distributions
> Version: $Revision$
> Last-Modified: $Date$
> Author: Robert T. McGibbon <rmcgibbo at gmail.com>, Nathaniel J. Smith
> <njs at pobox.com>
> Status: Draft
> Type: Process
> Content-Type: text/x-rst
> Created: 19-Jan-2016
> Post-History: 19-Jan-2016
>
>
> Abstract
> ========
>
> This PEP proposes the creation of a new platform tag for Python package built
> distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
> external dependencies limited restricted to a standardized subset of
> the Linux kernel and core userspace ABI. It proposes that PyPI support
> uploading and distributing Wheels with this platform tag, and that ``pip``
> support downloading and installing these packages on compatible platforms.
>
>
> Rationale
> =========
>
> Currently, distribution of binary Python extensions for Windows and OS X is
> straightforward. Developers and packagers build wheels, which are assigned
> platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload these
> wheels to PyPI. Users can download and install these wheels using tools such
> as ``pip``.
>
> For Linux, the situation is much more delicate. In general, compiled Python
> extension modules built on one Linux distribution will not work on other Linux
> distributions, or even on the same Linux distribution with different system
> libraries installed.
>
> Build tools using PEP 425 platform tags [1]_ do not track information about the
> particular Linux distribution or installed system libraries, and instead assign
> all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
> this ambiguity, there is no expectation that ``linux``-tagged built
> distributions compiled on one machine will work properly on another, and for
> this reason, PyPI has not permitted the uploading of wheels for Linux.
>
> It would be ideal if wheel packages could be compiled that would work on *any*
> linux system. But, because of the incredible diversity of Linux systems -- from
> PCs to Android to embedded systems with custom libcs -- this cannot
> be guaranteed in general.
>
> Instead, we define a standard subset of the kernel+core userspace ABI that,
> in practice, is compatible enough that packages conforming to this standard
> will work on *many* linux systems, including essentially all of the desktop
> and server distributions in common use. We know this because there are
> companies who have been distributing such widely-portable pre-compiled Python
> extension modules for Linux -- e.g. Enthought with Canopy [2]_ and Continuum
> Analytics with Anaconda [3]_.
>
> Building on the compability lessons learned from these companies, we thus
> define a baseline ``manylinux1`` platform tag for use by binary Python
> wheels, and introduce the implementation of preliminary tools to aid in the
> construction of these ``manylinux1`` wheels.
>
>
> Key Causes of Inter-Linux Binary Incompatibility
> ================================================
>
> To properly define a standard that will guarantee that wheel packages meeting
> this specification will operate on *many* linux platforms, it is necessary to
> understand the root causes which often prevent portability of pre-compiled
> binaries on Linux. The two key causes are dependencies on shared libraries
> which are not present on users' systems, and dependencies on particular
> versions of certain core libraries like ``glibc``.
>
>
> External Shared Libraries
> -------------------------
>
> Most desktop and server linux distributions come with a system package manager
> (examples include ``APT`` on Debian-based systems, ``yum`` on
> ``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
> responsibilities, the installation of shared libraries installed to system
> directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
> on one or more of these shared libraries, and thus function properly only on
> systems where the user has the proper libraries (and the proper
> versions thereof), either installed using their package manager, or installed
> manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
> to notify the runtime linker of the location of the depended-upon shared
> libraries.
>
>
> Versioning of Core Shared Libraries
> -----------------------------------
>
> Even if author or maintainers of a Python extension module with to use no
> external shared libraries, the modules will generally have a dynamic runtime
> dependency on the GNU C library, ``glibc``. While it is possible, statically
> linking ``glibc`` is usually a bad idea because of bloat, and because certain
> important C functions like ``dlopen()`` cannot be called from code that
> statically links ``glibc``. A runtime shared library dependency on a
> system-provided ``glibc`` is unavoidable in practice.
>
> The maintainers of the GNU C library follow a strict symbol versioning scheme
> for backward compatibility. This ensures that binaries compiled against an older
> version of ``glibc`` can run on systems that have a newer ``glibc``. The
> opposite is generally not true -- binaries compiled on newer Linux
> distributions tend to rely upon versioned functions in glibc that are not
> available on older systems.
>
> This generally prevents built distributions compiled on the latest Linux
> distributions from being portable.
>
>
> The ``manylinux1`` policy
> =========================
>
> For these reasons, to achieve broad portability, Python wheels
>
>   * should depend only on an extremely limited set of external shared
>     libraries; and
>   * should depend only on ``old`` symbol versions in those external shared
>     libraries.
>
> The ``manylinux1`` policy thus encompasses a standard for what the
> permitted external shared libraries a wheel may depend on, and the maximum
> depended-upon symbol versions therein.
>
> The permitted external shared libraries are: ::
>
>      libpanelw.so.5
>      libncursesw.so.5
>      libgcc_s.so.1
>      libstdc++.so.6
>      libm.so.6
>      libdl.so.2
>      librt.so.1
>      libcrypt.so.1
>      libc.so.6
>      libnsl.so.1
>      libutil.so.1
>      libpthread.so.0
>      libX11.so.6
>      libXext.so.6
>      libXrender.so.1
>      libICE.so.6
>      libSM.so.6
>      libGL.so.1
>      libgobject-2.0.so.0
>      libgthread-2.0.so.0
>      libglib-2.0.so.0
>
> On Debian-based systems, these libraries are provided by the packages ::
>
>      libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
>      libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0
>
> On RPM-based systems, these libraries are provided by the packages ::
>
>      ncurses libgcc libstdc++ glibc libXext libXrender
>      libICE libSM mesa-libGL glib2
>
> This list was compiled by checking the external shared library dependencies of
> the Canopy [1]_ and Anaconda [2]_ distributions, which both include a wide array
> of the most popular Python modules and have been confirmed in practice to work
> across a wide swath of Linux systems in the wild.
>
> For dependencies on externally-provided versioned symbols in the above shared
> libraries, the following symbol versions are permitted: ::
>
>      GLIBC <= 2.5
>      CXXABI <= 3.4.8
>      GLIBCXX <= 3.4.9
>      GCC <= 4.2.0
>
> These symbol versions were determined by inspecting the latest symbol version
> provided in the libraries distributed with CentOS 5, a Linux distribution
> released in April 2007. In practice, this means that Python wheels which conform
> to this policy should function on almost any linux distribution released after
> this date.
>
>
> Compilation and Tooling
> =======================
>
> To support the compilation of wheels meeting the ``manylinux1`` standard, we
> provide initial drafts of two tools.
>
> The first is a Docker image based on CentOS 5.11, which is recommended as an
> easy to use self-contained build box for compiling ``manylinux1`` wheels [4]_.
> Compiling on a more recently-released linux distribution will generally
> introduce dependencies on too-new versioned symbols. The image comes with a
> full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
> well as the latest releases of Python and pip.
>
> The second tool is a command line executable called ``auditwheel`` [5]_. First,
> it inspects all of the ELF files inside a wheel to check for dependencies on
> versioned symbols or external shared libraries, and verifies conformance with
> the ``manylinux1`` policy. This includes the ability to add the new platform
> tag to conforming wheels.
>
> In addition, ``auditwheel`` has the ability to automatically modify wheels that
> depend on external shared libraries by copying those shared libraries from
> the system into the wheel itself, and modifying the appropriate RPATH entries
> such that these libraries will be picked up at runtime. This accomplishes a
> similar result as if the libraries had been statically linked without requiring
> changes to the build system.
>
> Neither of these tools are necessary to build wheels which conform with the
> ``manylinux1`` policy. Similar results can usually be achieved by statically
> linking external dependencies and/or using certain inline assembly constructs
> to instruct the linker to prefer older symbol versions, however these tricks
> can be quite esoteric.
>
>
> Platform Detection for Installers
> =================================
>
> Because the ``manylinux1`` profile is already known to work for the many
> thousands of users of popular commercial Python distributions, we suggest that
> installation tools like ``pip`` should error on the side of assuming that a
> system *is* compatible, unless there is specific reason to think otherwise.
>
> We know of three main sources of potential incompatibility that are likely to
> arise in practice:
>
> * A linux distribution that is too old (e.g. RHEL 4)
> * A linux distribution that does not use glibc (e.g. Alpine Linux, which is
>    based on musl libc, or Android)
> * Eventually, in the future, there may exist distributions that break
>    compatibility with this profile
>
> To handle the first two cases, we propose the following simple and reliable
> check: ::
>
>      def have_glibc_version(major, minimum_minor):
>          import ctypes
>
>          process_namespace = ctypes.CDLL(None)
>          try:
>              gnu_get_libc_version = process_namespace.gnu_get_libc_version
>          except AttributeError:
>              # We are not linked to glibc.
>              return False
>
>          gnu_get_libc_version.restype = ctypes.c_char_p
>          version_str = gnu_get_libc_version()
>          # py2 / py3 compatibility:
>          if not isinstance(version_str, str):
>              version_str = version_str.decode("ascii")
>
>          version = [int(piece) for piece in version_str.split(".")]
>          assert len(version) == 2
>          if major != version[0]:
>              return False
>          if minimum_minor > version[1]:
>              return False
>          return True
>
>      # CentOS 5 uses glibc 2.5.
>      is_manylinux1_compatible = have_glibc_version(2, 5)
>
> To handle the third case, we propose the creation of a file
> ``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
> contents: ::
>
>     [manylinux1]
>     compatible = true
>
> where the supported values for the ``manylinux1.compatible`` entry are the
> same as those supported by the ConfigParser ``getboolean`` method.
>
> The proposed logic for ``pip`` or related tools, then, is:
>
> 0) If ``distutils.util.get_platform()`` does not start with the string
>     ``"linux"``, then assume the current system is not ``manylinux1``
>     compatible.
> 1) If ``/etc/python/compatibility.conf`` exists and contains a ``manylinux1``
>     key, then trust that.
> 2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
>     current system can handle ``manylinux1`` wheels.
> 3) Otherwise, assume that the current system cannot handle ``manylinux1``
>     wheels.
>
>
> Security Implications
> =====================
>
> One of the advantages of dependencies on centralized libraries in Linux is
> that bugfixes and security updates can be deployed system-wide, and
> applications which depend on on these libraries will automatically feel the
> effects of these patches when the underlying libraries are updated. This can
> be particularly important for security updates in packages communication
> across the network or cryptography.
>
> ``manylinux1`` wheels distributed through PyPI that bundle security-critical
> libraries like OpenSSL will thus assume responsibility for prompt updates in
> response disclosed vulnerabilities and patches. This closely parallels the
> security implications of the distribution of binary wheels on Windows that,
> because the platform lacks a system package manager, generally bundle their
> dependencies. In particular, because its lacks a stable ABI, OpenSSL cannot be
> included in the ``manylinux1`` profile.
>
>
> Rejected Alternatives
> =====================
>
> One alternative would be to provide separate platform tags for each Linux
> distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
> ``debian_jessie``, etc. Nothing in this proposal rules out the possibility of
> adding such platform tags in the future, or of further extensions to wheel
> metadata that would allow wheels to declare dependencies on external
> system-installed packages. However, such extensions would require substantially
> more work than this proposal, and still might not be appreciated by package
> developers who would prefer not to have to maintain multiple build environments
> and build multiple wheels in order to cover all the common Linux distributions.
> Therefore we consider such proposals to be out-of-scope for this PEP.
>
>
> References
> ==========
>
> .. [1] PEP 425 -- Compatibility Tags for Built Distributions
>     (https://www.python.org/dev/peps/pep-0425/)
> .. [2] Enthought Canopy Python Distribution
>     (https://store.enthought.com/downloads/)
> .. [3] Continuum Analytics Anaconda Python Distribution
>     (https://www.continuum.io/downloads)
> .. [4] manylinux1 docker image
>     (https://quay.io/repository/manylinux/manylinux)
> .. [5] auditwheel
>     (https://pypi.python.org/pypi/auditwheel)
>
> Copyright
> =========
>
> This document has been placed into the public domain.
>
> ..
>
>     Local Variables:
>     mode: indented-text
>     indent-tabs-mode: nil
>     sentence-end-double-space: t
>     fill-column: 70
>     coding: utf-8
>     End:
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160120/3ee9f1ad/attachment-0001.html>

From ncoghlan at gmail.com  Thu Jan 21 01:17:04 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Jan 2016 16:17:04 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <CADiSq7dnjq=5e+bAurii1H=wbneC4s6dEHeqtv1hmL6khmwUWw@mail.gmail.com>

On 21 January 2016 at 13:55, Nathaniel Smith <njs at pobox.com> wrote:
> Hi all,
>
> Here's a first draft of a PEP for the manylinux1 platform tag
> mentioned earlier, posted for feedback. Really Robert McGibbon should
> get the main credit for this, since he wrote it, and also the docker
> image and the amazing auditwheel tool linked below, but he asked me to
> do the honors of posting it :-).

Very nice!

Do you have a link to the text file version of that? I can assign it a
number and add it to the PEPs repo.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rmcgibbo at gmail.com  Thu Jan 21 01:39:31 2016
From: rmcgibbo at gmail.com (Robert McGibbon)
Date: Wed, 20 Jan 2016 22:39:31 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7dnjq=5e+bAurii1H=wbneC4s6dEHeqtv1hmL6khmwUWw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CADiSq7dnjq=5e+bAurii1H=wbneC4s6dEHeqtv1hmL6khmwUWw@mail.gmail.com>
Message-ID: <CAN4+E8FyJpx6LFo_KcPTpoky5tF8WYgRWmB+v5zqnfeh0yKacA@mail.gmail.com>

Hi Nick,

The text version is here:
https://raw.githubusercontent.com/manylinux/manylinux/master/pep-XXXX.rst

-Robert

On Wed, Jan 20, 2016 at 10:17 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 21 January 2016 at 13:55, Nathaniel Smith <njs at pobox.com> wrote:
> > Hi all,
> >
> > Here's a first draft of a PEP for the manylinux1 platform tag
> > mentioned earlier, posted for feedback. Really Robert McGibbon should
> > get the main credit for this, since he wrote it, and also the docker
> > image and the amazing auditwheel tool linked below, but he asked me to
> > do the honors of posting it :-).
>
> Very nice!
>
> Do you have a link to the text file version of that? I can assign it a
> number and add it to the PEPs repo.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160120/114e9af1/attachment.html>

From ncoghlan at gmail.com  Thu Jan 21 01:50:43 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Jan 2016 16:50:43 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAN4+E8FyJpx6LFo_KcPTpoky5tF8WYgRWmB+v5zqnfeh0yKacA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CADiSq7dnjq=5e+bAurii1H=wbneC4s6dEHeqtv1hmL6khmwUWw@mail.gmail.com>
 <CAN4+E8FyJpx6LFo_KcPTpoky5tF8WYgRWmB+v5zqnfeh0yKacA@mail.gmail.com>
Message-ID: <CADiSq7e6JbuO=igfv1jHpywwNjEUTeATvDt1zwaMWVznv4UqqA@mail.gmail.com>

On 21 January 2016 at 16:39, Robert McGibbon <rmcgibbo at gmail.com> wrote:
> Hi Nick,
>
> The text version is here:
> https://raw.githubusercontent.com/manylinux/manylinux/master/pep-XXXX.rst

Thanks - this is now PEP 513:
https://www.python.org/dev/peps/pep-0513/ (404 response caching on the
site is being its usual annoying self, but that will sort itself out
before too long)

In addition to assigning the PEP number, I also set the BDFL-Delegate
field (to me), fixed the PEP type (Process -> Informational) and set
the Discussions-To header (distutils-sig), so if you could copy those
changes back into your working copy, that would be great.

In terms of the content, it all looks reasonable to me personally, but
we'll wait and see what everyone else has to say (especially the PyPI
and pip developers).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Jan 21 01:51:38 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Jan 2016 16:51:38 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7e6JbuO=igfv1jHpywwNjEUTeATvDt1zwaMWVznv4UqqA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CADiSq7dnjq=5e+bAurii1H=wbneC4s6dEHeqtv1hmL6khmwUWw@mail.gmail.com>
 <CAN4+E8FyJpx6LFo_KcPTpoky5tF8WYgRWmB+v5zqnfeh0yKacA@mail.gmail.com>
 <CADiSq7e6JbuO=igfv1jHpywwNjEUTeATvDt1zwaMWVznv4UqqA@mail.gmail.com>
Message-ID: <CADiSq7cH1qvEEHwtjabzEHSun9j0639pEarpWwFDbt-nWVKVOg@mail.gmail.com>

On 21 January 2016 at 16:50, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 21 January 2016 at 16:39, Robert McGibbon <rmcgibbo at gmail.com> wrote:
>> Hi Nick,
>>
>> The text version is here:
>> https://raw.githubusercontent.com/manylinux/manylinux/master/pep-XXXX.rst
>
> Thanks - this is now PEP 513:
> https://www.python.org/dev/peps/pep-0513/ (404 response caching on the
> site is being its usual annoying self, but that will sort itself out
> before too long)

Apparently this is the "between writing that and hitting send" variant
of "before too long" :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mal at egenix.com  Thu Jan 21 04:03:27 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2016 10:03:27 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <56A09EDF.3000500@egenix.com>


Robert McGibbon: Thanks for writing up this PEP :-)

Some comments below...

On 21.01.2016 04:55, Nathaniel Smith wrote:
> The ``manylinux1`` policy
> =========================
> 
> For these reasons, to achieve broad portability, Python wheels
> 
>  * should depend only on an extremely limited set of external shared
>    libraries; and
>  * should depend only on ``old`` symbol versions in those external shared
>    libraries.
> 
> The ``manylinux1`` policy thus encompasses a standard for what the
> permitted external shared libraries a wheel may depend on, and the maximum
> depended-upon symbol versions therein.
> 
> The permitted external shared libraries are: ::
> 
>     libpanelw.so.5
>     libncursesw.so.5
>     libgcc_s.so.1
>     libstdc++.so.6
>     libm.so.6
>     libdl.so.2
>     librt.so.1
>     libcrypt.so.1
>     libc.so.6
>     libnsl.so.1
>     libutil.so.1
>     libpthread.so.0
>     libX11.so.6
>     libXext.so.6
>     libXrender.so.1
>     libICE.so.6
>     libSM.so.6
>     libGL.so.1
>     libgobject-2.0.so.0
>     libgthread-2.0.so.0
>     libglib-2.0.so.0

The list is good start, but limiting the possible external
references to only these libraries will make it impossible
to have manylinux1 wheels which link against other, similarly
old, but very common libraries, or alternatively rather
new ones, which are then optionally included via subpackage
instead of being mandatory.

At eGenix we have been tackling this problem for years with
our extensions and the approach that's been the most
successful was to simply use Linux build systems which
are at least 5 years old. In our case, that's openSUSE 11.3.

I think a better approach is to use the above list to
test for used library *versions* and then apply the tag
based on the findings.

If a package includes binaries which link to e.g.
later libc.so versions, it would be rejected. If it
includes other libraries not listed in the above listing,
that's fine, as long as these libraries also comply to
the version limitation.

What I'm getting at here is that incompatibilities are
not caused by libraries being absent on the system
(the package simply won't load, but that's not due to the
the package being incompatible to the platform, only due
to the system lacking a few packages), but instead by
having the packages use more recent versions of these
system libraries.

> Compilation and Tooling
> =======================
> 
> To support the compilation of wheels meeting the ``manylinux1`` standard, we
> provide initial drafts of two tools.
> 
> The first is a Docker image based on CentOS 5.11, which is recommended as an
> easy to use self-contained build box for compiling ``manylinux1`` wheels [4]_.
> Compiling on a more recently-released linux distribution will generally
> introduce dependencies on too-new versioned symbols. The image comes with a
> full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
> well as the latest releases of Python and pip.
> 
> The second tool is a command line executable called ``auditwheel`` [5]_. First,
> it inspects all of the ELF files inside a wheel to check for dependencies on
> versioned symbols or external shared libraries, and verifies conformance with
> the ``manylinux1`` policy. This includes the ability to add the new platform
> tag to conforming wheels.
> 
> In addition, ``auditwheel`` has the ability to automatically modify wheels that
> depend on external shared libraries by copying those shared libraries from
> the system into the wheel itself, and modifying the appropriate RPATH entries
> such that these libraries will be picked up at runtime. This accomplishes a
> similar result as if the libraries had been statically linked without requiring
> changes to the build system.

This approach has a few problems:

* Libraries typically depend on a lot more context than just
  the code that is provided in the libraries file, e.g. config
  files, external resources, other libraries which are loaded
  on demand, etc.

* By including the libraries in the wheel you are distributing
  the binary, which can lead to licensing problems, esp. with
  GPLed or LGPLed code.

> Neither of these tools are necessary to build wheels which conform with the
> ``manylinux1`` policy. Similar results can usually be achieved by statically
> linking external dependencies and/or using certain inline assembly constructs
> to instruct the linker to prefer older symbol versions, however these tricks
> can be quite esoteric.

Static linking only helps in very few cases, where the context
needed for the external library to work is minimal.

> Platform Detection for Installers
> =================================
> 
> Because the ``manylinux1`` profile is already known to work for the many
> thousands of users of popular commercial Python distributions, we suggest that
> installation tools like ``pip`` should error on the side of assuming that a
> system *is* compatible, unless there is specific reason to think otherwise.
> 
> We know of three main sources of potential incompatibility that are likely to
> arise in practice:
> 
> * A linux distribution that is too old (e.g. RHEL 4)
> * A linux distribution that does not use glibc (e.g. Alpine Linux, which is
>   based on musl libc, or Android)
> * Eventually, in the future, there may exist distributions that break
>   compatibility with this profile
> 
> To handle the first two cases, we propose the following simple and reliable
> check: ::
>     def have_glibc_version(major, minimum_minor):
>         [...]

It would be better to use platform.libc_ver() for this.

> ``manylinux1`` wheels distributed through PyPI that bundle security-critical
> libraries like OpenSSL will thus assume responsibility for prompt updates in
> response disclosed vulnerabilities and patches. This closely parallels the
> security implications of the distribution of binary wheels on Windows that,
> because the platform lacks a system package manager, generally bundle their
> dependencies. In particular, because its lacks a stable ABI, OpenSSL cannot be
> included in the ``manylinux1`` profile.

The OpenSSL ABI has been quite stable in recent years (unlike in the
days of 0.9.7 and earlier).

Since many libraries do link against OpenSSL (basically everything
that uses network connections nowadays), using the fixed scheme
outlined in the PEP would severely limited the usefulness.

By using the version based approach, we'd not run into this
problem and gain a lot more.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 21 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From rmcgibbo at gmail.com  Thu Jan 21 04:27:09 2016
From: rmcgibbo at gmail.com (Robert McGibbon)
Date: Thu, 21 Jan 2016 01:27:09 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A09EDF.3000500@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
Message-ID: <CAN4+E8GBv6zoAZstQp1SbGG0+bz6BzDb9GgCQgOFvObYwXC+iQ@mail.gmail.com>

After I posted the PEP link on social media, a friend of mine, Kyle
Beauchamp, asked: "I wonder if there are speed and correctness implications
to always reverting to the lowest common denominator of glibc."

For speed, I believe there is some consequence, in that the maximum gcc
version you can realistically use with glibc <= 2.5 is gcc 4.8.2, which is
presumably somewhat (although not much) slower than the latest gcc release.
As to correctness, it seems like a reasonable concern, and I don't know one
way or the other. I thought maybe someone on the list would know.

There are also potential compatibility issues in wheels that use certain
instruction set extensions (SSE 4, AVX, etc) that might not be available on
certain platforms. I think policies on this issue are better left up to
individual packages than set at a PEP-wide level, but it also may be worth
talking about.

-Robert


On Thu, Jan 21, 2016 at 1:03 AM, M.-A. Lemburg <mal at egenix.com> wrote:

>
> Robert McGibbon: Thanks for writing up this PEP :-)
>
> Some comments below...
>
> On 21.01.2016 04:55, Nathaniel Smith wrote:
> > The ``manylinux1`` policy
> > =========================
> >
> > For these reasons, to achieve broad portability, Python wheels
> >
> >  * should depend only on an extremely limited set of external shared
> >    libraries; and
> >  * should depend only on ``old`` symbol versions in those external shared
> >    libraries.
> >
> > The ``manylinux1`` policy thus encompasses a standard for what the
> > permitted external shared libraries a wheel may depend on, and the
> maximum
> > depended-upon symbol versions therein.
> >
> > The permitted external shared libraries are: ::
> >
> >     libpanelw.so.5
> >     libncursesw.so.5
> >     libgcc_s.so.1
> >     libstdc++.so.6
> >     libm.so.6
> >     libdl.so.2
> >     librt.so.1
> >     libcrypt.so.1
> >     libc.so.6
> >     libnsl.so.1
> >     libutil.so.1
> >     libpthread.so.0
> >     libX11.so.6
> >     libXext.so.6
> >     libXrender.so.1
> >     libICE.so.6
> >     libSM.so.6
> >     libGL.so.1
> >     libgobject-2.0.so.0
> >     libgthread-2.0.so.0
> >     libglib-2.0.so.0
>
> The list is good start, but limiting the possible external
> references to only these libraries will make it impossible
> to have manylinux1 wheels which link against other, similarly
> old, but very common libraries, or alternatively rather
> new ones, which are then optionally included via subpackage
> instead of being mandatory.
>
> At eGenix we have been tackling this problem for years with
> our extensions and the approach that's been the most
> successful was to simply use Linux build systems which
> are at least 5 years old. In our case, that's openSUSE 11.3.
>
> I think a better approach is to use the above list to
> test for used library *versions* and then apply the tag
> based on the findings.
>
> If a package includes binaries which link to e.g.
> later libc.so versions, it would be rejected. If it
> includes other libraries not listed in the above listing,
> that's fine, as long as these libraries also comply to
> the version limitation.
>
> What I'm getting at here is that incompatibilities are
> not caused by libraries being absent on the system
> (the package simply won't load, but that's not due to the
> the package being incompatible to the platform, only due
> to the system lacking a few packages), but instead by
> having the packages use more recent versions of these
> system libraries.
>
> > Compilation and Tooling
> > =======================
> >
> > To support the compilation of wheels meeting the ``manylinux1``
> standard, we
> > provide initial drafts of two tools.
> >
> > The first is a Docker image based on CentOS 5.11, which is recommended
> as an
> > easy to use self-contained build box for compiling ``manylinux1`` wheels
> [4]_.
> > Compiling on a more recently-released linux distribution will generally
> > introduce dependencies on too-new versioned symbols. The image comes
> with a
> > full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2)
> as
> > well as the latest releases of Python and pip.
> >
> > The second tool is a command line executable called ``auditwheel`` [5]_.
> First,
> > it inspects all of the ELF files inside a wheel to check for
> dependencies on
> > versioned symbols or external shared libraries, and verifies conformance
> with
> > the ``manylinux1`` policy. This includes the ability to add the new
> platform
> > tag to conforming wheels.
> >
> > In addition, ``auditwheel`` has the ability to automatically modify
> wheels that
> > depend on external shared libraries by copying those shared libraries
> from
> > the system into the wheel itself, and modifying the appropriate RPATH
> entries
> > such that these libraries will be picked up at runtime. This
> accomplishes a
> > similar result as if the libraries had been statically linked without
> requiring
> > changes to the build system.
>
> This approach has a few problems:
>
> * Libraries typically depend on a lot more context than just
>   the code that is provided in the libraries file, e.g. config
>   files, external resources, other libraries which are loaded
>   on demand, etc.
>
> * By including the libraries in the wheel you are distributing
>   the binary, which can lead to licensing problems, esp. with
>   GPLed or LGPLed code.
>
> > Neither of these tools are necessary to build wheels which conform with
> the
> > ``manylinux1`` policy. Similar results can usually be achieved by
> statically
> > linking external dependencies and/or using certain inline assembly
> constructs
> > to instruct the linker to prefer older symbol versions, however these
> tricks
> > can be quite esoteric.
>
> Static linking only helps in very few cases, where the context
> needed for the external library to work is minimal.
>
> > Platform Detection for Installers
> > =================================
> >
> > Because the ``manylinux1`` profile is already known to work for the many
> > thousands of users of popular commercial Python distributions, we
> suggest that
> > installation tools like ``pip`` should error on the side of assuming
> that a
> > system *is* compatible, unless there is specific reason to think
> otherwise.
> >
> > We know of three main sources of potential incompatibility that are
> likely to
> > arise in practice:
> >
> > * A linux distribution that is too old (e.g. RHEL 4)
> > * A linux distribution that does not use glibc (e.g. Alpine Linux, which
> is
> >   based on musl libc, or Android)
> > * Eventually, in the future, there may exist distributions that break
> >   compatibility with this profile
> >
> > To handle the first two cases, we propose the following simple and
> reliable
> > check: ::
> >     def have_glibc_version(major, minimum_minor):
> >         [...]
>
> It would be better to use platform.libc_ver() for this.
>
> > ``manylinux1`` wheels distributed through PyPI that bundle
> security-critical
> > libraries like OpenSSL will thus assume responsibility for prompt
> updates in
> > response disclosed vulnerabilities and patches. This closely parallels
> the
> > security implications of the distribution of binary wheels on Windows
> that,
> > because the platform lacks a system package manager, generally bundle
> their
> > dependencies. In particular, because its lacks a stable ABI, OpenSSL
> cannot be
> > included in the ``manylinux1`` profile.
>
> The OpenSSL ABI has been quite stable in recent years (unlike in the
> days of 0.9.7 and earlier).
>
> Since many libraries do link against OpenSSL (basically everything
> that uses network connections nowadays), using the fixed scheme
> outlined in the PEP would severely limited the usefulness.
>
> By using the version based approach, we'd not run into this
> problem and gain a lot more.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts (#1, Jan 21 2016)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> Python Database Interfaces ...           http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
> ________________________________________________________________________
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
>                       http://www.malemburg.com/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/de4b9614/attachment.html>

From ncoghlan at gmail.com  Thu Jan 21 04:31:16 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Jan 2016 19:31:16 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A09EDF.3000500@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
Message-ID: <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>

On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
> By using the version based approach, we'd not run into this
> problem and gain a lot more.

I think it's better to start with a small core that we *know* works,
then expand later, rather than trying to make the first iteration too
wide. The "manylinux1" tag itself is versioned (hence the "1" at the
end), so "manylinux2" may simply have *more* libraries defined, rather
than newer ones.

The key is that we only have one chance to make a good first
impression with binary Linux wheel support on PyPI, and we want that
to be positive for everyone:

* for publishers, going from "no Linux wheels" to "Linux wheels if you
have few external dependencies beyond glibc" is a big step up (it's
enough for a Cython accelerator module, for example, or a cffi wrapper
around a bundled library)
* for end users, we need to nigh certain that wheels built this way
will *just work*

Even with a small starting list of libraries defined, we're going to
end up with cases where the installed extension module will fail to
load, and end users will have to figure out what dependencies are
missing. The "external dependency specification" at
https://github.com/pypa/interoperability-peps/pull/30 would let pip
detect that at install time (rather the user finding out at runtime
when the module fails to load), but that will still leave the end user
to figure out how to get the external dependencies installed.

If Donald can provide the list of "most downloaded wheel files" for
other platforms, that could also be a useful guide as to how many
source builds may potentially already be avoided through the draft
"manylinux1" definition.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rmcgibbo at gmail.com  Thu Jan 21 04:57:50 2016
From: rmcgibbo at gmail.com (Robert McGibbon)
Date: Thu, 21 Jan 2016 01:57:50 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
Message-ID: <CAN4+E8H8KAde-nxNPqseecYORkff0cAATiYRhTtjyqRev-bEcw@mail.gmail.com>

On Thu, Jan 21, 2016 at 1:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> I think it's better to start with a small core that we *know* works,
> then expand later, rather than trying to make the first iteration too
> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
> end), so "manylinux2" may simply have *more* libraries defined, rather
> than newer ones.
>
> The key is that we only have one chance to make a good first
> impression with binary Linux wheel support on PyPI, and we want that
> to be positive for everyone:
>
> * for publishers, going from "no Linux wheels" to "Linux wheels if you
> have few external dependencies beyond glibc" is a big step up (it's
> enough for a Cython accelerator module, for example, or a cffi wrapper
> around a bundled library)
> * for end users, we need to nigh certain that wheels built this way
> will *just work*
>


In general, I see a tension between permissiveness and flexibility in the
policy
here, with good arguments on both sides. A restrictive policy (like the one
we
propose) will keep some wheels off PyPI that would work just fine on most
Linux boxes. But it will also ensure that fewer broken packages are
uploaded.

In my opinion, the packaging system we have currently works pretty well.
Adopting a loose policy could therefore be experienced as a regression for
users who type ``pip install <package>` and receive a broken binary wheel.
This is one of the reasons we thought that it would be safest to start small
and work incrementally.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/ef902724/attachment.html>

From rmcgibbo at gmail.com  Thu Jan 21 05:02:57 2016
From: rmcgibbo at gmail.com (Robert McGibbon)
Date: Thu, 21 Jan 2016 02:02:57 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAN4+E8H8KAde-nxNPqseecYORkff0cAATiYRhTtjyqRev-bEcw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <CAN4+E8H8KAde-nxNPqseecYORkff0cAATiYRhTtjyqRev-bEcw@mail.gmail.com>
Message-ID: <CAN4+E8EZ4hiQETjJDGD85h+WG9+xYmzMSYtrv2kpb6_E3igJcw@mail.gmail.com>

Another wrinkle: it also should be possible, I think, for wheels to include
binaries that are linked against shared libraries that are provided by other
wheels (that are part of the  install_requires), but I haven't thought much
about this. I know Nathaniel has been thinking about this type of thing for
a separate BLAS wheel package that, for example, numpy could depend
on.

I assume that this type of thing should be kosher within the manylinux1
policy, although the text of the draft PEP does not definitely address it.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/7ec9f84d/attachment.html>

From mal at egenix.com  Thu Jan 21 05:05:42 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2016 11:05:42 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
Message-ID: <56A0AD76.7090107@egenix.com>

On 21.01.2016 10:31, Nick Coghlan wrote:
> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>> By using the version based approach, we'd not run into this
>> problem and gain a lot more.
> 
> I think it's better to start with a small core that we *know* works,
> then expand later, rather than trying to make the first iteration too
> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
> end), so "manylinux2" may simply have *more* libraries defined, rather
> than newer ones.

My argument is that the file based approach taken by the PEP
is too limiting to actually make things work for a large
set of Python packages.

It will basically only work for packages that do not interface
to other external libraries (except for the few cases listed in
the PEP, e.g. X11, GL, which aren't always installed or
available either).

IMO, testing the versions of a set of libraries is a safer
approach. It's perfectly fine to have a few dependencies
not work in a module because an optional system package is not
installed, e.g. say a package comes with UIs written in
Qt and one in GTK.

pip could then warn about missing dependencies in the
installed packages.

Another detail we have found when external dependencies
is that some platforms use different names for the libraries,
e.g. RedHat has a tendency to use non-standard OpenSSL
library names (/lib64/libssl.so.10 instead of the more
common libssl.so.1.0.0).

> The key is that we only have one chance to make a good first
> impression with binary Linux wheel support on PyPI, and we want that
> to be positive for everyone:

Sure, but if we get the concept wrong, it'll be difficult
to switch later on and since there will always be libs not in the
set, we'll need to address this in some way.

> * for publishers, going from "no Linux wheels" to "Linux wheels if you
> have few external dependencies beyond glibc" is a big step up (it's
> enough for a Cython accelerator module, for example, or a cffi wrapper
> around a bundled library)
> * for end users, we need to nigh certain that wheels built this way
> will *just work*
> 
> Even with a small starting list of libraries defined, we're going to
> end up with cases where the installed extension module will fail to
> load, and end users will have to figure out what dependencies are
> missing. The "external dependency specification" at
> https://github.com/pypa/interoperability-peps/pull/30 would let pip
> detect that at install time (rather the user finding out at runtime
> when the module fails to load), but that will still leave the end user
> to figure out how to get the external dependencies installed.
> 
> If Donald can provide the list of "most downloaded wheel files" for
> other platforms, that could also be a useful guide as to how many
> source builds may potentially already be avoided through the draft
> "manylinux1" definition.

I still believe that installers are in a better position to
decide which binary files to install based on the findings
on the installation system. This is why we are using a more
flexible tag system in our prebuilt format:

http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/

In essence, the installer knows which files are available
and can then analyze the installation system to pick the
right binary. Tags can be added as necessary to address all
the different dimensions that need testing, e.g. whether the
binary runs on a Raspi2 or only a Raspi1.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 21 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From ncoghlan at gmail.com  Thu Jan 21 05:11:10 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Jan 2016 20:11:10 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A0AD76.7090107@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
Message-ID: <CADiSq7duwq_2-xG3jn8oCB6bfXiNud3u038Aq3dnFyNH0qcCfg@mail.gmail.com>

On 21 January 2016 at 20:05, M.-A. Lemburg <mal at egenix.com> wrote:
> On 21.01.2016 10:31, Nick Coghlan wrote:
>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>> By using the version based approach, we'd not run into this
>>> problem and gain a lot more.
>>
>> I think it's better to start with a small core that we *know* works,
>> then expand later, rather than trying to make the first iteration too
>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>> end), so "manylinux2" may simply have *more* libraries defined, rather
>> than newer ones.
>
> My argument is that the file based approach taken by the PEP
> is too limiting to actually make things work for a large
> set of Python packages.
>
> It will basically only work for packages that do not interface
> to other external libraries (except for the few cases listed in
> the PEP, e.g. X11, GL, which aren't always installed or
> available either).
>
> IMO, testing the versions of a set of libraries is a safer
> approach.

I still don't really understand what you mean by "testing the versions
of a set of libraries", but if you have the time available to propose
a competing PEP, that always leads to a stronger result than when we
only have only proposed approach to consider.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mal at egenix.com  Thu Jan 21 06:06:34 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2016 12:06:34 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7duwq_2-xG3jn8oCB6bfXiNud3u038Aq3dnFyNH0qcCfg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CADiSq7duwq_2-xG3jn8oCB6bfXiNud3u038Aq3dnFyNH0qcCfg@mail.gmail.com>
Message-ID: <56A0BBBA.3050603@egenix.com>

On 21.01.2016 11:11, Nick Coghlan wrote:
> On 21 January 2016 at 20:05, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>>
>> IMO, testing the versions of a set of libraries is a safer
>> approach.
> 
> I still don't really understand what you mean by "testing the versions
> of a set of libraries", but if you have the time available to propose
> a competing PEP, that always leads to a stronger result than when we
> only have only proposed approach to consider.

I think the PEP is fine as it is, just the restriction to test
for the library file names is something that would need to be
changed to implement the different approach:

"""
For these reasons, we define a set of library versions that
are supported by a wide range of Linux distributions. We therefore
pick library versions which have been around for at least 5 years.

When using these external libraries, Python wheels should
only depend on library versions listed in the section below.

Python wheels are free to depend on additional libraries
not included in this set, however, care should be taken
that these additional libraries do not depend on later
versions of the listed libraries, e.g. OpenSSL libraries
compiled against the C library versions listed below.

The ``manylinux1`` policy thus encompasses a standard for which
versions of these external shared libraries a wheel may depend on,
and the maximum depended-upon symbol versions therein.

Future versions of the manylinux policy may include more
libraries, or move on to later versions.

The permitted external shared libraries and versions for
``manylinux1``are: ::

    libpanelw.so.5
    libncursesw.so.5
    libgcc_s.so.1
    libstdc++.so.6
    ...
"""

This will still lead to cases where a package doesn't work
because of missing system packages, but at least they won't
fail due to mismatch in basic C library versions, which is the
most problematic case for users.

The PEP will also have to address to problems introduced
by versioned symbols in more recent Linux shared libs:
even though the library file names have not changed,
they may well include different support levels for the
various APIs, e.g. glibc 2.1, 2.2, 2.3, etc.

For our binaries, we have chosen to use a system where
this versioning has not yet been enabled for system libs.
We did this because we found using a library compiled
against a versioned lib on a system which comes with
an unversioned lib causes warnings to be issued.

For openSUSE the change was applied between the 11.3 and
11.4 releases.

Some references which show case the problem:
 - ?
http://stackoverflow.com/questions/137773/what-does-the-no-version-information-available-error-from-linux-dynamic-linker/156387#156387
 - ?
http://superuser.com/questions/305055/how-to-diagnosis-and-resolve-usr-lib64-libz-so-1-no-version-information-avail
 - ?
http://forums.opensuse.org/english/get-technical-help-here/applications/466547-google-chrome-issues.html


-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 21 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From leorochael at gmail.com  Thu Jan 21 08:30:05 2016
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Thu, 21 Jan 2016 11:30:05 -0200
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <CAMxX_oPBN5Zsuy_E-D3CE3FSzRKBzwPfH5_3ZT9zESp3F-bkuw@mail.gmail.com>

Hi Nathaniel and Robert,

This is a really nice proposal.

I would only like to see added to this proposal (or another one) that new
versions of installers should offer an option to opt in or out of binary
packages from specific sources.

Right now pip has the switches:

   - --no-binary <format_control>
   - --only-binary <format_control>

These switches (dis)allow downloading binaries for specific packages.

I think it would be nice to have the following options as well:

   - --no-binary-from <origin_control>
   - --only-binary-from <origin_control>

This way I could block binaries from PyPI but allow from my private index
where they were compiled with my favorite compiler or on the closest
platform possible to my production, or with my favorite compilation flags
present in the environment.

As Robert quoted, the "speed and correctness implications to always
reverting to the lowest common denominator of glibc." will result in some
deployments preferring to set up their own indexes or "find-links"
repositories for storing optimized binary packages while allowing PyPI to
provide the rest.

Alternatively, the `<format_control>` specification above could be expanded
to allow not only specifying package names, but package names with their
respective sources.

Regards,

Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/5a867ba6/attachment.html>

From ncoghlan at gmail.com  Thu Jan 21 08:39:48 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Jan 2016 23:39:48 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAMxX_oPBN5Zsuy_E-D3CE3FSzRKBzwPfH5_3ZT9zESp3F-bkuw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CAMxX_oPBN5Zsuy_E-D3CE3FSzRKBzwPfH5_3ZT9zESp3F-bkuw@mail.gmail.com>
Message-ID: <CADiSq7dYto7f2ErrSxO6T59R6nYaHYe+Oo5RRKastuxtSnsHtw@mail.gmail.com>

On 21 January 2016 at 23:30, Leonardo Rochael Almeida
<leorochael at gmail.com> wrote:
> I think it would be nice to have the following options as well:
>
> --no-binary-from <origin_control>
> --only-binary-from <origin_control>
>
> This way I could block binaries from PyPI but allow from my private index
> where they were compiled with my favorite compiler or on the closest
> platform possible to my production, or with my favorite compilation flags
> present in the environment.

I quite like that idea, but I don't think it needs to be linked to the
PEP - it can just be an enhancement request for pip (It applies just
as much to other platforms as it does Linux, and even for Linux,
private indices can already be used to host prebuilt Linux binaries).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From John-Whitlock at ieee.org  Thu Jan 21 09:39:46 2016
From: John-Whitlock at ieee.org (John Whitlock)
Date: Thu, 21 Jan 2016 08:39:46 -0600
Subject: [Distutils] Package classifiers - both major and minor Python
 versions?
Message-ID: <CAJ3y2QRxrSW2dER_2X-7yFJDpPrbNU6f79+p0ki7Kwn-bQayFw@mail.gmail.com>

A discussion about the Python language classifiers came up in a pull
request [1], and I couldn't find a definite answer. The question is -
should a packager specify the major Python versions, minor Python versions,
or both?

The Python Packaging User Guide's example [2] has both:

  # Specify the Python versions you support here. In particular, ensure
  # that you indicate whether you support Python 2, Python 3 or both.
  'Programming Language :: Python :: 2',
  'Programming Language :: Python :: 2.6',
  'Programming Language :: Python :: 2.7',
  'Programming Language :: Python :: 3',
  'Programming Language :: Python :: 3.2',
  'Programming Language :: Python :: 3.3',
  'Programming Language :: Python :: 3.4',

In the example, 'Programming Language :: Python :: 2' is a major version,
and 'Programming Language :: Python :: 2.7' is a minor version.

pyroma [3], which I use as a packaging linter, has insisted on both the
major and minor versions since the first release in 2011 [4].

These were added in 2008, but the announcement on this mailing list didn't
include guidance on usage [5].  I can't find any guidance in PEPs either.

Thanks for your advice,
John Whitlock

[1] https://github.com/zheller/flake8-quotes/pull/14/files#r50356768
[2]
https://python-packaging-user-guide.readthedocs.org/en/latest/distributing/#classifiers
[3] https://bitbucket.org/regebro/pyroma
[4]
https://bitbucket.org/regebro/pyroma/src/9addf825bc8fc3deaf09e724a16cb98b35c53c76/pyroma/tests.py?fileviewer=file-view-default
[5] https://mail.python.org/pipermail/distutils-sig/2008-October/010419.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/f8ac5eb4/attachment-0001.html>

From njs at pobox.com  Thu Jan 21 11:13:12 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jan 2016 08:13:12 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A0AD76.7090107@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
Message-ID: <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>

On Jan 21, 2016 2:07 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>
> On 21.01.2016 10:31, Nick Coghlan wrote:
> > On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
> >> By using the version based approach, we'd not run into this
> >> problem and gain a lot more.
> >
> > I think it's better to start with a small core that we *know* works,
> > then expand later, rather than trying to make the first iteration too
> > wide. The "manylinux1" tag itself is versioned (hence the "1" at the
> > end), so "manylinux2" may simply have *more* libraries defined, rather
> > than newer ones.
>
> My argument is that the file based approach taken by the PEP
> is too limiting to actually make things work for a large
> set of Python packages.
>
> It will basically only work for packages that do not interface
> to other external libraries (except for the few cases listed in
> the PEP, e.g. X11, GL, which aren't always installed or
> available either).

The list of allowed libraries is exactly the same list of libraries as are
required by the Anaconda python distribution, so we *know* that it works
for about a hundred different python packages, including lots of tricky
ones (the whole scientific stack), and had been tested by tens or hundreds
of thousands of users. (I posted a link above to some actual working,
compliant pyside and numpy packages, which we picked for testing because
they're particular interesting/difficult packages that need to interface to
external libraries.) Yes, various extra tricks are needed to get things
working on top of this base, including strategies for shipping libraries
that are not in the baseline set, but these are problems that can be solved
on a project-by-project basis, and don't need a PEP.

[...]
> Another detail we have found when external dependencies
> is that some platforms use different names for the libraries,
> e.g. RedHat has a tendency to use non-standard OpenSSL
> library names (/lib64/libssl.so.10 instead of the more
> common libssl.so.1.0.0).
>
> > The key is that we only have one chance to make a good first
> > impression with binary Linux wheel support on PyPI, and we want that
> > to be positive for everyone:
>
> Sure, but if we get the concept wrong, it'll be difficult
> to switch later on and since there will always be libs not in the
> set, we'll need to address this in some way.

There's no lock-in here -- any alternative approach just needs its own
platform tag. Pypi and pip can easily support multiple such tags at the
same time, if more sophisticated proposals arise in the future. In the mean
time, for packagers targeting manylinux is at least as easy as targeting
windows (which also provides very few libraries "for free").

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/a2dc7424/attachment.html>

From mal at egenix.com  Thu Jan 21 11:53:13 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 21 Jan 2016 17:53:13 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
Message-ID: <56A10CF9.8070801@egenix.com>

On 21.01.2016 17:13, Nathaniel Smith wrote:
> On Jan 21, 2016 2:07 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>>
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
> 
> The list of allowed libraries is exactly the same list of libraries as are
> required by the Anaconda python distribution, so we *know* that it works
> for about a hundred different python packages, including lots of tricky
> ones (the whole scientific stack), and had been tested by tens or hundreds
> of thousands of users. (I posted a link above to some actual working,
> compliant pyside and numpy packages, which we picked for testing because
> they're particular interesting/difficult packages that need to interface to
> external libraries.) Yes, various extra tricks are needed to get things
> working on top of this base, including strategies for shipping libraries
> that are not in the baseline set, but these are problems that can be solved
> on a project-by-project basis, and don't need a PEP.

And that's the problem: The set is limited to the needs
of the scientific community and there to the users of
one or two distributions only.

It doesn't address needs of others that e.g. use Qt or GTK as
basis for GUIs, people using OpenSSL for networking, people
using ImageMagick for processing images, or type libs for
type setting, or sound libs for doing sound processing,
codec libs for video processing, etc.

The idea to include the needed share libs in the wheel
goes completely against the idea of relying on a system
vendor to provide updates and security fixes. In some cases,
this may be reasonable, but as design approach, it's
not a good idea.

> [...]
>> Another detail we have found when external dependencies
>> is that some platforms use different names for the libraries,
>> e.g. RedHat has a tendency to use non-standard OpenSSL
>> library names (/lib64/libssl.so.10 instead of the more
>> common libssl.so.1.0.0).
>>
>>> The key is that we only have one chance to make a good first
>>> impression with binary Linux wheel support on PyPI, and we want that
>>> to be positive for everyone:
>>
>> Sure, but if we get the concept wrong, it'll be difficult
>> to switch later on and since there will always be libs not in the
>> set, we'll need to address this in some way.
> 
> There's no lock-in here -- any alternative approach just needs its own
> platform tag. Pypi and pip can easily support multiple such tags at the
> same time, if more sophisticated proposals arise in the future. In the mean
> time, for packagers targeting manylinux is at least as easy as targeting
> windows (which also provides very few libraries "for free").

Yes, there's no lock-in, there's lock-out :-)

We'd simply not allow people who have other requirements to
upload Linux wheels to PyPI and that's not really acceptable.

Right now, no one can upload Linux wheels, so that a fair
setup.

Using an approach where every single group first has to write
a PEP, get it accepted and have PyPI and pip patched before
they can upload wheels to PyPI does not read like a community
friendly approach.

I also can't imagine that we really want proliferation of
"linux" tags for various purposes or even for various
versions of library catalogs.

What we need is a system that provides a few dimensions
for various system specific differences (e.g. bitness,
architecture) and a recommendation for library
versions of a few very basic libraries to use when
compiling for those systems.

I believe the PEP is almost there, it just needs to use a
slightly different concept, since limiting the set of
allowed libraries does not provide the basis of an open
system which PyPI/pip/wheels need to be.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 21 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From donald at stufft.io  Thu Jan 21 12:18:59 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 21 Jan 2016 12:18:59 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
Message-ID: <9F10A851-ED9B-4C6C-A116-61017E468D78@stufft.io>


> On Jan 21, 2016, at 4:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> If Donald can provide the list of "most downloaded wheel files" for
> other platforms, that could also be a useful guide as to how many
> source builds may potentially already be avoided through the draft
> "manylinux1" definition.


SELECT
  COUNT(*) AS downloads,
  file.filename
FROM
  TABLE_DATE_RANGE( [long-stack-762:pypi.downloads], TIMESTAMP("20160114"), CURRENT_TIMESTAMP() )
WHERE
  file.type = 'bdist_wheel'
  AND NOT REGEXP_MATCH(file.filename, '^.*-none-any.whl$')
GROUP BY
  file.filename
ORDER BY
  downloads DESC
LIMIT
  1000


https://gist.github.com/dstufft/1dda9a9f87ee7121e0ee

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/58963c2d/attachment.sig>

From donald at stufft.io  Thu Jan 21 12:23:31 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 21 Jan 2016 12:23:31 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <9F10A851-ED9B-4C6C-A116-61017E468D78@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <9F10A851-ED9B-4C6C-A116-61017E468D78@stufft.io>
Message-ID: <D7809959-8B6A-40FD-BD87-25402692765B@stufft.io>


> On Jan 21, 2016, at 12:18 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Jan 21, 2016, at 4:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> 
>> If Donald can provide the list of "most downloaded wheel files" for
>> other platforms, that could also be a useful guide as to how many
>> source builds may potentially already be avoided through the draft
>> "manylinux1" definition.
> 


Or https://gist.github.com/dstufft/ea8a95580b022b233635 if you prefer it grouped by project.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/9b969f22/attachment.sig>

From donald at stufft.io  Thu Jan 21 12:32:11 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 21 Jan 2016 12:32:11 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <C628BE1F-59E9-4842-A2EE-300E42BE9594@stufft.io>


> On Jan 20, 2016, at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> The permitted external shared libraries are: ::
> 
>    libpanelw.so.5
>    libncursesw.so.5
>    libgcc_s.so.1
>    libstdc++.so.6
>    libm.so.6
>    libdl.so.2
>    librt.so.1
>    libcrypt.so.1
>    libc.so.6
>    libnsl.so.1
>    libutil.so.1
>    libpthread.so.0
>    libX11.so.6
>    libXext.so.6
>    libXrender.so.1
>    libICE.so.6
>    libSM.so.6
>    libGL.so.1
>    libgobject-2.0.so.0
>    libgthread-2.0.so.0
>    libglib-2.0.so.0


Forgive my, probably stupid, question? but why these libraries? Are these
libraries unable to be reasonably statically linked like glibc is?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/6c585587/attachment.sig>

From brett at python.org  Thu Jan 21 12:53:32 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Jan 2016 17:53:32 +0000
Subject: [Distutils] Package classifiers - both major and minor Python
 versions?
In-Reply-To: <CAJ3y2QRxrSW2dER_2X-7yFJDpPrbNU6f79+p0ki7Kwn-bQayFw@mail.gmail.com>
References: <CAJ3y2QRxrSW2dER_2X-7yFJDpPrbNU6f79+p0ki7Kwn-bQayFw@mail.gmail.com>
Message-ID: <CAP1=2W4RYmnsy41y144eAMYptNQge3XrbAPgSu+VNbeCBEOXcQ@mail.gmail.com>

On Thu, 21 Jan 2016 at 06:48 John Whitlock <John-Whitlock at ieee.org> wrote:

> A discussion about the Python language classifiers came up in a pull
> request [1], and I couldn't find a definite answer. The question is -
> should a packager specify the major Python versions, minor Python versions,
> or both?
>
> The Python Packaging User Guide's example [2] has both:
>
>   # Specify the Python versions you support here. In particular, ensure
>   # that you indicate whether you support Python 2, Python 3 or both.
>   'Programming Language :: Python :: 2',
>   'Programming Language :: Python :: 2.6',
>   'Programming Language :: Python :: 2.7',
>   'Programming Language :: Python :: 3',
>   'Programming Language :: Python :: 3.2',
>   'Programming Language :: Python :: 3.3',
>   'Programming Language :: Python :: 3.4',
>
> In the example, 'Programming Language :: Python :: 2' is a major version,
> and 'Programming Language :: Python :: 2.7' is a minor version.
>
> pyroma [3], which I use as a packaging linter, has insisted on both the
> major and minor versions since the first release in 2011 [4].
>
> These were added in 2008, but the announcement on this mailing list didn't
> include guidance on usage [5].  I can't find any guidance in PEPs either.
>

You should at least do the major versions, and if you are up for
maintaining them, then do the minor ones as well.

You want the major ones to tell people whether you even support Python 3 or
not. Various tools like caniusepython3 rely on at least the major version
classifier existing to programmatically know about Python 3 support.

Minor support is good to let people know what versions you have tested
against and officially support. This is especially useful right after a new
version of Python is released as it lets people know whether you have
tested against the new release yet or not. It also lets people know if you
have dropped support for an older version of Python.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/aa8172f5/attachment.html>

From njs at pobox.com  Thu Jan 21 13:43:24 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jan 2016 10:43:24 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <C628BE1F-59E9-4842-A2EE-300E42BE9594@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <C628BE1F-59E9-4842-A2EE-300E42BE9594@stufft.io>
Message-ID: <CAPJVwBkdsRFP65CT6aB_USrnUFpte=9ZvbWB0G5uWSC1O4a=zA@mail.gmail.com>

On Jan 21, 2016 9:32 AM, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> > On Jan 20, 2016, at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >
> > The permitted external shared libraries are: ::
> >
> >    libpanelw.so.5
> >    libncursesw.so.5
> >    libgcc_s.so.1
> >    libstdc++.so.6
> >    libm.so.6
> >    libdl.so.2
> >    librt.so.1
> >    libcrypt.so.1
> >    libc.so.6
> >    libnsl.so.1
> >    libutil.so.1
> >    libpthread.so.0
> >    libX11.so.6
> >    libXext.so.6
> >    libXrender.so.1
> >    libICE.so.6
> >    libSM.so.6
> >    libGL.so.1
> >    libgobject-2.0.so.0
> >    libgthread-2.0.so.0
> >    libglib-2.0.so.0
>
>
> Forgive my, probably stupid, question? but why these libraries? Are these
> libraries unable to be reasonably statically linked like glibc is?

It's not a stupid question at all :-).

What we want is a list of libraries that are present, and compatible, and
relevant, across the Linux distributions that most people use in practice.

Unfortunately, there isn't really any formal specification or published
data on this -- the only way to make such a list is to make a guess, start
distributing software to lots of people based on your guess, wait for bug
reports to come in, edit your list to avoid the problematic libraries, add
some more libraries to your list to cover the new packages you've added to
your distribution and where you think the risk is a worthwhile trade off
versus just static linking, then repeat until you stop getting bug reports.

This is expensive and time consuming and requires data we don't have, so we
delegated: the list in the PEP is exactly the set of libraries that
Continuum's Anaconda distribution links to and a subset of the libraries
that Enthought's Canopy links to [1]. They've both been going through the
loop above for years, with lots of packages, and lots of lots of downloads,
so we're piggybacking off their effort.

Here's the list of packages in Anaconda:
http://docs.continuum.io/anaconda/pkg-docs

-n

[1] why only a subset of Canopy's libraries? because when we analyzed the
latest release of Canopy we found several weird libraries that we knew must
be mistakes and being pulled in by code paths that were never used in
practice, so we decided to be conservative about trusting their library
list. We're in contact with their distribution folks and might add some
more libraries to this list if they come back to us and assure us that
those libraries have actually been tested.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/885267c6/attachment-0001.html>

From rmcgibbo at gmail.com  Thu Jan 21 14:16:47 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 11:16:47 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A10CF9.8070801@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
Message-ID: <CAN4+E8EXJQFE9ycUOBdu7njSV2o3Oh2P621x=NGZEJ9k5S2anA@mail.gmail.com>

On Thu, Jan 21, 2016 at 8:53 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> [...]
> What we need is a system that provides a few dimensions
> for various system specific differences (e.g. bitness,
> architecture) and a recommendation for library
> versions of a few very basic libraries to use when
> compiling for those systems.
>
> I believe the PEP is almost there, it just needs to use a
> slightly different concept, since limiting the set of
> allowed libraries does not provide the basis of an open
> system which PyPI/pip/wheels need to be.



Sorry, I still don't think I quite understand. Is your position essentially
that we should allow wheels to link against any system library included
in (for example) stock openSUSE 11.5? Or that we should allow wheels
to link against any library that can be installed into openSUSE 11.5 using
`yum install <library>`?

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/1f2448b7/attachment.html>

From njs at pobox.com  Thu Jan 21 14:37:06 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jan 2016 11:37:06 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A10CF9.8070801@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
Message-ID: <CAPJVwBk=WeEER=vjA8_yESukF7+raG9rUFHv8ogLiG1iYaK5aw@mail.gmail.com>

On Jan 21, 2016 8:53 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>
[...]
> And that's the problem: The set is limited to the needs
> of the scientific community and there to the users of
> one or two distributions only.
>
> It doesn't address needs of others that e.g. use Qt or GTK as
> basis for GUIs, people using OpenSSL for networking, people
> using ImageMagick for processing images, or type libs for
> type setting, or sound libs for doing sound processing,
> codec libs for video processing, etc.

I've pointed out several times now that our first test package was Qt
bindings, and Glyph told us last week that this proposal is exactly how the
cryptography package wants to handle their openssl dependency:
https://www.mail-archive.com/distutils-sig at python.org/msg23506.html
So this paragraph is just you making stuff up.

Is manylinux1 the perfect panacea for every package? Probably not. In
particular it's great for popular cross platform packages, because it works
now and means they can basically reuse the work that they're already doing
to make static windows and OSX wheels; it's less optimal for smaller
Linux-specific packages that might prefer to take more than of Linux's
unique package management functionality and only care about targeting one
or two distros.

> The idea to include the needed share libs in the wheel
> goes completely against the idea of relying on a system
> vendor to provide updates and security fixes. In some cases,
> this may be reasonable, but as design approach, it's
> not a good idea.
>
> > [...]
> >> Another detail we have found when external dependencies
> >> is that some platforms use different names for the libraries,
> >> e.g. RedHat has a tendency to use non-standard OpenSSL
> >> library names (/lib64/libssl.so.10 instead of the more
> >> common libssl.so.1.0.0).
> >>
> >>> The key is that we only have one chance to make a good first
> >>> impression with binary Linux wheel support on PyPI, and we want that
> >>> to be positive for everyone:
> >>
> >> Sure, but if we get the concept wrong, it'll be difficult
> >> to switch later on and since there will always be libs not in the
> >> set, we'll need to address this in some way.
> >
> > There's no lock-in here -- any alternative approach just needs its own
> > platform tag. Pypi and pip can easily support multiple such tags at the
> > same time, if more sophisticated proposals arise in the future. In the
mean
> > time, for packagers targeting manylinux is at least as easy as targeting
> > windows (which also provides very few libraries "for free").
>
> Yes, there's no lock-in, there's lock-out :-)
>
> We'd simply not allow people who have other requirements to
> upload Linux wheels to PyPI and that's not really acceptable.
>
> Right now, no one can upload Linux wheels, so that a fair
> setup.

The fairness that I'm more worried about is that right now Windows and OSX
users get wheels, and Linux users don't. Feature parity across these
platforms isn't everything, but it's a good start.

> Using an approach where every single group first has to write
> a PEP, get it accepted and have PyPI and pip patched before
> they can upload wheels to PyPI does not read like a community
> friendly approach.
>
> I also can't imagine that we really want proliferation of
> "linux" tags for various purposes or even for various
> versions of library catalogs.
>
> What we need is a system that provides a few dimensions
> for various system specific differences (e.g. bitness,
> architecture) and a recommendation for library
> versions of a few very basic libraries to use when
> compiling for those systems.
>
> I believe the PEP is almost there, it just needs to use a
> slightly different concept, since limiting the set of
> allowed libraries does not provide the basis of an open
> system which PyPI/pip/wheels need to be.

Like Nick, I'm still not sure what this "slightly different concept" you
keep referring to is.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/dcbc7f77/attachment.html>

From chris.barker at noaa.gov  Thu Jan 21 14:42:57 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 21 Jan 2016 11:42:57 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>

nice, idea, but....

    libX11.so.6
    libXext.so.6
    libXrender.so.1
    libGL.so.1

These are all X11, yes? pretty much any workstation will have these, but in
general, servers won't.

Someone on this thread suggested that that's OK -- don't expect a GUI
package to work on a linux box without a GUI. But some of these libs might
get used for stuff like back-end rendering, etc, so would be expected to
work on a headless box. I think Anaconda an Canopy have gotten away with
this because both of their user bases are primarily desktop data analysis
type of stuff -- not web services, web servers, etc.


Then there is the "additional libs" problem. Again, Anaconda and Canopy can
do this because they are providing those libs - lots of stuff you'd
generally expect to be there in a typical *nix system stuff like libpng,
libjpeg, who knows what they heck else.

So without a plan to provide all that stuff -- I"m not sure of the utility
of this -- how are you gong to get PIL/Pillow to work? statically link up
the ying-yang? Not sure the linux world will take to that.

Anyway, maybe we just need to try and see how it shakes out....

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/7da551d5/attachment-0001.html>

From chris.barker at noaa.gov  Thu Jan 21 14:54:24 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 21 Jan 2016 11:54:24 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk=WeEER=vjA8_yESukF7+raG9rUFHv8ogLiG1iYaK5aw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
 <CAPJVwBk=WeEER=vjA8_yESukF7+raG9rUFHv8ogLiG1iYaK5aw@mail.gmail.com>
Message-ID: <CALGmxEKc3ud3oZ2PxQqd2QrkPgv88FaOov9545yxCqKiVEA0zg@mail.gmail.com>

On Thu, Jan 21, 2016 at 11:37 AM, Nathaniel Smith <njs at pobox.com> wrote:

>  Glyph told us last week that this proposal is exactly how the
> cryptography package wants to handle their openssl dependency:
> https://www.mail-archive.com/distutils-sig at python.org/msg23506.html
>
> well, SSL is a pretty unique case --  there's one where controlling the
version of the lib, and having it be recent it critical.

We will have issues with all sorts of other "Pretty common, but can't count
on it" libs.

> Is manylinux1 the perfect panacea for every package? Probably not. In
> particular it's great for popular cross platform packages, because it works
> now and means they can basically reuse the work that
>
they're already doing to make static windows and OSX wheels;
>
except that static linking is a pain on LInux  -- the toolchain really
doesn't want you to do that :-) It's also not part of the culture.

Windows is "working" because of Chris Gohlke's heroic efforts.

OS-X is kind-of sort of working, because of Matthew Brett's also heroic
efforts.

But Anaconda and Canopy exist, and are popular, for a reason -- they solve
a very real problem, and many linux is only solving a very small part of
that problem -- the easy part.

Maybe there will be a Gohlke-like figure that will step up and build
statically linked wheels for all sorts of stuff -- but is that the end-game
we want anyway? everyting statically linked?

One plus -- with Docker and CI systems, it's getting pretty easy to set up
a build sandbox that only has the manylinux libs on it -- so not too hard
to automate and test your builds....

> The idea to include the needed share libs in the wheel
> > goes completely against the idea of relying on a system
> > vendor to provide updates and security fixes. In some cases,
> > this may be reasonable, but as design approach, it's
> > not a good idea.
>
Is this any different than static linking -- probably not. And that's
pretty much what I mean by the culture os dynamic linking on Linux.

On Windows, dll hell is such that we've all accepted that we're going to
need to statically link and provide dlls.

On OS-X, at least the base system is pretty well defined --  though I sure
wish they'd supply more of what really should be basic libs.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/5c4215d9/attachment.html>

From rmcgibbo at gmail.com  Thu Jan 21 14:55:05 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 11:55:05 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
Message-ID: <CAN4+E8HevADw+=EZWfGAwAZJPkHY2zobBMGAj8ZJOzrtMou=0w@mail.gmail.com>

On Jan 21, 2016 11:43 AM, "Chris Barker" <chris.barker at noaa.gov> wrote:
>
> nice, idea, but....
>
>     libX11.so.6
>     libXext.so.6
>     libXrender.so.1
>     libGL.so.1
>
> These are all X11, yes? pretty much any workstation will have these, but
in general, servers won't.
>
> Someone on this thread suggested that that's OK -- don't expect a GUI
package to work on a linux box without a GUI. But some of these libs might
get used for stuff like back-end rendering, etc, so would be expected to
work on a headless box. I think Anaconda an Canopy have gotten away with
this because both of their user bases are primarily desktop data analysis
type of stuff -- not web services, web servers, etc.

I can only speak for myself and my team, but we use Anaconda on servers on
a daily basis, including with libraries like matplotlib to generate images
that are displayed over a web service.

I believe this is a pretty common use case, especially with popular apps
like Jupyter servers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/d4cb3949/attachment.html>

From chris.barker at noaa.gov  Thu Jan 21 14:57:33 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 21 Jan 2016 11:57:33 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAH6Pt5qYxa-Z90MVE4_cLL+N73rhiUmfq4Nb8jO9C=oM_55FJQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAH6Pt5qYxa-Z90MVE4_cLL+N73rhiUmfq4Nb8jO9C=oM_55FJQ@mail.gmail.com>
Message-ID: <CALGmxE+VgFBpx82DSY9GNUEeq=CNRXbKFHCDM7yzw9G5amy6eQ@mail.gmail.com>

On Thu, Jan 21, 2016 at 11:48 AM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> > So without a plan to provide all that stuff -- I"m not sure of the
> utility
> > of this -- how are you gong to get PIL/Pillow to work? statically link up
> > the ying-yang? Not sure the linux world will take to that.
>
> That's exactly how the current OSX Pillow wheels work, and they've
> been working fine for a while now.  There just aren't that many
> libraries to worry about for the vast majority of packages.
>

well, that was really intended to be only an example. And OS-X provides
more basic libs than manylinux anyway (zlib, freetype, either png or jpeg,
can't remember which).

The library list got long enough to drive me crazy -- I guess you've got
more patience than I have. Tried to build any OSGEO stuff lately?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/264dc1f5/attachment.html>

From p.f.moore at gmail.com  Thu Jan 21 14:58:15 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 21 Jan 2016 19:58:15 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk=WeEER=vjA8_yESukF7+raG9rUFHv8ogLiG1iYaK5aw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
 <CAPJVwBk=WeEER=vjA8_yESukF7+raG9rUFHv8ogLiG1iYaK5aw@mail.gmail.com>
Message-ID: <CACac1F8WQvWheoM1viTTbxbTdyxTfY5_YxaQ-vMtLXX43EjwWg@mail.gmail.com>

On 21 January 2016 at 19:37, Nathaniel Smith <njs at pobox.com> wrote:
>> Right now, no one can upload Linux wheels, so that a fair
>> setup.
>
> The fairness that I'm more worried about is that right now Windows and OSX
> users get wheels, and Linux users don't. Feature parity across these
> platforms isn't everything, but it's a good start.

100% agreed. I don't use Linux, but as a Windows user, the fact that
Linux users are excluded from using wheels saddens me because it means
that without support for the large base of Linux users we still don't
have a good binary distribution story.

The story isn't perfect on Windows or OSX either. Let's not let a
quest for perfection stand in the way of helping people get stuff
done.

>> Using an approach where every single group first has to write
>> a PEP, get it accepted and have PyPI and pip patched before
>> they can upload wheels to PyPI does not read like a community
>> friendly approach.

The valid Linux tags are defined in a pep, and coded in pip. But
experience has shown that those tags are not a usable solution. Fixing
that *requires* a PEP and a change to pip. There's no way round that.
But equally, there's no reason we can't have more than one PEP/change.
Just because someone has done the work and offered a PEP doesn't
preclude anyone else doing so to. Contrariwise, if Nathaniel's PEP
wasn't around, that wouldn't stop anyone with an alternative proposal
from having to write a PEP.

So I'm not sure what you're saying here. That the PEP process in
general is not community friendly? That writing a PEP that turns out
to be insufficient in one small area is somehow a huge issue for the
community? That the wheel spec should never have been subject to the
PEP process? That we should let people upload wheels with the
insufficiently precise "linux" tag to PyPI and let the users sort out
the resulting mess?

OTOH, if you're suggesting that people might be put off proposing
alternatives to the manylinux suggestion because of the grief
Nathaniel is getting over what is (IMO) a practical, well-researched,
and field tested suggestion, then you're possibly right. But the
easiest way to fix that is to accept that it's a good proposal
(possibly only an initial step, but that's fine) and approve it,
rather than requiring it to be perfect (as opposed to merely
significantly better than what we now have).

FWIW, the proposal has a solid +1 from me.

Paul

From chris.barker at noaa.gov  Thu Jan 21 14:59:33 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 21 Jan 2016 11:59:33 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxEJ4y0ZzYU3POQinzSdjPijYwZQJ4un6XskSbjRZDpBFXQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAN4+E8HevADw+=EZWfGAwAZJPkHY2zobBMGAj8ZJOzrtMou=0w@mail.gmail.com>
 <CALGmxEJ4y0ZzYU3POQinzSdjPijYwZQJ4un6XskSbjRZDpBFXQ@mail.gmail.com>
Message-ID: <CALGmxELkoYj_LTxFBhyaCn9tB1ostJ7XgNc4yso0A00cTo=CHA@mail.gmail.com>

> On Thu, Jan 21, 2016 at 11:55 AM, Robert T. McGibbon <rmcgibbo at gmail.com>
> wrote:
>
>>
>> > These are all X11, yes? pretty much any workstation will have these,
>> but in general, servers won't.
>> >
>> > Someone on this thread suggested that that's OK -- don't expect a GUI
>> package to work on a linux box without a GUI. But some of these libs might
>> get used for stuff like back-end rendering, etc, so would be expected to
>> work on a headless box. I think Anaconda an Canopy have gotten away with
>> this because both of their user bases are primarily desktop data analysis
>> type of stuff -- not web services, web servers, etc.
>>
>> I can only speak for myself and my team, but we use Anaconda on servers
>> on a daily basis, including with libraries like matplotlib to generate
>> images that are displayed over a web service.
>>
>> I believe this is a pretty common use case, especially with popular apps
>> like Jupyter servers.
>>
> yup -- and Bokeh -- getting more popular -- do you put X11 on your servers?
>
>
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/31189786/attachment.html>

From rmcgibbo at gmail.com  Thu Jan 21 15:02:44 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 12:02:44 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxEKc3ud3oZ2PxQqd2QrkPgv88FaOov9545yxCqKiVEA0zg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
 <CAPJVwBk=WeEER=vjA8_yESukF7+raG9rUFHv8ogLiG1iYaK5aw@mail.gmail.com>
 <CALGmxEKc3ud3oZ2PxQqd2QrkPgv88FaOov9545yxCqKiVEA0zg@mail.gmail.com>
Message-ID: <CAN4+E8ExLEPLUCXKkG1+0bh9hC+QjH9oZyEAgDC+JCad_XgiEA@mail.gmail.com>

On Jan 21, 2016 11:55 AM, "Chris Barker" <chris.barker at noaa.gov> wrote:
>
> On Thu, Jan 21, 2016 at 11:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>>  Glyph told us last week that this proposal is exactly how the
cryptography package wants to handle their openssl dependency:
https://www.mail-archive.com/distutils-sig at python.org/msg23506.html
>
> well, SSL is a pretty unique case --  there's one where controlling the
version of the lib, and having it be recent it critical.
>
> We will have issues with all sorts of other "Pretty common, but can't
count on it" libs.
>>
>> Is manylinux1 the perfect panacea for every package? Probably not. In
particular it's great for popular cross platform packages, because it works
now and means they can basically reuse the work that
>>
>> they're already doing to make static windows and OSX wheels;
>
> except that static linking is a pain on LInux  -- the toolchain really
doesn't want you to do that :-) It's also not part of the culture.
>
> Windows is "working" because of Chris Gohlke's heroic efforts.
>
> OS-X is kind-of sort of working, because of Matthew Brett's also heroic
efforts.
>
> But Anaconda and Canopy exist, and are popular, for a reason -- they
solve a very real problem, and many linux is only solving a very small part
of that problem -- the easy part.
>
> Maybe there will be a Gohlke-like figure that will step up and build
statically linked wheels for all sorts of stuff -- but is that the end-game
we want anyway? everyting statically linked?
>
> One plus -- with Docker and CI systems, it's getting pretty easy to set
up a build sandbox that only has the manylinux libs on it -- so not too
hard to automate and test your builds....
>
>> > The idea to include the needed share libs in the wheel
>> > goes completely against the idea of relying on a system
>> > vendor to provide updates and security fixes. In some cases,
>> > this may be reasonable, but as design approach, it's
>> > not a good idea.
>
> Is this any different than static linking -- probably not. And that's
pretty much what I mean by the culture os dynamic linking on Linux.

The difference between static linking and vendoring the shared libraries
into the wheel using ``auditwheel repair`` is essentially just that the
second option is much easier because it doesn't require modifications to
the build system. Other than that, they're about the same.

This is why Nathaniel was able to make a PySide wheel in 5 minutes. I don't
think heroic efforts are really required.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/8b889012/attachment.html>

From doko at ubuntu.com  Thu Jan 21 14:54:49 2016
From: doko at ubuntu.com (Matthias Klose)
Date: Thu, 21 Jan 2016 20:54:49 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
Message-ID: <56A13789.9050000@ubuntu.com>

On 21.01.2016 17:13, Nathaniel Smith wrote:
> On Jan 21, 2016 2:07 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>>
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>
> The list of allowed libraries is exactly the same list of libraries as are
> required by the Anaconda python distribution, so we *know* that it works
> for about a hundred different python packages, including lots of tricky
> ones (the whole scientific stack), and had been tested by tens or hundreds
> of thousands of users.

so this is x86_64-linux-gnu. Any other architectures?  Any reason to choose gcc 
4.8.2 which is known for it's defects?  This whole thing looks like an Anaconda 
marketing PEP ...

Matthias


From donald at stufft.io  Thu Jan 21 15:08:24 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 21 Jan 2016 15:08:24 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBkdsRFP65CT6aB_USrnUFpte=9ZvbWB0G5uWSC1O4a=zA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <C628BE1F-59E9-4842-A2EE-300E42BE9594@stufft.io>
 <CAPJVwBkdsRFP65CT6aB_USrnUFpte=9ZvbWB0G5uWSC1O4a=zA@mail.gmail.com>
Message-ID: <F2A63794-7815-4611-89F6-2986FCA71DB6@stufft.io>


> On Jan 21, 2016, at 1:43 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> On Jan 21, 2016 9:32 AM, "Donald Stufft" <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> >
> >
> > > On Jan 20, 2016, at 10:55 PM, Nathaniel Smith <njs at pobox.com <mailto:njs at pobox.com>> wrote:
> > >
> > > The permitted external shared libraries are: ::
> > >
> > >    libpanelw.so.5
> > >    libncursesw.so.5
> > >    libgcc_s.so.1
> > >    libstdc++.so.6
> > >    libm.so.6
> > >    libdl.so.2
> > >    librt.so.1
> > >    libcrypt.so.1
> > >    libc.so.6
> > >    libnsl.so.1
> > >    libutil.so.1
> > >    libpthread.so.0
> > >    libX11.so.6
> > >    libXext.so.6
> > >    libXrender.so.1
> > >    libICE.so.6
> > >    libSM.so.6
> > >    libGL.so.1
> > >    libgobject-2.0.so.0
> > >    libgthread-2.0.so.0
> > >    libglib-2.0.so.0
> >
> >
> > Forgive my, probably stupid, question? but why these libraries? Are these
> > libraries unable to be reasonably statically linked like glibc is?
> 
> It's not a stupid question at all :-).
> 
> What we want is a list of libraries that are present, and compatible, and relevant, across the Linux distributions that most people use in practice.
> 
> Unfortunately, there isn't really any formal specification or published data on this -- the only way to make such a list is to make a guess, start distributing software to lots of people based on your guess, wait for bug reports to come in, edit your list to avoid the problematic libraries, add some more libraries to your list to cover the new packages you've added to your distribution and where you think the risk is a worthwhile trade off versus just static linking, then repeat until you stop getting bug reports.
> 
> This is expensive and time consuming and requires data we don't have, so we delegated: the list in the PEP is exactly the set of libraries that Continuum's Anaconda distribution links to and a subset of the libraries that Enthought's Canopy links to [1]. They've both been going through the loop above for years, with lots of packages, and lots of lots of downloads, so we're piggybacking off their effort.
> 
> Here's the list of packages in Anaconda:
> http://docs.continuum.io/anaconda/pkg-docs <http://docs.continuum.io/anaconda/pkg-docs>
> 

I guess my underlying question is, if we?re considering static linking (or shipping the .so dll style) to be good enough for everything not on this list, why are these specific packages on the list? Why are we not selecting the absolute bare minimum packages that you *cannot* reasonably static link or ship the .so?


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/0eb6cb82/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/0eb6cb82/attachment-0001.sig>

From rmcgibbo at gmail.com  Thu Jan 21 15:11:56 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 12:11:56 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A13789.9050000@ubuntu.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A13789.9050000@ubuntu.com>
Message-ID: <CAN4+E8HiT2ORxgx=diXSn49h-EixZHX_fUUKmiezN8M6rxMnrg@mail.gmail.com>

On Jan 21, 2016 12:04 PM, "Matthias Klose" <doko at ubuntu.com> wrote:

> so this is x86_64-linux-gnu. Any other architectures?  Any reason to
choose gcc 4.8.2 which is known for it's defects?

I'm not aware of those defects. Can you share more information?

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/5630ce2f/attachment.html>

From njs at pobox.com  Thu Jan 21 15:14:35 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jan 2016 12:14:35 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A13789.9050000@ubuntu.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A13789.9050000@ubuntu.com>
Message-ID: <CAPJVwBkB7WpejNzSE=-h8+O_o=m-A-TAZwGkPbSCmdcox5Uatw@mail.gmail.com>

On Thu, Jan 21, 2016 at 11:54 AM, Matthias Klose <doko at ubuntu.com> wrote:
> On 21.01.2016 17:13, Nathaniel Smith wrote:
>>
>> On Jan 21, 2016 2:07 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>>>
>>>
>>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>>>
>>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>>
>>>>> By using the version based approach, we'd not run into this
>>>>> problem and gain a lot more.
>>>>
>>>>
>>>> I think it's better to start with a small core that we *know* works,
>>>> then expand later, rather than trying to make the first iteration too
>>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>>> than newer ones.
>>>
>>>
>>> My argument is that the file based approach taken by the PEP
>>> is too limiting to actually make things work for a large
>>> set of Python packages.
>>>
>>> It will basically only work for packages that do not interface
>>> to other external libraries (except for the few cases listed in
>>> the PEP, e.g. X11, GL, which aren't always installed or
>>> available either).
>>
>>
>> The list of allowed libraries is exactly the same list of libraries as are
>> required by the Anaconda python distribution, so we *know* that it works
>> for about a hundred different python packages, including lots of tricky
>> ones (the whole scientific stack), and had been tested by tens or hundreds
>> of thousands of users.
>
>
> so this is x86_64-linux-gnu. Any other architectures?

That's by far the dominant architecture for Linux workstations and
servers, so that's what we've focused on, yeah. I assume that
mainstream glibc-using distros on x86-32 and ARM are similar; unless
someone wants to go do the research somehow then I think the simplest
way forward is to proceed on the assumption that the same spec will
work, and then fix it up if/when it turns out to break.

> Any reason to choose
> gcc 4.8.2 which is known for it's defects?

It's the most recent version of gcc that's able to target CentOS 5
(thanks to RH's backporting efforts in their "devtoolset" releases),
and CentOS 5 is the target baseline that's currently used by
approximately everyone who distributes universal linux binaries
(google e.g. "holy build box").

> This whole thing looks like an
> Anaconda marketing PEP ...

Anaconda's main selling point is that they provide binaries that "just
work", and pip doesn't. Not sure how improving pip to be a stronger
competitor to Anaconda is Anaconda marketing.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From doko at ubuntu.com  Thu Jan 21 15:18:22 2016
From: doko at ubuntu.com (Matthias Klose)
Date: Thu, 21 Jan 2016 21:18:22 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <56A13D0E.5000605@ubuntu.com>

On 21.01.2016 04:55, Nathaniel Smith wrote:

the choice of compiler is questionable.  Just a pick into a release series. Not 
even the last released version on this release series.  Is this a good choice? 
Maybe for x86_64 and i386, but not for anything else.

> The permitted external shared libraries are: ::
>
>      libpanelw.so.5
>      libncursesw.so.5

this doesn't include libtinfo, dependency of libncursesw.

>      libgcc_s.so.1
>      libstdc++.so.6

if you insist on a version built from before GCC 5, you are in trouble with the 
libstdc++ ABI. So maybe link that one statically.

>      libgobject-2.0.so.0
>      libgthread-2.0.so.0
>      libglib-2.0.so.0

while glib2.0 is somehow frozen, what will you do if these change the soname?

libgfortran is missing from this list while later down you mention gfortran. 
This will include libquadmath too.

Any reason to not list libz?

> Compilation and Tooling
> =======================

so how are people supposed to build these wheels?  will you provide a 
development distribution, or do you "trust" people building such wheels?

> Platform Detection for Installers
> =================================
>
> Because the ``manylinux1`` profile is already known to work for the many
> thousands of users of popular commercial Python distributions, we suggest that
> installation tools like ``pip`` should error on the side of assuming that a
> system *is* compatible, unless there is specific reason to think otherwise.
>
> We know of three main sources of potential incompatibility that are likely to
> arise in practice:
>
> * A linux distribution that is too old (e.g. RHEL 4)
> * A linux distribution that does not use glibc (e.g. Alpine Linux, which is
>    based on musl libc, or Android)
> * Eventually, in the future, there may exist distributions that break
>    compatibility with this profile

add: "A Linux distribution that is built with clang", e.g. Mageia (libc++ 
instead of libstdc++).

> Security Implications
> =====================
>
> One of the advantages of dependencies on centralized libraries in Linux is
> that bugfixes and security updates can be deployed system-wide, and
> applications which depend on on these libraries will automatically feel the
> effects of these patches when the underlying libraries are updated. This can
> be particularly important for security updates in packages communication
> across the network or cryptography.
>
> ``manylinux1`` wheels distributed through PyPI that bundle security-critical
> libraries like OpenSSL will thus assume responsibility for prompt updates in
> response disclosed vulnerabilities and patches. This closely parallels the
> security implications of the distribution of binary wheels on Windows that,
> because the platform lacks a system package manager, generally bundle their
> dependencies. In particular, because its lacks a stable ABI, OpenSSL cannot be
> included in the ``manylinux1`` profile.

so you rely on the python build to provide this and access OpenSSL through the 
standard library?


 From my point of view this draft is too much influenced by Anaconda and their 
needs.  It doesn't talk at all about interaction with other system libraries, or 
interaction with extensions provided by distributions.

Matthias


From donald at stufft.io  Thu Jan 21 15:21:55 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 21 Jan 2016 15:21:55 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBkB7WpejNzSE=-h8+O_o=m-A-TAZwGkPbSCmdcox5Uatw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A13789.9050000@ubuntu.com>
 <CAPJVwBkB7WpejNzSE=-h8+O_o=m-A-TAZwGkPbSCmdcox5Uatw@mail.gmail.com>
Message-ID: <8BCA8DD9-B71E-41A5-AB4E-2B7A19209557@stufft.io>


> On Jan 21, 2016, at 3:14 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> On Thu, Jan 21, 2016 at 11:54 AM, Matthias Klose <doko at ubuntu.com> wrote:
>> On 21.01.2016 17:13, Nathaniel Smith wrote:
>>> 
>>> On Jan 21, 2016 2:07 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>>>> 
>>>> 
>>>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>>>> 
>>>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>>> 
>>>>>> By using the version based approach, we'd not run into this
>>>>>> problem and gain a lot more.
>>>>> 
>>>>> 
>>>>> I think it's better to start with a small core that we *know* works,
>>>>> then expand later, rather than trying to make the first iteration too
>>>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>>>> than newer ones.
>>>> 
>>>> 
>>>> My argument is that the file based approach taken by the PEP
>>>> is too limiting to actually make things work for a large
>>>> set of Python packages.
>>>> 
>>>> It will basically only work for packages that do not interface
>>>> to other external libraries (except for the few cases listed in
>>>> the PEP, e.g. X11, GL, which aren't always installed or
>>>> available either).
>>> 
>>> 
>>> The list of allowed libraries is exactly the same list of libraries as are
>>> required by the Anaconda python distribution, so we *know* that it works
>>> for about a hundred different python packages, including lots of tricky
>>> ones (the whole scientific stack), and had been tested by tens or hundreds
>>> of thousands of users.
>> 
>> 
>> so this is x86_64-linux-gnu. Any other architectures?
> 
> That's by far the dominant architecture for Linux workstations and
> servers, so that's what we've focused on, yeah. I assume that
> mainstream glibc-using distros on x86-32 and ARM are similar; unless
> someone wants to go do the research somehow then I think the simplest
> way forward is to proceed on the assumption that the same spec will
> work, and then fix it up if/when it turns out to break.

Numbers!

Here?s the # of downloads from PyPI in the last week or so that we can identify as a linux machine using pip and what the value of their platform.machine() is: https://caremad.io/s/SIxbkCB82C/

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/c030028a/attachment-0001.sig>

From matthew.brett at gmail.com  Thu Jan 21 14:05:27 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 21 Jan 2016 11:05:27 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A0AD76.7090107@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
Message-ID: <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>

Hi,

On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 21.01.2016 10:31, Nick Coghlan wrote:
>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>> By using the version based approach, we'd not run into this
>>> problem and gain a lot more.
>>
>> I think it's better to start with a small core that we *know* works,
>> then expand later, rather than trying to make the first iteration too
>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>> end), so "manylinux2" may simply have *more* libraries defined, rather
>> than newer ones.
>
> My argument is that the file based approach taken by the PEP
> is too limiting to actually make things work for a large
> set of Python packages.
>
> It will basically only work for packages that do not interface
> to other external libraries (except for the few cases listed in
> the PEP, e.g. X11, GL, which aren't always installed or
> available either).
>
> IMO, testing the versions of a set of libraries is a safer
> approach. It's perfectly fine to have a few dependencies
> not work in a module because an optional system package is not
> installed, e.g. say a package comes with UIs written in
> Qt and one in GTK.

Please forgive my slowness, but I don't understand exactly what you
mean.  Can you give a specific example?

Say my package depends on libpng.

Call the machine I'm installing on the client machine.

Are you saying that, when I build a wheel, I should specify to the
wheel what versions of libpng I can tolerate on the the client
machine, and if if the client does have a compatible version, then pip
should raise an error, perhaps with a useful message about how to get
libpng?

If you do mean that, how do you want the PEP changed?

Best,

Matthew

From matthew.brett at gmail.com  Thu Jan 21 14:48:16 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 21 Jan 2016 11:48:16 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
Message-ID: <CAH6Pt5qYxa-Z90MVE4_cLL+N73rhiUmfq4Nb8jO9C=oM_55FJQ@mail.gmail.com>

Hi,

On Thu, Jan 21, 2016 at 11:42 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> nice, idea, but....
>
>     libX11.so.6
>     libXext.so.6
>     libXrender.so.1
>     libGL.so.1
>
> These are all X11, yes? pretty much any workstation will have these, but in
> general, servers won't.
>
> Someone on this thread suggested that that's OK -- don't expect a GUI
> package to work on a linux box without a GUI. But some of these libs might
> get used for stuff like back-end rendering, etc, so would be expected to
> work on a headless box. I think Anaconda an Canopy have gotten away with
> this because both of their user bases are primarily desktop data analysis
> type of stuff -- not web services, web servers, etc.

Someone administering a headless server will surely be able to cope
with that problem, and the trade-off of convenience for the large
majority of users seems like a good one to me.

> Then there is the "additional libs" problem. Again, Anaconda and Canopy can
> do this because they are providing those libs - lots of stuff you'd
> generally expect to be there in a typical *nix system stuff like libpng,
> libjpeg, who knows what they heck else.
>
> So without a plan to provide all that stuff -- I"m not sure of the utility
> of this -- how are you gong to get PIL/Pillow to work? statically link up
> the ying-yang? Not sure the linux world will take to that.

That's exactly how the current OSX Pillow wheels work, and they've
been working fine for a while now.  There just aren't that many
libraries to worry about for the vast majority of packages.

Cheers,

Matthew

From matthew.brett at gmail.com  Thu Jan 21 14:22:11 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 21 Jan 2016 11:22:11 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
Message-ID: <CAH6Pt5oADi=qu=zpEfu8Dnf7+PMpwAePeQsbpjTNQM1abGzDiQ@mail.gmail.com>

On Thu, Jan 21, 2016 at 11:05 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>>
>> IMO, testing the versions of a set of libraries is a safer
>> approach. It's perfectly fine to have a few dependencies
>> not work in a module because an optional system package is not
>> installed, e.g. say a package comes with UIs written in
>> Qt and one in GTK.
>
> Please forgive my slowness, but I don't understand exactly what you
> mean.  Can you give a specific example?
>
> Say my package depends on libpng.
>
> Call the machine I'm installing on the client machine.
>
> Are you saying that, when I build a wheel, I should specify to the
> wheel what versions of libpng I can tolerate on the the client
> machine, and if if the client does have a compatible version, then pip
> should raise an error, perhaps with a useful message about how to get
> libpng?

Sorry, slowness any typos - corrected:

Are you saying that, when I build a wheel, I should specify to the
wheel what versions of libpng I can tolerate on the the client
machine, and if the client does _not_ have a compatible version, then pip
should raise an error, perhaps with a useful message about how to get
libpng?

Best again,

Matthew

From matthew.brett at gmail.com  Thu Jan 21 15:03:51 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 21 Jan 2016 12:03:51 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxE+VgFBpx82DSY9GNUEeq=CNRXbKFHCDM7yzw9G5amy6eQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAH6Pt5qYxa-Z90MVE4_cLL+N73rhiUmfq4Nb8jO9C=oM_55FJQ@mail.gmail.com>
 <CALGmxE+VgFBpx82DSY9GNUEeq=CNRXbKFHCDM7yzw9G5amy6eQ@mail.gmail.com>
Message-ID: <CAH6Pt5qf2JvO2pONqE5QcXzf7p+M0NwwvuoN8FY7sWExP6up8Q@mail.gmail.com>

Hi,

On Thu, Jan 21, 2016 at 11:57 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Thu, Jan 21, 2016 at 11:48 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> > So without a plan to provide all that stuff -- I"m not sure of the
>> > utility
>> > of this -- how are you gong to get PIL/Pillow to work? statically link
>> > up
>> > the ying-yang? Not sure the linux world will take to that.
>>
>> That's exactly how the current OSX Pillow wheels work, and they've
>> been working fine for a while now.  There just aren't that many
>> libraries to worry about for the vast majority of packages.
>
>
> well, that was really intended to be only an example. And OS-X provides more
> basic libs than manylinux anyway (zlib, freetype, either png or jpeg, can't
> remember which).
>
> The library list got long enough to drive me crazy -- I guess you've got
> more patience than I have. Tried to build any OSGEO stuff lately?

I'm sure there are packages that would be hard to build, but for the
vast majority of packages, getting a build recipe is a one-time only
job which might (in bad cases) take a day or so, and then can be
maintained from time to time by the package maintainer.   The Pillow
wheel builder is a good example - I built the prototype, but the
Pillow guys own and maintain it now.

I don't think it's sensible to veto all linux wheels because there are
some packages that will be hard to build.

Cheers,

Matthew

From rmcgibbo at gmail.com  Thu Jan 21 16:33:02 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 13:33:02 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <F2A63794-7815-4611-89F6-2986FCA71DB6@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <C628BE1F-59E9-4842-A2EE-300E42BE9594@stufft.io>
 <CAPJVwBkdsRFP65CT6aB_USrnUFpte=9ZvbWB0G5uWSC1O4a=zA@mail.gmail.com>
 <F2A63794-7815-4611-89F6-2986FCA71DB6@stufft.io>
Message-ID: <CAN4+E8H+Y2RTaU3otpkWR6ZABa1hqcvPHNy5fWwN-iDdYuwz1Q@mail.gmail.com>

On Thu, Jan 21, 2016 at 12:08 PM, Donald Stufft <donald at stufft.io> wrote:
>
> I guess my underlying question is, if we?re considering static linking (or
> shipping the .so dll style) to be good enough for everything not on this
> list, why are these specific packages on the list? Why are we not selecting
> the absolute bare minimum packages that you *cannot* reasonably static link
> or ship the .so?
>

This is a fair question. The principle, practical reason is that we
followed the lead of what other projects have done here for distributing
cross-distro binaries, especially Anaconda and Canopy.

I also just looked at the external libraries required by the portable
firefox Linux binaries (
https://www.mozilla.org/en-US/firefox/43.0.4/system-requirements/). The
additional shared libraries that the firefox pre-compiled binaries require
that are not included in our list are

    libXcomposite.so.1
    libXdamage.so.1
    libXfixes.so.3
    libXt.so.6
    libasound.so.2
    libatk-1.0.so.0
    libcairo.so.2
    libdbus-1.so.3
    libdbus-glib-1.so.2
    libfontconfig.so.1
    libfreetype.so.6
    libgdk-x11-2.0.so.0
    libgdk_pixbuf-2.0.so.0
    libgio-2.0.so.0
    libgmodule-2.0.so.0
    libgtk-x11-2.0.so.0
    libpango-1.0.so.0
    libpangocairo-1.0.so.0
    libpangoft2-1.0.so.0


I would be open to including some of these libraries in the manylinux1
policy, or in a subsequent update (manylinux2, etc).

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/b8027bdb/attachment-0001.html>

From njs at pobox.com  Thu Jan 21 16:33:59 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jan 2016 13:33:59 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A13D0E.5000605@ubuntu.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A13D0E.5000605@ubuntu.com>
Message-ID: <CAPJVwBkuLgMHJCKAzWrFuESX0i0LASOQtHaBPp8g7RCm9ibftg@mail.gmail.com>

On Thu, Jan 21, 2016 at 12:18 PM, Matthias Klose <doko at ubuntu.com> wrote:
> On 21.01.2016 04:55, Nathaniel Smith wrote:
>
> the choice of compiler is questionable.  Just a pick into a release series.
> Not even the last released version on this release series.  Is this a good
> choice? Maybe for x86_64 and i386, but not for anything else.

There's no mandatory compiler. The rule is basically: "if your wheel
works when given access to CentOS 5's versions of the following
packages: ..., then your wheel is manylinux1 compatible". Any method
for achieving that is fair game :-). The RH devtoolset fork of gcc 4.8
happens to be the only C++11-supporting compiler that we know of that
can produce binaries that accomplish that, but if you have another
compiler that works better for you then go for it.

>> The permitted external shared libraries are: ::
>>
>>      libpanelw.so.5
>>      libncursesw.so.5
>
>
> this doesn't include libtinfo, dependency of libncursesw.
>
>>      libgcc_s.so.1
>>      libstdc++.so.6
>
>
> if you insist on a version built from before GCC 5, you are in trouble with
> the libstdc++ ABI. So maybe link that one statically.

The libstdc++ ABI is forward compatible across the GCC 4/GCC 5
transition; the exception is if you want to pass C++ stdlib objects
directly between code compiled with GCC 4 and code compiled with GCC
5. This is extremely rare, since Python packages almost never provide
a C++ ABI to other Python packages. (I don't know of any examples of
this.) If it does come up then those package authors will have to
figure out how they want to handle it, yes, possibly via picking a
more modern libstdc++ and agreeing to all use it together.

>>      libgobject-2.0.so.0
>>      libgthread-2.0.so.0
>>      libglib-2.0.so.0
>
>
> while glib2.0 is somehow frozen, what will you do if these change the
> soname?
>
> libgfortran is missing from this list while later down you mention gfortran.
> This will include libquadmath too.

libgfortran is missing intentionally, because it turns out from
experience that lots of end-user systems don't have it installed out
of the box. You'll notice that the numpy wheels I posted earlier
include libgfortran inside them.

(libquadmath is optional though -- you can build gfortran with
--disable-libquadmath-support and get a toolchain that generates
binaries that don't depend on libquadmath.)

> Any reason to not list libz?
>
>> Compilation and Tooling
>> =======================
>
>
> so how are people supposed to build these wheels?  will you provide a
> development distribution, or do you "trust" people building such wheels?

We currently provide a docker image (docker pull
quay.io/manylinux/manylinux) and a tool that checks wheels for
compliance and can in many cases automatically vendor any needed
shared libraries (pip3 install auditwheel). In the end of course we
can't force people to use these tools -- there's nothing that will
actually stop someone from renaming a Windows wheel to say
"manylinux1" in the platform tag and uploading it somewhere :-) -- but
our experience with OS X wheels is that if you provide good docs and
tools then people will generally get it right.

Package authors generally don't set out to build broken packages that
frustrate users ;-)

>> Platform Detection for Installers
>> =================================
>>
>> Because the ``manylinux1`` profile is already known to work for the many
>> thousands of users of popular commercial Python distributions, we suggest
>> that
>> installation tools like ``pip`` should error on the side of assuming that
>> a
>> system *is* compatible, unless there is specific reason to think
>> otherwise.
>>
>> We know of three main sources of potential incompatibility that are likely
>> to
>> arise in practice:
>>
>> * A linux distribution that is too old (e.g. RHEL 4)
>> * A linux distribution that does not use glibc (e.g. Alpine Linux, which
>> is
>>    based on musl libc, or Android)
>> * Eventually, in the future, there may exist distributions that break
>>    compatibility with this profile
>
>
> add: "A Linux distribution that is built with clang", e.g. Mageia (libc++
> instead of libstdc++).

Interesting point. It looks like from some quick googling that even
Mageia does provide a libstdc++6 package, though, presumably because
of these kinds of compatibility issues?

>> Security Implications
>> =====================
>>
>> One of the advantages of dependencies on centralized libraries in Linux is
>> that bugfixes and security updates can be deployed system-wide, and
>> applications which depend on on these libraries will automatically feel
>> the
>> effects of these patches when the underlying libraries are updated. This
>> can
>> be particularly important for security updates in packages communication
>> across the network or cryptography.
>>
>> ``manylinux1`` wheels distributed through PyPI that bundle
>> security-critical
>> libraries like OpenSSL will thus assume responsibility for prompt updates
>> in
>> response disclosed vulnerabilities and patches. This closely parallels the
>> security implications of the distribution of binary wheels on Windows
>> that,
>> because the platform lacks a system package manager, generally bundle
>> their
>> dependencies. In particular, because its lacks a stable ABI, OpenSSL
>> cannot be
>> included in the ``manylinux1`` profile.
>
>
> so you rely on the python build to provide this and access OpenSSL through
> the standard library?

That is one strategy that's available to packages who need access to
OpenSSL, yes.

> From my point of view this draft is too much influenced by Anaconda and
> their needs.  It doesn't talk at all about interaction with other system
> libraries, or interaction with extensions provided by distributions.

It's true that it doesn't do those things, just like there's no PEP
talking about how pip on Windows should interact with nuget or how pip
on OS X should interact with homebrew. This is a first step, the
perfect is the enemy of the good, etc.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Thu Jan 21 16:42:07 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jan 2016 13:42:07 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
Message-ID: <CAPJVwB=PjEk7RXiwx3piwQvOYyAUYHwwa8MgCr+XDvzRK26SKw@mail.gmail.com>

On Thu, Jan 21, 2016 at 1:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I think it's better to start with a small core that we *know* works,
> then expand later, rather than trying to make the first iteration too
> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
> end), so "manylinux2" may simply have *more* libraries defined, rather
> than newer ones.

We wouldn't even necessarily need to bump the version number, since
the main thing the "manylinux1" tag does is serve as a notice to pip
about whether a wheel is likely to work on a given system. If after
experience we realize that in fact there are more libraries that would
work on those same systems, then we can release a "manylinux 1.1"
document that updates the guidance to package authors but continues to
use the "manylinux1" tag and leaves the same code in pip. (And ditto
if we realize that there are libraries on the list that shouldn't be.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From solipsis at pitrou.net  Thu Jan 21 16:58:29 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Jan 2016 22:58:29 +0100
Subject: [Distutils] draft PEP: manylinux1
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
Message-ID: <20160121225829.7504847c@fsol>

On Thu, 21 Jan 2016 11:42:57 -0800
Chris Barker <chris.barker at noaa.gov> wrote:
> nice, idea, but....
> 
>     libX11.so.6
>     libXext.so.6
>     libXrender.so.1
>     libGL.so.1
> 
> These are all X11, yes? pretty much any workstation will have these, but in
> general, servers won't.

For the record, not having or not running a X11 server doesn't mean you
don't have a subset of the corresponding *libraries*.

For example, I have a Debian jessie server and, while there's no
X11-related executable on the machine, I still have the following
libraries installed:

$ dpkg -l | grep X
[...]
ii  libpangox-1.0-0:amd64            0.0.2-5                       amd64        pango library X backend
ii  libpixman-1-0:amd64              0.32.6-3                      amd64        pixel-manipulation library for X and cairo
ii  libpod-latex-perl                0.61-1                        all          module to convert Pod data to formatted LaTeX
ii  libx11-6:amd64                   2:1.6.2-3                     amd64        X11 client-side library
ii  libx11-data                      2:1.6.2-3                     all          X11 client-side library
ii  libxau6:amd64                    1:1.0.8-1                     amd64        X11 authorisation library
ii  libxcb-render0:amd64             1.10-3+b1                     amd64        X C Binding, render extension
ii  libxcb-shm0:amd64                1.10-3+b1                     amd64        X C Binding, shm extension
ii  libxcb1:amd64                    1.10-3+b1                     amd64        X C Binding
ii  libxdmcp6:amd64                  1:1.1.1-1+b1                  amd64        X11 Display Manager Control Protocol library
ii  libxext6:amd64                   2:1.3.3-1                     amd64        X11 miscellaneous extension library
ii  libxft2:amd64                    2.3.2-1                       amd64        FreeType-based font drawing library for X
ii  libxml2:amd64                    2.9.1+dfsg1-5+deb8u1          amd64        GNOME XML library
ii  libxpm4:amd64                    1:3.5.11-1+b1                 amd64        X11 pixmap library
ii  libxrender1:amd64                1:0.9.8-1+b1                  amd64        X Rendering Extension client library
ii  xkb-data                         2.12-1                        all          X Keyboard Extension (XKB) configuration data
[...]

Regards

Antoine.



From cournape at gmail.com  Thu Jan 21 17:04:41 2016
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 21 Jan 2016 22:04:41 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
Message-ID: <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>

On Thu, Jan 21, 2016 at 7:42 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> nice, idea, but....
>
>     libX11.so.6
>     libXext.so.6
>     libXrender.so.1
>     libGL.so.1
>
> These are all X11, yes? pretty much any workstation will have these, but
> in general, servers won't.
>

Those would be required by GUI packages. People who know how to install
headless systems would know how to handle errors because of missing sonames.

I can talk from experience that this problem does not happen often.

Note that this set of libraries was built from both what anaconda and
canopy do

>
> Someone on this thread suggested that that's OK -- don't expect a GUI
> package to work on a linux box without a GUI. But some of these libs might
> get used for stuff like back-end rendering, etc, so would be expected to
> work on a headless box. I think Anaconda an Canopy have gotten away with
> this because both of their user bases are primarily desktop data analysis
> type of stuff -- not web services, web servers, etc.
>

This is not true for us at Enthought, and I would be surprised if it were
for anaconda.


> So without a plan to provide all that stuff -- I"m not sure of the utility
> of this -- how are you gong to get PIL/Pillow to work? statically link up
> the ying-yang? Not sure the linux world will take to that.
>

We can explain how things work in details for some packages, but the main
rationale for the PEP list is that this is a list that works in practice.
It has worked well for us at Enthought for many years, and it has for
(Ana)conda as well. Between both distributions, we are talking about
millions of installs over the year, on many different systems.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/bb3ee383/attachment-0001.html>

From cournape at gmail.com  Thu Jan 21 17:08:13 2016
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 21 Jan 2016 22:08:13 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
Message-ID: <CAGY4rcXDn2s1UuQFr=1BvJ-PsDhgTi0BwOPpQP3MSStEx+Eo+A@mail.gmail.com>

On Thu, Jan 21, 2016 at 10:04 PM, David Cournapeau <cournape at gmail.com>
wrote:

>
>
> On Thu, Jan 21, 2016 at 7:42 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
>
>> nice, idea, but....
>>
>>     libX11.so.6
>>     libXext.so.6
>>     libXrender.so.1
>>     libGL.so.1
>>
>> These are all X11, yes? pretty much any workstation will have these, but
>> in general, servers won't.
>>
>
> Those would be required by GUI packages. People who know how to install
> headless systems would know how to handle errors because of missing sonames.
>

I would also mention that at Enthought, we have a fairly comprehensive
testing policy for our packages (e.g. running the test suite of a package
at every build as much as possible). We do the testing under headless
environments on every platform as much as possible, and it works for maybe
95 % of the packages (including things like matplotlib, etc...). E.g. on
Linux, most packages work well with a framebuffer if you only care about
offline rendering or testing.

David

I can talk from experience that this problem does not happen often.
>
> Note that this set of libraries was built from both what anaconda and
> canopy do
>
>>
>> Someone on this thread suggested that that's OK -- don't expect a GUI
>> package to work on a linux box without a GUI. But some of these libs might
>> get used for stuff like back-end rendering, etc, so would be expected to
>> work on a headless box. I think Anaconda an Canopy have gotten away with
>> this because both of their user bases are primarily desktop data analysis
>> type of stuff -- not web services, web servers, etc.
>>
>
> This is not true for us at Enthought, and I would be surprised if it were
> for anaconda.
>
>
>> So without a plan to provide all that stuff -- I"m not sure of the
>> utility of this -- how are you gong to get PIL/Pillow to work? statically
>> link up the ying-yang? Not sure the linux world will take to that.
>>
>
> We can explain how things work in details for some packages, but the main
> rationale for the PEP list is that this is a list that works in practice.
> It has worked well for us at Enthought for many years, and it has for
> (Ana)conda as well. Between both distributions, we are talking about
> millions of installs over the year, on many different systems.
>
> David
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/4e9dad33/attachment.html>

From cournape at gmail.com  Thu Jan 21 17:11:32 2016
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 21 Jan 2016 22:11:32 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A13D0E.5000605@ubuntu.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A13D0E.5000605@ubuntu.com>
Message-ID: <CAGY4rcXZB=0y+r8Xe_sFy1QF6V2wV1zCy7v+m7+HWaEUoTtkLg@mail.gmail.com>

On Thu, Jan 21, 2016 at 8:18 PM, Matthias Klose <doko at ubuntu.com> wrote:

> On 21.01.2016 04:55, Nathaniel Smith wrote:
>
> the choice of compiler is questionable.  Just a pick into a release
> series. Not even the last released version on this release series.  Is this
> a good choice? Maybe for x86_64 and i386, but not for anything else.
>
> The permitted external shared libraries are: ::
>>
>>      libpanelw.so.5
>>      libncursesw.so.5
>>
>
> this doesn't include libtinfo, dependency of libncursesw.
>
>      libgcc_s.so.1
>>      libstdc++.so.6
>>
>
> if you insist on a version built from before GCC 5, you are in trouble
> with the libstdc++ ABI. So maybe link that one statically.
>
>      libgobject-2.0.so.0
>>      libgthread-2.0.so.0
>>      libglib-2.0.so.0
>>
>
> while glib2.0 is somehow frozen, what will you do if these change the
> soname?
>
> libgfortran is missing from this list while later down you mention
> gfortran. This will include libquadmath too.
>
> Any reason to not list libz?
>
> Compilation and Tooling
>> =======================
>>
>
> so how are people supposed to build these wheels?  will you provide a
> development distribution, or do you "trust" people building such wheels?
>
> Platform Detection for Installers
>> =================================
>>
>> Because the ``manylinux1`` profile is already known to work for the many
>> thousands of users of popular commercial Python distributions, we suggest
>> that
>> installation tools like ``pip`` should error on the side of assuming that
>> a
>> system *is* compatible, unless there is specific reason to think
>> otherwise.
>>
>> We know of three main sources of potential incompatibility that are
>> likely to
>> arise in practice:
>>
>> * A linux distribution that is too old (e.g. RHEL 4)
>> * A linux distribution that does not use glibc (e.g. Alpine Linux, which
>> is
>>    based on musl libc, or Android)
>> * Eventually, in the future, there may exist distributions that break
>>    compatibility with this profile
>>
>
> add: "A Linux distribution that is built with clang", e.g. Mageia (libc++
> instead of libstdc++).
>
> Security Implications
>> =====================
>>
>> One of the advantages of dependencies on centralized libraries in Linux is
>> that bugfixes and security updates can be deployed system-wide, and
>> applications which depend on on these libraries will automatically feel
>> the
>> effects of these patches when the underlying libraries are updated. This
>> can
>> be particularly important for security updates in packages communication
>> across the network or cryptography.
>>
>> ``manylinux1`` wheels distributed through PyPI that bundle
>> security-critical
>> libraries like OpenSSL will thus assume responsibility for prompt updates
>> in
>> response disclosed vulnerabilities and patches. This closely parallels the
>> security implications of the distribution of binary wheels on Windows
>> that,
>> because the platform lacks a system package manager, generally bundle
>> their
>> dependencies. In particular, because its lacks a stable ABI, OpenSSL
>> cannot be
>> included in the ``manylinux1`` profile.
>>
>
> so you rely on the python build to provide this and access OpenSSL through
> the standard library?
>
>
> From my point of view this draft is too much influenced by Anaconda and
> their needs.  It doesn't talk at all about interaction with other system
> libraries, or interaction with extensions provided by distributions.


FWIW, the list of libraries and the dockerfile was originally built from
communications I had w/ Nathaniel and Matthew Brett, and I work for a
continuum's competitor, so you can be fairly confident that there is no
hidden agenda.

David



>
> Matthias
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/fbbd1bf3/attachment.html>

From nate at bx.psu.edu  Thu Jan 21 17:22:02 2016
From: nate at bx.psu.edu (Nate Coraor)
Date: Thu, 21 Jan 2016 17:22:02 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <CALT861Hiped2pqtx9wGqweL0jXe9f_zMs7_4_bAJfOZc4giTxw@mail.gmail.com>

Nathaniel, Robert, I'm really excited to see how quickly you're making
progress. A few comments below as I haven't had a chance to catch up on the
day's discussion:

On Wed, Jan 20, 2016 at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:

> Building on the compability lessons learned from these companies, we thus
> define a baseline ``manylinux1`` platform tag for use by binary Python
> wheels, and introduce the implementation of preliminary tools to aid in the
> construction of these ``manylinux1`` wheels.
>

Just a standards question: does this still require an update to PEP 425, or
would the definition of the  manylinux1 platform here supersede that
section of 425?

* Eventually, in the future, there may exist distributions that break
>   compatibility with this profile
>


> To handle the third case, we propose the creation of a file
> ``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
> contents: ::
>
>    [manylinux1]
>    compatible = true
>
> where the supported values for the ``manylinux1.compatible`` entry are the
> same as those supported by the ConfigParser ``getboolean`` method.
>

Could this instead use the more powerful json-based syntax proposed by Nick
here:

    https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html

I have already implemented support for this in pip and wheel.


> Security Implications
> =====================
>
> One of the advantages of dependencies on centralized libraries in Linux is
> that bugfixes and security updates can be deployed system-wide, and
> applications which depend on on these libraries will automatically feel the
> effects of these patches when the underlying libraries are updated. This
> can
> be particularly important for security updates in packages communication
> across the network or cryptography.
>
> ``manylinux1`` wheels distributed through PyPI that bundle
> security-critical
> libraries like OpenSSL will thus assume responsibility for prompt updates
> in
> response disclosed vulnerabilities and patches. This closely parallels the
> security implications of the distribution of binary wheels on Windows that,
> because the platform lacks a system package manager, generally bundle their
> dependencies. In particular, because its lacks a stable ABI, OpenSSL
> cannot be
> included in the ``manylinux1`` profile.
>

I appreciate that this was addressed. I don't want to be responsible for
keeping the versions of these things up to date. So instead, I made a
docker-based build system that builds a ton of wheels on different
distros/versions using a common definition. But I appreciate that it's a
bit heavy and not everyone will prefer this.

One piece that is not yet complete in the work I've done so far is actually
ensuring that the external dependencies are installed, and providing some
feedback on what's missing. But that can be done.


> Rejected Alternatives
> =====================
>
> One alternative would be to provide separate platform tags for each Linux
> distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
> ``debian_jessie``, etc. Nothing in this proposal rules out the possibility
> of
> adding such platform tags in the future, or of further extensions to wheel
> metadata that would allow wheels to declare dependencies on external
> system-installed packages. However, such extensions would require
> substantially
> more work than this proposal, and still might not be appreciated by package
> developers who would prefer not to have to maintain multiple build
> environments
> and build multiple wheels in order to cover all the common Linux
> distributions.
> Therefore we consider such proposals to be out-of-scope for this PEP.
>

;)

For anyone who's interested, the next release of Galaxy (a popular
bioinformatics framework for running tools, workflows, etc.), due next
week, will ship with our modified pip that includes support for
distro/version-specific platform tags in wheels. All but one of our
dependent package's wheels are built with the generic `linux_x86_64` tag on
Debian Squeeze and will work with most distros, though, so we're basically
doing a "loose" version of manylinux1. Only our psycopg2 wheels are built
per-distro/version. I'm happy to see a more rigid definition for what we're
doing with the "generic" ones, this is certainly necessary should support
for generic Linux wheels ever be allowed into PyPI. manylinux1 and this PEP
seem to me like the right idea to do this.

We build these distro/version wheels using a modified wheel and the
aforementioned docker-based build system, called Starforge. Here's where
all of it lives:

Galaxy: https://github.com/galaxyproject/galaxy
Starforge: https://github.com/galaxyproject/starforge
"Galaxy pip": https://github.com/natefoo/pip/tree/linux-wheels
"Galaxy wheel": https://bitbucket.org/natefoo/wheel

I say all of this (again) because I think there's still a place for the
work we've done even if manylinux1 is accepted. Both manylinux1 *and* more
specific distro/version platform tags can happily coexist. So I want to
publicize the fact that this work is already done and in use (although I
know there are some things that would need to be done before it could be
upstreamed).

Basically I'm trying to determine whether there's any interest in this from
the pip and PyPI developers. If so, I'd be happy to write a PEP that
complements the one written by Robert and Nathaniel so that both types of
platform tags and wheels using them can be supported.

--nate
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/656c22f9/attachment-0001.html>

From njs at pobox.com  Thu Jan 21 17:38:36 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jan 2016 14:38:36 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <F2A63794-7815-4611-89F6-2986FCA71DB6@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <C628BE1F-59E9-4842-A2EE-300E42BE9594@stufft.io>
 <CAPJVwBkdsRFP65CT6aB_USrnUFpte=9ZvbWB0G5uWSC1O4a=zA@mail.gmail.com>
 <F2A63794-7815-4611-89F6-2986FCA71DB6@stufft.io>
Message-ID: <CAPJVwBkwfw9Y+ragt2YuBmx-PdwXgfub_UWSotWyyikRMR+R7g@mail.gmail.com>

On Jan 21, 2016 12:08 PM, "Donald Stufft" <donald at stufft.io> wrote:
>
>
>> On Jan 21, 2016, at 1:43 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Jan 21, 2016 9:32 AM, "Donald Stufft" <donald at stufft.io> wrote:
>> >
>> >
>> > > On Jan 20, 2016, at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> > >
>> > > The permitted external shared libraries are: ::
>> > >
>> > >    libpanelw.so.5
>> > >    libncursesw.so.5
>> > >    libgcc_s.so.1
>> > >    libstdc++.so.6
>> > >    libm.so.6
>> > >    libdl.so.2
>> > >    librt.so.1
>> > >    libcrypt.so.1
>> > >    libc.so.6
>> > >    libnsl.so.1
>> > >    libutil.so.1
>> > >    libpthread.so.0
>> > >    libX11.so.6
>> > >    libXext.so.6
>> > >    libXrender.so.1
>> > >    libICE.so.6
>> > >    libSM.so.6
>> > >    libGL.so.1
>> > >    libgobject-2.0.so.0
>> > >    libgthread-2.0.so.0
>> > >    libglib-2.0.so.0
>> >
>> >
>> > Forgive my, probably stupid, question? but why these libraries? Are
these
>> > libraries unable to be reasonably statically linked like glibc is?
>>
>> It's not a stupid question at all :-).
>>
>> What we want is a list of libraries that are present, and compatible,
and relevant, across the Linux distributions that most people use in
practice.
>>
>> Unfortunately, there isn't really any formal specification or published
data on this -- the only way to make such a list is to make a guess, start
distributing software to lots of people based on your guess, wait for bug
reports to come in, edit your list to avoid the problematic libraries, add
some more libraries to your list to cover the new packages you've added to
your distribution and where you think the risk is a worthwhile trade off
versus just static linking, then repeat until you stop getting bug reports.
>>
>> This is expensive and time consuming and requires data we don't have, so
we delegated: the list in the PEP is exactly the set of libraries that
Continuum's Anaconda distribution links to and a subset of the libraries
that Enthought's Canopy links to [1]. They've both been going through the
loop above for years, with lots of packages, and lots of lots of downloads,
so we're piggybacking off their effort.
>>
>> Here's the list of packages in Anaconda:
>> http://docs.continuum.io/anaconda/pkg-docs
>>
>>
>
> I guess my underlying question is, if we?re considering static linking
(or shipping the .so dll style) to be good enough for everything not on
this list, why are these specific packages on the list? Why are we not
selecting the absolute bare minimum packages that you *cannot* reasonably
static link or ship the .so?

So, there are tradeoffs here of course. Some libraries are more or less
difficult to statically link, and vendoring libraries does have costs in
terms of maintenance, download sizes, memory usage, and so forth... it's
complicated and varies from library to library. I suspect that even the
people who have been dealing with these problems full time for years can't
necessarily remember all the details of what happened that made them take
particular decisions. (Both Anaconda and Canopy have made the decision to
vendor libz, even though you'd think that libz would be a prime candidate
for getting from the package manager, being both ubiquitous and
historically notable for all the problems that vendoring it has caused in
the past. Why do they do this? I don't know, and I haven't shipped millions
of Linux binary packages so my guesses probably aren't worth that much
:-).) The list probably isn't perfect, but we do know that it will work,
which is an advantage that other lists don't have.

The other thing is the Mason-Dixon principle: we have to draw this
<censored> line somewhere ;-).

You're suggesting more conservative, Marc-Andre and Matthais want it to be
more liberal, and of course any self respecting Linux geek has a guess
about exactly which libraries should be included (myself included). This
bikeshed won't paint itself! So rather than open the door to endless petty
debate and fiddling, just sticking with a known good list is pretty
attractive.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/f600254b/attachment.html>

From njs at pobox.com  Thu Jan 21 18:31:32 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jan 2016 15:31:32 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALT861Hiped2pqtx9wGqweL0jXe9f_zMs7_4_bAJfOZc4giTxw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALT861Hiped2pqtx9wGqweL0jXe9f_zMs7_4_bAJfOZc4giTxw@mail.gmail.com>
Message-ID: <CAPJVwB=kVbh+o-bHqbCookKLtfFLNKpfbPyWVC2tyzgSHVK-Tw@mail.gmail.com>

On Thu, Jan 21, 2016 at 2:22 PM, Nate Coraor <nate at bx.psu.edu> wrote:
> Nathaniel, Robert, I'm really excited to see how quickly you're making
> progress. A few comments below as I haven't had a chance to catch up on the
> day's discussion:
>
> On Wed, Jan 20, 2016 at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Building on the compability lessons learned from these companies, we thus
>> define a baseline ``manylinux1`` platform tag for use by binary Python
>> wheels, and introduce the implementation of preliminary tools to aid in
>> the
>> construction of these ``manylinux1`` wheels.
>
>
> Just a standards question: does this still require an update to PEP 425, or
> would the definition of the  manylinux1 platform here supersede that section
> of 425?

I guess this is a pure process question, so I'll defer to Nick...

>> * Eventually, in the future, there may exist distributions that break
>>   compatibility with this profile
>
>
>>
>> To handle the third case, we propose the creation of a file
>> ``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
>> contents: ::
>>
>>    [manylinux1]
>>    compatible = true
>>
>> where the supported values for the ``manylinux1.compatible`` entry are the
>> same as those supported by the ConfigParser ``getboolean`` method.
>
>
> Could this instead use the more powerful json-based syntax proposed by Nick
> here:
>
>     https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html
>
> I have already implemented support for this in pip and wheel.

Totally happy to change the compatibility.cfg stuff -- the version in
the PEP was written in about 5 minutes in hopes of sparking discussion
:-).

Some other questions:
1) is this going to work for multi-arch (binaries for multiple cpu
architectures sharing a single /etc)? Multiple interpreters? I guess
the advantage of Nick's design is that it's scoped by the value of
distutils.util.get_platform(), so multi-arch installs could have
different values -- a distro could declare that their x86-64 python
builds are manylinux1 compatible but their i386 python builds aren't.
Maybe it would be even better if the files were
/etc/python/binary-compatibility/linux_x86_64.cfg etc., so that the
different .cfg files could be shipped in each architecture's package
without colliding. OTOH I don't know if any of this is very useful in
practice.
2) in theory one could imaging overriding this on a per-venv,
per-user, or per-system level; which of these are useful to support?
(Per-system seems like the most obvious, since an important use case
will be distros setting this flag on behalf of their users.)

There is one feature that I do think is important in the PEP 513
draft, and that Nick's suggestion from July doesn't have: in the PEP
513 design, the manylinux1 flag can be true, false, or unspecified,
and this is independent of other compatibility settings; in Nick's
proposal there's an exhaustive list of all compatible tags, and
everything not on that list is assumed to be incompatible. Where this
matters is when we want to release manylinux2. At this point we'll
want pip to use some autodetection logic on old distros that were
released before manylinux2, while respecting the compatibility flag on
newer distros that do know about manylinux2. This requires a tri-state
setting with "not specified" as a valid value.

[...]
>> Rejected Alternatives
>> =====================
>>
>> One alternative would be to provide separate platform tags for each Linux
>> distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
>> ``debian_jessie``, etc. Nothing in this proposal rules out the possibility
>> of
>> adding such platform tags in the future, or of further extensions to wheel
>> metadata that would allow wheels to declare dependencies on external
>> system-installed packages. However, such extensions would require
>> substantially
>> more work than this proposal, and still might not be appreciated by
>> package
>> developers who would prefer not to have to maintain multiple build
>> environments
>> and build multiple wheels in order to cover all the common Linux
>> distributions.
>> Therefore we consider such proposals to be out-of-scope for this PEP.
>
>
> ;)
>
> For anyone who's interested, the next release of Galaxy (a popular
> bioinformatics framework for running tools, workflows, etc.), due next week,
> will ship with our modified pip that includes support for
> distro/version-specific platform tags in wheels. All but one of our
> dependent package's wheels are built with the generic `linux_x86_64` tag on
> Debian Squeeze and will work with most distros, though, so we're basically
> doing a "loose" version of manylinux1. Only our psycopg2 wheels are built
> per-distro/version. I'm happy to see a more rigid definition for what we're
> doing with the "generic" ones, this is certainly necessary should support
> for generic Linux wheels ever be allowed into PyPI. manylinux1 and this PEP
> seem to me like the right idea to do this.
>
> We build these distro/version wheels using a modified wheel and the
> aforementioned docker-based build system, called Starforge. Here's where all
> of it lives:
>
> Galaxy: https://github.com/galaxyproject/galaxy
> Starforge: https://github.com/galaxyproject/starforge
> "Galaxy pip": https://github.com/natefoo/pip/tree/linux-wheels
> "Galaxy wheel": https://bitbucket.org/natefoo/wheel
>
> I say all of this (again) because I think there's still a place for the work
> we've done even if manylinux1 is accepted. Both manylinux1 *and* more
> specific distro/version platform tags can happily coexist. So I want to
> publicize the fact that this work is already done and in use (although I
> know there are some things that would need to be done before it could be
> upstreamed).
>
> Basically I'm trying to determine whether there's any interest in this from
> the pip and PyPI developers. If so, I'd be happy to write a PEP that
> complements the one written by Robert and Nathaniel so that both types of
> platform tags and wheels using them can be supported.

I don't have anything in particular to say about this part, except
that it's really cool :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From rmcgibbo at gmail.com  Thu Jan 21 20:08:28 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 17:08:28 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <D7809959-8B6A-40FD-BD87-25402692765B@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <9F10A851-ED9B-4C6C-A116-61017E468D78@stufft.io>
 <D7809959-8B6A-40FD-BD87-25402692765B@stufft.io>
Message-ID: <CAN4+E8GarTCcDhcUd4SdnX_Oq0uomfBFAgYpCn3iQ+JG+tVnjg@mail.gmail.com>

On Thu, Jan 21, 2016 at 9:23 AM, Donald Stufft <donald at stufft.io> wrote:
>
> >> On Jan 21, 2016, at 4:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> If Donald can provide the list of "most downloaded wheel files" for
> >> other platforms, that could also be a useful guide as to how many
> >> source builds may potentially already be avoided through the draft
> >> "manylinux1" definition.
> >
>
> Or https://gist.github.com/dstufft/ea8a95580b022b233635 if you prefer it
> grouped by project.



I went through this list and compiled manylinux1 wheels for each of the top
15 projects in the
list (py35).

The wheels are here, if you're interested
http://stanford.edu/~rmcgibbo/wheelhouse/. The amount
of work was pretty small -- the complete dockerfile for this, building off
from the
quay.io/manylinux/manylinux image mentioned in the pep draft is here:
https://github.com/rmcgibbo/manylinux/blob/popular/build-popular/Dockerfile

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/bd12b1cf/attachment-0001.html>

From chris.barker at noaa.gov  Thu Jan 21 21:47:47 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Thu, 21 Jan 2016 18:47:47 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
Message-ID: <-4166805547148809032@unknownmsgid>

> So without a plan to provide all that stuff -- I"m not sure of the utility
> of this -- how are you gong to get PIL/Pillow to work? statically link up
> the ying-yang? Not sure the linux world will take to that.
>

We can explain how things work in details for some packages, but the main
rationale for the PEP list is that this is a list that works in practice.
It has worked well for us at Enthought for many years, and it has for
(Ana)conda as well.


My point was that it works in those cases because Enthought and Continuum
provide a bunch of (often hard to build) packages that provide all the
extra stuff.

Maybe the community will spring forth and do that -- I'm skeptical because
I tried to to that for years for OS-X and it was just too much to do. And
the infrastructure was there.

Before pip and wheel there were mpkgs on OS-X, and repo's of toms for Linux
years before that -- but always the result of a couple people's heroic
efforts.

Maybe the infrastructure has improved, and the community grown enough, that
this will all work. We'll see.

CHB



Between both distributions, we are talking about millions of installs over
the year, on many different systems.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/4475d060/attachment.html>

From chris.barker at noaa.gov  Thu Jan 21 22:19:28 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Thu, 21 Jan 2016 19:19:28 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAN4+E8GarTCcDhcUd4SdnX_Oq0uomfBFAgYpCn3iQ+JG+tVnjg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <9F10A851-ED9B-4C6C-A116-61017E468D78@stufft.io>
 <D7809959-8B6A-40FD-BD87-25402692765B@stufft.io>
 <CAN4+E8GarTCcDhcUd4SdnX_Oq0uomfBFAgYpCn3iQ+JG+tVnjg@mail.gmail.com>
Message-ID: <8042213482852003691@unknownmsgid>

I went through this list and compiled manylinux1 wheels for each of the top
15 projects in the
list (py35).

The wheels are here, if you're interested
http://stanford.edu/~rmcgibbo/wheelhouse


Cool! Are the non-manylinux dependencies all statically linked?

-CHB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/116a067c/attachment.html>

From rmcgibbo at gmail.com  Thu Jan 21 22:27:08 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 19:27:08 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <-4166805547148809032@unknownmsgid>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
Message-ID: <CAN4+E8E4UZy9jwbrSYvWdUfcCapmhKbq+ppJEhXRPHqTXnJ2XQ@mail.gmail.com>

On Thu, Jan 21, 2016 at 6:47 PM, Chris Barker - NOAA Federal <
chris.barker at noaa.gov> wrote:


> Maybe the infrastructure has improved, and the community grown enough,
> that this will all work. We'll see.
>


Yeah, that's my hope too. Currently, the community lacks the permissions to
upload Linux wheels to PyPI. Given
that, it's no surprise that the community hasn't formed yet. I'm hopeful
that in the future, if this PEP is accepted
and we can make the tooling and documentation excellent, uploading Linux
wheels can start to become a
standard part of the PyPI release cycle for package maintainers.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/3df5d057/attachment.html>

From donald at stufft.io  Thu Jan 21 22:29:46 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 21 Jan 2016 22:29:46 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAN4+E8E4UZy9jwbrSYvWdUfcCapmhKbq+ppJEhXRPHqTXnJ2XQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAN4+E8E4UZy9jwbrSYvWdUfcCapmhKbq+ppJEhXRPHqTXnJ2XQ@mail.gmail.com>
Message-ID: <CBB146FE-B62F-4A95-8B11-9F132F8FA87F@stufft.io>


> On Jan 21, 2016, at 10:27 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> 
> I'm hopeful that in the future, if this PEP is accepted
> and we can make the tooling and documentation excellent, uploading Linux wheels can start to become a
> standard part of the PyPI release cycle for package maintainers.


Longer term, I really want to (but have put zero effort into trying to plan it out beyond pie in the sky dreams) have it so that maintainers can publish a sdist to PyPI, and have PyPI automatically build wheels for you. This is a long way down the road though.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/bddbbe8f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/bddbbe8f/attachment.sig>

From rmcgibbo at gmail.com  Thu Jan 21 22:30:24 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 19:30:24 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <8042213482852003691@unknownmsgid>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <9F10A851-ED9B-4C6C-A116-61017E468D78@stufft.io>
 <D7809959-8B6A-40FD-BD87-25402692765B@stufft.io>
 <CAN4+E8GarTCcDhcUd4SdnX_Oq0uomfBFAgYpCn3iQ+JG+tVnjg@mail.gmail.com>
 <8042213482852003691@unknownmsgid>
Message-ID: <CAN4+E8EEiKW=tj4QMkopGHo==6V1xsoUEFk+r3McsxH=rcEghg@mail.gmail.com>

On Thu, Jan 21, 2016 at 7:19 PM, Chris Barker - NOAA Federal <
chris.barker at noaa.gov> wrote:

Cool! Are the non-manylinux dependencies all statically linked?
>


No, they're not statically linked. They're vendored inside the wheel using
``auditwheel repair``.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/4aa8d459/attachment-0001.html>

From ncoghlan at gmail.com  Thu Jan 21 22:32:49 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2016 13:32:49 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A10CF9.8070801@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
Message-ID: <CADiSq7cX39B=LP1Ssxq09f8AX4YWTMgxOqU6JoG1jjo+0+f87w@mail.gmail.com>

On 22 January 2016 at 02:53, M.-A. Lemburg <mal at egenix.com> wrote:
> On 21.01.2016 17:13, Nathaniel Smith wrote:
>> On Jan 21, 2016 2:07 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>>>
>>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>> By using the version based approach, we'd not run into this
>>>>> problem and gain a lot more.
>>>>
>>>> I think it's better to start with a small core that we *know* works,
>>>> then expand later, rather than trying to make the first iteration too
>>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>>> than newer ones.
>>>
>>> My argument is that the file based approach taken by the PEP
>>> is too limiting to actually make things work for a large
>>> set of Python packages.
>>>
>>> It will basically only work for packages that do not interface
>>> to other external libraries (except for the few cases listed in
>>> the PEP, e.g. X11, GL, which aren't always installed or
>>> available either).
>>
>> The list of allowed libraries is exactly the same list of libraries as are
>> required by the Anaconda python distribution, so we *know* that it works
>> for about a hundred different python packages, including lots of tricky
>> ones (the whole scientific stack), and had been tested by tens or hundreds
>> of thousands of users. (I posted a link above to some actual working,
>> compliant pyside and numpy packages, which we picked for testing because
>> they're particular interesting/difficult packages that need to interface to
>> external libraries.) Yes, various extra tricks are needed to get things
>> working on top of this base, including strategies for shipping libraries
>> that are not in the baseline set, but these are problems that can be solved
>> on a project-by-project basis, and don't need a PEP.
>
> And that's the problem: The set is limited to the needs
> of the scientific community and there to the users of
> one or two distributions only.
>
> It doesn't address needs of others that e.g. use Qt or GTK as
> basis for GUIs, people using OpenSSL for networking, people
> using ImageMagick for processing images, or type libs for
> type setting, or sound libs for doing sound processing,
> codec libs for video processing, etc.

This is fine - at the moment *everyone* is locked out from publishing
Linux wheels to PyPI, so I'm entirely OK with biting off incremental
chunks that meet the needs of different sections of the community,
rather than trying to maintain an ever-expanding one-size-fits-all
platform definition.

However, it does suggest a possible alternative approach to naming
these compatibility subsets: what if the name of this particular
platform compatibility tag was something like "linux-sciabi1", rather
than "manylinux1"?

That way, if someone later wanted to propose "linux-guiabi1" or
"linux-audioabi1" or "linux-videoabi1", that could be done.

The auditwheel utility is already designed to support this "multiple
compatibility policy" approach, and the idea of using Docker-based
build environments also lends itself well to that model.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rmcgibbo at gmail.com  Thu Jan 21 22:35:09 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 19:35:09 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CBB146FE-B62F-4A95-8B11-9F132F8FA87F@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAN4+E8E4UZy9jwbrSYvWdUfcCapmhKbq+ppJEhXRPHqTXnJ2XQ@mail.gmail.com>
 <CBB146FE-B62F-4A95-8B11-9F132F8FA87F@stufft.io>
Message-ID: <CAN4+E8ErSj6FnMp6j=frT+kmHC7Muppo_VmknXpcM5LPjYsWNw@mail.gmail.com>

On Thu, Jan 21, 2016 at 7:29 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Jan 21, 2016, at 10:27 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
> wrote:
>
> I'm hopeful that in the future, if this PEP is accepted
> and we can make the tooling and documentation excellent, uploading Linux
> wheels can start to become a
> standard part of the PyPI release cycle for package maintainers.
>
>
>
> Longer term, I really want to (but have put zero effort into trying to
> plan it out beyond pie in the sky dreams) have it so that maintainers can
> publish a sdist to PyPI, and have PyPI automatically build wheels for you.
> This is a long way down the road though.
>


Yes, absolutely! I think this will actually not be _too_ difficult for
Linux (because docker). The challenges for Windows
and OS X are more significant.  For the open source communities that I'm
involved in, Travis-CI has really made everyone much more aware and
comfortable with Linux container-based web services that compile our
packages, so
a PyPI wheel farm seems very much within reach over the next year or so.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/0b230639/attachment.html>

From rmcgibbo at gmail.com  Thu Jan 21 22:45:04 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 21 Jan 2016 19:45:04 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7cX39B=LP1Ssxq09f8AX4YWTMgxOqU6JoG1jjo+0+f87w@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
 <CADiSq7cX39B=LP1Ssxq09f8AX4YWTMgxOqU6JoG1jjo+0+f87w@mail.gmail.com>
Message-ID: <CAN4+E8FiJsyERnBFSUx5jF4eZDpXCPtyS-ayoRKSCCO0YH6HSg@mail.gmail.com>

On Thu, Jan 21, 2016 at 7:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> However, it does suggest a possible alternative approach to naming
> these compatibility subsets: what if the name of this particular
> platform compatibility tag was something like "linux-sciabi1", rather
> than "manylinux1"?
>

That's an interesting idea, but I personally don't see the manylinux1 list
as particularly
"scientific". If anything, I'd call it "minimal".


> That way, if someone later wanted to propose "linux-guiabi1" or
> "linux-audioabi1" or "linux-videoabi1", that could be done.


This would be something, but if we want to have Linux binary wheels that
tightly integrate
with system libraries for certain use cases, the *really *valuable thing
would be https://github.com/pypa/interoperability-peps/pull/30/files, more
so than specific ABI tags, IMO.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/412d111e/attachment.html>

From ncoghlan at gmail.com  Thu Jan 21 22:52:29 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2016 13:52:29 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwB=kVbh+o-bHqbCookKLtfFLNKpfbPyWVC2tyzgSHVK-Tw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALT861Hiped2pqtx9wGqweL0jXe9f_zMs7_4_bAJfOZc4giTxw@mail.gmail.com>
 <CAPJVwB=kVbh+o-bHqbCookKLtfFLNKpfbPyWVC2tyzgSHVK-Tw@mail.gmail.com>
Message-ID: <CADiSq7cBFCvLv5=kZuVfJ9ECAW8tHKXu3ZqW-BmKhktuY3hFAg@mail.gmail.com>

On 22 January 2016 at 09:31, Nathaniel Smith <njs at pobox.com> wrote:
> On Thu, Jan 21, 2016 at 2:22 PM, Nate Coraor <nate at bx.psu.edu> wrote:
>> Nathaniel, Robert, I'm really excited to see how quickly you're making
>> progress. A few comments below as I haven't had a chance to catch up on the
>> day's discussion:
>>
>> On Wed, Jan 20, 2016 at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>
>>> Building on the compability lessons learned from these companies, we thus
>>> define a baseline ``manylinux1`` platform tag for use by binary Python
>>> wheels, and introduce the implementation of preliminary tools to aid in
>>> the
>>> construction of these ``manylinux1`` wheels.
>>
>>
>> Just a standards question: does this still require an update to PEP 425, or
>> would the definition of the  manylinux1 platform here supersede that section
>> of 425?
>
> I guess this is a pure process question, so I'll defer to Nick...

I've finally started working on changing the change proposal process,
so I'm going to say "No" :)

I'll start a separate thread about that, since there are some areas
where I'd be interested in people's feedback before I get to far into
it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri Jan 22 01:58:02 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2016 16:58:02 +1000
Subject: [Distutils] Changing the way we use the PEP process (redux)
Message-ID: <CADiSq7dTAW3VDn2G2q0HJmHQW65EVqQU42kG137qHtM8dcjWQQ@mail.gmail.com>

I've posted about this idea to the list before, but this time I've
finally started working on it and have a concrete plan to discuss :)

The basic idea:

* I want to progressively move the active interoperability
specifications out of PEPs and into a subsection of
packaging.python.org
* packaging PEPs would then become a tool for changing those
specifications, rather than the specifications themselves
* the description of this process would be captured along with the
rest of the PyPA operational docs at pypa.io
* there's a draft PR to start down this path at
https://github.com/pypa/pypa.io/pull/12

That PR provides an example of the potential benefits of this approach
- it's able to state which parts of PEP 345 have been superseded by
other PEPs, and also note the "Provides-Extra" field which exists as a
de facto standard, despite not being formally adopted through the PEP
process.

However, having written the draft PR entirely against pypa.io, I've
now gone back to thinking packaging.python.org would be a better fit
for the actual hosting of the agreed specifications - the "python.org"
domain is a vastly better known one than "pypa.io", and being on a
subdomain of python.org more clearly establishes these
interoperability specifications as the Python Packaging counterparts
to the Python Language Reference and Python Library Reference.

So my next iteration will be split into two PRs: one for pypa.io
defining the specification management process, and one for
packaging.python.org adding a specifications section

Once those changes are merged, we'll end up with additional lower
overhead ways to handle minor updates to the specifications: pure
clarifications can be handled through PRs and issues against
packaging.python.org, minor updates and/or updating the specifications
to match real world practices can be handled through a distutils-sig
discussion, while larger more complex (or more controversial) updates
will still need to go through the PEP process.

The additional background:

For folks that haven't used the PEP process to work on CPython itself,
here's the way that works:

- major or controversial proposals get a standards track PEP
- that gets debated/discussed on python-dev (perhaps with a
preliminary discussion on python-ideas or one of the SIGs)
- if Accepted, the relevant changes get made to CPython, including the
language and library reference
- the PEP is marked Final
- at this point, CPython and its docs are the source of authoritative
info, NOT the PEP
- future minor updates are handled as tracker issues, with the full
PEP process only invoked again for major or controversial changes

The key point there is that once the PEP is marked Final it becomes a
*historical document*, so there's no need to have a
meta-change-management process for the PEP itself.

It started out that distutils used PEPs at least in something
resembling the same way, since the standard library's distutils was
the reference implementation, and packaging standards evolved at the
same pace of the rest of the standard library. We broke that model
when we moved to using the independently developed pip and setuptools
as the reference implementations.

We've since been using the PEP process in a way a bit more like the
way IETF RFC's work, and I think we can all agree that's been a pretty
clumsy and horrible way to run things - the PEP process really wasn't
designed to be used that way, and it shows.

The approach I'm proposing we switch to gets us back to something much
closer to the way CPython uses the PEP process, which should help both
folks trying to figure out the *current* approaches to
interoperability handling, as well as those of us trying to work on
improvements to those standards.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From matthew.brett at gmail.com  Fri Jan 22 02:04:09 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 21 Jan 2016 23:04:09 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAN4+E8FiJsyERnBFSUx5jF4eZDpXCPtyS-ayoRKSCCO0YH6HSg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
 <CADiSq7cX39B=LP1Ssxq09f8AX4YWTMgxOqU6JoG1jjo+0+f87w@mail.gmail.com>
 <CAN4+E8FiJsyERnBFSUx5jF4eZDpXCPtyS-ayoRKSCCO0YH6HSg@mail.gmail.com>
Message-ID: <CAH6Pt5rdYUT+S-KaE8Jk8BnSSw+UrRnyewTdG3TDYQr3eBStDA@mail.gmail.com>

Hi,

On Thu, Jan 21, 2016 at 7:45 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> On Thu, Jan 21, 2016 at 7:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>
>> However, it does suggest a possible alternative approach to naming
>> these compatibility subsets: what if the name of this particular
>> platform compatibility tag was something like "linux-sciabi1", rather
>> than "manylinux1"?
>
>
> That's an interesting idea, but I personally don't see the manylinux1 list
> as particularly
> "scientific". If anything, I'd call it "minimal".

Yes, I agree, I don't think 'linux-sciabi1" would differentiate this
from other ways of building wheels.  For example, I can't see why this
wouldn't be a perfectly reasonable way to proceed for someone doing
audio or video.   The difference that "manylinux" was designed to
capture is the idea of having a single set of wheels for many versions
of Linux, rather than wheels specific to particular distributions or
packaged versions of external libraries.

Cheers,

Matthew

From marius at gedmin.as  Fri Jan 22 02:06:13 2016
From: marius at gedmin.as (Marius Gedminas)
Date: Fri, 22 Jan 2016 09:06:13 +0200
Subject: [Distutils] Package classifiers - both major and minor Python
 versions?
In-Reply-To: <CAP1=2W4RYmnsy41y144eAMYptNQge3XrbAPgSu+VNbeCBEOXcQ@mail.gmail.com>
References: <CAJ3y2QRxrSW2dER_2X-7yFJDpPrbNU6f79+p0ki7Kwn-bQayFw@mail.gmail.com>
 <CAP1=2W4RYmnsy41y144eAMYptNQge3XrbAPgSu+VNbeCBEOXcQ@mail.gmail.com>
Message-ID: <20160122070613.GB5531@platonas>

On Thu, Jan 21, 2016 at 05:53:32PM +0000, Brett Cannon wrote:
> On Thu, 21 Jan 2016 at 06:48 John Whitlock <John-Whitlock at ieee.org> wrote:
> 
> > A discussion about the Python language classifiers came up in a pull
> > request [1], and I couldn't find a definite answer. The question is -
> > should a packager specify the major Python versions, minor Python versions,
> > or both?
> >
> > The Python Packaging User Guide's example [2] has both:
> >
> >   # Specify the Python versions you support here. In particular, ensure
> >   # that you indicate whether you support Python 2, Python 3 or both.
> >   'Programming Language :: Python :: 2',
> >   'Programming Language :: Python :: 2.6',
> >   'Programming Language :: Python :: 2.7',
> >   'Programming Language :: Python :: 3',
> >   'Programming Language :: Python :: 3.2',
> >   'Programming Language :: Python :: 3.3',
> >   'Programming Language :: Python :: 3.4',
> >
> > In the example, 'Programming Language :: Python :: 2' is a major version,
> > and 'Programming Language :: Python :: 2.7' is a minor version.
> >
> > pyroma [3], which I use as a packaging linter, has insisted on both the
> > major and minor versions since the first release in 2011 [4].
> >
> > These were added in 2008, but the announcement on this mailing list didn't
> > include guidance on usage [5].  I can't find any guidance in PEPs either.
> >
> 
> You should at least do the major versions, and if you are up for
> maintaining them, then do the minor ones as well.
> 
> You want the major ones to tell people whether you even support Python 3 or
> not. Various tools like caniusepython3 rely on at least the major version
> classifier existing to programmatically know about Python 3 support.

Are these tools unable to realize that supporting a particular minor
version implies support for the corresponding major version?

Marius Gedminas
-- 
HOST SYSTEM RESPONDING, PROBABLY UP...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/dc890911/attachment.sig>

From fred at fdrake.net  Fri Jan 22 02:34:27 2016
From: fred at fdrake.net (Fred Drake)
Date: Fri, 22 Jan 2016 02:34:27 -0500
Subject: [Distutils] Package classifiers - both major and minor Python
 versions?
In-Reply-To: <20160122070613.GB5531@platonas>
References: <CAJ3y2QRxrSW2dER_2X-7yFJDpPrbNU6f79+p0ki7Kwn-bQayFw@mail.gmail.com>
 <CAP1=2W4RYmnsy41y144eAMYptNQge3XrbAPgSu+VNbeCBEOXcQ@mail.gmail.com>
 <20160122070613.GB5531@platonas>
Message-ID: <CAFT4OTFO5-PvpyQSRe3A9VO=-peqJU1P=j252TJaBitg==r25Q@mail.gmail.com>

On Fri, Jan 22, 2016 at 2:06 AM, Marius Gedminas <marius at gedmin.as> wrote:
> Are these tools unable to realize that supporting a particular minor
> version implies support for the corresponding major version?

Should supporting 2.7 indicate support for all 2.x releases?

My take is that classifiers should be selected based on intent,
especially given the lack of documentation on how each should be
interpreted.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From chris.jerdonek at gmail.com  Fri Jan 22 02:57:12 2016
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Thu, 21 Jan 2016 23:57:12 -0800
Subject: [Distutils] Package classifiers - both major and minor Python
 versions?
In-Reply-To: <CAFT4OTFO5-PvpyQSRe3A9VO=-peqJU1P=j252TJaBitg==r25Q@mail.gmail.com>
References: <CAJ3y2QRxrSW2dER_2X-7yFJDpPrbNU6f79+p0ki7Kwn-bQayFw@mail.gmail.com>
 <CAP1=2W4RYmnsy41y144eAMYptNQge3XrbAPgSu+VNbeCBEOXcQ@mail.gmail.com>
 <20160122070613.GB5531@platonas>
 <CAFT4OTFO5-PvpyQSRe3A9VO=-peqJU1P=j252TJaBitg==r25Q@mail.gmail.com>
Message-ID: <CAOTb1weEs17LsqOWnfKkSbnHQS76PLkPgyLbQQiXvBnJosfe2Q@mail.gmail.com>

On Thu, Jan 21, 2016 at 11:34 PM, Fred Drake <fred at fdrake.net> wrote:
> On Fri, Jan 22, 2016 at 2:06 AM, Marius Gedminas <marius at gedmin.as> wrote:
>> Are these tools unable to realize that supporting a particular minor
>> version implies support for the corresponding major version?
>
> Should supporting 2.7 indicate support for all 2.x releases?

No, I think he's asking why someone should need to list "Python 2" as
a classifier if they're already listing, say, "Python 2.7."  This is
assuming that "Python 2" means that _some_ 2.x version is supported
rather than all, which is what I had assumed it to mean (and similarly
for "Python 3").

If "Python 2" really means that all 2.x versions are supported as
opposed to just some, then that should also be clarified (and
similarly for "Python 3").

--Chris


>
> My take is that classifiers should be selected based on intent,
> especially given the lack of documentation on how each should be
> interpreted.
>
>
>   -Fred
>
> --
> Fred L. Drake, Jr.    <fred at fdrake.net>
> "A storm broke loose in my mind."  --Albert Einstein
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From ncoghlan at gmail.com  Fri Jan 22 03:05:19 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2016 18:05:19 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAH6Pt5rdYUT+S-KaE8Jk8BnSSw+UrRnyewTdG3TDYQr3eBStDA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
 <CADiSq7cX39B=LP1Ssxq09f8AX4YWTMgxOqU6JoG1jjo+0+f87w@mail.gmail.com>
 <CAN4+E8FiJsyERnBFSUx5jF4eZDpXCPtyS-ayoRKSCCO0YH6HSg@mail.gmail.com>
 <CAH6Pt5rdYUT+S-KaE8Jk8BnSSw+UrRnyewTdG3TDYQr3eBStDA@mail.gmail.com>
Message-ID: <CADiSq7d68Lz1JfrutcGRcirWAAZSz4uTrG7emR8gwzwjXevmkA@mail.gmail.com>

On 22 January 2016 at 17:04, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
> On Thu, Jan 21, 2016 at 7:45 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
>> On Thu, Jan 21, 2016 at 7:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> However, it does suggest a possible alternative approach to naming
>>> these compatibility subsets: what if the name of this particular
>>> platform compatibility tag was something like "linux-sciabi1", rather
>>> than "manylinux1"?
>>
>> That's an interesting idea, but I personally don't see the manylinux1 list
>> as particularly
>> "scientific". If anything, I'd call it "minimal".
>
> Yes, I agree, I don't think 'linux-sciabi1" would differentiate this
> from other ways of building wheels.  For example, I can't see why this
> wouldn't be a perfectly reasonable way to proceed for someone doing
> audio or video.   The difference that "manylinux" was designed to
> capture is the idea of having a single set of wheels for many versions
> of Linux, rather than wheels specific to particular distributions or
> packaged versions of external libraries.

Yeah, it was just an idea to potentially address MAL's concerns
regarding scope. However, I think the other replies to the thread have
adequately addressed that, and we can continue deferring the question
of scope increases to manylinux2 after seeing how far the current list
and "auditwheel repair" can get us.

The PEP should also be explicit that this does reintroduce the
bundling problem that distro unbundling policies were designed to
address, but:

1. In these days of automated continuous intregration & deployment
pipelines, publishing new versions and updating dependencies is easier
than it was when those policies were defined
2. Folks remain free to use "--no-binary" if they want to force local
builds rather than using pre-built wheel files
3. The popularity of container based deployment and "immutable
infrastructure" models involve substantial bundling at the application
layer anyway
4. This PEP doesn't rule out the idea of offering more targeted
binaries for particular Linux distributions

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Fri Jan 22 04:26:08 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 22 Jan 2016 09:26:08 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAH6Pt5rdYUT+S-KaE8Jk8BnSSw+UrRnyewTdG3TDYQr3eBStDA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAPJVwBkwdpV_Q1fJU4Y5wQ21LN0wDZzYuw5vE2abT_boDu2Hag@mail.gmail.com>
 <56A10CF9.8070801@egenix.com>
 <CADiSq7cX39B=LP1Ssxq09f8AX4YWTMgxOqU6JoG1jjo+0+f87w@mail.gmail.com>
 <CAN4+E8FiJsyERnBFSUx5jF4eZDpXCPtyS-ayoRKSCCO0YH6HSg@mail.gmail.com>
 <CAH6Pt5rdYUT+S-KaE8Jk8BnSSw+UrRnyewTdG3TDYQr3eBStDA@mail.gmail.com>
Message-ID: <CACac1F_Y0vJzCDchFRY0KGqpsZSW_JZkNbY1+BgoMYp4pRrOgg@mail.gmail.com>

On 22 January 2016 at 07:04, Matthew Brett <matthew.brett at gmail.com> wrote:
>> That's an interesting idea, but I personally don't see the manylinux1 list
>> as particularly
>> "scientific". If anything, I'd call it "minimal".
>
> Yes, I agree, I don't think 'linux-sciabi1" would differentiate this
> from other ways of building wheels.  For example, I can't see why this
> wouldn't be a perfectly reasonable way to proceed for someone doing
> audio or video.   The difference that "manylinux" was designed to
> capture is the idea of having a single set of wheels for many versions
> of Linux, rather than wheels specific to particular distributions or
> packaged versions of external libraries.

Experience with Christoph Gohlke's binary distributions on Windows
suggests that a significant majority of non-scientific uses are
perfectly well served by the sort of package list that scientific
users would need. And I suspect that not all Enthought/Anaconda users
are scientists, either.

So I'd rather that the tag was based on capability rather than
community / intended use. On that basis, "linux-minimal1" sounds fine
to me.

Paul

From mal at egenix.com  Fri Jan 22 04:33:11 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 22 Jan 2016 10:33:11 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
Message-ID: <56A1F757.4050107@egenix.com>

On 21.01.2016 20:05, Matthew Brett wrote:
> Hi,
> 
> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>>
>> IMO, testing the versions of a set of libraries is a safer
>> approach. It's perfectly fine to have a few dependencies
>> not work in a module because an optional system package is not
>> installed, e.g. say a package comes with UIs written in
>> Qt and one in GTK.
> 
> Please forgive my slowness, but I don't understand exactly what you
> mean.  Can you give a specific example?
> 
> Say my package depends on libpng.
> 
> Call the machine I'm installing on the client machine.
> 
> Are you saying that, when I build a wheel, I should specify to the
> wheel what versions of libpng I can tolerate on the the client
> machine, and if if the client does have a compatible version, then pip
> should raise an error, perhaps with a useful message about how to get
> libpng?
> 
> If you do mean that, how do you want the PEP changed?

I already posted a change proposal earlier on in the thread.
I'll repeat it here (with a minor enhancements):

"""
The ``manylinux1`` policy
=========================

For these reasons, we define a set of library versions that
are supported by a wide range of Linux distributions. We therefore
pick library versions which have been around for at least 5 years.

When using these external libraries, Python wheels should
only depend on library versions listed in the section below.

Python wheels are free to depend on additional libraries
not included in this set, however, care should be taken
that these additional libraries do not depend on later
versions of the listed libraries, e.g. OpenSSL libraries
compiled against the C library versions listed below.

The ``manylinux1`` policy thus encompasses a standard for which
versions of these external shared libraries a wheel may depend on,
and the maximum depended-upon symbol versions therein.

Future versions of the manylinux policy may include more
libraries, or move on to later versions.

The permitted external shared libraries versions for
``manylinux1``are: ::

    libgcc_s.so.1
    libstdc++.so.6
    ... only include the basic set of libs, no GUI or curses ...

"""

The idea is to not pin down the set of usable external libraries,
but instead pin down a set of versions for the most important
libraries wheels will depend on and then let the wheels use
other external libraries as necessary without version checks.

In more details:

If you want a wheel to run on many Linux distributions, you have
to make sure that the basic lib C and a few other utility
libraries are available and compatible with the ones you used
to build the wheel.

This can be addressed by defining a set of important libraries
and corresponding versions. You do not have to limit the overall
set of usable libraries for this, since less commonly used
libraries will usually have to be installed separately anyway.

For example, if a package needs a specific version of libpng,
the package author can document this and the user can then make
sure to install that particular version.

The PEP should only be concerned with the basic set of
libraries you typically need for a wheel, not any of the
less common ones. The X11 libs for example do not have to
be version pinned for the manylinux tag, since they are not
essential for the vast majority of Python packages (and here
I'm talking about the thousands of packages on PyPI, not the
few hundred mentioned earlier in the thread, which are
covered by Anaconda and Canopy).

By defining "manylinux1" in such a way you get:

 * broad compatibility of wheel files on Linux

 * full flexibility of wheels interfacing or wrapping
   to other external libraries not covered in the PEP

 * no lock-out of package authors who would like to push
   wheel files for their packages to PyPI, but happen to
   use libraries not in the predefined list of the original
   draft PEP

I left out the other details I mentioned (symbol versioning
and dealing with different architectures) to focus on
pinning libraries vs. pinning versions for now. Later on,
we'd have to apply a similar strategy to other platforms
as well, e.g. *BSD, AIX, Solaris, etc.

Since we're targeting Linux, it may be helpful to base
the list of libraries and versions on the Linux Standard
Base (LSB), since one of the main ideas behind the LSB is
binary compatibility between distributions:

http://refspecs.linuxfoundation.org/lsb.shtml

If people on this list still can't see the benefit of
just pinning down versions of specific library files
for "manylinux1" over limiting the set of allowed libraries
and forcing people to embed any other libs in the wheel files,
I guess I'll just have to write up a competing or additional
tag PEP to enable all package authors to publish wheels for
Linux on PyPI, but this won't happen until February.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 22 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From njs at pobox.com  Fri Jan 22 05:03:09 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 22 Jan 2016 02:03:09 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A1F757.4050107@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
Message-ID: <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>

On Fri, Jan 22, 2016 at 1:33 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 21.01.2016 20:05, Matthew Brett wrote:
>> Hi,
>>
>> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>> By using the version based approach, we'd not run into this
>>>>> problem and gain a lot more.
>>>>
>>>> I think it's better to start with a small core that we *know* works,
>>>> then expand later, rather than trying to make the first iteration too
>>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>>> than newer ones.
>>>
>>> My argument is that the file based approach taken by the PEP
>>> is too limiting to actually make things work for a large
>>> set of Python packages.
>>>
>>> It will basically only work for packages that do not interface
>>> to other external libraries (except for the few cases listed in
>>> the PEP, e.g. X11, GL, which aren't always installed or
>>> available either).
>>>
>>> IMO, testing the versions of a set of libraries is a safer
>>> approach. It's perfectly fine to have a few dependencies
>>> not work in a module because an optional system package is not
>>> installed, e.g. say a package comes with UIs written in
>>> Qt and one in GTK.
>>
>> Please forgive my slowness, but I don't understand exactly what you
>> mean.  Can you give a specific example?
>>
>> Say my package depends on libpng.
>>
>> Call the machine I'm installing on the client machine.
>>
>> Are you saying that, when I build a wheel, I should specify to the
>> wheel what versions of libpng I can tolerate on the the client
>> machine, and if if the client does have a compatible version, then pip
>> should raise an error, perhaps with a useful message about how to get
>> libpng?
>>
>> If you do mean that, how do you want the PEP changed?
>
> I already posted a change proposal earlier on in the thread.
> I'll repeat it here (with a minor enhancements):

Okay, I think I get it now. I'll try to repeat back to summarize and
see if I have understood your proposal correctly:

In the PEP 513 "manylinux1" approach, when users do 'pip install foo',
then one of three things happens:
1) they get a working foo and are immediately good-to-go, or
2) pip says "I'm sorry, there's no compatible wheel", or
3) something else happens, in which case this is a bug, and the spec
provides some framework to help us determine whether this is a bug in
the wheel, a bug in pip, or a bug in the spec.

In your approach, users do 'pip install foo', and then pip installs
the wheel, and then when they try to use the wheel they get an error
message from the dynamic linker about missing libraries, and then the
user has to read the docs or squint at these error messages in order
to figure out what set of apt-get / yum / pacman / ... commands they
need to run in order to make foo work. (And possibly there is no such
combination of commands that will actually work, because e.g. the
wheel was linked against Debian's version of libbar.so.7 and Fedora's
version of libbar.so.7 turns out to have an incompatible ABI, or
Fedora simply doesn't provide a libbar.so.7 package at all.)

I won't express any opinion on your alternative PEP with its own
platform tag without reading it, but we're not going to change PEP 513
to work this way.

>  * no lock-out of package authors who would like to push
>    wheel files for their packages to PyPI, but happen to
>    use libraries not in the predefined list of the original
>    draft PEP

https://mail.python.org/pipermail/distutils-sig/2016-January/028050.html

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Fri Jan 22 05:42:59 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2016 20:42:59 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A1F757.4050107@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
Message-ID: <CADiSq7d-4pr-j2bSu=x+yBgNZvVWWdGUYs0TbgY6Xd4qQMvjiA@mail.gmail.com>

On 22 January 2016 at 19:33, M.-A. Lemburg <mal at egenix.com> wrote:
> For example, if a package needs a specific version of libpng,
> the package author can document this and the user can then make
> sure to install that particular version.

The assumption that any given Python user will know how to do this is
not a reasonable assumption in 2016.

If a publisher wants to bundle a particular version of libpng, they
can. If (as is also entirely reasonable) they don't want to assume the
associated responsibilities for responding to CVEs, then they can
stick with source distributions, or target more specific Linux
versions (as previously discussed in the context of Nate Coraor's
Starforge work)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mal at egenix.com  Fri Jan 22 05:48:33 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 22 Jan 2016 11:48:33 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
Message-ID: <56A20901.1050300@egenix.com>

On 22.01.2016 11:03, Nathaniel Smith wrote:
> On Fri, Jan 22, 2016 at 1:33 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 21.01.2016 20:05, Matthew Brett wrote:
>>> Hi,
>>>
>>> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>>> By using the version based approach, we'd not run into this
>>>>>> problem and gain a lot more.
>>>>>
>>>>> I think it's better to start with a small core that we *know* works,
>>>>> then expand later, rather than trying to make the first iteration too
>>>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>>>> than newer ones.
>>>>
>>>> My argument is that the file based approach taken by the PEP
>>>> is too limiting to actually make things work for a large
>>>> set of Python packages.
>>>>
>>>> It will basically only work for packages that do not interface
>>>> to other external libraries (except for the few cases listed in
>>>> the PEP, e.g. X11, GL, which aren't always installed or
>>>> available either).
>>>>
>>>> IMO, testing the versions of a set of libraries is a safer
>>>> approach. It's perfectly fine to have a few dependencies
>>>> not work in a module because an optional system package is not
>>>> installed, e.g. say a package comes with UIs written in
>>>> Qt and one in GTK.
>>>
>>> Please forgive my slowness, but I don't understand exactly what you
>>> mean.  Can you give a specific example?
>>>
>>> Say my package depends on libpng.
>>>
>>> Call the machine I'm installing on the client machine.
>>>
>>> Are you saying that, when I build a wheel, I should specify to the
>>> wheel what versions of libpng I can tolerate on the the client
>>> machine, and if if the client does have a compatible version, then pip
>>> should raise an error, perhaps with a useful message about how to get
>>> libpng?
>>>
>>> If you do mean that, how do you want the PEP changed?
>>
>> I already posted a change proposal earlier on in the thread.
>> I'll repeat it here (with a minor enhancements):
> 
> Okay, I think I get it now. I'll try to repeat back to summarize and
> see if I have understood your proposal correctly:
> 
> In the PEP 513 "manylinux1" approach, when users do 'pip install foo',
> then one of three things happens:
> 1) they get a working foo and are immediately good-to-go, or
> 2) pip says "I'm sorry, there's no compatible wheel", or
> 3) something else happens, in which case this is a bug, and the spec
> provides some framework to help us determine whether this is a bug in
> the wheel, a bug in pip, or a bug in the spec.
> 
> In your approach, users do 'pip install foo', and then pip installs
> the wheel, and then when they try to use the wheel they get an error
> message from the dynamic linker about missing libraries, and then the
> user has to read the docs or squint at these error messages in order
> to figure out what set of apt-get / yum / pacman / ... commands they
> need to run in order to make foo work. (And possibly there is no such
> combination of commands that will actually work, because e.g. the
> wheel was linked against Debian's version of libbar.so.7 and Fedora's
> version of libbar.so.7 turns out to have an incompatible ABI, or
> Fedora simply doesn't provide a libbar.so.7 package at all.)

pip could be made to check the wheel for missing library
dependencies in order to provide help with cases where
additional packages are needed, but overall, yes, that's
the way it should work, IMO.

It's better to have wheels than not to have them, since
installing an additional system package is by far easier
than trying to compile packages from source (this will
usually also require additional -dev packages to be
installed).

>>  * no lock-out of package authors who would like to push
>>    wheel files for their packages to PyPI, but happen to
>>    use libraries not in the predefined list of the original
>>    draft PEP
>
> https://mail.python.org/pipermail/distutils-sig/2016-January/028050.html

Embedding additional libraries in the wheels files to overcome
deficiencies in the PEP design simply doesn't feel right
to me.

People who rely on Linux distributions want to continue
to do so and get regular updates for system packages from
their system vendor. Having wheel files override these
system packages by including libs directly in the wheel
silently breaks this expectation, potentially opening
up the installations for security holes, difficult to
track bugs and possible version conflicts with already
loaded versions of the shared libs.

IMO, that's much worse than having to install additional
system packages to make a Python wheel work.

The embedding approach also creates licensing problems,
since those libs may be under different licenses than the
package itself. And of course, it increases the size of the
wheel files, causing more bandwidth to be necessary,
more disk space to be used for wheel caches, etc.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 22 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From donald at stufft.io  Fri Jan 22 06:25:46 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 22 Jan 2016 06:25:46 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A20901.1050300@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
Message-ID: <4FBA4669-A5D9-4B65-8F2D-CE835A016660@stufft.io>


> On Jan 22, 2016, at 5:48 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> Embedding additional libraries in the wheels files to overcome
> deficiencies in the PEP design simply doesn't feel right
> to me.
> 
> People who rely on Linux distributions want to continue
> to do so and get regular updates for system packages from
> their system vendor. Having wheel files override these
> system packages by including libs directly in the wheel
> silently breaks this expectation, potentially opening
> up the installations for security holes, difficult to
> track bugs and possible version conflicts with already
> loaded versions of the shared libs.
> 
> IMO, that's much worse than having to install additional
> system packages to make a Python wheel work.
> 
> The embedding approach also creates licensing problems,
> since those libs may be under different licenses than the
> package itself. And of course, it increases the size of the
> wheel files, causing more bandwidth to be necessary,
> more disk space to be used for wheel caches, etc.

I think there are a few things here, but this is not my area of expertise so I
could be wrong. As I understand it, The manylinux platform definition is
largely going to be a documentation effort and there isn't going to be much in
the way of enforcement. That means that people who build wheels against the
manylinux platform tag are free to really do whatever they want even if it
doesn't strictly match the definition of the manylinux platform. The difference
here is that if you link against something that isn't included in the set of
libraries, and that subsequently breaks due to an ABI incompatability, that's
not a pip bug or a manylinux bug, that's a packaging bug with that particular
library and they'll have to decide how they want to resolve it (if they want
to resolve it). So you'll be free to link to anything you want, but you get to
keep both pieces if it breaks and it's outside this defined set of libraries.

I also agree that it's OK for users to have to ``apt-get`` (or whatever) a
particular library to use something and we don't have to *only* rely on items
that are installed as part of a "standard" linux base system. However, what is
not OK (IMO) is for the PEP to bless something that has a high chance of ending
up with ABI issues rather than "need to apt-get install" issues. For instance,
even if you compile against a sufficiently old copy of OpenSSL, OpenSSL (to my
understanding) does not have a stable ABI and you cannot take something
compiled against OpenSSL on CentOS 5.reallyold and expect it to work on say
Arch Linux.

So I think there's an explicit list of packages that we know will generally
work as long as you build against a sufficiently old copy of them and outside
of that it's really a big unknown in general if a particular library can be
used in this way or not. We obviously can't enumerate the list of every
possible C library that has a stable ABI that can sanely be used cross distro
but I think it's reasonable to list some sort of base minimum here, and if
people experiment with stepping outside the standard list and can come to us
and show "hey, I tried it with xyz library, we've gotten X installs and no
complaints" we can then possibly expand the definition of the manylinux
platform to include that library and move that project from depending on
undefined behavior to defined behavior.

Thinking of it in terms of a C like "undefined behavior" is probably a
reasonable way of doing it. Linking against a system provided library that is
on this list is a defined behavior of the manylinux "platform", linking against
something else is undefined and may or may not work. At some level, once you've
gotten to the point you're using pip to manage some set of your packages it
doesn't really matter if that set of things you're pulling from PyPI includes
a C library or not. If you're relying on say psycopg2 it's not clear to me that
libpq *needs* to be getting security any more than psycopg2 itself does and so
you'll need some method of solving that problem for your Python level
dependencies anyways.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/2c238b66/attachment.sig>

From ncoghlan at gmail.com  Fri Jan 22 06:29:23 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2016 21:29:23 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A20901.1050300@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
Message-ID: <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>

On 22 January 2016 at 20:48, M.-A. Lemburg <mal at egenix.com> wrote:
> People who rely on Linux distributions want to continue
> to do so and get regular updates for system packages from
> their system vendor. Having wheel files override these
> system packages by including libs directly in the wheel
> silently breaks this expectation, potentially opening
> up the installations for security holes, difficult to
> track bugs and possible version conflicts with already
> loaded versions of the shared libs.

For the time being, these users should either pass the "--no-binary"
option to pip, ask their distro to provide an index of pre-built wheel
files for that distro (once we have the distro-specific wheel tagging
PEP sorted out), or else ask their distro to update system Python
packages in a more timely fashion (or all of the above).

> IMO, that's much worse than having to install additional
> system packages to make a Python wheel work.
>
> The embedding approach also creates licensing problems,
> since those libs may be under different licenses than the
> package itself. And of course, it increases the size of the
> wheel files, causing more bandwidth to be necessary,
> more disk space to be used for wheel caches, etc.

Then *don't publish manylinux wheel files*. Manylinux is, by design, a
platform+publisher-silo model, very similar to the way smart phone
operating systems work, and the way Windows applications and (I
believe) Mac App Bundles work. It is anti-thetical to the traditional
tightly coupled shared everything model adopted by Linux distributions
(where all the binaries are also generally built by a common central
build system).

There is a different model, which could be tagged as (for example)
"integratedlinux1", which is the one you propose. That wouldn't be
viable from a UX perspective without an external dependency
description system like the one Tennessee proposed in
https://github.com/pypa/interoperability-peps/pull/30, but that
approach requires a lot more development work before it could be
adopted.

>From the point of view of future-proofing PEP 513 against having such
an alternative available in the future, the main question that would
need to be considered is how tools would decide download priority
between a distro-specific wheel, an integrated linux wheel, and a
linux wheel with bundled dependencies.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri Jan 22 06:42:02 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2016 21:42:02 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <4FBA4669-A5D9-4B65-8F2D-CE835A016660@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <4FBA4669-A5D9-4B65-8F2D-CE835A016660@stufft.io>
Message-ID: <CADiSq7dqe6antwegHBDfgbhX7t9RK_+KauDJY3PKxVNt56Nu5Q@mail.gmail.com>

On 22 January 2016 at 21:25, Donald Stufft <donald at stufft.io> wrote:
> Thinking of it in terms of a C like "undefined behavior" is probably a
> reasonable way of doing it. Linking against a system provided library that is
> on this list is a defined behavior of the manylinux "platform", linking against
> something else is undefined and may or may not work. At some level, once you've
> gotten to the point you're using pip to manage some set of your packages it
> doesn't really matter if that set of things you're pulling from PyPI includes
> a C library or not. If you're relying on say psycopg2 it's not clear to me that
> libpq *needs* to be getting security any more than psycopg2 itself does and so
> you'll need some method of solving that problem for your Python level
> dependencies anyways.

It also wouldn't surprise me if CVE trackers like requires.io and
versioneye.com gained the ability to search wheel files for embedded
dependencies and flag outdated and vulnerable ones.

However, it's a good point that PyPI won't be running auditwheel to
*force* compliance with the "no external dependencies outside the
defined set" guideline - while a service like pythonwheels.com could
potentially be set up independently of PyPI to run auditwheel on
manylinux wheels, PyPI itself wouldn't do it.

An external scan like that could actually be a useful way of defining
manylinux2 in the future - scanning popular manylinux wheel downloads
for both embedded libraries and for external dependencies outside the
defined set.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From olivier.grisel at ensta.org  Fri Jan 22 06:43:42 2016
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Fri, 22 Jan 2016 12:43:42 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <-4166805547148809032@unknownmsgid>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
Message-ID: <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>

2016-01-22 3:47 GMT+01:00 Chris Barker - NOAA Federal <chris.barker at noaa.gov>:
>
> Maybe the community will spring forth and do that -- I'm skeptical because I
> tried to to that for years for OS-X and it was just too much to do. And the
> infrastructure was there.
>
> Before pip and wheel there were mpkgs on OS-X, and repo's of toms for Linux
> years before that -- but always the result of a couple people's heroic
> efforts.
>
> Maybe the infrastructure has improved, and the community grown enough, that
> this will all work. We'll see.

I think the infrastructure has improved. For instance I invested some
time and effort to provide a template configuration to build C/C++
compiled extensions for windows wheels on top of the free AppVeyor.com
CI platform [1].

Since then this build configuration has been integrated in the python
packaging documentation [2] and I had the opportunity to present that
work at PyCon [3] (and PyCon FR) last year. Now the community of
project maintainers has picked it up. I can count more than 300
projects using this build setup on github. I have very little work to
do to help them maintain that infra-strucuture nowadays. Even the
configuration upgrade to make it work with MSVC 2015 / Python 3.5 was
contributed to my repo before I could find the time to investigate the
issue my-self.

My point is that once we have clearly defined best-practices for
packaging and convenient tools to build the packages automatically and
test that they work as expected (e.g. free hosted CI that support
running an old centos-based docker container), I am rather confident
that the community will do the work.

It's mostly a matter of providing good tools and education resources
(documentation and example configuration templates).

[1] https://github.com/ogrisel/python-appveyor-demo/
[2] https://www.appveyor.com/docs/packaging-artifacts
[3] https://www.youtube.com/watch?v=d-p6lJJObLU

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel

From donald at stufft.io  Fri Jan 22 06:55:13 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 22 Jan 2016 06:55:13 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
Message-ID: <AB651C36-AC25-4EAA-A350-A5339E62F138@stufft.io>


> On Jan 22, 2016, at 6:29 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 22 January 2016 at 20:48, M.-A. Lemburg <mal at egenix.com> wrote:
>> People who rely on Linux distributions want to continue
>> to do so and get regular updates for system packages from
>> their system vendor. Having wheel files override these
>> system packages by including libs directly in the wheel
>> silently breaks this expectation, potentially opening
>> up the installations for security holes, difficult to
>> track bugs and possible version conflicts with already
>> loaded versions of the shared libs.
> 
> For the time being, these users should either pass the "--no-binary"
> option to pip, ask their distro to provide an index of pre-built wheel
> files for that distro (once we have the distro-specific wheel tagging
> PEP sorted out), or else ask their distro to update system Python
> packages in a more timely fashion (or all of the above).

I should note, once we have some solution to the fact that "linux, 64bit" is
way too generic of a platform tag for the general case I plan to allow the
current super generic platform tags to be uploaded as well to PyPI. We don't
try to prevent you from ever being able to release a broken package, we just
want to make it reasonable that you can do the right thing. In other words, as
long as the tooling makes it possible to do the right thing, the fact that you
can also generate packaging bugs in your project (as opposed to pip doing the
wrong thing) isn't a problem. So if people want to do something that isn't
manylinux and don't want to claim to be a manylinux wheel, they'll be free to
use the current linux tags as well.

> 
>> IMO, that's much worse than having to install additional
>> system packages to make a Python wheel work.
>> 
>> The embedding approach also creates licensing problems,
>> since those libs may be under different licenses than the
>> package itself. And of course, it increases the size of the
>> wheel files, causing more bandwidth to be necessary,
>> more disk space to be used for wheel caches, etc.
> 
> Then *don't publish manylinux wheel files*. Manylinux is, by design, a
> platform+publisher-silo model, very similar to the way smart phone
> operating systems work, and the way Windows applications and (I
> believe) Mac App Bundles work. It is anti-thetical to the traditional
> tightly coupled shared everything model adopted by Linux distributions
> (where all the binaries are also generally built by a common central
> build system).
> 
> There is a different model, which could be tagged as (for example)
> "integratedlinux1", which is the one you propose. That wouldn't be
> viable from a UX perspective without an external dependency
> description system like the one Tennessee proposed in
> https://github.com/pypa/interoperability-peps/pull/30, but that
> approach requires a lot more development work before it could be
> adopted.
> 
> From the point of view of future-proofing PEP 513 against having such
> an alternative available in the future, the main question that would
> need to be considered is how tools would decide download priority
> between a distro-specific wheel, an integrated linux wheel, and a
> linux wheel with bundled dependencies.

I think that this could probably just be left up to the individual tools? We
already have to decide between multiple candidate wheels, this is just another
thing to look at.

> 
> Regards,
> Nick.
> 
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/0c82ecb6/attachment.sig>

From mal at egenix.com  Fri Jan 22 07:07:25 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 22 Jan 2016 13:07:25 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <4FBA4669-A5D9-4B65-8F2D-CE835A016660@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <4FBA4669-A5D9-4B65-8F2D-CE835A016660@stufft.io>
Message-ID: <56A21B7D.7010008@egenix.com>

On 22.01.2016 12:25, Donald Stufft wrote:
> 
>> On Jan 22, 2016, at 5:48 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> Embedding additional libraries in the wheels files to overcome
>> deficiencies in the PEP design simply doesn't feel right
>> to me.
>>
>> People who rely on Linux distributions want to continue
>> to do so and get regular updates for system packages from
>> their system vendor. Having wheel files override these
>> system packages by including libs directly in the wheel
>> silently breaks this expectation, potentially opening
>> up the installations for security holes, difficult to
>> track bugs and possible version conflicts with already
>> loaded versions of the shared libs.
>>
>> IMO, that's much worse than having to install additional
>> system packages to make a Python wheel work.
>>
>> The embedding approach also creates licensing problems,
>> since those libs may be under different licenses than the
>> package itself. And of course, it increases the size of the
>> wheel files, causing more bandwidth to be necessary,
>> more disk space to be used for wheel caches, etc.
> 
> I think there are a few things here, but this is not my area of expertise so I
> could be wrong. As I understand it, The manylinux platform definition is
> largely going to be a documentation effort and there isn't going to be much in
> the way of enforcement. That means that people who build wheels against the
> manylinux platform tag are free to really do whatever they want even if it
> doesn't strictly match the definition of the manylinux platform. The difference
> here is that if you link against something that isn't included in the set of
> libraries, and that subsequently breaks due to an ABI incompatability, that's
> not a pip bug or a manylinux bug, that's a packaging bug with that particular
> library and they'll have to decide how they want to resolve it (if they want
> to resolve it). So you'll be free to link to anything you want, but you get to
> keep both pieces if it breaks and it's outside this defined set of libraries.

Hmm, if that were the reading, things would look a lot brighter,
but if PyPI will start to only support uploading manylinux wheels
for Linux platforms, you essentially have the effect that the
PEP ends up defining the set of allowed external libraries and forces
package authors to embed any other external libraries into the
wheel file - or not be able to upload wheel files for Linux at all.

This can hardly be in the interest of Python users who don't want
to use wheel embedded system libraries on their Linux system and
most likely also don't expect wheel files to ship alternative
versions with them in the first place.

If we'd lift the ban of "linux_*" tagged wheels on PyPI at
the same time we allow "manylinux" wheels, that'd remove a lot
of my concerns.

In that case, I'd just like to see a way to tell pip not to install
manylinux wheels with embedded system libraries, or simply outright
reject embedded system libraries in manylinux wheel files.

> I also agree that it's OK for users to have to ``apt-get`` (or whatever) a
> particular library to use something and we don't have to *only* rely on items
> that are installed as part of a "standard" linux base system. However, what is
> not OK (IMO) is for the PEP to bless something that has a high chance of ending
> up with ABI issues rather than "need to apt-get install" issues. For instance,
> even if you compile against a sufficiently old copy of OpenSSL, OpenSSL (to my
> understanding) does not have a stable ABI and you cannot take something
> compiled against OpenSSL on CentOS 5.reallyold and expect it to work on say
> Arch Linux.

True. There will always be incompatibilities out there which
cannot be addressed with a one-fits-all approach. For those
cases, vendor specific wheels would need to be created.

> So I think there's an explicit list of packages that we know will generally
> work as long as you build against a sufficiently old copy of them and outside
> of that it's really a big unknown in general if a particular library can be
> used in this way or not. We obviously can't enumerate the list of every
> possible C library that has a stable ABI that can sanely be used cross distro
> but I think it's reasonable to list some sort of base minimum here, and if
> people experiment with stepping outside the standard list and can come to us
> and show "hey, I tried it with xyz library, we've gotten X installs and no
> complaints" we can then possibly expand the definition of the manylinux
> platform to include that library and move that project from depending on
> undefined behavior to defined behavior.
> 
> Thinking of it in terms of a C like "undefined behavior" is probably a
> reasonable way of doing it. Linking against a system provided library that is
> on this list is a defined behavior of the manylinux "platform", linking against
> something else is undefined and may or may not work. At some level, once you've
> gotten to the point you're using pip to manage some set of your packages it
> doesn't really matter if that set of things you're pulling from PyPI includes
> a C library or not. If you're relying on say psycopg2 it's not clear to me that
> libpq *needs* to be getting security any more than psycopg2 itself does and so
> you'll need some method of solving that problem for your Python level
> dependencies anyways.

You need both: solving issues at the Python level and at the
system level.

However, system vendors will often be a lot faster with updates
than package authors, simply because it's their business model,
so as user you will want to benefit from those updates and
not have to rely on the package author to ship new wheel files.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 22 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From p.f.moore at gmail.com  Fri Jan 22 07:20:52 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 22 Jan 2016 12:20:52 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <AB651C36-AC25-4EAA-A350-A5339E62F138@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
 <AB651C36-AC25-4EAA-A350-A5339E62F138@stufft.io>
Message-ID: <CACac1F8hm7YbqMCDNuD+HGQmte0DfUt3AFXPsuV+CdDyYzVqPw@mail.gmail.com>

On 22 January 2016 at 11:55, Donald Stufft <donald at stufft.io> wrote:
>> From the point of view of future-proofing PEP 513 against having such
>> an alternative available in the future, the main question that would
>> need to be considered is how tools would decide download priority
>> between a distro-specific wheel, an integrated linux wheel, and a
>> linux wheel with bundled dependencies.
>
> I think that this could probably just be left up to the individual tools? We
> already have to decide between multiple candidate wheels, this is just another
> thing to look at.

The compatibility tag spec (PEP 425) does note that choosing between
candidate implementations is a tool-specific implementation detail (at
https://www.python.org/dev/peps/pep-0425/#id14). It seems to me that
this simply comes under the point of "It is recommended that
installers try to choose the most feature complete built distribution
available (the one most specific to the installation environment)..."

At the moment, the process is effective but rudimentary (we don't
really expect more than one compatible wheel that isn't pure-python).
If people start seeing multiple potential wheels and want to have the
option to control the priorities, it's probably better to develop a
solution via pip feature requests, and when things stabilise, document
the solution in a standard if it's appropriate (i.e., it's *not* just
pip configuration options).

Paul

From fred at fdrake.net  Fri Jan 22 08:47:51 2016
From: fred at fdrake.net (Fred Drake)
Date: Fri, 22 Jan 2016 08:47:51 -0500
Subject: [Distutils] Package classifiers - both major and minor Python
 versions?
In-Reply-To: <CAOTb1weEs17LsqOWnfKkSbnHQS76PLkPgyLbQQiXvBnJosfe2Q@mail.gmail.com>
References: <CAJ3y2QRxrSW2dER_2X-7yFJDpPrbNU6f79+p0ki7Kwn-bQayFw@mail.gmail.com>
 <CAP1=2W4RYmnsy41y144eAMYptNQge3XrbAPgSu+VNbeCBEOXcQ@mail.gmail.com>
 <20160122070613.GB5531@platonas>
 <CAFT4OTFO5-PvpyQSRe3A9VO=-peqJU1P=j252TJaBitg==r25Q@mail.gmail.com>
 <CAOTb1weEs17LsqOWnfKkSbnHQS76PLkPgyLbQQiXvBnJosfe2Q@mail.gmail.com>
Message-ID: <CAFT4OTG9BSLgD9mqC=mCECkFwwDinfenshVZhL6G7s5UG7uMtA@mail.gmail.com>

On Fri, Jan 22, 2016 at 2:57 AM, Chris Jerdonek
<chris.jerdonek at gmail.com> wrote:
> If "Python 2" really means that all 2.x versions are supported as
> opposed to just some, then that should also be clarified (and
> similarly for "Python 3").

Bingo!  That's the main point, I think.  Because there's no single
reference for what it means to apply any particular classifier, we're
not going to get a lot of mileage from them.

At this point, classifiers are still pretty much recreational.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From blowekamp at mail.nih.gov  Fri Jan 22 11:26:14 2016
From: blowekamp at mail.nih.gov (Lowekamp, Bradley (NIH/NLM/LHC) [C])
Date: Fri, 22 Jan 2016 16:26:14 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
Message-ID: <DA2915A530468E47B3C4845298E6EB035E522B53@msgb06.nih.gov>

Hello,

I noticed that libpython is missing from the lists of dependent libraries. Also the ?manylinux? Docker image has it?s Python versions compiled with libpython static.

Does this mean that we must do static linking against libpython?
If so, can?t this cause problems with mixing Python objects from different patch versions at runtime? 

I checked Anaconda, and they are using a shared Python library.

Thanks,
Brad
________________________________________
From: Nathaniel Smith [njs at pobox.com]
Sent: Wednesday, January 20, 2016 10:55 PM
To: distutils-sig
Subject: [Distutils] draft PEP: manylinux1

Hi all,

Here's a first draft of a PEP for the manylinux1 platform tag
mentioned earlier, posted for feedback. Really Robert McGibbon should
get the main credit for this, since he wrote it, and also the docker
image and the amazing auditwheel tool linked below, but he asked me to
do the honors of posting it :-).

BTW, if anyone wants to try this out, there are some test
"manylinux1-compatible" wheels at
  https://vorpus.org/~njs/tmp/manylinux-test-wheels/repaired
for PySide (i.e. Qt) and numpy (using openblas). They should be
installable on any ordinary linux system with:
  pip install --no-index -f
https://vorpus.org/~njs/tmp/manylinux-test-wheels/repaired $PKG
(Note that this may require a reasonably up-to-date pip -- e.g. the
one in Debian is too old, which confused me for a bit.)

(How they were created: docker run -it quay.io/manylinux/manylinux
bash; install conda because to get builds of Qt and OpenBLAS because I
was too lazy to do it myself; pip wheel PySide / pip wheel numpy;
auditwheel repair <the resulting wheels>, which copies in all the
dependencies to make the wheels self-contained. Just proof-of-concept
for now, but they seem to work.)

----

PEP: XXXX
Title: A Platform Tag for Portable Linux Built Distributions
Version: $Revision$
Last-Modified: $Date$
Author: Robert T. McGibbon <rmcgibbo at gmail.com>, Nathaniel J. Smith
<njs at pobox.com>
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 19-Jan-2016
Post-History: 19-Jan-2016


Abstract
========

This PEP proposes the creation of a new platform tag for Python package built
distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
external dependencies limited restricted to a standardized subset of
the Linux kernel and core userspace ABI. It proposes that PyPI support
uploading and distributing Wheels with this platform tag, and that ``pip``
support downloading and installing these packages on compatible platforms.


Rationale
=========

Currently, distribution of binary Python extensions for Windows and OS X is
straightforward. Developers and packagers build wheels, which are assigned
platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload these
wheels to PyPI. Users can download and install these wheels using tools such
as ``pip``.

For Linux, the situation is much more delicate. In general, compiled Python
extension modules built on one Linux distribution will not work on other Linux
distributions, or even on the same Linux distribution with different system
libraries installed.

Build tools using PEP 425 platform tags [1]_ do not track information about the
particular Linux distribution or installed system libraries, and instead assign
all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
this ambiguity, there is no expectation that ``linux``-tagged built
distributions compiled on one machine will work properly on another, and for
this reason, PyPI has not permitted the uploading of wheels for Linux.

It would be ideal if wheel packages could be compiled that would work on *any*
linux system. But, because of the incredible diversity of Linux systems -- from
PCs to Android to embedded systems with custom libcs -- this cannot
be guaranteed in general.

Instead, we define a standard subset of the kernel+core userspace ABI that,
in practice, is compatible enough that packages conforming to this standard
will work on *many* linux systems, including essentially all of the desktop
and server distributions in common use. We know this because there are
companies who have been distributing such widely-portable pre-compiled Python
extension modules for Linux -- e.g. Enthought with Canopy [2]_ and Continuum
Analytics with Anaconda [3]_.

Building on the compability lessons learned from these companies, we thus
define a baseline ``manylinux1`` platform tag for use by binary Python
wheels, and introduce the implementation of preliminary tools to aid in the
construction of these ``manylinux1`` wheels.


Key Causes of Inter-Linux Binary Incompatibility
================================================

To properly define a standard that will guarantee that wheel packages meeting
this specification will operate on *many* linux platforms, it is necessary to
understand the root causes which often prevent portability of pre-compiled
binaries on Linux. The two key causes are dependencies on shared libraries
which are not present on users' systems, and dependencies on particular
versions of certain core libraries like ``glibc``.


External Shared Libraries
-------------------------

Most desktop and server linux distributions come with a system package manager
(examples include ``APT`` on Debian-based systems, ``yum`` on
``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
responsibilities, the installation of shared libraries installed to system
directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
on one or more of these shared libraries, and thus function properly only on
systems where the user has the proper libraries (and the proper
versions thereof), either installed using their package manager, or installed
manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
to notify the runtime linker of the location of the depended-upon shared
libraries.


Versioning of Core Shared Libraries
-----------------------------------

Even if author or maintainers of a Python extension module with to use no
external shared libraries, the modules will generally have a dynamic runtime
dependency on the GNU C library, ``glibc``. While it is possible, statically
linking ``glibc`` is usually a bad idea because of bloat, and because certain
important C functions like ``dlopen()`` cannot be called from code that
statically links ``glibc``. A runtime shared library dependency on a
system-provided ``glibc`` is unavoidable in practice.

The maintainers of the GNU C library follow a strict symbol versioning scheme
for backward compatibility. This ensures that binaries compiled against an older
version of ``glibc`` can run on systems that have a newer ``glibc``. The
opposite is generally not true -- binaries compiled on newer Linux
distributions tend to rely upon versioned functions in glibc that are not
available on older systems.

This generally prevents built distributions compiled on the latest Linux
distributions from being portable.


The ``manylinux1`` policy
=========================

For these reasons, to achieve broad portability, Python wheels

 * should depend only on an extremely limited set of external shared
   libraries; and
 * should depend only on ``old`` symbol versions in those external shared
   libraries.

The ``manylinux1`` policy thus encompasses a standard for what the
permitted external shared libraries a wheel may depend on, and the maximum
depended-upon symbol versions therein.

The permitted external shared libraries are: ::

    libpanelw.so.5
    libncursesw.so.5
    libgcc_s.so.1
    libstdc++.so.6
    libm.so.6
    libdl.so.2
    librt.so.1
    libcrypt.so.1
    libc.so.6
    libnsl.so.1
    libutil.so.1
    libpthread.so.0
    libX11.so.6
    libXext.so.6
    libXrender.so.1
    libICE.so.6
    libSM.so.6
    libGL.so.1
    libgobject-2.0.so.0
    libgthread-2.0.so.0
    libglib-2.0.so.0

On Debian-based systems, these libraries are provided by the packages ::

    libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
    libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0

On RPM-based systems, these libraries are provided by the packages ::

    ncurses libgcc libstdc++ glibc libXext libXrender
    libICE libSM mesa-libGL glib2

This list was compiled by checking the external shared library dependencies of
the Canopy [1]_ and Anaconda [2]_ distributions, which both include a wide array
of the most popular Python modules and have been confirmed in practice to work
across a wide swath of Linux systems in the wild.

For dependencies on externally-provided versioned symbols in the above shared
libraries, the following symbol versions are permitted: ::

    GLIBC <= 2.5
    CXXABI <= 3.4.8
    GLIBCXX <= 3.4.9
    GCC <= 4.2.0

These symbol versions were determined by inspecting the latest symbol version
provided in the libraries distributed with CentOS 5, a Linux distribution
released in April 2007. In practice, this means that Python wheels which conform
to this policy should function on almost any linux distribution released after
this date.


Compilation and Tooling
=======================

To support the compilation of wheels meeting the ``manylinux1`` standard, we
provide initial drafts of two tools.

The first is a Docker image based on CentOS 5.11, which is recommended as an
easy to use self-contained build box for compiling ``manylinux1`` wheels [4]_.
Compiling on a more recently-released linux distribution will generally
introduce dependencies on too-new versioned symbols. The image comes with a
full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
well as the latest releases of Python and pip.

The second tool is a command line executable called ``auditwheel`` [5]_. First,
it inspects all of the ELF files inside a wheel to check for dependencies on
versioned symbols or external shared libraries, and verifies conformance with
the ``manylinux1`` policy. This includes the ability to add the new platform
tag to conforming wheels.

In addition, ``auditwheel`` has the ability to automatically modify wheels that
depend on external shared libraries by copying those shared libraries from
the system into the wheel itself, and modifying the appropriate RPATH entries
such that these libraries will be picked up at runtime. This accomplishes a
similar result as if the libraries had been statically linked without requiring
changes to the build system.

Neither of these tools are necessary to build wheels which conform with the
``manylinux1`` policy. Similar results can usually be achieved by statically
linking external dependencies and/or using certain inline assembly constructs
to instruct the linker to prefer older symbol versions, however these tricks
can be quite esoteric.


Platform Detection for Installers
=================================

Because the ``manylinux1`` profile is already known to work for the many
thousands of users of popular commercial Python distributions, we suggest that
installation tools like ``pip`` should error on the side of assuming that a
system *is* compatible, unless there is specific reason to think otherwise.

We know of three main sources of potential incompatibility that are likely to
arise in practice:

* A linux distribution that is too old (e.g. RHEL 4)
* A linux distribution that does not use glibc (e.g. Alpine Linux, which is
  based on musl libc, or Android)
* Eventually, in the future, there may exist distributions that break
  compatibility with this profile

To handle the first two cases, we propose the following simple and reliable
check: ::

    def have_glibc_version(major, minimum_minor):
        import ctypes

        process_namespace = ctypes.CDLL(None)
        try:
            gnu_get_libc_version = process_namespace.gnu_get_libc_version
        except AttributeError:
            # We are not linked to glibc.
            return False

        gnu_get_libc_version.restype = ctypes.c_char_p
        version_str = gnu_get_libc_version()
        # py2 / py3 compatibility:
        if not isinstance(version_str, str):
            version_str = version_str.decode("ascii")

        version = [int(piece) for piece in version_str.split(".")]
        assert len(version) == 2
        if major != version[0]:
            return False
        if minimum_minor > version[1]:
            return False
        return True

    # CentOS 5 uses glibc 2.5.
    is_manylinux1_compatible = have_glibc_version(2, 5)

To handle the third case, we propose the creation of a file
``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
contents: ::

   [manylinux1]
   compatible = true

where the supported values for the ``manylinux1.compatible`` entry are the
same as those supported by the ConfigParser ``getboolean`` method.

The proposed logic for ``pip`` or related tools, then, is:

0) If ``distutils.util.get_platform()`` does not start with the string
   ``"linux"``, then assume the current system is not ``manylinux1``
   compatible.
1) If ``/etc/python/compatibility.conf`` exists and contains a ``manylinux1``
   key, then trust that.
2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
   current system can handle ``manylinux1`` wheels.
3) Otherwise, assume that the current system cannot handle ``manylinux1``
   wheels.


Security Implications
=====================

One of the advantages of dependencies on centralized libraries in Linux is
that bugfixes and security updates can be deployed system-wide, and
applications which depend on on these libraries will automatically feel the
effects of these patches when the underlying libraries are updated. This can
be particularly important for security updates in packages communication
across the network or cryptography.

``manylinux1`` wheels distributed through PyPI that bundle security-critical
libraries like OpenSSL will thus assume responsibility for prompt updates in
response disclosed vulnerabilities and patches. This closely parallels the
security implications of the distribution of binary wheels on Windows that,
because the platform lacks a system package manager, generally bundle their
dependencies. In particular, because its lacks a stable ABI, OpenSSL cannot be
included in the ``manylinux1`` profile.


Rejected Alternatives
=====================

One alternative would be to provide separate platform tags for each Linux
distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
``debian_jessie``, etc. Nothing in this proposal rules out the possibility of
adding such platform tags in the future, or of further extensions to wheel
metadata that would allow wheels to declare dependencies on external
system-installed packages. However, such extensions would require substantially
more work than this proposal, and still might not be appreciated by package
developers who would prefer not to have to maintain multiple build environments
and build multiple wheels in order to cover all the common Linux distributions.
Therefore we consider such proposals to be out-of-scope for this PEP.


References
==========

.. [1] PEP 425 -- Compatibility Tags for Built Distributions
   (https://www.python.org/dev/peps/pep-0425/)
.. [2] Enthought Canopy Python Distribution
   (https://store.enthought.com/downloads/)
.. [3] Continuum Analytics Anaconda Python Distribution
   (https://www.continuum.io/downloads)
.. [4] manylinux1 docker image
   (https://quay.io/repository/manylinux/manylinux)
.. [5] auditwheel
   (https://pypi.python.org/pypi/auditwheel)

Copyright
=========

This document has been placed into the public domain.

..

   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:


--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig

From donald at stufft.io  Fri Jan 22 13:10:55 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 22 Jan 2016 13:10:55 -0500
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
Message-ID: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>

PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
supposed to be:

    This option is the name of the tool used to invoke the installation.

However, nothing has really ever implemented it and it's gone largely ignored
until just recently pip 8.0 started writing the INSTALLER file into the
metadata directories with a value of "pip".

I'd like to propose adding a special cased value to add to the installer file
that will tell projects like pip that this particular installed thing is being
managed by someone else, and we should keep our hands off of it. According to
PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
since nobody has ever implemented it, we could expand that so that it so you
can also have a special value, of "dpkg (system)" or maybe that's not worth it
and we could just have "system" as a special value.

The benefit of doing this, is that with a special value in that file that says
"this file belongs to the OS", then pip could start looking for that file and
require a --force flag before it modifies any files belonging to that project.
Then distributors like Debian, Fedora, etc could simply write out the INSTALLER
file with the correct value, and pip would start to respect their files by
default.

Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
PEP or can we just amend PEP 376 to add this extra bit of information?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/a13318ba/attachment-0001.sig>

From p.f.moore at gmail.com  Fri Jan 22 13:22:41 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 22 Jan 2016 18:22:41 +0000
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
Message-ID: <CACac1F8NmaNx1O97QP_Z1Kxq81LuQ=i8D4DVXPctDZddiDDk6g@mail.gmail.com>

On 22 January 2016 at 18:10, Donald Stufft <donald at stufft.io> wrote:
> The benefit of doing this, is that with a special value in that file that says
> "this file belongs to the OS", then pip could start looking for that file and
> require a --force flag before it modifies any files belonging to that project.
> Then distributors like Debian, Fedora, etc could simply write out the INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.
>
> Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

Using "system" as the name of the "installer" sounds like it conforms
pretty well to PEP 376 as it stands.
Paul

From fred at fdrake.net  Fri Jan 22 13:28:15 2016
From: fred at fdrake.net (Fred Drake)
Date: Fri, 22 Jan 2016 13:28:15 -0500
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
Message-ID: <CAFT4OTHRrmC3Yi4Vk6vXmK2w57heYSuyP6uxYrcejty9M1+80g@mail.gmail.com>

On Fri, Jan 22, 2016 at 1:10 PM, Donald Stufft <donald at stufft.io> wrote:
> Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

Identifying something special like "system" doesn't seem bad, and
conforms (assuming PEP 376 really meant to include "+" at the end of
the regular expression).

If tools find the INSTALLER file and it identifies some other tool,
though, it shouldn't matter if there's a special "system" value.
Debian systems could use "dpkg" or "apt", RPM-based systems could use
"rpm" or "yum", and it sounds like the new pip would be just as happy
to do the right thing.

I don't think a new or updated PEP is needed.  I'm glad to see this
being addressed.  Thanks for all your hard work, Donald!


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From graffatcolmingov at gmail.com  Fri Jan 22 13:31:00 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Fri, 22 Jan 2016 12:31:00 -0600
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
Message-ID: <CAN-Kwu3sQU7cDQbKWtwD0ThJoBBAFcpxi_ZWDKAHaGy8BSB2Kw@mail.gmail.com>

On Fri, Jan 22, 2016 at 12:10 PM, Donald Stufft <donald at stufft.io> wrote:
> PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
> supposed to be:
>
>     This option is the name of the tool used to invoke the installation.
>
> However, nothing has really ever implemented it and it's gone largely ignored
> until just recently pip 8.0 started writing the INSTALLER file into the
> metadata directories with a value of "pip".
>
> I'd like to propose adding a special cased value to add to the installer file
> that will tell projects like pip that this particular installed thing is being
> managed by someone else, and we should keep our hands off of it. According to
> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
> since nobody has ever implemented it, we could expand that so that it so you
> can also have a special value, of "dpkg (system)" or maybe that's not worth it
> and we could just have "system" as a special value.

Why not use something like JSON to allow us to have a little more
information about the installer, e.g.,

{"name": "pip", "system": true, ...}

> The benefit of doing this, is that with a special value in that file that says
> "this file belongs to the OS", then pip could start looking for that file and
> require a --force flag before it modifies any files belonging to that project.
> Then distributors like Debian, Fedora, etc could simply write out the INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.
>
> Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

I think amending the PEP makes sense.

From njs at pobox.com  Fri Jan 22 13:46:34 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 22 Jan 2016 10:46:34 -0800
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
Message-ID: <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>

On Jan 22, 2016 10:11 AM, "Donald Stufft" <donald at stufft.io> wrote:
>
> PEP 376 added a file to the .dist-info directory called "INSTALLER" which
was
> supposed to be:
>
>     This option is the name of the tool used to invoke the installation.
>
> However, nothing has really ever implemented it and it's gone largely
ignored
> until just recently pip 8.0 started writing the INSTALLER file into the
> metadata directories with a value of "pip".
>
> I'd like to propose adding a special cased value to add to the installer
file
> that will tell projects like pip that this particular installed thing is
being
> managed by someone else, and we should keep our hands off of it.
According to
> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I
think
> since nobody has ever implemented it, we could expand that so that it so
you
> can also have a special value, of "dpkg (system)" or maybe that's not
worth it
> and we could just have "system" as a special value.

I think we want more than just "system", because the same user could have
some packages managed by dpkg and some by conda, both of which have their
own dependency resolution mechanisms that are outside pip's and could get
broken if pip removes stuff willy-nilly. And when pip errors out, you want
to be able to explain to the user "this package is managed by conda, and
using pip on it may break your conda setup..." versus "this package is
managed by Debian, and using pip on it may break your Debian setup...".

(Actually I'm not sure what the status these days is of mixing pip and
conda -- they've gotten somewhat better at handling it. Is the proposed
behavior in pip when it sees this flag something that distribution
maintainers have asked for? Are they present in this thread?)

> The benefit of doing this, is that with a special value in that file that
says
> "this file belongs to the OS", then pip could start looking for that file
and
> require a --force flag before it modifies any files belonging to that
project.
> Then distributors like Debian, Fedora, etc could simply write out the
INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.

I'd like a little more clarity on exactly what circumstances justify
setting this flag. If I write a new python package manager, then should I
set this flag on all my packages because I don't trust anyone else to get
things right? :-)

Maybe the relevant thing is what I said above, that there is some system
tracking these files that is not using the dist-info directory as its
source-of-truth about what's installed, dependencies, etc.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/45cb42b0/attachment.html>

From robertc at robertcollins.net  Fri Jan 22 14:09:47 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Sat, 23 Jan 2016 08:09:47 +1300
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
Message-ID: <CAJ3HoZ2H8-cE+UNftcoDX2MwqcSnNQQCp-Si+MxDHsZK3fi-9A@mail.gmail.com>

On 23 January 2016 at 07:10, Donald Stufft <donald at stufft.io> wrote:
> PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
> supposed to be:
>
>     This option is the name of the tool used to invoke the installation.
>
> However, nothing has really ever implemented it and it's gone largely ignored
> until just recently pip 8.0 started writing the INSTALLER file into the
> metadata directories with a value of "pip".
>
> I'd like to propose adding a special cased value to add to the installer file
> that will tell projects like pip that this particular installed thing is being
> managed by someone else, and we should keep our hands off of it. According to
> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
> since nobody has ever implemented it, we could expand that so that it so you
> can also have a special value, of "dpkg (system)" or maybe that's not worth it
> and we could just have "system" as a special value.
>
> The benefit of doing this, is that with a special value in that file that says
> "this file belongs to the OS", then pip could start looking for that file and
> require a --force flag before it modifies any files belonging to that project.
> Then distributors like Debian, Fedora, etc could simply write out the INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.
>
> Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

I think asking other distros to export the installing information to
us is a good thing.

I think requiring a force flag to replace files installed by another
packaging tool is an unbreakme option as things stand, and so I'm very
concerned about the resulting user experience.

If someone runs 'sudo pip install -U pip' they are already signalling
precisely what they want, and as a user I very much doubt that:

"Refusing to uninstall pip because it was not installed by pip"

will make even 5% of them pause longer than required to look up the
--force-alternative-installer flag (or whatever it is) and pass it in.

The only reason we're touching these other installer installed files
is because there is no location on the Python and exec paths where we
can install the new pip without overwriting the one dpkg (or conda or
whatever) installed. If there was such a place we could just write
(and upgrade things) there and not have to deal with shadowing, or
ever uninstalling dpkg etc installed files. In the absence of such a
place, removing the other-install-system's installed files is the
right thing to do when a user asks us to, without needing an
additional option.

We could add an option folk could set to turn *on* such a safety belt
- but I don't know who it would /really/ be helping. (I'm fairly sure
I know the places folk think it would help, I just don't agree that it
would :)).

Having a interface to the system package manager as has been
previously mooted - and I'm going to get back to that - might help a
little, but uninstalls are quite different to installs. Uninstalls get
blocked by things like 'app-foo requires requests', and I would be
super suprised (and delighted!) if we were able to override that when
upgrading requests via pip :)

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud

From donald at stufft.io  Fri Jan 22 14:28:52 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 22 Jan 2016 14:28:52 -0500
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
 <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>
Message-ID: <A915D6CD-9D80-46DC-BA4E-EDB2C10A2AAD@stufft.io>


> On Jan 22, 2016, at 1:46 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> On Jan 22, 2016 10:11 AM, "Donald Stufft" <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> >
> > PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
> > supposed to be:
> >
> >     This option is the name of the tool used to invoke the installation.
> >
> > However, nothing has really ever implemented it and it's gone largely ignored
> > until just recently pip 8.0 started writing the INSTALLER file into the
> > metadata directories with a value of "pip".
> >
> > I'd like to propose adding a special cased value to add to the installer file
> > that will tell projects like pip that this particular installed thing is being
> > managed by someone else, and we should keep our hands off of it. According to
> > PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
> > since nobody has ever implemented it, we could expand that so that it so you
> > can also have a special value, of "dpkg (system)" or maybe that's not worth it
> > and we could just have "system" as a special value.
> 
> I think we want more than just "system", because the same user could have some packages managed by dpkg and some by conda, both of which have their own dependency resolution mechanisms that are outside pip's and could get broken if pip removes stuff willy-nilly. And when pip errors out, you want to be able to explain to the user "this package is managed by conda, and using pip on it may break your conda setup..." versus "this package is managed by Debian, and using pip on it may break your Debian setup...".
> 
> (Actually I'm not sure what the status these days is of mixing pip and conda -- they've gotten somewhat better at handling it. Is the proposed behavior in pip when it sees this flag something that distribution maintainers have asked for? Are they present in this thread?)
> 
> 

Yea, that?s why I thought about dpkg (system) or system(Debian) or something. The main reason I can think of for preferring ?system? is if we don?t want to change the valid characters for a value in this file. Then you can have system(Debian) and system(Conda) and everything will work just fine.
> > The benefit of doing this, is that with a special value in that file that says
> > "this file belongs to the OS", then pip could start looking for that file and
> > require a --force flag before it modifies any files belonging to that project.
> > Then distributors like Debian, Fedora, etc could simply write out the INSTALLER
> > file with the correct value, and pip would start to respect their files by
> > default.
> 
> I'd like a little more clarity on exactly what circumstances justify setting this flag. If I write a new python package manager, then should I set this flag on all my packages because I don't trust anyone else to get things right? :-)
> 
> Maybe the relevant thing is what I said above, that there is some system tracking these files that is not using the dist-info directory as its source-of-truth about what's installed, dependencies, etc.
> 
> -n
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/3cd580b9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/3cd580b9/attachment.sig>

From donald at stufft.io  Fri Jan 22 14:41:21 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 22 Jan 2016 14:41:21 -0500
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <CAJ3HoZ2H8-cE+UNftcoDX2MwqcSnNQQCp-Si+MxDHsZK3fi-9A@mail.gmail.com>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
 <CAJ3HoZ2H8-cE+UNftcoDX2MwqcSnNQQCp-Si+MxDHsZK3fi-9A@mail.gmail.com>
Message-ID: <969B5D87-6272-4F41-9836-FAB76DD25D19@stufft.io>


> On Jan 22, 2016, at 2:09 PM, Robert Collins <robertc at robertcollins.net> wrote:
> 
> I think requiring a force flag to replace files installed by another
> packaging tool is an unbreakme option as things stand, and so I'm very
> concerned about the resulting user experience.

I don't think the current behavior makes *any* sense and represents nothing but
a footgun where you end up with two systems fighting over what the content of a
particular file is.

Currently there are two sort of scenarios where something like this comes up:

Debuntu
-------

Current behavior is we'll uninstall the file sitting at
/usr/lib/pythonX.Y/dist-packages/* and then we'll go ahead and install our own
files into /usr/local/lib/pythonX.Y/dist-packages/*. Then, the next time that
package gets an update the files are put back into place by dpkg into
/usr/lib/pythonX.Y/dist-packages/* and we're finally at a consistent state
where both package managers know the state of their respective systems.

In the "you need a flag" future world, we would just skip uninstalling the
files from /usr/lib/pythonX.Y/dist-packages/* and just install our own files
into /usr/local/lib/pythonX.Y/dist-packages. Thus we never touch anything that
isn't owned by us and Debian's internal systems stay consistent. If someone did
only ``pip uninstall requests`` and that was located in the OS files, then we
would refuse to uninstall it, directing them to the platform tooling instead.


Everyone Else
-------------

The /usr/lib and /usr/local/lib split is Debian specific (though at some point
I want to bring it to upstream Python) so on other systems you have a single
location, typically in /usr/lib/.../site-packages where both the OS and pip
will be trying to modify files. Right now this is pure chaos, if
python-requests is installed by the system, pip will come along and uninstall
it then upgrade requests, then the next time an update is released, the system
tooling will then come along and write all over the files that pip dropped
into place, possibly corrupting things depending on how their patching works.
Even for just ``pip uninstall foo``, it's really only uninstall until the
next time that package gets updated.

In the "you need a flag future", if python-requests is installed by the OS and
it's in the same location that *we* want to write a file to, then we'll error
out and require a --force flag. If it isn't already installed and nothing is
in our way, then we'll go ahead and install to that location. This isn't as
nice as the Debuntu world, but nothing will be without the patch that Debian
has.


Essentially though, ``pip install -U anything-installed-by-the-os`` is at best
a no-op that deletes files for no good reason (Debuntu) or it's a field full of
footguns where you're going to have two independent systems shitting all over
each others files trying to do what they think the user wants.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/f000044e/attachment.sig>

From rmcgibbo at gmail.com  Fri Jan 22 15:18:43 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Fri, 22 Jan 2016 12:18:43 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <DA2915A530468E47B3C4845298E6EB035E522B53@msgb06.nih.gov>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <DA2915A530468E47B3C4845298E6EB035E522B53@msgb06.nih.gov>
Message-ID: <CAN4+E8HxMVTdRH=xo_jjXtuVKr_jBZLiKMUUJ_wJj3eJdxRZdA@mail.gmail.com>

On Jan 22, 2016 9:04 AM, "Lowekamp, Bradley (NIH/NLM/LHC) [C]" <
blowekamp at mail.nih.gov> wrote:
>
> Hello,
>
> I noticed that libpython is missing from the lists of dependent
libraries. Also the ?manylinux? Docker image has it?s Python versions
compiled with libpython static.
>
> Does this mean that we must do static linking against libpython?

This is a bug/imprecision in the PEP. Manylinux1 wheels *can* link against
libpython (the appropriate version for whatever python they're targeting),
and the latest version of the docker image uses a shared libpython now.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/353fc74b/attachment-0001.html>

From p.f.moore at gmail.com  Fri Jan 22 15:38:47 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 22 Jan 2016 20:38:47 +0000
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
 <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>
Message-ID: <CACac1F-kH2zivN_8aWb-9X5UtVue3oa3LfP8LNmniqzaoTqd-A@mail.gmail.com>

On 22 January 2016 at 18:46, Nathaniel Smith <njs at pobox.com> wrote:
>> I'd like to propose adding a special cased value to add to the installer
>> file
>> that will tell projects like pip that this particular installed thing is
>> being
>> managed by someone else, and we should keep our hands off of it. According
>> to
>> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I
>> think
>> since nobody has ever implemented it, we could expand that so that it so
>> you
>> can also have a special value, of "dpkg (system)" or maybe that's not
>> worth it
>> and we could just have "system" as a special value.
>
> I think we want more than just "system", because the same user could have
> some packages managed by dpkg and some by conda, both of which have their
> own dependency resolution mechanisms that are outside pip's and could get
> broken if pip removes stuff willy-nilly. And when pip errors out, you want
> to be able to explain to the user "this package is managed by conda, and
> using pip on it may break your conda setup..." versus "this package is
> managed by Debian, and using pip on it may break your Debian setup...".

Well, I would expect conda to be specifying "conda" in the INSTALLER
file. That's what the file is intended for, surely? To record what
tool installed the package?

Equally, there seems to me to be no reason dpkg couldn't use "dpkg",
and yum use "yum", etc. I just see Donald's suggestion of using
"system" as a simple flag saying "the OS" in a generic way for
distributions that don't (for some reason) *want* to name the specific
installer in the INSTALLER file. Well, that and the fact that they
could have done that for years now, so being able to say "just put
"system" in there and be done with it please!" is a nice easy message
to give to distributions.

Honestly, this seems to me to be a bit of a non-discussion.

My recollection of PEP 376 and the discussions around INSTALLER are
that it was simply a place where install tools could say "hey, it was
me that did this". It never got used largely because nobody seemed to
care about having that information. Now pip is starting to care, but
as far as I can see there are only 3 cases pip should care about:

1. The file says "pip". Do just as we do now, we control that package.
2. The file says something *other* than pip. We don't own the files,
take appropriate action. Warn, require an extra flag, whatever - we
can use what's in the file to say "files owned by FOO" but other than
that there's no reason we should behave differently depending on what
specific non-pip value we see.
3. There's no INSTALLER file. This is the awkward one, as we can't
(yet) say that this means the file isn't owned by pip. In a few years
maybe, but for now we have to assume it could be either of the above
cases.

So the message should be "if you want pip to be careful with your
files, put something (probably your installer name, but in practice
anything other than "pip") in INSTALLER".

Longer term, other tools might taking the same approach as pip, at
which point not using the same name as other tools becomes useful - at
that point, a generic term like "system" could be a bad choice...
Paul

From donald at stufft.io  Fri Jan 22 15:45:59 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 22 Jan 2016 15:45:59 -0500
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <CACac1F-kH2zivN_8aWb-9X5UtVue3oa3LfP8LNmniqzaoTqd-A@mail.gmail.com>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
 <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>
 <CACac1F-kH2zivN_8aWb-9X5UtVue3oa3LfP8LNmniqzaoTqd-A@mail.gmail.com>
Message-ID: <94AB2EFE-FFC8-4D30-9691-28597D8D5E24@stufft.io>


> On Jan 22, 2016, at 3:38 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 22 January 2016 at 18:46, Nathaniel Smith <njs at pobox.com> wrote:
>>> I'd like to propose adding a special cased value to add to the installer
>>> file
>>> that will tell projects like pip that this particular installed thing is
>>> being
>>> managed by someone else, and we should keep our hands off of it. According
>>> to
>>> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I
>>> think
>>> since nobody has ever implemented it, we could expand that so that it so
>>> you
>>> can also have a special value, of "dpkg (system)" or maybe that's not
>>> worth it
>>> and we could just have "system" as a special value.
>> 
>> I think we want more than just "system", because the same user could have
>> some packages managed by dpkg and some by conda, both of which have their
>> own dependency resolution mechanisms that are outside pip's and could get
>> broken if pip removes stuff willy-nilly. And when pip errors out, you want
>> to be able to explain to the user "this package is managed by conda, and
>> using pip on it may break your conda setup..." versus "this package is
>> managed by Debian, and using pip on it may break your Debian setup...".
> 
> Well, I would expect conda to be specifying "conda" in the INSTALLER
> file. That's what the file is intended for, surely? To record what
> tool installed the package?
> 
> Equally, there seems to me to be no reason dpkg couldn't use "dpkg",
> and yum use "yum", etc. I just see Donald's suggestion of using
> "system" as a simple flag saying "the OS" in a generic way for
> distributions that don't (for some reason) *want* to name the specific
> installer in the INSTALLER file. Well, that and the fact that they
> could have done that for years now, so being able to say "just put
> "system" in there and be done with it please!" is a nice easy message
> to give to distributions.
> 
> Honestly, this seems to me to be a bit of a non-discussion.
> 
> My recollection of PEP 376 and the discussions around INSTALLER are
> that it was simply a place where install tools could say "hey, it was
> me that did this". It never got used largely because nobody seemed to
> care about having that information. Now pip is starting to care, but
> as far as I can see there are only 3 cases pip should care about:
> 
> 1. The file says "pip". Do just as we do now, we control that package.
> 2. The file says something *other* than pip. We don't own the files,
> take appropriate action. Warn, require an extra flag, whatever - we
> can use what's in the file to say "files owned by FOO" but other than
> that there's no reason we should behave differently depending on what
> specific non-pip value we see.
> 3. There's no INSTALLER file. This is the awkward one, as we can't
> (yet) say that this means the file isn't owned by pip. In a few years
> maybe, but for now we have to assume it could be either of the above
> cases.
> 
> So the message should be "if you want pip to be careful with your
> files, put something (probably your installer name, but in practice
> anything other than "pip") in INSTALLER".
> 
> Longer term, other tools might taking the same approach as pip, at
> which point not using the same name as other tools becomes useful - at
> that point, a generic term like "system" could be a bad choice...
> Paul

Hmm, in my head the three cases were more like:

1) The installed project is managed by a Python level tool which uses the Python metadata as it?s database. Thus if you?re this kind of tool too, then you can muck around here because things will be fine.
2) The installed project is managed by something that isn?t a Python level tool, and it has it?s own database and the .(egg|dist)-info files are not understood by it.
3) We don?t know what is managing the project.

The first would be things like, pip, easy_install, and distil. These can all easily interopt with each other because they?re all going to lay down .egg-info or .dist-info files and read those same files. The second is things like Conda, dpkg, yum, etc which are going to treat .egg-info or .dist-info files as just another opaque file that is included in it?s package that it has to lay down.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/21a47a54/attachment.sig>

From p.f.moore at gmail.com  Fri Jan 22 16:11:58 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 22 Jan 2016 21:11:58 +0000
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <94AB2EFE-FFC8-4D30-9691-28597D8D5E24@stufft.io>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
 <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>
 <CACac1F-kH2zivN_8aWb-9X5UtVue3oa3LfP8LNmniqzaoTqd-A@mail.gmail.com>
 <94AB2EFE-FFC8-4D30-9691-28597D8D5E24@stufft.io>
Message-ID: <CACac1F-3S5ugvyK97nA-Ym-w3Tr4HXseLyzP6nYBEn8JAbMRZg@mail.gmail.com>

On 22 January 2016 at 20:45, Donald Stufft <donald at stufft.io> wrote:
> Hmm, in my head the three cases were more like:
>
> 1) The installed project is managed by a Python level tool which uses the Python metadata as it?s database. Thus if you?re this kind of tool too, then you can muck around here because things will be fine.
> 2) The installed project is managed by something that isn?t a Python level tool, and it has it?s own database and the .(egg|dist)-info files are not understood by it.
> 3) We don?t know what is managing the project.
>
> The first would be things like, pip, easy_install, and distil. These can all easily interopt with each other because they?re all going to lay down .egg-info or .dist-info files and read those same files. The second is things like Conda, dpkg, yum, etc which are going to treat .egg-info or .dist-info files as just another opaque file that is included in it?s package that it has to lay down.

Good point - maybe that's how it *should* have been. My recollection
of the discussions at the time was "put your tool name in here" and
nobody really thought about the possibility of more than one tool
co-operating. This was in the bad old days of "my tool is better than
yours" flamewars :-(

Switching to something like that model probably does need a PEP. But
INSTALLER doesn't seem the right concept for that - it's more like
wanting a file that defines how the project metadata is stored -
"Metadata 2.0" or "OS" or "Conda" or whatever.

Paul

From njs at pobox.com  Fri Jan 22 17:17:12 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 22 Jan 2016 14:17:12 -0800
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <A915D6CD-9D80-46DC-BA4E-EDB2C10A2AAD@stufft.io>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
 <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>
 <A915D6CD-9D80-46DC-BA4E-EDB2C10A2AAD@stufft.io>
Message-ID: <CAPJVwB=5uO5j_4qhnSfwQPC8sF7dEJKQUQRZdbyhutNqeGmFqQ@mail.gmail.com>

On Fri, Jan 22, 2016 at 11:28 AM, Donald Stufft <donald at stufft.io> wrote:

>
> On Jan 22, 2016, at 1:46 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
> On Jan 22, 2016 10:11 AM, "Donald Stufft" <donald at stufft.io> wrote:
> >
> > PEP 376 added a file to the .dist-info directory called "INSTALLER"
> which was
> > supposed to be:
> >
> >     This option is the name of the tool used to invoke the installation.
> >
> > However, nothing has really ever implemented it and it's gone largely
> ignored
> > until just recently pip 8.0 started writing the INSTALLER file into the
> > metadata directories with a value of "pip".
> >
> > I'd like to propose adding a special cased value to add to the installer
> file
> > that will tell projects like pip that this particular installed thing is
> being
> > managed by someone else, and we should keep our hands off of it.
> According to
> > PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I
> think
> > since nobody has ever implemented it, we could expand that so that it so
> you
> > can also have a special value, of "dpkg (system)" or maybe that's not
> worth it
> > and we could just have "system" as a special value.
>
> I think we want more than just "system", because the same user could have
> some packages managed by dpkg and some by conda, both of which have their
> own dependency resolution mechanisms that are outside pip's and could get
> broken if pip removes stuff willy-nilly. And when pip errors out, you want
> to be able to explain to the user "this package is managed by conda, and
> using pip on it may break your conda setup..." versus "this package is
> managed by Debian, and using pip on it may break your Debian setup...".
>
> (Actually I'm not sure what the status these days is of mixing pip and
> conda -- they've gotten somewhat better at handling it. Is the proposed
> behavior in pip when it sees this flag something that distribution
> maintainers have asked for? Are they present in this thread?)
>
>
> Yea, that?s why I thought about dpkg (system) or system(Debian) or
> something. The main reason I can think of for preferring ?system? is if we
> don?t want to change the valid characters for a value in this file. Then
> you can have system(Debian) and system(Conda) and everything will work just
> fine.
>

Maybe to simplify the discussion we should just forget about INSTALLER,
leave it in peace to be what the PEP says ("fyi this is the last program to
touch this, in case anyone cares"), and add a new file with whatever
syntax/semantics make sense. Filenames are cheap and plentiful, and no
reason to complicate this discussion with hypothetical backcompat worries.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org <http://vorpus.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160122/3ba52add/attachment-0001.html>

From fungi at yuggoth.org  Fri Jan 22 19:45:28 2016
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Sat, 23 Jan 2016 00:45:28 +0000
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <CAJ3HoZ2H8-cE+UNftcoDX2MwqcSnNQQCp-Si+MxDHsZK3fi-9A@mail.gmail.com>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
 <CAJ3HoZ2H8-cE+UNftcoDX2MwqcSnNQQCp-Si+MxDHsZK3fi-9A@mail.gmail.com>
Message-ID: <20160123004527.GH2343@yuggoth.org>

On 2016-01-23 08:09:47 +1300 (+1300), Robert Collins wrote:
[...]
> Having a interface to the system package manager as has been
> previously mooted - and I'm going to get back to that - might help a
> little, but uninstalls are quite different to installs. Uninstalls get
> blocked by things like 'app-foo requires requests', and I would be
> super suprised (and delighted!) if we were able to override that when
> upgrading requests via pip :)

Add to this the fact that whatever system-distributed files pip
removes/replaces will get quietly overwritten again the next time
you update your system packages and there's a new version of
whatever you've replaced... perhaps being able to set a hold state
through the distro package manager from pip at that point for
whatever package owned the files it replaced would be a helpful
middle ground (I know how I'd go about it on Debian derivatives,
less so on all those "other" distros).
-- 
Jeremy Stanley

From ncoghlan at gmail.com  Fri Jan 22 22:26:51 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Jan 2016 13:26:51 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <56A21B7D.7010008@egenix.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <4FBA4669-A5D9-4B65-8F2D-CE835A016660@stufft.io>
 <56A21B7D.7010008@egenix.com>
Message-ID: <CADiSq7fvqng6DBFz1D3j6iF9TvKMZp-QVYBxtH_Zd67qrA3AeQ@mail.gmail.com>

On 22 January 2016 at 22:07, M.-A. Lemburg <mal at egenix.com> wrote:
> However, system vendors will often be a lot faster with updates
> than package authors, simply because it's their business model,
> so as user you will want to benefit from those updates and
> not have to rely on the package author to ship new wheel files.

This is true for the subset of packages monitored by distro security
response teams, but there's a *lot* of software not currently packaged
for Linux distros that never will be as more attention is given to the
"rebuild the world on demand" model that elastic cloud computing and
fast internet connections enable.

My fundamental concern is that if a package author publishes a distro
dependent wheel file, pip attempts to install it, and it doesn't work,
the reaction for many users is going to be "Python packaging is
broken", not "the packaging of this particular package is broken".

However, moving the "generic linux wheels are ignored by default"
behaviour to pip-the-client, rather than enforcing it as a restriction
on PyPI uploads could definitely be a reasonable alternative way of
addressing that concern.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri Jan 22 22:52:57 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Jan 2016 13:52:57 +1000
Subject: [Distutils] PEP 376, the INSTALLER file, and system packages
In-Reply-To: <CAPJVwB=5uO5j_4qhnSfwQPC8sF7dEJKQUQRZdbyhutNqeGmFqQ@mail.gmail.com>
References: <D199ECE2-20FA-4C21-A536-8E34542D0144@stufft.io>
 <CAPJVwB=O98x9+cDVE6zM15+w9rkrGgUO1nUg2kQEU5xEJS8GKg@mail.gmail.com>
 <A915D6CD-9D80-46DC-BA4E-EDB2C10A2AAD@stufft.io>
 <CAPJVwB=5uO5j_4qhnSfwQPC8sF7dEJKQUQRZdbyhutNqeGmFqQ@mail.gmail.com>
Message-ID: <CADiSq7dh3XPTs8WrQq7Rn_EMLdWdX8FJdBLtKmg5eCajut1_Vg@mail.gmail.com>

On 23 January 2016 at 08:17, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Jan 22, 2016 at 11:28 AM, Donald Stufft <donald at stufft.io> wrote:
>> Yea, that?s why I thought about dpkg (system) or system(Debian) or
>> something. The main reason I can think of for preferring ?system? is if we
>> don?t want to change the valid characters for a value in this file. Then you
>> can have system(Debian) and system(Conda) and everything will work just
>> fine.
>
> Maybe to simplify the discussion we should just forget about INSTALLER,
> leave it in peace to be what the PEP says ("fyi this is the last program to
> touch this, in case anyone cares"), and add a new file with whatever
> syntax/semantics make sense. Filenames are cheap and plentiful, and no
> reason to complicate this discussion with hypothetical backcompat worries.

Right, that was my theory in implementing INSTALLER just as PEP 376
defined it - having pip write "pip" to that file is enough to let us
determine:

1. Controlled in a pip-compatible way
2. Controlled in a pip-incompatible way
3. pip compatibility not specified

Paul's right that at the time the assumption was each tool would just
write *its own* name into the file, but given the other changes that
have happened since then, it now makes sense that other tools that are
fully interoperable with another more-widely used installer may choose
to write that installer's name instead of their own, and the only
states we *really* care about from a UX perspective are "definitely
compatible", "definitely incompatible" and "don't know".

We may decide to do something more sophisticated later, but that would
be as part of someone finding the roundtuits to actually design a
plugin system for delegating to system package managers when running
as root.

>From a downstream perspective, the main thing we need to know in order
to choose a suitable value to put into that file is how pip decides to
use whatever we write there. For example, if pip were to emit a
warning message looking something like:

"<project> is not managed by pip, skipping (use '<INSTALLER content>'
or '--force' to update it)"

Then the distro could update our packaging policies such that we write
our respective package management command names into the INSTALLER
files when creating system packages.

(As far as the regex defining the permitted content goes,
appropriately caveating PEP 376 to better match the reality of how
current generation tools work is one of my main motivations for
revising the specification process)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mal at egenix.com  Sat Jan 23 05:44:36 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 23 Jan 2016 11:44:36 +0100
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7fvqng6DBFz1D3j6iF9TvKMZp-QVYBxtH_Zd67qrA3AeQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <4FBA4669-A5D9-4B65-8F2D-CE835A016660@stufft.io>
 <56A21B7D.7010008@egenix.com>
 <CADiSq7fvqng6DBFz1D3j6iF9TvKMZp-QVYBxtH_Zd67qrA3AeQ@mail.gmail.com>
Message-ID: <56A35994.8050801@egenix.com>

On 23.01.2016 04:26, Nick Coghlan wrote:
> On 22 January 2016 at 22:07, M.-A. Lemburg <mal at egenix.com> wrote:
>> However, system vendors will often be a lot faster with updates
>> than package authors, simply because it's their business model,
>> so as user you will want to benefit from those updates and
>> not have to rely on the package author to ship new wheel files.
> 
> This is true for the subset of packages monitored by distro security
> response teams, but there's a *lot* of software not currently packaged
> for Linux distros that never will be as more attention is given to the
> "rebuild the world on demand" model that elastic cloud computing and
> fast internet connections enable.
> 
> My fundamental concern is that if a package author publishes a distro
> dependent wheel file, pip attempts to install it, and it doesn't work,
> the reaction for many users is going to be "Python packaging is
> broken", not "the packaging of this particular package is broken".

I think helping the user to identify where the problem
originates is certainly possible.

This can be done in a generic way by e.g. having pip or the wheel
package scan the wheel file for shared libraries using ldd and
finding potentially missing libs.

Or we could define a special package post install entry point which
pip calls to have the wheel itself check the system it was installed
on for missing system packages or any other post install actions
that need to take place before the wheel file can be used, e.g.
setting up initial config files.

> However, moving the "generic linux wheels are ignored by default"
> behaviour to pip-the-client, rather than enforcing it as a restriction
> on PyPI uploads could definitely be a reasonable alternative way of
> addressing that concern.

I don't think that's the right strategy. There are certainly ways
to improve error reporting for Python packaging (see above), but
outright rejecting generic wheels is not a good approach, IMO.

The wheel system is not yet complete, but until it is, using
a "we want to protect the user from failing wheels" approach
is not going to help much, since we're just replacing this
with a "we'll let the user handle failing source installations"
approach instead - with the main reason apparently being that
we want to avoid putting the user blame for failures on PyPI,
pip or wheels.

This ignores that fact that generic wheels have a much
better chance of success than source installations of the same
package on the same system (it's rather unlikely that the user
will have the foo-dev package installed with the corresponding
foo binary package).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 23 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From vano at mail.mipt.ru  Sat Jan 23 16:21:27 2016
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Sun, 24 Jan 2016 01:21:27 +0400
Subject: [Distutils] Source code for wrapper EXEs
Message-ID: <56A3EED7.40106@mail.mipt.ru>

Where can I find sibj for cli*.exe/gui*.exe? In the published source 
code 
(https://pypi.python.org/packages/source/s/setuptools/setuptools-19.4.tar.gz#md5=c5a7d90c1e0acf8c4ec5c2bf31bc25b5) 
and even the BitBucket repo, they are already in compiled form.

--
Regards,
Ivan

From contact at ionelmc.ro  Sat Jan 23 18:32:47 2016
From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=)
Date: Sun, 24 Jan 2016 01:32:47 +0200
Subject: [Distutils] Source code for wrapper EXEs
In-Reply-To: <56A3EED7.40106@mail.mipt.ru>
References: <56A3EED7.40106@mail.mipt.ru>
Message-ID: <CANkHFr-rBmsveZA8MO9fmKiJb6Pr2OKGeJ1o3d9PPrH1dtQd4Q@mail.gmail.com>

These are the sources:
https://bitbucket.org/pypa/setuptools/src/tip/launcher.c


Thanks,
-- Ionel Cristian M?rie?, http://blog.ionelmc.ro

On Sat, Jan 23, 2016 at 11:21 PM, Ivan Pozdeev via Distutils-SIG <
distutils-sig at python.org> wrote:

> Where can I find sibj for cli*.exe/gui*.exe? In the published source code (
> https://pypi.python.org/packages/source/s/setuptools/setuptools-19.4.tar.gz#md5=c5a7d90c1e0acf8c4ec5c2bf31bc25b5)
> and even the BitBucket repo, they are already in compiled form.
>
> --
> Regards,
> Ivan
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160124/3e05b0fc/attachment.html>

From chris.barker at noaa.gov  Sat Jan 23 21:19:12 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Sat, 23 Jan 2016 18:19:12 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
Message-ID: <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>

OK,

I'll try to stop being emotional here :-)

2016-01-22 3:47 GMT+01:00 Chris Barker - NOAA Federal <chris.barker at noaa.gov
> >:
> >
> >  I'm skeptical because I
> > tried to to that for years for OS-X and it was just too much to do. And
> the
> > infrastructure was there.
>


> My point is that once we have clearly defined best-practices for
> packaging and convenient tools to build the packages automatically and
> test that they work as expected (e.g. free hosted CI that support
> running an old centos-based docker container), I am rather confident
> that the community will do the work.
>

OK -- here is the emotional part -- I worked for years to try to get
support to:

"clearly defined best-practices for packaging and convenient tools to build
the packages automatically"

Primarily for OS-X. I got zero support -- nada -- nothing. Really. A
handful of people did their own thing to support the community, but no
cooperation of standards -- each package built was done with a lot of hand
work and each in its own way. So when i found the conda community working
on common tools and methods, it was very refreshing.

But OK -- maybe times have changed.

By the way, I'm in the middle of some build hell with conda -- it's doing
some really weird stuff with finding shared libs provided by other conda
packages -- so maybe I'm open to looking at wheels again :-)

But my concern is not the base libs -- that will get Linux on par with
Windows and OS-X. My concern is with third party libs, what's the plan
there?

1)  each package that needs a third partly lib statically links it in.
2)  each package that needs a third partly lib provides it, linked with a
realtive path (IIUC, that's how most Windows packages are done.
3) We establish some standard for providing binary libs as wheels, so that
other packages can depend on them and link to them.

1) is a pain int he %^# with gcc and linux, which really likes to
dynamically link

2) seems to have been made pretty easy with auditwheel -- nice!

3) seems like the "proper" way to go. somehow making everyone figure out
how to build and shop those deps, and then bloating the wheels and
installations and binaries feels wrong to me.

We've been using the example of, say, libpng, in this discussion --that's a
good example, because it's pretty commonly used, but not part of all base
distributions. but it's also pretty easy to build and pretty small.

So let's look at a couple examples I've dealt with:

The netcdf / hdf5 stack -- there is a pynetcd4 package, which requires the
c netcdf lib, which requires libhdf5, which requires libcurl, zlib, (anda
few others, I think) non-trivial to build and ship. Also, there are at
least two other commonly used python packages I know of that use hdf5:
pytables and h5py.

So it would be really nice if all these python packages didn't need to
solve the build issues and ship all these libs themselves, resulting in
possibly incompatible version all loaded into python all at once. Oh, and
as i happens, I've got my obscure python package that uses a bunch of C
code that also needs libnetcdf....

Then there is the OpenGIS stack: there is the geos lib and proj4 lib, that
most others things are built on. There is the GDAL/OGR lib, that requires
those, plus (optionally), a lot of other ugly libs (this is ugly to build,
believe me). And GDAL comes with its own python bindings, but there is also
shapely, that wraps geos, and pyproj4 that wraps proj4, and fiona that
wraps OGR, and .... A big interconnected web. [oh, fiona and shapely also
required numpy at the binary level...)

I can only imagine that the image processing or video or audio processing
communities have similar piles of interconnected packages. (I know I've
tried, unsuccessfully, to get FFMPEG to work...)

So all this requires, I think, (3) to get anywhere -- is the community
ready to support such an effort? And this going to requires some more
tooling, too.

Somewhere on this thread, someone suggested there may be a videolinuxapi,
or some such. Perhaps the better way to to have a core base (such as
manylinux), and then a handful of binary lib collections as a single wheel:

osgeowheel
hdf-wheel
audio-wheel
image-wheel

hmm -- kind of liking that idea, actually.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160123/8fbdf7f7/attachment.html>

From rmcgibbo at gmail.com  Sat Jan 23 21:31:50 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Sat, 23 Jan 2016 18:31:50 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
Message-ID: <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>

On Sat, Jan 23, 2016 at 6:19 PM, Chris Barker <chris.barker at noaa.gov> wrote:

>
> 1)  each package that needs a third partly lib statically links it in.
> 2)  each package that needs a third partly lib provides it, linked with a
> relative path (IIUC, that's how most Windows packages are done.
> 3) We establish some standard for providing binary libs as wheels, so that
> other packages can depend on them and link to them.
>

In my view, *all* of these are valid options. I think much of this will
need to be worked out by the communities -- especially if individual
packages and subcommunities decide to take the option (3) approach. I hope
this PEP will enable the communities involved in OpenGIS, audio processing,
image processing, etc to work out the solutions that work for them and
their users.

Perhaps one thing that is missing from the PEP is an explicit statement
that option (3) is compatible with the manylinux1 tag -- bundling is a
valid solution, but it's not the *only* solution.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160123/7bfd35a4/attachment-0001.html>

From p.f.moore at gmail.com  Sun Jan 24 03:22:27 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 24 Jan 2016 08:22:27 +0000
Subject: [Distutils] Source code for wrapper EXEs
In-Reply-To: <CANkHFr-rBmsveZA8MO9fmKiJb6Pr2OKGeJ1o3d9PPrH1dtQd4Q@mail.gmail.com>
References: <56A3EED7.40106@mail.mipt.ru>
 <CANkHFr-rBmsveZA8MO9fmKiJb6Pr2OKGeJ1o3d9PPrH1dtQd4Q@mail.gmail.com>
Message-ID: <CACac1F_4ymm88CW6Hnc8eytMaRjhidn_L+6cYcCRCm5bcYkkYQ@mail.gmail.com>

On 23 January 2016 at 23:32, Ionel Cristian M?rie? <contact at ionelmc.ro> wrote:
> These are the sources:
> https://bitbucket.org/pypa/setuptools/src/tip/launcher.c

And the distil launchers (used by pip for wheel installs) are at
https://bitbucket.org/vinay.sajip/simple_launcher

Paul

From ncoghlan at gmail.com  Sun Jan 24 07:08:47 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Jan 2016 22:08:47 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
Message-ID: <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>

On 24 January 2016 at 12:31, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> On Sat, Jan 23, 2016 at 6:19 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>>
>> 1)  each package that needs a third partly lib statically links it in.
>> 2)  each package that needs a third partly lib provides it, linked with a
>> relative path (IIUC, that's how most Windows packages are done.
>> 3) We establish some standard for providing binary libs as wheels, so that
>> other packages can depend on them and link to them.
>
> In my view, all of these are valid options. I think much of this will need
> to be worked out by the communities -- especially if individual packages and
> subcommunities decide to take the option (3) approach. I hope this PEP will
> enable the communities involved in OpenGIS, audio processing, image
> processing, etc to work out the solutions that work for them and their
> users.
>
> Perhaps one thing that is missing from the PEP is an explicit statement that
> option (3) is compatible with the manylinux1 tag -- bundling is a valid
> solution, but it's not the *only* solution.

I've long resisted the notion of defining our own cross-distro
platform ABI, but the Docker build environment that was put together
for the manylinux project has made me realise that doing that may not
be as hellish in a post-Docker world as it would have been in a
pre-Docker world. (Since we can go with the specification + reference
implementation approach that CPython has used so successfully for so
long, rather than having to have the build environment and ABI
specification be entirely exhaustive).

On Windows and Mac OS X, our binary compatibility policies for wheel
files are actually pretty loose - it's "be binary compatible with the
python.org builds for that platform, including linking against the
appropriate C standard library", and that's about it. Upgrades to
those ABIs are then driven by CPython switching to newer base
compatibility levels (dropping end-of-life versions on the Windows
side [1], and updating to new deployment target macros on the Mac OS X
side). Folks with external dependencies either bundle them, skip
publishing wheel files, or just let them fail at import time if the
external dependency is missing. (Neither platform has an anti-bundling
culture, though, so I assume a lot of folks go with the first option
over the last one)

If the aim is to bring Linux wheel support in line with Windows and
Mac OS X, then rather than defining a *new* compatibility tag (which
would require new pip clients to process), perhaps we could instead
adopt a similarly loose policy on what the existing generic "linux"
tag means as we have for Windows and Mac OS X: it could just mean
wheel files that are binary compatible with the Python binaries in the
"manylinux" build environment. The difference would then just be that
the target Linux ABI would be defined by PyPA and the manylinux
developers, rather than by python-dev.

In terms of the concerns regarding the age of gcc needed to target
CentOS 5.11, it would be good to know just what nominating CentOS 6.x
as the baseline ABI instead would buy us - CentOS 5 is going on 9
years old (released 2007) and stopped receiving full updates back in
2014 [2], while RHEL/CentOS 6 is just over 5 years old and has another
year of full updates left. The CentOS 6 ABI should still be old enough
to be compatible with the Debian 6 ABI (current stable is Debian 8),
as well as the Ubuntu 12.04 LTS ABI (Ubuntu 16.04 LTS is due out in a
few months).

Cheers,
Nick.

[1] https://www.python.org/dev/peps/pep-0011/#microsoft-windows
[2] https://wiki.centos.org/About/Product

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Sun Jan 24 11:39:58 2016
From: donald at stufft.io (Donald Stufft)
Date: Sun, 24 Jan 2016 11:39:58 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
Message-ID: <A34523E3-B716-4E2C-9DA5-577AD9DA7E90@stufft.io>


> On Jan 24, 2016, at 7:08 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 24 January 2016 at 12:31, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
>> On Sat, Jan 23, 2016 at 6:19 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>>> 
>>> 1)  each package that needs a third partly lib statically links it in.
>>> 2)  each package that needs a third partly lib provides it, linked with a
>>> relative path (IIUC, that's how most Windows packages are done.
>>> 3) We establish some standard for providing binary libs as wheels, so that
>>> other packages can depend on them and link to them.
>> 
>> In my view, all of these are valid options. I think much of this will need
>> to be worked out by the communities -- especially if individual packages and
>> subcommunities decide to take the option (3) approach. I hope this PEP will
>> enable the communities involved in OpenGIS, audio processing, image
>> processing, etc to work out the solutions that work for them and their
>> users.
>> 
>> Perhaps one thing that is missing from the PEP is an explicit statement that
>> option (3) is compatible with the manylinux1 tag -- bundling is a valid
>> solution, but it's not the *only* solution.
> 
> I've long resisted the notion of defining our own cross-distro
> platform ABI, but the Docker build environment that was put together
> for the manylinux project has made me realise that doing that may not
> be as hellish in a post-Docker world as it would have been in a
> pre-Docker world. (Since we can go with the specification + reference
> implementation approach that CPython has used so successfully for so
> long, rather than having to have the build environment and ABI
> specification be entirely exhaustive).
> 
> On Windows and Mac OS X, our binary compatibility policies for wheel
> files are actually pretty loose - it's "be binary compatible with the
> python.org builds for that platform, including linking against the
> appropriate C standard library", and that's about it. Upgrades to
> those ABIs are then driven by CPython switching to newer base
> compatibility levels (dropping end-of-life versions on the Windows
> side [1], and updating to new deployment target macros on the Mac OS X
> side). Folks with external dependencies either bundle them, skip
> publishing wheel files, or just let them fail at import time if the
> external dependency is missing. (Neither platform has an anti-bundling
> culture, though, so I assume a lot of folks go with the first option
> over the last one)
> 
> If the aim is to bring Linux wheel support in line with Windows and
> Mac OS X, then rather than defining a *new* compatibility tag (which
> would require new pip clients to process), perhaps we could instead
> adopt a similarly loose policy on what the existing generic "linux"
> tag means as we have for Windows and Mac OS X: it could just mean
> wheel files that are binary compatible with the Python binaries in the
> "manylinux" build environment. The difference would then just be that
> the target Linux ABI would be defined by PyPA and the manylinux
> developers, rather than by python-dev.
> 
> In terms of the concerns regarding the age of gcc needed to target
> CentOS 5.11, it would be good to know just what nominating CentOS 6.x
> as the baseline ABI instead would buy us - CentOS 5 is going on 9
> years old (released 2007) and stopped receiving full updates back in
> 2014 [2], while RHEL/CentOS 6 is just over 5 years old and has another
> year of full updates left. The CentOS 6 ABI should still be old enough
> to be compatible with the Debian 6 ABI (current stable is Debian 8),
> as well as the Ubuntu 12.04 LTS ABI (Ubuntu 16.04 LTS is due out in a
> few months).
> 

It's probably not worth it to try and reuse the existing linux tags. Prior to
the release of pip 8, pip had a hardcoded check to not trust linux wheels from
PyPI, even if one was allowed to be uploaded, because we weren't sure if some
change was going to be required in pip to support whatever solution we come up
with for linux wheels and we didn't want a bunch of old pip's to suddenly start
getting wheels they didn't know how to install. After thinking about it more, I
decided that installing wheels are pretty simple and it's unlikely pip is going
to have to change anything to support that particular tag, so in pip 8 I
removed that hack. So it's been less than a week that a version of pip has
existed that would install the generic linux platform anyways.

There is a similar sort of project to try and make it easy to build cross
distro compatible C/C++ things, called the "Holy Build Box" [1] so that
certainly lends some extra weight behind the whole idea.

Another thing to consider is applicablity outside of Python. When I was talking
about the manylinux thing with some friends, one of them mentioned that they
were interested in the same thing, but for the Rust language. If we define the
manylinux platform, it may make sense to promote it to something that isn't so
Python specific and define two things, the manylinux platform (possibly we can
buy some domain name for it or something), and then a PEP that just integrates
that platform with Python. Might not be worth it though.

[1] https://phusion.github.io/holy-build-box/

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160124/44b20a9d/attachment.sig>

From njs at pobox.com  Sun Jan 24 17:32:01 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 24 Jan 2016 14:32:01 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
Message-ID: <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>

On Sun, Jan 24, 2016 at 4:08 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 24 January 2016 at 12:31, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
>> On Sat, Jan 23, 2016 at 6:19 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>>>
>>> 1)  each package that needs a third partly lib statically links it in.
>>> 2)  each package that needs a third partly lib provides it, linked with a
>>> relative path (IIUC, that's how most Windows packages are done.
>>> 3) We establish some standard for providing binary libs as wheels, so that
>>> other packages can depend on them and link to them.
>>
>> In my view, all of these are valid options. I think much of this will need
>> to be worked out by the communities -- especially if individual packages and
>> subcommunities decide to take the option (3) approach. I hope this PEP will
>> enable the communities involved in OpenGIS, audio processing, image
>> processing, etc to work out the solutions that work for them and their
>> users.
>>
>> Perhaps one thing that is missing from the PEP is an explicit statement that
>> option (3) is compatible with the manylinux1 tag -- bundling is a valid
>> solution, but it's not the *only* solution.
>
> I've long resisted the notion of defining our own cross-distro
> platform ABI, but the Docker build environment that was put together
> for the manylinux project has made me realise that doing that may not
> be as hellish in a post-Docker world as it would have been in a
> pre-Docker world. (Since we can go with the specification + reference
> implementation approach that CPython has used so successfully for so
> long, rather than having to have the build environment and ABI
> specification be entirely exhaustive).
>
> On Windows and Mac OS X, our binary compatibility policies for wheel
> files are actually pretty loose - it's "be binary compatible with the
> python.org builds for that platform, including linking against the
> appropriate C standard library", and that's about it. Upgrades to
> those ABIs are then driven by CPython switching to newer base
> compatibility levels (dropping end-of-life versions on the Windows
> side [1], and updating to new deployment target macros on the Mac OS X
> side). Folks with external dependencies either bundle them, skip
> publishing wheel files, or just let them fail at import time if the
> external dependency is missing. (Neither platform has an anti-bundling
> culture, though, so I assume a lot of folks go with the first option
> over the last one)
>
> If the aim is to bring Linux wheel support in line with Windows and
> Mac OS X, then rather than defining a *new* compatibility tag (which
> would require new pip clients to process), perhaps we could instead
> adopt a similarly loose policy on what the existing generic "linux"
> tag means as we have for Windows and Mac OS X: it could just mean
> wheel files that are binary compatible with the Python binaries in the
> "manylinux" build environment. The difference would then just be that
> the target Linux ABI would be defined by PyPA and the manylinux
> developers, rather than by python-dev.

It's an option I guess, though Donald's message below makes it rather
less attractive :-). The other thing is that as compared to Windows or
OS X, it requires much more attention to actually meet the target
Linux ABI -- on Windows or OS X an out-of-the-box build for a simple
project will more-often-than-not legitimately meet the ABI, and if you
can make a package that also works on your office-mate's computer then
it will probably work everywhere. On Linux, the way glibc versioning
works means that just doing the obvious 'pip wheel' call will
basically never give you a wheel that meets the ABI, and testing on
your office-mate's computer proves nothing (except that you're both
running Ubuntu 15.10 or whatever). Also, there's a huge quantity of
existing linux-tagged wheels out there that definitely don't meet the
ABI.

> In terms of the concerns regarding the age of gcc needed to target
> CentOS 5.11, it would be good to know just what nominating CentOS 6.x
> as the baseline ABI instead would buy us - CentOS 5 is going on 9
> years old (released 2007) and stopped receiving full updates back in
> 2014 [2], while RHEL/CentOS 6 is just over 5 years old and has another
> year of full updates left. The CentOS 6 ABI should still be old enough
> to be compatible with the Debian 6 ABI (current stable is Debian 8),
> as well as the Ubuntu 12.04 LTS ABI (Ubuntu 16.04 LTS is due out in a
> few months).

AFAICT everyone I've found publishing info on distributing generic
Linux binaries is currently using CentOS 5 as their target -- not just
manylinux1, but also the Holy Build Box / "travelling ruby" folks,
Firefox (not sure exactly what they're using but it seems to be <=
CentOS 5), etc. I guess bumping up to CentOS 6 would be trivial enough
-- just keep the same library list and bump up the minimum version
requirements for glibc / libgcc / libstdc++ -- but I think we'd be
pioneers here, and that's something we might not want to be at the
same time that we're first dipping our toes into the water :-).

GCC 4.8 was released in 2013; it's not actually terribly old. It has
decent C++11 support, and it's sufficient to compile things like LLVM
and Qt and Firefox. (Compare to Windows, where anyone building py27
wheels gets to use MSVC 2008, which doesn't even know C99.) So I'd be
inclined to stick with CentOS 5 for now, and gather some experience
while waiting to see how far it can go before it breaks.

The one thing that does give me pause is that whenever we *do* decide
to switch to manylinux2, then it's going to be a big drag to wait for
a whole pip release/upgrade cycle -- Debian unstable is still shipping
pip 1.5.6 (released May 2014) :-(. And when it comes to wheel
compatibility tags and pip upgrades, the UX is really terrible: if pip
is too old to recognize the provided .wheels, then it doesn't tell the
user "hey, you should upgrade me" or otherwise provide some hint that
there might be a trivial solution to this problem; instead it just
silently downloads the source and attempts to build it (and quite
often blows up after 30 pegging the CPU for 30 minutes or something).

I guess one way to square this circle would be for pip to have some
logic that checks for manylinux[0-9]+ platform tags, and if it sees a
wheel like this with a platform tag that post-dates its own release,
AND the only other option is to build from source, then it tells the
user "hey, there's an *excellent* chance that there's a new pip that
could give you a wheel right now -- what do you want me to do?". Or we
could even make it fail-open rather than fail-closed, like:

If pip knows about manylinux 1..n, then given wheels for manylinux (n
-1), n, and (n+1), then it should have the preference ordering:
  n > (n - 1) > (n + 1)
i.e., for known platform tags we prefer newer platform tags to older
ones; for unknown platform tags from the future, we optimistically
assume that they'll probably work (since the whole idea of the
manylinux tags is that they will work almost everywhere), but we
prefer known tags to unknown tags, so that we only install the
manylinux(n+1) wheel if nothing else is available. (And print some
message saying what we're doing.)

...well, or maybe just erroring out when it sees the future and asking
the user to help would be good enough :-). This would impose the
requirement going forward that we'd have to wait for a pip release
with support for manylinuxN before allowing manylinuxN onto PyPI, but
that doesn't seem too onerous.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From donald at stufft.io  Sun Jan 24 17:56:33 2016
From: donald at stufft.io (Donald Stufft)
Date: Sun, 24 Jan 2016 17:56:33 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
Message-ID: <68CE5246-592B-4812-AA82-3DCD2A470D0F@stufft.io>


> On Jan 24, 2016, at 5:32 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> The one thing that does give me pause is that whenever we *do* decide
> to switch to manylinux2, then it's going to be a big drag to wait for
> a whole pip release/upgrade cycle -- Debian unstable is still shipping
> pip 1.5.6 (released May 2014) :-(. And when it comes to wheel
> compatibility tags and pip upgrades, the UX is really terrible: if pip
> is too old to recognize the provided .wheels, then it doesn't tell the
> user "hey, you should upgrade me" or otherwise provide some hint that
> there might be a trivial solution to this problem; instead it just
> silently downloads the source and attempts to build it (and quite
> often blows up after 30 pegging the CPU for 30 minutes or something).

Ever since 6.0 pip now implicitly warns users on every invocation if
there is a newer version of pip available on PyPI (assuming they are
not in some constrained environment that cannot access pypi.python.og).

I fully expect Debian to upgrade to pip 8.0 in the near future in sid,
it obviously want make it back to Jesse though except via back ports. This
is mostly an outlier I think, pip 6+ really ramped up the number of things
that pip bundles, plus with it getting bundled with pip it created a bit
of a contentious situation with Debian. I believe that the excellent Barry
Warsaw is close to having that ironed out and pip 8.0 ready to land, and
future upgrades should be pretty painless.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160124/a523feef/attachment.sig>

From glyph at twistedmatrix.com  Sun Jan 24 18:06:59 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Sun, 24 Jan 2016 15:06:59 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <68CE5246-592B-4812-AA82-3DCD2A470D0F@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <68CE5246-592B-4812-AA82-3DCD2A470D0F@stufft.io>
Message-ID: <EEC6604C-1AC2-4DF8-81A2-02E75A97BA0B@twistedmatrix.com>


> On Jan 24, 2016, at 2:56 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> ... the excellent Barry Warsaw ...

Distro work is often thankless (especially here) so I just wanted to echo this: glory to the FLUFL, may his reign be <> short.

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160124/19c39543/attachment.html>

From njs at pobox.com  Sun Jan 24 19:21:01 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 24 Jan 2016 16:21:01 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
Message-ID: <CAPJVwBnxChZeXnixePQ0ftCPjVGLStpErCD6E60UnBDzM8BstA@mail.gmail.com>

On Sat, Jan 23, 2016 at 6:19 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> OK,
>
> I'll try to stop being emotional here :-)
>
>> 2016-01-22 3:47 GMT+01:00 Chris Barker - NOAA Federal
>> <chris.barker at noaa.gov>:
>> >
>> >  I'm skeptical because I
>> > tried to to that for years for OS-X and it was just too much to do. And
>> > the
>> > infrastructure was there.
>
>
>>
>> My point is that once we have clearly defined best-practices for
>> packaging and convenient tools to build the packages automatically and
>> test that they work as expected (e.g. free hosted CI that support
>> running an old centos-based docker container), I am rather confident
>> that the community will do the work.
>
>
> OK -- here is the emotional part -- I worked for years to try to get support
> to:
>
> "clearly defined best-practices for packaging and convenient tools to build
> the packages automatically"
>
> Primarily for OS-X. I got zero support -- nada -- nothing. Really. A handful
> of people did their own thing to support the community, but no cooperation
> of standards -- each package built was done with a lot of hand work and each
> in its own way. So when i found the conda community working on common tools
> and methods, it was very refreshing.

I hear that :-/. But it feels like there is some momentum building
these days -- before the only people who cared about the scientific
stack + wheels were people doing Mac development, so there are nice
Mac docs like:
   https://github.com/MacPython/wiki/wiki/Wheel-building
and nothing more general. But now we've got Linux and Windows both
ramping up, to the point that I'm having trouble juggling stuff across
the different mailing lists and trying to keep people in the loop.
So... obviously the solution is to create yet another mailing list
:-).

Maybe we need wheel-builders-sig? Their mandate would be to hash out
things like how to build binary-libraries-wrapped-up-in-wheels, share
knowledge about the minutiae of linker behavior on different platforms
(oh god there's so much minutiae), maintain tools like delocate and
auditwheel (and whatever the equivalent will be for windows... and do
we really need 3 different tools?), collect knowledge from where it's
scattered now and put it into the guide at packaging.python.org [1],
etc.? It seems a bit outside distutils-sig's focus in practice, since
this would all be about third-party tools and individual package
authors as opposed to distutils-sig's focus on writing
interoperability PEPs and maintaining the core python.org-affiliated
infrastructure like PyPI / setuptools / pip.

-n

[1] currently the "official guide" to building binary wheels is this
:-): https://packaging.python.org/en/latest/extensions/#publishing-binary-extensions

-- 
Nathaniel J. Smith -- https://vorpus.org

From donald at stufft.io  Sun Jan 24 19:23:33 2016
From: donald at stufft.io (Donald Stufft)
Date: Sun, 24 Jan 2016 19:23:33 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBnxChZeXnixePQ0ftCPjVGLStpErCD6E60UnBDzM8BstA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAPJVwBnxChZeXnixePQ0ftCPjVGLStpErCD6E60UnBDzM8BstA@mail.gmail.com>
Message-ID: <22E08F8F-A24A-4AD4-A7D3-E699774A751D@stufft.io>


> On Jan 24, 2016, at 7:21 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> Maybe we need wheel-builders-sig? Their mandate would be to hash out
> things like how to build binary-libraries-wrapped-up-in-wheels, share
> knowledge about the minutiae of linker behavior on different platforms
> (oh god there's so much minutiae), maintain tools like delocate and
> auditwheel (and whatever the equivalent will be for windows... and do
> we really need 3 different tools?), collect knowledge from where it's
> scattered now and put it into the guide at packaging.python.org [1],
> etc.? It seems a bit outside distutils-sig's focus in practice, since
> this would all be about third-party tools and individual package
> authors as opposed to distutils-sig's focus on writing
> interoperability PEPs and maintaining the core python.org-affiliated
> infrastructure like PyPI / setuptools / pip.

I?m not about to tell someone they *have* to use distutils-sig, but I
think at this point it?s perfectly reasonable to discuss things of
this nature here as well. distutils-sig would really be more accurate
if it was named packaging-sig, but historical reasons ;)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160124/5a10b417/attachment.sig>

From ncoghlan at gmail.com  Sun Jan 24 21:04:53 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 25 Jan 2016 12:04:53 +1000
Subject: [Distutils] Changing the way we use the PEP process (redux)
In-Reply-To: <CADiSq7dTAW3VDn2G2q0HJmHQW65EVqQU42kG137qHtM8dcjWQQ@mail.gmail.com>
References: <CADiSq7dTAW3VDn2G2q0HJmHQW65EVqQU42kG137qHtM8dcjWQQ@mail.gmail.com>
Message-ID: <CADiSq7fPtrQ3nmO5SC0uwgOuf_N_5Po9uJr6W8_Q=A8WC1choQ@mail.gmail.com>

On 22 January 2016 at 16:58, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I've posted about this idea to the list before, but this time I've
> finally started working on it and have a concrete plan to discuss :)
>
> The basic idea:
>
> * I want to progressively move the active interoperability
> specifications out of PEPs and into a subsection of
> packaging.python.org
> * packaging PEPs would then become a tool for changing those
> specifications, rather than the specifications themselves
> * the description of this process would be captured along with the
> rest of the PyPA operational docs at pypa.io
> * there's a draft PR to start down this path at
> https://github.com/pypa/pypa.io/pull/12
>
> That PR provides an example of the potential benefits of this approach
> - it's able to state which parts of PEP 345 have been superseded by
> other PEPs, and also note the "Provides-Extra" field which exists as a
> de facto standard, despite not being formally adopted through the PEP
> process.
>
> However, having written the draft PR entirely against pypa.io, I've
> now gone back to thinking packaging.python.org would be a better fit
> for the actual hosting of the agreed specifications - the "python.org"
> domain is a vastly better known one than "pypa.io", and being on a
> subdomain of python.org more clearly establishes these
> interoperability specifications as the Python Packaging counterparts
> to the Python Language Reference and Python Library Reference.
>
> So my next iteration will be split into two PRs: one for pypa.io
> defining the specification management process, and one for
> packaging.python.org adding a specifications section

I implemented the split into two PRs:

Process PR: https://github.com/pypa/pypa.io/pull/14
Specifications PR: https://github.com/pypa/python-packaging-user-guide/pull/213

Absent any strong outcry, I'll merge these tomorrow, and we can
continue to tweak the spec maintenance process from there.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Sun Jan 24 22:30:57 2016
From: donald at stufft.io (Donald Stufft)
Date: Sun, 24 Jan 2016 22:30:57 -0500
Subject: [Distutils] Changing the way we use the PEP process (redux)
In-Reply-To: <CADiSq7fPtrQ3nmO5SC0uwgOuf_N_5Po9uJr6W8_Q=A8WC1choQ@mail.gmail.com>
References: <CADiSq7dTAW3VDn2G2q0HJmHQW65EVqQU42kG137qHtM8dcjWQQ@mail.gmail.com>
 <CADiSq7fPtrQ3nmO5SC0uwgOuf_N_5Po9uJr6W8_Q=A8WC1choQ@mail.gmail.com>
Message-ID: <B4D1F6AD-EA2A-44DA-A668-6604DB7A5778@stufft.io>

Will we start moving the actual specifications into packaging.python.org, or will they stay in the PEP repository? I?m not sure I can tell from your two PRs currently.

> On Jan 24, 2016, at 9:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 22 January 2016 at 16:58, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> I've posted about this idea to the list before, but this time I've
>> finally started working on it and have a concrete plan to discuss :)
>> 
>> The basic idea:
>> 
>> * I want to progressively move the active interoperability
>> specifications out of PEPs and into a subsection of
>> packaging.python.org
>> * packaging PEPs would then become a tool for changing those
>> specifications, rather than the specifications themselves
>> * the description of this process would be captured along with the
>> rest of the PyPA operational docs at pypa.io
>> * there's a draft PR to start down this path at
>> https://github.com/pypa/pypa.io/pull/12
>> 
>> That PR provides an example of the potential benefits of this approach
>> - it's able to state which parts of PEP 345 have been superseded by
>> other PEPs, and also note the "Provides-Extra" field which exists as a
>> de facto standard, despite not being formally adopted through the PEP
>> process.
>> 
>> However, having written the draft PR entirely against pypa.io, I've
>> now gone back to thinking packaging.python.org would be a better fit
>> for the actual hosting of the agreed specifications - the "python.org"
>> domain is a vastly better known one than "pypa.io", and being on a
>> subdomain of python.org more clearly establishes these
>> interoperability specifications as the Python Packaging counterparts
>> to the Python Language Reference and Python Library Reference.
>> 
>> So my next iteration will be split into two PRs: one for pypa.io
>> defining the specification management process, and one for
>> packaging.python.org adding a specifications section
> 
> I implemented the split into two PRs:
> 
> Process PR: https://github.com/pypa/pypa.io/pull/14
> Specifications PR: https://github.com/pypa/python-packaging-user-guide/pull/213
> 
> Absent any strong outcry, I'll merge these tomorrow, and we can
> continue to tweak the spec maintenance process from there.
> 
> Cheers,
> Nick.
> 
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160124/096e1dc2/attachment.sig>

From ncoghlan at gmail.com  Sun Jan 24 23:17:03 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 25 Jan 2016 14:17:03 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <22E08F8F-A24A-4AD4-A7D3-E699774A751D@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAPJVwBnxChZeXnixePQ0ftCPjVGLStpErCD6E60UnBDzM8BstA@mail.gmail.com>
 <22E08F8F-A24A-4AD4-A7D3-E699774A751D@stufft.io>
Message-ID: <CADiSq7fD4ZLPc6wCwrRgFFs9BKs9sdXfeL+39PWKV1nYe1_iiA@mail.gmail.com>

On 25 January 2016 at 10:23, Donald Stufft <donald at stufft.io> wrote:
>
>> On Jan 24, 2016, at 7:21 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Maybe we need wheel-builders-sig? Their mandate would be to hash out
>> things like how to build binary-libraries-wrapped-up-in-wheels, share
>> knowledge about the minutiae of linker behavior on different platforms
>> (oh god there's so much minutiae), maintain tools like delocate and
>> auditwheel (and whatever the equivalent will be for windows... and do
>> we really need 3 different tools?), collect knowledge from where it's
>> scattered now and put it into the guide at packaging.python.org [1],
>> etc.? It seems a bit outside distutils-sig's focus in practice, since
>> this would all be about third-party tools and individual package
>> authors as opposed to distutils-sig's focus on writing
>> interoperability PEPs and maintaining the core python.org-affiliated
>> infrastructure like PyPI / setuptools / pip.
>
> I?m not about to tell someone they *have* to use distutils-sig, but I
> think at this point it?s perfectly reasonable to discuss things of
> this nature here as well.

I agree with this - while I wouldn't call any aspect of software
distribution easy, binary compatibility is one of the hardest, and we
want *more* knowledge sharing in that regard, rather than continue to
maintain the divide between "folks who care about binary
compatibility" and "folks who wish everyone would just stop creating
extension modules already because pure Python modules are so much
easier to deal with" :)

Keeping the discussions centralised also creates more opportunities
for serendipitous collaboration, where folks notice that they can
potentially help out with what someone else is working on.

> distutils-sig would really be more accurate
> if it was named packaging-sig, but historical reasons ;)

I kind of look forward to a distant future where we *do* decide to
rename it, as that will mean all of the more pressing concerns have
already been knocked off the todo list :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jan 24 23:37:58 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 25 Jan 2016 14:37:58 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
Message-ID: <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>

On 25 January 2016 at 08:32, Nathaniel Smith <njs at pobox.com> wrote:
> On Sun, Jan 24, 2016 at 4:08 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> If the aim is to bring Linux wheel support in line with Windows and
>> Mac OS X, then rather than defining a *new* compatibility tag (which
>> would require new pip clients to process), perhaps we could instead
>> adopt a similarly loose policy on what the existing generic "linux"
>> tag means as we have for Windows and Mac OS X: it could just mean
>> wheel files that are binary compatible with the Python binaries in the
>> "manylinux" build environment. The difference would then just be that
>> the target Linux ABI would be defined by PyPA and the manylinux
>> developers, rather than by python-dev.
>
> It's an option I guess, though Donald's message below makes it rather
> less attractive :-).

Yeah, I didn't know about the client side block in older versions of
pip, so we may as well stick with the custom tag rather than trying to
use the existing one.

> The other thing is that as compared to Windows or
> OS X, it requires much more attention to actually meet the target
> Linux ABI -- on Windows or OS X an out-of-the-box build for a simple
> project will more-often-than-not legitimately meet the ABI, and if you
> can make a package that also works on your office-mate's computer then
> it will probably work everywhere. On Linux, the way glibc versioning
> works means that just doing the obvious 'pip wheel' call will
> basically never give you a wheel that meets the ABI, and testing on
> your office-mate's computer proves nothing (except that you're both
> running Ubuntu 15.10 or whatever).

That does raise a question for the PEP: should it be proposing changes
to the default behaviour of "pip wheel" and "bdist_wheel" on Linux?

Even if the answer is "No", then the PEP should probably explain why not.

>> In terms of the concerns regarding the age of gcc needed to target
>> CentOS 5.11, it would be good to know just what nominating CentOS 6.x
>> as the baseline ABI instead would buy us - CentOS 5 is going on 9
>> years old (released 2007) and stopped receiving full updates back in
>> 2014 [2], while RHEL/CentOS 6 is just over 5 years old and has another
>> year of full updates left. The CentOS 6 ABI should still be old enough
>> to be compatible with the Debian 6 ABI (current stable is Debian 8),
>> as well as the Ubuntu 12.04 LTS ABI (Ubuntu 16.04 LTS is due out in a
>> few months).
>
> AFAICT everyone I've found publishing info on distributing generic
> Linux binaries is currently using CentOS 5 as their target -- not just
> manylinux1, but also the Holy Build Box / "travelling ruby" folks,
> Firefox (not sure exactly what they're using but it seems to be <=
> CentOS 5), etc. I guess bumping up to CentOS 6 would be trivial enough
> -- just keep the same library list and bump up the minimum version
> requirements for glibc / libgcc / libstdc++ -- but I think we'd be
> pioneers here, and that's something we might not want to be at the
> same time that we're first dipping our toes into the water :-).
>
> GCC 4.8 was released in 2013; it's not actually terribly old. It has
> decent C++11 support, and it's sufficient to compile things like LLVM
> and Qt and Firefox. (Compare to Windows, where anyone building py27
> wheels gets to use MSVC 2008, which doesn't even know C99.) So I'd be
> inclined to stick with CentOS 5 for now, and gather some experience
> while waiting to see how far it can go before it breaks.

That argument makes sense to me, so this is possibly worth another
note as a "deferred to a later manylinux update" topic, but otherwise
doesn't affect the PEP.

> The one thing that does give me pause is that whenever we *do* decide
> to switch to manylinux2, then it's going to be a big drag to wait for
> a whole pip release/upgrade cycle -- Debian unstable is still shipping
> pip 1.5.6 (released May 2014) :-(. And when it comes to wheel
> compatibility tags and pip upgrades, the UX is really terrible: if pip
> is too old to recognize the provided .wheels, then it doesn't tell the
> user "hey, you should upgrade me" or otherwise provide some hint that
> there might be a trivial solution to this problem; instead it just
> silently downloads the source and attempts to build it (and quite
> often blows up after 30 pegging the CPU for 30 minutes or something).
>
> I guess one way to square this circle would be for pip to have some
> logic that checks for manylinux[0-9]+ platform tags, and if it sees a
> wheel like this with a platform tag that post-dates its own release,
> AND the only other option is to build from source, then it tells the
> user "hey, there's an *excellent* chance that there's a new pip that
> could give you a wheel right now -- what do you want me to do?".

As Donald noted, pip already emits a "Your pip is old" warning at
startup when new versions are available from PyPI.

> Or we
> could even make it fail-open rather than fail-closed, like:
>
> If pip knows about manylinux 1..n, then given wheels for manylinux (n
> -1), n, and (n+1), then it should have the preference ordering:
>   n > (n - 1) > (n + 1)
> i.e., for known platform tags we prefer newer platform tags to older
> ones; for unknown platform tags from the future, we optimistically
> assume that they'll probably work (since the whole idea of the
> manylinux tags is that they will work almost everywhere), but we
> prefer known tags to unknown tags, so that we only install the
> manylinux(n+1) wheel if nothing else is available. (And print some
> message saying what we're doing.)
>
> ...well, or maybe just erroring out when it sees the future and asking
> the user to help would be good enough :-). This would impose the
> requirement going forward that we'd have to wait for a pip release
> with support for manylinuxN before allowing manylinuxN onto PyPI, but
> that doesn't seem too onerous.

For Windows and Mac OS X, dropping support for old platforms is
effectively coupled to the CPython release cycle - a Python 3.5 wheel
is less likely to work on Windows XP than a 3.3 one, for example,
since 3.5 doesn't support XP, while 3.3 does.

We could potentially consider something similar for manylinux: scoping
the evolution of the definition by corresponding CPython release,
rather than giving it its own version number.

So the first release of manylinux would define the ABI for Linux
wheels for 3.5 and earlier (including the 2.x series), and then 3.6
would be the next chance to consider revising it (e.g. by bumping up
the base ABI from CentOS 5 to CentOS 6).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jan 24 23:54:21 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 25 Jan 2016 14:54:21 +1000
Subject: [Distutils] Changing the way we use the PEP process (redux)
In-Reply-To: <B4D1F6AD-EA2A-44DA-A668-6604DB7A5778@stufft.io>
References: <CADiSq7dTAW3VDn2G2q0HJmHQW65EVqQU42kG137qHtM8dcjWQQ@mail.gmail.com>
 <CADiSq7fPtrQ3nmO5SC0uwgOuf_N_5Po9uJr6W8_Q=A8WC1choQ@mail.gmail.com>
 <B4D1F6AD-EA2A-44DA-A668-6604DB7A5778@stufft.io>
Message-ID: <CADiSq7es0tR_hijUFhOKxq_+8CVRETzUg92qKUKopVH3SP5W9Q@mail.gmail.com>

On 25 January 2016 at 13:30, Donald Stufft <donald at stufft.io> wrote:
> Will we start moving the actual specifications into packaging.python.org, or will they stay in the PEP repository? I?m not sure I can tell from your two PRs currently.

The initial PRs are aimed more at documenting the status quo, so the
current crop of specifications won't move, but may gain
packaging.python.org annotations explaining how reality differs from
the nominal description in the PEP (I already did this for the PEP 345
metadata spec, and to a lesser degree for the PEP 376 installation
database spec). I figure that's useful even if we *don't* change the
process, so it's a good place to start.

The next step would then be to create a new PR against pypa.io
proposing a change to the spec handling process to make it so that as
the various specs get updated they become self-contained reference
documents on packaging.python.org, with the PEPs being more focused on
making the case for the changes relative to the previous revision (or
for the addition of a new specification to the specification set when
that's what's happening).

I'm still not entirely sure what that PR is going to look like though,
which is why I decided to start with making the status quo easier to
follow, rather than jumping straight to the proposed change. (I may
also still use a PEP 376 update to trial run the process, since that
has a lot of "not actually implemented" bits and pieces that obscure
the main aspects of the dist-info directory and the RECORD metadata
file)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rmcgibbo at gmail.com  Mon Jan 25 16:51:03 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Mon, 25 Jan 2016 13:51:03 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
Message-ID: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>

Hi,

This PEP is an updated version of the draft manylinux1 PEP posted to this
list a couple days
ago by Nathaniel and myself. The changes reflect the discussion on the list
(thanks to everyone
for all of the feedback), and generally go to the clarity and precision of
the text.

HTML version: https://github.com/manylinux/manylinux/blob/master/pep-513.rst

-Robert

----

PEP: 513
Title: A Platform Tag for Portable Linux Built Distributions
Version: $Revision$
Last-Modified: $Date$
Author: Robert T. McGibbon <rmcgibbo at gmail.com>, Nathaniel J. Smith
<njs at pobox.com>
BDFL-Delegate: Nick Coghlan <ncoghlan at gmail.com>
Discussions-To: Distutils SIG <distutils-sig at python.org>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 19-Jan-2016
Post-History: 19-Jan-2016, 25-Jan-2016


Abstract
========

This PEP proposes the creation of a new platform tag for Python package built
distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
external dependencies limited to a standardized, restricted subset of
the Linux kernel and core userspace ABI. It proposes that PyPI support
uploading and distributing wheels with this platform tag, and that ``pip``
support downloading and installing these packages on compatible platforms.


Rationale
=========

Currently, distribution of binary Python extensions for Windows and OS X is
straightforward. Developers and packagers build wheels [1]_ [2]_, which are
assigned platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload
these wheels to PyPI. Users can download and install these wheels using tools
such as ``pip``.

For Linux, the situation is much more delicate. In general, compiled Python
extension modules built on one Linux distribution will not work on other Linux
distributions, or even on different machines running the same Linux
distribution with different system libraries installed.

Build tools using PEP 425 platform tags [3]_ do not track information about the
particular Linux distribution or installed system libraries, and instead assign
all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
this ambiguity, there is no expectation that ``linux``-tagged built
distributions compiled on one machine will work properly on another, and for
this reason, PyPI has not permitted the uploading of wheels for Linux.

It would be ideal if wheel packages could be compiled that would work on *any*
linux system. But, because of the incredible diversity of Linux systems -- from
PCs to Android to embedded systems with custom libcs -- this cannot
be guaranteed in general.

Instead, we define a standard subset of the kernel+core userspace ABI that,
in practice, is compatible enough that packages conforming to this standard
will work on *many* linux systems, including essentially all of the desktop
and server distributions in common use. We know this because there are
companies who have been distributing such widely-portable pre-compiled Python
extension modules for Linux -- e.g. Enthought with Canopy [4]_ and Continuum
Analytics with Anaconda [5]_.

Building on the compability lessons learned from these companies, we thus
define a baseline ``manylinux1`` platform tag for use by binary Python
wheels, and introduce the implementation of preliminary tools to aid in the
construction of these ``manylinux1`` wheels.


Key Causes of Inter-Linux Binary Incompatibility
================================================

To properly define a standard that will guarantee that wheel packages meeting
this specification will operate on *many* linux platforms, it is necessary to
understand the root causes which often prevent portability of pre-compiled
binaries on Linux. The two key causes are dependencies on shared libraries
which are not present on users' systems, and dependencies on particular
versions of certain core libraries like ``glibc``.


External Shared Libraries
-------------------------

Most desktop and server linux distributions come with a system package manager
(examples include ``APT`` on Debian-based systems, ``yum`` on
``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
responsibilities, the installation of shared libraries installed to system
directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
on one or more of these shared libraries, and thus function properly only on
systems where the user has the proper libraries (and the proper
versions thereof), either installed using their package manager, or installed
manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
to notify the runtime linker of the location of the depended-upon shared
libraries.


Versioning of Core Shared Libraries
-----------------------------------

Even if the developers a Python extension module wish to use no
external shared libraries, the modules will generally have a dynamic runtime
dependency on the GNU C library, ``glibc``. While it is possible, statically
linking ``glibc`` is usually a bad idea because certain important C functions
like ``dlopen()`` cannot be called from code that statically links ``glibc``. A
runtime shared library dependency on a system-provided ``glibc`` is unavoidable
in practice.

The maintainers of the GNU C library follow a strict symbol versioning scheme
for backward compatibility. This ensures that binaries compiled against an older
version of ``glibc`` can run on systems that have a newer ``glibc``. The
opposite is generally not true -- binaries compiled on newer Linux
distributions tend to rely upon versioned functions in ``glibc`` that are not
available on older systems.

This generally prevents wheels compiled on the latest Linux distributions
from being portable.


The ``manylinux1`` policy
=========================

For these reasons, to achieve broad portability, Python wheels

* should depend only on an extremely limited set of external shared
  libraries; and
* should depend only on "old" symbol versions in those external shared
  libraries; and
* should depend only on a widely-compatible kernel ABI.

To be eligible for the ``manylinux1`` platform tag, a Python wheel must
therefore both (a) contain binary executables and compiled code that links
*only* to libraries (other than the appropriate ``libpython`` library, which is
always a permitted dependency consistent with the PEP 425 ABI tag) with SONAMEs
included in the following list: ::

    libpanelw.so.5
    libncursesw.so.5
    libgcc_s.so.1
    libstdc++.so.6
    libm.so.6
    libdl.so.2
    librt.so.1
    libcrypt.so.1
    libc.so.6
    libnsl.so.1
    libutil.so.1
    libpthread.so.0
    libX11.so.6
    libXext.so.6
    libXrender.so.1
    libICE.so.6
    libSM.so.6
    libGL.so.1
    libgobject-2.0.so.0
    libgthread-2.0.so.0
    libglib-2.0.so.0

and (b), work on a stock CentOS 5.11 [6]_ system that contains the system
package manager's provided versions of these libraries.

Because CentOS 5 is only available for x86_64 and i386 architectures,
these are the only architectures currently supported by the ``manylinux1``
policy.

On Debian-based systems, these libraries are provided by the packages ::

    libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
    libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0

On RPM-based systems, these libraries are provided by the packages ::

    ncurses libgcc libstdc++ glibc libXext libXrender
    libICE libSM mesa-libGL glib2

This list was compiled by checking the external shared library dependencies of
the Canopy [4]_ and Anaconda [5]_ distributions, which both include a wide array
of the most popular Python modules and have been confirmed in practice to work
across a wide swath of Linux systems in the wild.

Many of the permitted system libraries listed above use symbol versioning
schemes for backward compatibility. The latest symbol versions provided with
the CentOS 5.11 versions of these libraries are: ::

    GLIBC_2.5
    CXXABI_3.4.8
    GLIBCXX_3.4.9
    GCC_4.2.0

Therefore, as a consequence of requirement (b), any wheel that depends on
versioned symbols from the above shared libraries may depend only on symbols
with the following versions: ::

    GLIBC <= 2.5
    CXXABI <= 3.4.8
    GLIBCXX <= 3.4.9
    GCC <= 4.2.0

These recommendations are the outcome of the relevant discussions in January
2016 [7]_, [8]_.

Note that in our recommendations below, we do not suggest that ``pip``
or PyPI should attempt to check for and enforce the details of this
policy (just as they don't check for and enforce the details of
existing platform tags like ``win32``). The text above is provided (a)
as advice to package builders, and (b) as a method for allocating
blame if a given wheel doesn't work on some system: if it satisfies
the policy above, then this is a bug in the spec or the installation
tool; if it does not satisfy the policy above, then it's a bug in the
wheel. One useful consequence of this approach is that it leaves open
the possibility of further updates and tweaks as we gain more
experience, e.g., we could have a "manylinux 1.1" policy which targets
the same systems and uses the same ``manylinux1`` platform tag (and
thus requires no further changes to ``pip`` or PyPI), but that adjusts
the list above to remove libraries that have turned out to be
problematic or add libraries that have turned out to be safe.


Compilation of Compliant Wheels
===============================

The way glibc, libgcc, and libstdc++ manage their symbol versioning
means that in practice, the compiler toolchains that most developers
use to do their daily work are incapable of building
``manylinux1``-compliant wheels. Therefore we do not attempt to change
the default behavior of ``pip wheel`` / ``bdist_wheel``: they will
continue to generate regular ``linux_*`` platform tags, and developers
who wish to use them to generate ``manylinux1``-tagged wheels will
have to change the tag as a second post-processing step.

To support the compilation of wheels meeting the ``manylinux1`` standard, we
provide initial drafts of two tools.


Docker Image
------------

The first tool is a Docker image based on CentOS 5.11, which is recommended as
an easy to use self-contained build box for compiling ``manylinux1`` wheels
[9]_. Compiling on a more recently-released linux distribution will generally
introduce dependencies on too-new versioned symbols. The image comes with a
full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
well as the latest releases of Python and ``pip``.

Auditwheel
----------

The second tools is a command line executable called ``auditwheel`` [10]_ that
may aid in package maintainers in dealing with third-party external
dependencies.

There are at least three methods for building wheels that use third-party
external libraries in a way that meets the above policy.

1. The third-party libraries can be statically linked.
2. The third-party shared libraries can be distributed in
   separate packages on PyPI which are depended upon by the wheel.
3. The third-party shared libraries can be bundled inside the wheel
   libraries, linked with a relative path.

All of these are valid option which may be effectively used by different
packages and communities. Statically linking generally requires
package-specific modifications to the build system, and distributing
third-party dependencies on PyPI may require some coordination of the
community of users of the package.

As an often-automatic alternative to these options, we introduce ``auditwheel``.
The tool inspects all of the ELF files  inside a wheel to check for
dependencies on versioned symbols or external  shared libraries, and verifies
conformance with the ``manylinux1`` policy. This  includes the ability to add
the new platform tag to conforming wheels. More importantly, ``auditwheel`` has
the ability to automatically modify wheels that depend on external shared
libraries by copying those shared libraries from the system into the wheel
itself, and modifying the appropriate ``RPATH`` entries such that these
libraries will be picked up at runtime. This accomplishes a similar result as
if the libraries had been statically linked without requiring changes to the
build system. Packagers are advised that bundling, like static linking, may
implicate copyright concerns.


Bundled Wheels on Linux
=======================

While we acknowledge many approaches for dealing with third-party library
dependencies within ``manylinux1`` wheels, we recognize that the ``manylinux1``
policy encourages bundling external dependencies, a practice
which runs counter to  the package management policies of many linux
distributions' system package  managers [11]_, [12]_. The primary purpose of
this is cross-distro compatibility.  Furthermore, ``manylinux1`` wheels on PyPI
occupy a different  niche than the Python packages available through the
system package manager.

The decision in this PEP to encourage departure from general Linux distribution
unbundling policies is informed by the following concerns:

1. In these days of automated continuous integration and deployment
   pipelines, publishing new versions and updating dependencies is easier
   than it was when those policies were defined.
2. ``pip`` users remain free to use the ``"--no-binary"`` option if they want
   to force local builds rather than using pre-built wheel files.
3. The popularity of modern container based deployment and "immutable
   infrastructure" models involve substantial bundling at the application
   layer anyway.
4. Distribution of bundled wheels through PyPI is currently the norm for
   Windows and OS X.
5. This PEP doesn't rule out the idea of offering more targeted binaries for
   particular Linux distributions in the future.

The model described in this PEP is most ideally suited for cross-platform
Python packages, because it means they can reuse much of the
work that they're already doing to make static Windows and OS X wheels. We
recognize that it is less optimal for Linux-specific packages that might
prefer to interact more closely with Linux's unique package management
functionality and only care about targeting a small set of particular distos.


Security Implications
---------------------

One of the advantages of dependencies on centralized libraries in Linux is
that bugfixes and security updates can be deployed system-wide, and
applications which depend on these libraries will automatically feel the
effects of these patches when the underlying libraries are updated. This can
be particularly important for security updates in packages engaged in
communication across the network or cryptography.

``manylinux1`` wheels distributed through PyPI that bundle security-critical
libraries like OpenSSL will thus assume responsibility for prompt updates in
response disclosed vulnerabilities and patches. This closely parallels the
security implications of the distribution of binary wheels on Windows that,
because the platform lacks a system package manager, generally bundle their
dependencies. In particular, because it lacks a stable ABI, OpenSSL cannot be
included in the ``manylinux1`` profile.



Platform Detection for Installers
=================================

Above, we defined what it means for a *wheel* to be
``manylinux1``-compatible. Here we discuss what it means for a *Python
installation* to be ``manylinux1``-compatible. In particular, this is
important for tools like ``pip`` to know when deciding whether or not
they should consider ``manylinux1``-tagged wheels for installation.

Because the ``manylinux1`` profile is already known to work for the
many thousands of users of popular commercial Python distributions, we
suggest that installation tools should error on the side of assuming
that a system *is* compatible, unless there is specific reason to
think otherwise.

We know of three main sources of potential incompatibility that are likely to
arise in practice:

* A linux distribution that is too old (e.g. RHEL 4)
* A linux distribution that does not use ``glibc`` (e.g. Alpine Linux, which is
  based on musl ``libc``, or Android)
* Eventually, in the future, there may exist distributions that break
  compatibility with this profile

To handle the first two cases, we propose the following simple and reliable
check: ::

    def have_glibc_version(major, minimum_minor):
        import ctypes

        process_namespace = ctypes.CDLL(None)
        try:
            gnu_get_libc_version = process_namespace.gnu_get_libc_version
        except AttributeError:
            # We are not linked to glibc.
            return False

        gnu_get_libc_version.restype = ctypes.c_char_p
        version_str = gnu_get_libc_version()
        # py2 / py3 compatibility:
        if not isinstance(version_str, str):
            version_str = version_str.decode("ascii")

        version = [int(piece) for piece in version_str.split(".")]
        assert len(version) == 2
        if major != version[0]:
            return False
        if minimum_minor > version[1]:
            return False
        return True

    # CentOS 5 uses glibc 2.5.
    is_manylinux1_compatible = have_glibc_version(2, 5)

To handle the third case, we propose the creation of a file
``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
contents: ::

   [manylinux1]
   compatible = true

where the supported values for the ``manylinux1.compatible`` entry are the
same as those supported by the ConfigParser ``getboolean`` method.

The proposed logic for ``pip`` or related tools, then, is:

0) If ``distutils.util.get_platform()`` does not start with the string
   ``"linux"``, then assume the current system is not ``manylinux1``
   compatible.
1) If ``/etc/python/compatibility.conf`` exists and contains a ``manylinux1``
   key, then trust that.
2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
   current system can handle ``manylinux1`` wheels.
3) Otherwise, assume that the current system cannot handle ``manylinux1``
   wheels.


PyPI Support
============

PyPI should permit wheels containing the ``manylinux1`` platform tag to be
uploaded. PyPI should not attempt to formally verify that wheels containing
the ``manylinux1`` platform tag adhere to the ``manylinux1`` policy described
in this document. This verification tasks should be left to other tools, like
``auditwheel``, that are developed separately.


Rejected Alternatives
=====================

One alternative would be to provide separate platform tags for each Linux
distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
``debian_jessie``, etc. Nothing in this proposal rules out the possibility of
adding such platform tags in the future, or of further extensions to wheel
metadata that would allow wheels to declare dependencies on external
system-installed packages. However, such extensions would require substantially
more work than this proposal, and still might not be appreciated by package
developers who would prefer not to have to maintain multiple build environments
and build multiple wheels in order to cover all the common Linux distributions.
Therefore we consider such proposals to be out-of-scope for this PEP.


Future updates
==============

We anticipate that at some point in the future there will be a
``manylinux2`` specifying a more modern baseline environment (perhaps
based on CentOS 6), and someday a ``manylinux3`` and so forth, but we
defer specifying these until we have more experience with the initial
``manylinux1`` proposal.


References
==========

.. [1] PEP 0427 -- The Wheel Binary Package Format 1.0
   (https://www.python.org/dev/peps/pep-0427/)
.. [2] PEP 0491 -- The Wheel Binary Package Format 1.9
   (https://www.python.org/dev/peps/pep-0491/)
.. [3] PEP 425 -- Compatibility Tags for Built Distributions
   (https://www.python.org/dev/peps/pep-0425/)
.. [4] Enthought Canopy Python Distribution
   (https://store.enthought.com/downloads/)
.. [5] Continuum Analytics Anaconda Python Distribution
   (https://www.continuum.io/downloads)
.. [6] CentOS 5.11 Release Notes
   (https://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.11)
.. [7] manylinux-discuss mailing list discussion
   (https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U)
.. [8] distutils-sig discussion
   (https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html)
.. [9] manylinux1 docker image
   (https://quay.io/repository/manylinux/manylinux)
.. [10] auditwheel tool
   (https://pypi.python.org/pypi/auditwheel)
.. [11] Fedora Bundled Software Policy
   (https://fedoraproject.org/wiki/Bundled_Software_policy)
.. [12] Debian Policy Manual -- 4.13: Convenience copies of code
    (https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles)


Copyright
=========

This document has been placed into the public domain.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160125/538ef532/attachment-0001.html>

From njs at pobox.com  Mon Jan 25 22:58:47 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 25 Jan 2016 19:58:47 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
Message-ID: <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>

Hi all,

I think the PEP is pretty close to ready, but there's one remaining
minor-but-substantive technical point that got mostly skipped in the
previous discussion, and that I'm a little concerned about and wanted
to highlight. This is the mechanism by which a distribution/user can
explicitly specify their manylinux1-compatibility (overriding the
default heuristic rule):

On Mon, Jan 25, 2016 at 1:51 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
[...]
> To handle the third case, we propose the creation of a file
> ``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
> contents: ::
>
>    [manylinux1]
>    compatible = true
>
> where the supported values for the ``manylinux1.compatible`` entry are the
> same as those supported by the ConfigParser ``getboolean`` method.
>
> The proposed logic for ``pip`` or related tools, then, is:
>
> 0) If ``distutils.util.get_platform()`` does not start with the string
>    ``"linux"``, then assume the current system is not ``manylinux1``
>    compatible.
> 1) If ``/etc/python/compatibility.conf`` exists and contains a
> ``manylinux1``
>    key, then trust that.
> 2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
>    current system can handle ``manylinux1`` wheels.
> 3) Otherwise, assume that the current system cannot handle ``manylinux1``
>    wheels.

As stated, this proposal has a few problems:
- it doesn't key off of architecture -- e.g. on Debian a supported
(though weird) configuration would be to have both x86_64 and arm
builds of CPython installed into the same filesystem, and probably it
should be possible to specify the compatibility flag separately for
these two interpreters. If there's just one file in /etc, that isn't
possible.
- it's a single global setting, that applies even to non-global python
installs -- e.g. if I'm using a distribution which has compatible=True
in their /etc/python/compatibility.conf, and then in my home directory
I install a copy of CPython linked against musl-libc, then too bad,
the configuration file still applies and I'm going to get broken
wheels installed into my home directory environment.

Basically, in general, manylinux1-compatibility is not a property of a
system as a whole; it's a property of a single python interpreter
environment within that system. (I guess one can also imagine it
having different values within the same distro for a build of
python2.7 versus python3.4 versus pypy.)

So what would should we do? Some options that occur to me; none of
them obviously right :-):

- Make the configuration file lookup rules more elaborate -- e.g.,
look for a file called $WHEEL_ABI_TAG.cfg, and check first next to the
interpreter itself (re-using the search rule for pyvenv.cfg defined in
PEP 405) and then in /etc if that fails.

- Since this is a property of a particular python environment, bake it
into the environment itself, e.g. as a special variable
sys._manylinux1 that distributors can patch into place if they need to
override this.

- Just forget about the whole override concept for now; switch to
unconditionally checking for glibc version as the only rule, and wait
for someone to run into a case where they need the override feature
before adding it back.

Thoughts? I'd particularly like to hear from anyone associated with
the distros here, since this feature is intended to try and make them
happy :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Mon Jan 25 23:25:12 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 25 Jan 2016 20:25:12 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
Message-ID: <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>

On Sun, Jan 24, 2016 at 8:37 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
[...]
>> ...well, or maybe just erroring out when it sees the future and asking
>> the user to help would be good enough :-). This would impose the
>> requirement going forward that we'd have to wait for a pip release
>> with support for manylinuxN before allowing manylinuxN onto PyPI, but
>> that doesn't seem too onerous.
>
> For Windows and Mac OS X, dropping support for old platforms is
> effectively coupled to the CPython release cycle - a Python 3.5 wheel
> is less likely to work on Windows XP than a 3.3 one, for example,
> since 3.5 doesn't support XP, while 3.3 does.
>
> We could potentially consider something similar for manylinux: scoping
> the evolution of the definition by corresponding CPython release,
> rather than giving it its own version number.
>
> So the first release of manylinux would define the ABI for Linux
> wheels for 3.5 and earlier (including the 2.x series), and then 3.6
> would be the next chance to consider revising it (e.g. by bumping up
> the base ABI from CentOS 5 to CentOS 6).

The problem with this is that python 2.7 is going to be supported and
widely used until well past the EOL of CentOS 5, and maybe even past
the EOL of CentOS 6 (in 2020). On Linux, unlike Windows, the whole
system ABI underneath Python evolves in a backwards-compatible way, so
there's no anchor like the MSVC CRT that's tying us to old ABIs.

(And on OS X, the OS X version number is encoded as part of the
platform tag, similar to the proposal for manylinux1/2/...)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From rmcgibbo at gmail.com  Mon Jan 25 23:37:59 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Mon, 25 Jan 2016 20:37:59 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
Message-ID: <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>

I agree that this is an important detail. I generally use machines that
have many different Python interpreters installed (some distro-provided and
others in my home directory), and can easily imagine wanting them to have
different behavior w.r.t. manylinux1 wheels.

Perhaps the option could be put in site.py, or somewhere in
lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution here
is.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160125/a03638f3/attachment.html>

From rmcgibbo at gmail.com  Mon Jan 25 23:43:51 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Mon, 25 Jan 2016 20:43:51 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
Message-ID: <CAN4+E8FTpWf+qQN9S3BqDdZo3pShty-gkYA5s3+a2DXoBw98Lw@mail.gmail.com>

Alternatively, perhaps this question could be delegated to the pip
maintainers, for pip to store and maintain this configuration option
itself, perhaps by using its cache (for example, Linux pip already stores
some caches in ~/.cache/pip)?

-Robert

On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
wrote:

> I agree that this is an important detail. I generally use machines that
> have many different Python interpreters installed (some distro-provided and
> others in my home directory), and can easily imagine wanting them to have
> different behavior w.r.t. manylinux1 wheels.
>
> Perhaps the option could be put in site.py, or somewhere in
> lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution here
> is.
>
> -Robert
>



-- 
-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160125/a6a4fd7e/attachment-0001.html>

From njs at pobox.com  Tue Jan 26 00:38:13 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 25 Jan 2016 21:38:13 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7fD4ZLPc6wCwrRgFFs9BKs9sdXfeL+39PWKV1nYe1_iiA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAPJVwBnxChZeXnixePQ0ftCPjVGLStpErCD6E60UnBDzM8BstA@mail.gmail.com>
 <22E08F8F-A24A-4AD4-A7D3-E699774A751D@stufft.io>
 <CADiSq7fD4ZLPc6wCwrRgFFs9BKs9sdXfeL+39PWKV1nYe1_iiA@mail.gmail.com>
Message-ID: <CAPJVwBme-y3+PVFXoFJntwoaGVLFvGksVcAVEj7k6gVjY=aKbQ@mail.gmail.com>

On Sun, Jan 24, 2016 at 8:17 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 25 January 2016 at 10:23, Donald Stufft <donald at stufft.io> wrote:
>>
>>> On Jan 24, 2016, at 7:21 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>
>>> Maybe we need wheel-builders-sig? Their mandate would be to hash out
>>> things like how to build binary-libraries-wrapped-up-in-wheels, share
>>> knowledge about the minutiae of linker behavior on different platforms
>>> (oh god there's so much minutiae), maintain tools like delocate and
>>> auditwheel (and whatever the equivalent will be for windows... and do
>>> we really need 3 different tools?), collect knowledge from where it's
>>> scattered now and put it into the guide at packaging.python.org [1],
>>> etc.? It seems a bit outside distutils-sig's focus in practice, since
>>> this would all be about third-party tools and individual package
>>> authors as opposed to distutils-sig's focus on writing
>>> interoperability PEPs and maintaining the core python.org-affiliated
>>> infrastructure like PyPI / setuptools / pip.
>>
>> I?m not about to tell someone they *have* to use distutils-sig, but I
>> think at this point it?s perfectly reasonable to discuss things of
>> this nature here as well.
>
> I agree with this - while I wouldn't call any aspect of software
> distribution easy, binary compatibility is one of the hardest, and we
> want *more* knowledge sharing in that regard, rather than continue to
> maintain the divide between "folks who care about binary
> compatibility" and "folks who wish everyone would just stop creating
> extension modules already because pure Python modules are so much
> easier to deal with" :)
>
> Keeping the discussions centralised also creates more opportunities
> for serendipitous collaboration, where folks notice that they can
> potentially help out with what someone else is working on.

I'm definitely in favor of centralising this kind of knowledge and
initiative upstream with the core Python community. The whole
manylinux concept falls into this category of "hey let's move take
this experience with scientific packages and port it upstream" (and
btw if/when PEP 513 is accepted we should perhaps consider moving the
related tools into the pypa/ namespace?).

I guess my concern, though, is that distutils-sig has historically
been a rather contentious place. It's definitely not as bad these days
as its old reputation would suggest (much thanks to everyone who's
helped make that happen!). But I guess some amount of this is
intrinsic to its nature: the core infrastructure like distutils, pip,
PyPI, interoperability PEPs, is stuff that people with wildly varying
backgrounds and goals all *have* to use, and so we're kinda all
trapped here together. In a way, this kind of contention is a good
thing, because it forces this core infrastructure to do a better job
of serving all its many stakeholders. But OTOH I'm not sure it's very
conducive to making quick collaborative progress on focused topics
that don't need broad community consensus. (I'm not sure where
day-to-day discussion of routine changes to pip and warehouse and
setuptools is happening, but I do notice that it doesn't seem to be
here. And similarly there's a reason that the initial hashing out of
the manylinux stuff happened off-list -- if only because you didn't
need to see the blow-by-blow of our futzing about making docker work
;-).)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From chris.barker at noaa.gov  Tue Jan 26 01:29:44 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Mon, 25 Jan 2016 22:29:44 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
Message-ID: <-4219403055735535576@unknownmsgid>

(e.g. by bumping up
>> the base ABI from CentOS 5 to CentOS 6).
>
> The problem with this is that python 2.7 is going to be supported and
> widely used until well past the EOL of CentOS 5, and maybe even past
> the EOL of CentOS 6

Given that we're starting now ( not a year or two ago) and it'll take
a while for it to really catch on, we should go CentOS 6 ( or
equivalent ) now?

CentOS5 was released in 2007! That is a pretty long time in computing.

Just a thought, we'll be locked into it for a while, yes?

-CHB


> (in 2020). On Linux, unlike Windows, the whole
> system ABI underneath Python evolves in a backwards-compatible way, so
> there's no anchor like the MSVC CRT that's tying us to old ABIs.
>
> (And on OS X, the OS X version number is encoded as part of the
> platform tag, similar to the proposal for manylinux1/2/...)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From rmcgibbo at gmail.com  Tue Jan 26 01:49:22 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Mon, 25 Jan 2016 22:49:22 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <-4219403055735535576@unknownmsgid>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
Message-ID: <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>

On Mon, Jan 25, 2016 at 10:29 PM, Chris Barker - NOAA Federal <
chris.barker at noaa.gov> wrote:

> Given that we're starting now ( not a year or two ago) and it'll take
> a while for it to really catch on, we should go CentOS 6 ( or
> equivalent ) now?
>
> CentOS5 was released in 2007! That is a pretty long time in computing.
>

I understand the concern, but I think we should follow the lead of the
other projects
that have been doing portable linux binaries (holy build box, traveling
ruby, portable-pypy,
firefox, enthought, continuum) for some time, all based on CentOS 5. At
some point things
like C++17 support will be important and I agree that we'll need to update
the base spec,
but in the meantime, I don't see this as a problem where we should be the
first mover.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160125/1ff92804/attachment.html>

From njs at pobox.com  Tue Jan 26 04:02:50 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 01:02:50 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8FTpWf+qQN9S3BqDdZo3pShty-gkYA5s3+a2DXoBw98Lw@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAN4+E8FTpWf+qQN9S3BqDdZo3pShty-gkYA5s3+a2DXoBw98Lw@mail.gmail.com>
Message-ID: <CAPJVwBk54mD6pLXY6PbvkL-3iwjm6hcP6ynJGGBPW46gb=2hhA@mail.gmail.com>

On Mon, Jan 25, 2016 at 8:43 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> Alternatively, perhaps this question could be delegated to the pip
> maintainers, for pip to store and maintain this configuration option itself,
> perhaps by using its cache (for example, Linux pip already stores some
> caches in ~/.cache/pip)?

It could, but (a) it's probably nice if there's a standard, b/c in
theory pip-competitors might exist and need access to the same
information, (b) whatever the mechanism is it should probably be
something that gives python distributors a way to set the default
instead of forcing users to configure it manually (and distributors
don't in general control pip or ~/.anything), and (c) even if we
delegate the decision to the pip maintainers they'll probably just
tell us to figure it out and submit a PR for them to decide on ;-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Tue Jan 26 05:36:26 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 26 Jan 2016 20:36:26 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
Message-ID: <CADiSq7eS0=JpA2a0nQ-sP21abZ7HtoraV5Kf-DjnLKkJsm=hMQ@mail.gmail.com>

On 26 January 2016 at 16:49, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> On Mon, Jan 25, 2016 at 10:29 PM, Chris Barker - NOAA Federal
> <chris.barker at noaa.gov> wrote:
>>
>> Given that we're starting now ( not a year or two ago) and it'll take
>> a while for it to really catch on, we should go CentOS 6 ( or
>> equivalent ) now?
>>
>> CentOS5 was released in 2007! That is a pretty long time in computing.
>
> I understand the concern, but I think we should follow the lead of the other
> projects
> that have been doing portable linux binaries (holy build box, traveling
> ruby, portable-pypy,
> firefox, enthought, continuum) for some time, all based on CentOS 5. At some
> point things
> like C++17 support will be important and I agree that we'll need to update
> the base spec,
> but in the meantime, I don't see this as a problem where we should be the
> first mover.

I was discussing this with some of the folks that are responsible for
defining the RHEL (and hence CentOS) ABI, and they pointed out that
the main near term advantage of targeting CentOS 6 over CentOS 5 is
that it means that it would be possible to analyse binaries built that
way with the libabigail tools, including abicompat:
https://sourceware.org/libabigail/manual/abicompat.html

If I understand the problem correctly, the CentOS 5 gcc toolchain is
old enough that it simply doesn't emit the info libabigail needs in
order to work.

So if we went down the CentOS 6+ path, it should make it possible to
take binaries built on the reference environment and use abicompat to
check them against libraries from another distro, and vice-versa.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From olivier.grisel at ensta.org  Tue Jan 26 05:41:48 2016
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Tue, 26 Jan 2016 11:41:48 +0100
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
Message-ID: <CAFvE7K5aX9D9_mXGqhgNCmYUPjHK61oHg=sDJxrrbet8V0g5kQ@mail.gmail.com>

Maybe we could devise some syntax for /etc/python/compatibility.conf
to state that the manylinux1 entry is only valid for Python
interpreters such as distutils.util.get_platform() == 'linux-x86_64'?

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel

From solipsis at pitrou.net  Tue Jan 26 06:44:11 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Jan 2016 12:44:11 +0100
Subject: [Distutils] draft PEP: manylinux1
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CADiSq7eS0=JpA2a0nQ-sP21abZ7HtoraV5Kf-DjnLKkJsm=hMQ@mail.gmail.com>
Message-ID: <20160126124411.222ba104@fsol>

On Tue, 26 Jan 2016 20:36:26 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> If I understand the problem correctly, the CentOS 5 gcc toolchain is
> old enough that it simply doesn't emit the info libabigail needs in
> order to work.

If you build on CentOS 5, you certainly want to use the RH developer
toolset 2 which gives you a modern-ish toolchain (gcc 4.8.2, IIRC).

Regards

Antoine.



From tritium-list at sdamon.com  Tue Jan 26 06:50:15 2016
From: tritium-list at sdamon.com (Alexander Walters)
Date: Tue, 26 Jan 2016 06:50:15 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <20160126124411.222ba104@fsol>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CADiSq7eS0=JpA2a0nQ-sP21abZ7HtoraV5Kf-DjnLKkJsm=hMQ@mail.gmail.com>
 <20160126124411.222ba104@fsol>
Message-ID: <56A75D77.5070302@sdamon.com>

Isnt the entire point of using centos 5 to use an ancient toolchain?

On 1/26/2016 06:44, Antoine Pitrou wrote:
> On Tue, 26 Jan 2016 20:36:26 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> If I understand the problem correctly, the CentOS 5 gcc toolchain is
>> old enough that it simply doesn't emit the info libabigail needs in
>> order to work.
> If you build on CentOS 5, you certainly want to use the RH developer
> toolset 2 which gives you a modern-ish toolchain (gcc 4.8.2, IIRC).
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From solipsis at pitrou.net  Tue Jan 26 06:55:09 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Jan 2016 12:55:09 +0100
Subject: [Distutils] draft PEP: manylinux1
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CADiSq7eS0=JpA2a0nQ-sP21abZ7HtoraV5Kf-DjnLKkJsm=hMQ@mail.gmail.com>
 <20160126124411.222ba104@fsol> <56A75D77.5070302@sdamon.com>
Message-ID: <20160126125509.4020be31@fsol>

On Tue, 26 Jan 2016 06:50:15 -0500
Alexander Walters <tritium-list at sdamon.com> wrote:
> Isnt the entire point of using centos 5 to use an ancient toolchain?

No, the point is to link against an ancient glibc. The toolchain can be
modern (and actually has to if you want to compile e.g. C++11 code).

Regards

Antoine.



From olivier.grisel at ensta.org  Tue Jan 26 07:22:47 2016
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Tue, 26 Jan 2016 13:22:47 +0100
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAFvE7K5aX9D9_mXGqhgNCmYUPjHK61oHg=sDJxrrbet8V0g5kQ@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAFvE7K5aX9D9_mXGqhgNCmYUPjHK61oHg=sDJxrrbet8V0g5kQ@mail.gmail.com>
Message-ID: <CAFvE7K7--6boumZg_Wzjb-Xrh_0Mq-rpZ_g5Er9R4e-EC0VYwQ@mail.gmail.com>

2016-01-26 11:41 GMT+01:00 Olivier Grisel <olivier.grisel at ensta.org>:
> Maybe we could devise some syntax for /etc/python/compatibility.conf
> to state that the manylinux1 entry is only valid for Python
> interpreters such as distutils.util.get_platform() == 'linux-x86_64'?

Actually this won't work. I just try to debootstrap ubuntu trusty i386
on top of ubuntu trusty amd64 and I get the following behavior in
python3 inside the i386 chroot:

>>> from distutils.util import get_platform
>>> get_platform()
'linux-x86_64'

>>> import platform
>>> platform.machine()
'x86_64'

>>> import sys
>>> sys.maxsize > 2**32
False

So this is actually a 32 bit Python declared as running on a
linux-x86_64 platform (even though everything is i386 inside the
chroot)...

I get the exact same behavior when installing the 32 bit miniconda on
ubuntu trusty amd64.

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel

From cournape at gmail.com  Tue Jan 26 08:01:03 2016
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 26 Jan 2016 13:01:03 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <20160126124411.222ba104@fsol>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CADiSq7eS0=JpA2a0nQ-sP21abZ7HtoraV5Kf-DjnLKkJsm=hMQ@mail.gmail.com>
 <20160126124411.222ba104@fsol>
Message-ID: <CAGY4rcVHBjEGsW=cbj5DL38wZVAuA-UyibyYP_zb-B9YFAEgfA@mail.gmail.com>

On Tue, Jan 26, 2016 at 11:44 AM, Antoine Pitrou <solipsis at pitrou.net>
wrote:

> On Tue, 26 Jan 2016 20:36:26 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > If I understand the problem correctly, the CentOS 5 gcc toolchain is
> > old enough that it simply doesn't emit the info libabigail needs in
> > order to work.
>
> If you build on CentOS 5, you certainly want to use the RH developer
> toolset 2 which gives you a modern-ish toolchain (gcc 4.8.2, IIRC).
>

Indeed, C++11 is the main reason why I added devtoolsset 2 in the
`Dockerfile` that was part of what originated the manylinux effort.

David


> Regards
>
> Antoine.
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/7f126e4c/attachment.html>

From ncoghlan at gmail.com  Tue Jan 26 08:17:00 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 26 Jan 2016 23:17:00 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <20160126124411.222ba104@fsol>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CADiSq7eS0=JpA2a0nQ-sP21abZ7HtoraV5Kf-DjnLKkJsm=hMQ@mail.gmail.com>
 <20160126124411.222ba104@fsol>
Message-ID: <CADiSq7c6LNmaKqFdnDy545xcxaYBtDQk_kEzRv97a_3apgCU_w@mail.gmail.com>

On 26 January 2016 at 21:44, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 26 Jan 2016 20:36:26 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> If I understand the problem correctly, the CentOS 5 gcc toolchain is
>> old enough that it simply doesn't emit the info libabigail needs in
>> order to work.
>
> If you build on CentOS 5, you certainly want to use the RH developer
> toolset 2 which gives you a modern-ish toolchain (gcc 4.8.2, IIRC).

Yeah, that's the part I haven't clarified yet - whether it was just
the base CentOS toolchain that was too old, or devtoolset-2.

If abicompat et al all work with devtoolset-2, then it definitely
makes sense to stick with the CentOS 5 baseline for now.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Tue Jan 26 12:16:16 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 26 Jan 2016 12:16:16 -0500
Subject: [Distutils] Non English Speaking Users of PyPI - I need Help!
Message-ID: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>

Hello!

As many of you are aware there has been an effort to replace the current PyPI with a new, improved PyPI. This project has been codenamed Warehouse and has been progressing nicely. However we?ve run into a bit of an issue when deciding what to support that we?re not feeling super qualified to make an informed decision on.

The new PyPI is going to support translated content (for the UI elements, not for what people upload to there), although we will not launch with any translations actually added besides English. Currently the translation engine we?re using (l20n.js) does not support anything but ?Evergreen? browsers (browsers that constantly and automatically update) which means we don?t have support for older versions of IE. My question to anyone who is, or is familiar with places where English isn?t the native language, how big of a deal is this if we only support newer browsers for translations?

If you can weigh in on the issue for this (https://github.com/pypa/warehouse/issues/881) that would be great! If you know someone who might have a good insight, please pass this along to them as well.

Thanks!

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/f2607fa0/attachment.sig>

From solipsis at pitrou.net  Tue Jan 26 13:20:14 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Jan 2016 19:20:14 +0100
Subject: [Distutils] Non English Speaking Users of PyPI - I need Help!
References: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>
Message-ID: <20160126192014.4fee6147@fsol>

On Tue, 26 Jan 2016 12:16:16 -0500
Donald Stufft <donald at stufft.io> wrote:
> 
> As many of you are aware there has been an effort to replace the current PyPI with a new, improved PyPI. This project has been codenamed Warehouse and has been progressing nicely. However we?ve run into a bit of an issue when deciding what to support that we?re not feeling super qualified to make an informed decision on.
> 
> The new PyPI is going to support translated content (for the UI elements, not for what people upload to there), although we will not launch with any translations actually added besides English. Currently the translation engine we?re using (l20n.js) does not support anything but ?Evergreen? browsers (browsers that constantly and automatically update) which means we don?t have support for older versions of IE. My question to anyone who is, or is familiar with places where English isn?t the native language, how big of a deal is this if we only support newer browsers for translations?
> 
> If you can weigh in on the issue for this (https://github.com/pypa/warehouse/issues/881) that would be great! If you know someone who might have a good insight, please pass this along to them as well.

Not answering your question, but needing Javascript on the
client to support L10n sounds like a weird decision (although Mozilla
seems to be pushing this... how surprising).  Every bit of client-side
Javascript tends to make Web pages slower and it tends to accumulate
into the current bloated mess that is the modern Web.  For static text
this really doesn't sound warranted.

(not to mention that mutating the body text after the HTML has loaded
may also produce a poor user experience, depending on various
conditions. And the native English speakers who develop the software
on top-grade machines will probably not notice it, thinking everything
is fine.)

As for your question, though, I would expect some of the less proficient
English speakers to also have outdated hardware or software installs,
especially in poor countries or very humble social environments.

Regards

Antoine.



From rmcgibbo at gmail.com  Tue Jan 26 13:59:12 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Tue, 26 Jan 2016 10:59:12 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
Message-ID: <CAN4+E8Hxn7tZ-3wL1gs6riYDKBf-CHGbaiYbfQ3+wRP7mRDfyQ@mail.gmail.com>

On Mon, Jan 25, 2016 at 1:51 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
wrote:

>
> HTML version:
> https://github.com/manylinux/manylinux/blob/master/pep-513.rst
>

Nick, at some point would it be possible to put update the version in the
PEPs repo to
this version?

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/c61e0113/attachment.html>

From donald at stufft.io  Tue Jan 26 14:10:28 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 26 Jan 2016 14:10:28 -0500
Subject: [Distutils] Non English Speaking Users of PyPI - I need Help!
In-Reply-To: <20160126192014.4fee6147@fsol>
References: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>
 <20160126192014.4fee6147@fsol>
Message-ID: <4F847476-62D9-4294-B9AF-E9B5D5A8EB8E@stufft.io>


> On Jan 26, 2016, at 1:20 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> On Tue, 26 Jan 2016 12:16:16 -0500
> Donald Stufft <donald at stufft.io> wrote:
>> 
>> As many of you are aware there has been an effort to replace the current PyPI with a new, improved PyPI. This project has been codenamed Warehouse and has been progressing nicely. However we?ve run into a bit of an issue when deciding what to support that we?re not feeling super qualified to make an informed decision on.
>> 
>> The new PyPI is going to support translated content (for the UI elements, not for what people upload to there), although we will not launch with any translations actually added besides English. Currently the translation engine we?re using (l20n.js) does not support anything but ?Evergreen? browsers (browsers that constantly and automatically update) which means we don?t have support for older versions of IE. My question to anyone who is, or is familiar with places where English isn?t the native language, how big of a deal is this if we only support newer browsers for translations?
>> 
>> If you can weigh in on the issue for this (https://github.com/pypa/warehouse/issues/881) that would be great! If you know someone who might have a good insight, please pass this along to them as well.
> 
> Not answering your question, but needing Javascript on the
> client to support L10n sounds like a weird decision (although Mozilla
> seems to be pushing this... how surprising).  Every bit of client-side
> Javascript tends to make Web pages slower and it tends to accumulate
> into the current bloated mess that is the modern Web.  For static text
> this really doesn't sound warranted.
> 
> (not to mention that mutating the body text after the HTML has loaded
> may also produce a poor user experience, depending on various
> conditions. And the native English speakers who develop the software
> on top-grade machines will probably not notice it, thinking everything
> is fine.)
> 
> As for your question, though, I would expect some of the less proficient
> English speakers to also have outdated hardware or software installs,
> especially in poor countries or very humble social environments.
> 

So the reason for wanting to use L20n (and forgive me, English is the only
language I speak so a lot of this is based off of my, possibly wrong,
understanding of how other languages work) is because it is a lot more powerful
than the traditional gettext based solutions.

One such problem I believe is the lack of variants in the older tools like
gettext. I think at best you can get singular/plural but I think that other
languages have a whole host of different things that they need to vary their
grammar based on. An example from the L20n website is:

    <brandShortName {
        *nominative: "Aurora",
        genitive: "Aurore",
        dative: "Aurori",
        accusative: "Auroro",
        locative: "Aurori",
        instrumental: "Auroro"
    }>
    <aboutOld "O brskalniku {{ brandShortName }}">
    <about "O {{ brandShortName.locative }}">

Where that would allow you to have the brand name (in this exampel) translated
based on the items in that first list. This is powerful enough to support
choosing it based on something you pass into the translation engine from the
application being translated as well.

Another example of this is when you need to adjust the translation based on
the gender of the subject (though I don't tihnk we'll use this on PyPI since
we're unlikely to ever collect that information), but L20n makes this possible
if you pass the gender into the translation engine like:

    # Thing that gets passed in
    {
        "user": {
            "name": "Jane",
            "followers": 1337,
            "gender": "feminine"
        }
    }

    # Translation Snippet
    <shared[$user.gender] {
        masculine: "{{ $user.name }} shared your post to his {{ $user.followers }} follower(s).",
        feminine: "{{ $user.name }}  shared your post to her {{ $user.followers }} follower(s).",
        *default: "{{ $user.name }} shared your post to their {{ $user.followers }} follower(s)."
    }>

In addition to that, L20n also natively understands HTML which makes it a bit
easier to work with. In a traditional gettext based system, if you wanted to
do something like translate a string of text that contains a link to something
you'd need to do something like this:

    'This is a sentence that has an embedded <a href="%(url)s">link</a>'

Then you need to expect your translators to correctly generate that HTML,
including and classes or style information that is in it (and if you alter that
all translations need to be updated to fix it).

However, in L20n you can simple do something like this:

    <p data-l10n-id="mySentence">This is a sentence that has an embedded <a href="https://../">link</a></p>

Then when your translators go to translate it, they only need to do:

    <mySentence "This is a translated sentence with a <a>link</a>">

They never need to worry about matching the exact HTML, they just need to worry
about marking the structure correctly. The one downside to this, is that there
is not currently any way for them to *reorder* the HTML elements (like if you
have two links) which I'm not sure how big of a deal that is.

Finally, on the L20n vs gettext side, L20n forces you to define IDs for your
translations instead of reusing the source (generally English) text as your
ID. This means that people can be free to tweak the English text in ways that
do not alter the semantics of the statement without having that affect the
other translations, as long as the ID stays the same all existing translations
will continue to be used.

Now, all of the above could be written serverside in Python and not require
anything of the end user's browser. We're currently using L20n.js instead of
something serverside for a few reasons.

The biggest and most obvious reason is because L20n.js is currently the only
implementation of L20n that exists, so moving to L20n at the server side would
require us to devote time to writing that instead of working on Warehouse
itself.

Another reason is that L20n.js also allows you to do what they call "responsive
translations". Essentially it allows you to have variants of your translation
based on properties of the end user's machine such as operating system, window
size, time of day, etc. This makes it easy to have a translation say, switch
between a longer form of a translation when running full screen in a large
browser or a shorter form when running on a small smart phone screen without
having a fairly common (I think?) problem where the source text and the
translated text are vastly different in length.

The final reason is that by moving translation into the client side we can
increase the chance that for any particular page a user visits they will be
served directly out of a Fastly POP located closely to them instead of needing
to round trip from the Fastly POP located closely to them, to the Fastly POP
located in Ashburn, Virgnia, USA, to the PyPI servers located in another DC in
Virgnia. Instead of needing to cache and serve a different variant of the page
for every single language, we can instead have a single variant of the page
for all languages, and a single language file for each language that is used
for all pages (much like CSS) which will increase the cache hit ration. This
will also make it more likely that if the PyPI origin servers go down, that the
user won't get an error response (since serving out of cache never hits the
backend servers, and Fastly is configured to serve stale responses from the
cache on the case of an Error). Even for users of lesser used languages that
might not have their language file cached, if the PyPI servers are down,
they'll still get the English variant served from cache but if their language
file isn't cache and PyPI is down, they just won't get it translated, falling
back to the english version in that case.

Anyways, I don't know if all of these reasons are good enough reasons to impose
the browser requirement on our users. On paper it certainly appears to me like
they are, giving people better, higher quality translations seems to be a good
thing to me. However, I don't know how bad the limitations of gettext are in
practice. I will say that under 1% of our views (not users, views) on PyPI
currently come from browser set to something other than English AND which
l20n.js does not support but it's possible that this is a chicken/egg situation
where we're not getting a lot of traffic from those users because PyPI isn't
translated in a way that they can use. I am also unsure how the fact that the
majority of PyPI's content will still be in English (since this is just for the
UI, not the content) affects all of this, but there are projects out there
where the content is definitely not English (though I am unsure what languages
they are, the ones I've seen use some sort of Asian looking lettering).

Hopefully this was useful information!

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/23eb43e3/attachment-0001.sig>

From tritium-list at sdamon.com  Tue Jan 26 14:11:18 2016
From: tritium-list at sdamon.com (Alexander Walters)
Date: Tue, 26 Jan 2016 14:11:18 -0500
Subject: [Distutils] Non English Speaking Users of PyPI - I need Help!
In-Reply-To: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>
References: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>
Message-ID: <56A7C4D6.1050908@sdamon.com>

How...does the client side text interact with screen readers?  Is that 
an issue?  It sounds like an odd thing to do in the first place...

On 1/26/2016 12:16, Donald Stufft wrote:
> Hello!
>
> As many of you are aware there has been an effort to replace the current PyPI with a new, improved PyPI. This project has been codenamed Warehouse and has been progressing nicely. However we?ve run into a bit of an issue when deciding what to support that we?re not feeling super qualified to make an informed decision on.
>
> The new PyPI is going to support translated content (for the UI elements, not for what people upload to there), although we will not launch with any translations actually added besides English. Currently the translation engine we?re using (l20n.js) does not support anything but ?Evergreen? browsers (browsers that constantly and automatically update) which means we don?t have support for older versions of IE. My question to anyone who is, or is familiar with places where English isn?t the native language, how big of a deal is this if we only support newer browsers for translations?
>
> If you can weigh in on the issue for this (https://github.com/pypa/warehouse/issues/881) that would be great! If you know someone who might have a good insight, please pass this along to them as well.
>
> Thanks!
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/9ab4d873/attachment.html>

From donald at stufft.io  Tue Jan 26 14:15:44 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 26 Jan 2016 14:15:44 -0500
Subject: [Distutils] Non English Speaking Users of PyPI - I need Help!
In-Reply-To: <56A7C4D6.1050908@sdamon.com>
References: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>
 <56A7C4D6.1050908@sdamon.com>
Message-ID: <7B368FD2-2DF4-4967-96A8-4EDCFC65D63A@stufft.io>


> On Jan 26, 2016, at 2:11 PM, Alexander Walters <tritium-list at sdamon.com> wrote:
> 
> How...does the client side text interact with screen readers?  Is that an issue?  It sounds like an odd thing to do in the first place?

I don?t know for sure! According to http://a11yproject.com/posts/myth-screen-readers-dont-use-javascript/ <http://a11yproject.com/posts/myth-screen-readers-dont-use-javascript/> the majority of screen readers are perfectly capable of using javascript. It?s unclear what standard of JS they support though. To be clear though, (incase it wasn?t) the fallback if the translation engine is unable to run isn?t no text, it?s English text.


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/9fb10d35/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/9fb10d35/attachment.sig>

From solipsis at pitrou.net  Tue Jan 26 14:27:46 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Jan 2016 20:27:46 +0100
Subject: [Distutils] Non English Speaking Users of PyPI - I need Help!
In-Reply-To: <4F847476-62D9-4294-B9AF-E9B5D5A8EB8E@stufft.io>
References: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>
 <20160126192014.4fee6147@fsol>
 <4F847476-62D9-4294-B9AF-E9B5D5A8EB8E@stufft.io>
Message-ID: <20160126202746.14f6f4b8@fsol>

On Tue, 26 Jan 2016 14:10:28 -0500
Donald Stufft <donald at stufft.io> wrote:
> 
> Hopefully this was useful information!

It was certainly comprehensive. Thank you :-)

About gettext's shortcomings... I think it's so widely used that it's
probably possible to work around them whenever they appear. I'm not
saying the gettext API is pretty, though (especially the clumsy catalog
management commands).

By the way, the Babel library on PyPI may at least render some of that
stuff a bit easier to deal with.

Regards

Antoine.

From chris.barker at noaa.gov  Tue Jan 26 14:47:16 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 26 Jan 2016 11:47:16 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
Message-ID: <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>

On Mon, Jan 25, 2016 at 10:49 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
wrote:

> On Mon, Jan 25, 2016 at 10:29 PM, Chris Barker - NOAA Federal <
> chris.barker at noaa.gov> wrote:
>
>> Given that we're starting now ( not a year or two ago) and it'll take
>> a while for it to really catch on, we should go CentOS 6 ( or
>> equivalent ) now?
>>
>> CentOS5 was released in 2007! That is a pretty long time in computing.
>>
>
> I understand the concern, but I think we should follow the lead of the
> other projects
> that have been doing portable linux binaries (holy build box, traveling
> ruby, portable-pypy,
> firefox, enthought, continuum) for some time, all based on CentOS 5.
>

That's the point -- they have been doing it for some time -- some time ago,
you really wouls want a version that old.

The question is -- how many systems are there in teh wild now that are
older than CentOS6? I have no idea how to even find out. But if we're
starting somethign new, why start with what was appropriate 2+ years ago?

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/f617c791/attachment-0001.html>

From carl at oddbird.net  Tue Jan 26 15:15:26 2016
From: carl at oddbird.net (Carl Meyer)
Date: Tue, 26 Jan 2016 13:15:26 -0700
Subject: [Distutils] Non English Speaking Users of PyPI - I need Help!
In-Reply-To: <4F847476-62D9-4294-B9AF-E9B5D5A8EB8E@stufft.io>
References: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>
 <20160126192014.4fee6147@fsol>
 <4F847476-62D9-4294-B9AF-E9B5D5A8EB8E@stufft.io>
Message-ID: <56A7D3DE.8030101@oddbird.net>

Hi Donald,

Thanks for the pointer to l20n, I hadn't seen it before; as a standard
data format for translations it certainly looks much more flexible and
nicer to use than gettext! I'm curious about this, though:

On 01/26/2016 12:10 PM, Donald Stufft wrote:
> Now, all of the above could be written serverside in Python and not require
> anything of the end user's browser. We're currently using L20n.js instead of
> something serverside for a few reasons.
> 
> The biggest and most obvious reason is because L20n.js is currently the only
> implementation of L20n that exists, so moving to L20n at the server side would
> require us to devote time to writing that instead of working on Warehouse
> itself.

What about https://github.com/l20n/python-l20n ? It's hosted under the
same org, so it looks official-ish. Is it incomplete/unstable/untested?

(Of course the cache-related and responsive-translations arguments in
favor of l20n.js would still apply, so I'm not saying you should use
python-l20n, I'm mostly just curious whether it's usable.)

Carl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/df64e922/attachment.sig>

From donald at stufft.io  Tue Jan 26 15:52:30 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 26 Jan 2016 15:52:30 -0500
Subject: [Distutils] Non English Speaking Users of PyPI - I need Help!
In-Reply-To: <56A7D3DE.8030101@oddbird.net>
References: <5E5D297A-4503-453E-8C1B-4BFD3370BB08@stufft.io>
 <20160126192014.4fee6147@fsol>
 <4F847476-62D9-4294-B9AF-E9B5D5A8EB8E@stufft.io>
 <56A7D3DE.8030101@oddbird.net>
Message-ID: <7B425DA2-B993-4F04-9773-7A3831E5EA0B@stufft.io>


> On Jan 26, 2016, at 3:15 PM, Carl Meyer <carl at oddbird.net> wrote:
> 
> Hi Donald,
> 
> Thanks for the pointer to l20n, I hadn't seen it before; as a standard
> data format for translations it certainly looks much more flexible and
> nicer to use than gettext! I'm curious about this, though:
> 
> On 01/26/2016 12:10 PM, Donald Stufft wrote:
>> Now, all of the above could be written serverside in Python and not require
>> anything of the end user's browser. We're currently using L20n.js instead of
>> something serverside for a few reasons.
>> 
>> The biggest and most obvious reason is because L20n.js is currently the only
>> implementation of L20n that exists, so moving to L20n at the server side would
>> require us to devote time to writing that instead of working on Warehouse
>> itself.
> 
> What about https://github.com/l20n/python-l20n ? It's hosted under the
> same org, so it looks official-ish. Is it incomplete/unstable/untested?
> 
> (Of course the cache-related and responsive-translations arguments in
> favor of l20n.js would still apply, so I'm not saying you should use
> python-l20n, I'm mostly just curious whether it's usable.)
> 
> Carl
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

I?m told it?s a non functioning experiment for a previous version of L20n?s syntax and is effectively dead and useless.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/483e6197/attachment.sig>

From rmcgibbo at gmail.com  Tue Jan 26 16:04:12 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Tue, 26 Jan 2016 13:04:12 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAFvE7K7--6boumZg_Wzjb-Xrh_0Mq-rpZ_g5Er9R4e-EC0VYwQ@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAFvE7K5aX9D9_mXGqhgNCmYUPjHK61oHg=sDJxrrbet8V0g5kQ@mail.gmail.com>
 <CAFvE7K7--6boumZg_Wzjb-Xrh_0Mq-rpZ_g5Er9R4e-EC0VYwQ@mail.gmail.com>
Message-ID: <CAN4+E8EE50fw_vQtrrbtf8wSoAmUwq-PPd4XahF6MKrq_+Axww@mail.gmail.com>

The file pip uses to record its latest check for being out of date (on
linux ~/.cache/pip/selfcheck.json) is a json file that maps the path to the
interpreter's sys.prefix to some metadata about the installation

e.g.

{"/home/rmcgibbo/opt/python-3.5/":
{"last_check":"2016-01-12T00:09:02Z","pypi_version":"7.1.2"}, ...}

Perhaps a similar model should be used for /etc/python/compatibility.cfg.
That is, the configuration option should be keyed on a per-interpreter
basis.

[/usr]
manylinux1 compatible = true

[/path/to/armv7l/python]
manylinux1 compatible = false


-Robert



On Tue, Jan 26, 2016 at 4:22 AM, Olivier Grisel <olivier.grisel at ensta.org>
wrote:

> 2016-01-26 11:41 GMT+01:00 Olivier Grisel <olivier.grisel at ensta.org>:
> > Maybe we could devise some syntax for /etc/python/compatibility.conf
> > to state that the manylinux1 entry is only valid for Python
> > interpreters such as distutils.util.get_platform() == 'linux-x86_64'?
>
> Actually this won't work. I just try to debootstrap ubuntu trusty i386
> on top of ubuntu trusty amd64 and I get the following behavior in
> python3 inside the i386 chroot:
>
> >>> from distutils.util import get_platform
> >>> get_platform()
> 'linux-x86_64'
>
> >>> import platform
> >>> platform.machine()
> 'x86_64'
>
> >>> import sys
> >>> sys.maxsize > 2**32
> False
>
> So this is actually a 32 bit Python declared as running on a
> linux-x86_64 platform (even though everything is i386 inside the
> chroot)...
>
> I get the exact same behavior when installing the 32 bit miniconda on
> ubuntu trusty amd64.
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
>



-- 
-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/f3375695/attachment.html>

From njs at pobox.com  Tue Jan 26 16:37:03 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 13:37:03 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8EE50fw_vQtrrbtf8wSoAmUwq-PPd4XahF6MKrq_+Axww@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAFvE7K5aX9D9_mXGqhgNCmYUPjHK61oHg=sDJxrrbet8V0g5kQ@mail.gmail.com>
 <CAFvE7K7--6boumZg_Wzjb-Xrh_0Mq-rpZ_g5Er9R4e-EC0VYwQ@mail.gmail.com>
 <CAN4+E8EE50fw_vQtrrbtf8wSoAmUwq-PPd4XahF6MKrq_+Axww@mail.gmail.com>
Message-ID: <CAPJVwBkiJk-qLg=AH9Evh9iJVxEefaDEUA8RUpvHPMhDHy-sKA@mail.gmail.com>

On Jan 26, 2016 1:04 PM, "Robert T. McGibbon" <rmcgibbo at gmail.com> wrote:
>
> The file pip uses to record its latest check for being out of date (on
linux ~/.cache/pip/selfcheck.json) is a json file that maps the path to the
interpreter's sys.prefix to some metadata about the installation
>
> e.g.
>
> {"/home/rmcgibbo/opt/python-3.5/":
{"last_check":"2016-01-12T00:09:02Z","pypi_version":"7.1.2"}, ...}
>
> Perhaps a similar model should be used for /etc/python/compatibility.cfg.
That is, the configuration option should be keyed on a per-interpreter
basis.
>
> [/usr]
> manylinux1 compatible = true
>
> [/path/to/armv7l/python]
> manylinux1 compatible = false

Interesting idea. Any thoughts on how to propagate this information from
parent envs into virtualenvs?

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/d57b0946/attachment-0001.html>

From rmcgibbo at gmail.com  Tue Jan 26 17:06:53 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Tue, 26 Jan 2016 14:06:53 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBkiJk-qLg=AH9Evh9iJVxEefaDEUA8RUpvHPMhDHy-sKA@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAFvE7K5aX9D9_mXGqhgNCmYUPjHK61oHg=sDJxrrbet8V0g5kQ@mail.gmail.com>
 <CAFvE7K7--6boumZg_Wzjb-Xrh_0Mq-rpZ_g5Er9R4e-EC0VYwQ@mail.gmail.com>
 <CAN4+E8EE50fw_vQtrrbtf8wSoAmUwq-PPd4XahF6MKrq_+Axww@mail.gmail.com>
 <CAPJVwBkiJk-qLg=AH9Evh9iJVxEefaDEUA8RUpvHPMhDHy-sKA@mail.gmail.com>
Message-ID: <CAN4+E8EprigfbesApaut6i5u+gKc0ei=Q46ci3sN=rmtAzK1pw@mail.gmail.com>

Virtualenvs have `sys.base_prefix`, so perhaps they could just check
whether the key exists for their parent environment?

-Robert

On Tue, Jan 26, 2016 at 1:37 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Jan 26, 2016 1:04 PM, "Robert T. McGibbon" <rmcgibbo at gmail.com> wrote:
> >
> > The file pip uses to record its latest check for being out of date (on
> linux ~/.cache/pip/selfcheck.json) is a json file that maps the path to the
> interpreter's sys.prefix to some metadata about the installation
> >
> > e.g.
> >
> > {"/home/rmcgibbo/opt/python-3.5/":
> {"last_check":"2016-01-12T00:09:02Z","pypi_version":"7.1.2"}, ...}
> >
> > Perhaps a similar model should be used for
> /etc/python/compatibility.cfg. That is, the configuration option should be
> keyed on a per-interpreter basis.
> >
> > [/usr]
> > manylinux1 compatible = true
> >
> > [/path/to/armv7l/python]
> > manylinux1 compatible = false
>
> Interesting idea. Any thoughts on how to propagate this information from
> parent envs into virtualenvs?
>
> -n
>



-- 
-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/e7d8d6cc/attachment.html>

From njs at pobox.com  Tue Jan 26 17:13:25 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 14:13:25 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8EprigfbesApaut6i5u+gKc0ei=Q46ci3sN=rmtAzK1pw@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAFvE7K5aX9D9_mXGqhgNCmYUPjHK61oHg=sDJxrrbet8V0g5kQ@mail.gmail.com>
 <CAFvE7K7--6boumZg_Wzjb-Xrh_0Mq-rpZ_g5Er9R4e-EC0VYwQ@mail.gmail.com>
 <CAN4+E8EE50fw_vQtrrbtf8wSoAmUwq-PPd4XahF6MKrq_+Axww@mail.gmail.com>
 <CAPJVwBkiJk-qLg=AH9Evh9iJVxEefaDEUA8RUpvHPMhDHy-sKA@mail.gmail.com>
 <CAN4+E8EprigfbesApaut6i5u+gKc0ei=Q46ci3sN=rmtAzK1pw@mail.gmail.com>
Message-ID: <CAPJVwBmg40Ww4DNsPh4LxHmnnoW0wTraApJiCFME8WXYhC8+pA@mail.gmail.com>

On Tue, Jan 26, 2016 at 2:06 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> Virtualenvs have `sys.base_prefix`, so perhaps they could just check whether
> the key exists for their parent environment?

I think only shiny-new "venvs" have this, not old-school virtualenvs:

~$ /usr/bin/virtualenv -p /usr/bin/python2.7 tmp-env
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in tmp-env/bin/python2.7
Also creating executable in tmp-env/bin/python
Installing setuptools, pip...done.
~$ tmp-env/bin/python
Python 2.7.11 (default, Dec  9 2015, 00:29:25)
[GCC 5.3.1 20151205] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.base_prefix
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'base_prefix'

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From donald at stufft.io  Tue Jan 26 17:23:14 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 26 Jan 2016 17:23:14 -0500
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBmg40Ww4DNsPh4LxHmnnoW0wTraApJiCFME8WXYhC8+pA@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAFvE7K5aX9D9_mXGqhgNCmYUPjHK61oHg=sDJxrrbet8V0g5kQ@mail.gmail.com>
 <CAFvE7K7--6boumZg_Wzjb-Xrh_0Mq-rpZ_g5Er9R4e-EC0VYwQ@mail.gmail.com>
 <CAN4+E8EE50fw_vQtrrbtf8wSoAmUwq-PPd4XahF6MKrq_+Axww@mail.gmail.com>
 <CAPJVwBkiJk-qLg=AH9Evh9iJVxEefaDEUA8RUpvHPMhDHy-sKA@mail.gmail.com>
 <CAN4+E8EprigfbesApaut6i5u+gKc0ei=Q46ci3sN=rmtAzK1pw@mail.gmail.com>
 <CAPJVwBmg40Ww4DNsPh4LxHmnnoW0wTraApJiCFME8WXYhC8+pA@mail.gmail.com>
Message-ID: <35C31531-F7C3-43DF-AD95-FFEBC6A1E7D2@stufft.io>

> 
> On Jan 26, 2016, at 5:13 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> On Tue, Jan 26, 2016 at 2:06 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
>> Virtualenvs have `sys.base_prefix`, so perhaps they could just check whether
>> the key exists for their parent environment?
> 
> I think only shiny-new "venvs" have this, not old-school virtualenvs:


virtualenv?s have real_prefix.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/b7c59ed9/attachment.sig>

From njs at pobox.com  Tue Jan 26 17:36:01 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 14:36:01 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
Message-ID: <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>

On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> I agree that this is an important detail. I generally use machines that have
> many different Python interpreters installed (some distro-provided and
> others in my home directory), and can easily imagine wanting them to have
> different behavior w.r.t. manylinux1 wheels.
>
> Perhaps the option could be put in site.py, or somewhere in
> lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution here
> is.

On further thought, the site.py solution has sorta grown on me...
basically the PEP text would look something like:

"If there exists an attribute sys._manylinux1_compatible, then
bool(sys._manylinux1_compatible) determines whether the given
interpreter should be considered manylinux1-compatible. By default,
upstream Python builds will not provide any variable with this name,
but a distributor may create it if they choose (e.g. from
sitecustomize.py)."

It's not exactly pretty, but it's kinda elegant: it neatly solves the
problem of binding the configuration to an individual python
environment, allows it to be set from site.py or sitecustomize.py or
from a user's PYTHONSTARTUP or usercustomize.py or even by a local
patch to the interpreter, it's naturally inherited across
virtualenv/venvs, can be checked in very little code, and can be
specified in very few words.

I guess we can bikeshed about whether 'sys' is the appropriate place :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From xav.fernandez at gmail.com  Tue Jan 26 18:03:33 2016
From: xav.fernandez at gmail.com (Xavier Fernandez)
Date: Wed, 27 Jan 2016 00:03:33 +0100
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
Message-ID: <CAA4OSDD9WtGU-nJCJtkwyephWaQJ8L9FZuxRu5nv6YvkjMo5-Q@mail.gmail.com>

Hello,

The site.py solution seems appropriate but I'd be more in favor of a
`sys.supported_platform_tags =['manylinux1']` attribute, with an order of
preferences to ease the wheel selection when several such tags will be
added.

Regards,
Xavier
Le 26 janv. 2016 23:36, "Nathaniel Smith" <njs at pobox.com> a ?crit :

> On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
> wrote:
> > I agree that this is an important detail. I generally use machines that
> have
> > many different Python interpreters installed (some distro-provided and
> > others in my home directory), and can easily imagine wanting them to have
> > different behavior w.r.t. manylinux1 wheels.
> >
> > Perhaps the option could be put in site.py, or somewhere in
> > lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution here
> > is.
>
> On further thought, the site.py solution has sorta grown on me...
> basically the PEP text would look something like:
>
> "If there exists an attribute sys._manylinux1_compatible, then
> bool(sys._manylinux1_compatible) determines whether the given
> interpreter should be considered manylinux1-compatible. By default,
> upstream Python builds will not provide any variable with this name,
> but a distributor may create it if they choose (e.g. from
> sitecustomize.py)."
>
> It's not exactly pretty, but it's kinda elegant: it neatly solves the
> problem of binding the configuration to an individual python
> environment, allows it to be set from site.py or sitecustomize.py or
> from a user's PYTHONSTARTUP or usercustomize.py or even by a local
> patch to the interpreter, it's naturally inherited across
> virtualenv/venvs, can be checked in very little code, and can be
> specified in very few words.
>
> I guess we can bikeshed about whether 'sys' is the appropriate place :-)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/145652de/attachment-0001.html>

From njs at pobox.com  Tue Jan 26 18:41:55 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 15:41:55 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAA4OSDD9WtGU-nJCJtkwyephWaQJ8L9FZuxRu5nv6YvkjMo5-Q@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CAA4OSDD9WtGU-nJCJtkwyephWaQJ8L9FZuxRu5nv6YvkjMo5-Q@mail.gmail.com>
Message-ID: <CAPJVwBmh5S5i7rhtNaLBAskgKeWYD36Rih=U7NZPYduVmGdMYA@mail.gmail.com>

Hi Xavier,

On Tue, Jan 26, 2016 at 3:03 PM, Xavier Fernandez
<xav.fernandez at gmail.com> wrote:
> Hello,
>
> The site.py solution seems appropriate but I'd be more in favor of a
> `sys.supported_platform_tags =['manylinux1']` attribute, with an order of
> preferences to ease the wheel selection when several such tags will be
> added.

I talked a little about this here:
    https://mail.python.org/pipermail/distutils-sig/2016-January/028049.html

Basically my only concern is that we want some way to distinguish
between three states: "this interpreter definitely supports
manylinux1", "this interpreter definitely doesn't support manylinux1",
and "we don't know whether this interpreter supports manylinux1" (and
pip should fall back on the heuristics described in the PEP). The most
straightforward way of implementing your suggestion would be to check

    "manylinux1" in sys.supported_platform_tags

but this only gives us two possible outcomes. If it's not in there,
then we don't know whether that's because it's known not to work, or
because it's been left unspecified. (And similar issues will
presumably arise in the future for manylinux2, etc.)

I guess a simple extension would be:

    sys.supported_platform_tags = ["manylinux2_x86_64"]
    sys.unsupported_platform_tags = set(["manylinux1_x86_64"])

where the former gives a preferred ordering on known-supported tags,
and the latter gives a set of tags that definitely don't work, and for
any tags that aren't mentioned in either place then pip gets to figure
out for itself what it wants to do. Of course this still leaves the
problem of deciding where in the preference ordering these implicit
tags should be placed...

All this may be a bit over-engineered / YAGNI for now, given that for
now the exhaustive set of platform tags and their preference ordering
will be

  manylinux1_x86_64 > linux_x86_64

(or ditto for i386). :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Tue Jan 26 18:56:43 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 15:56:43 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
Message-ID: <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>

On Tue, Jan 26, 2016 at 11:47 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Mon, Jan 25, 2016 at 10:49 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
> wrote:
>>
>> On Mon, Jan 25, 2016 at 10:29 PM, Chris Barker - NOAA Federal
>> <chris.barker at noaa.gov> wrote:
>>>
>>> Given that we're starting now ( not a year or two ago) and it'll take
>>> a while for it to really catch on, we should go CentOS 6 ( or
>>> equivalent ) now?
>>>
>>> CentOS5 was released in 2007! That is a pretty long time in computing.
>>
>>
>> I understand the concern, but I think we should follow the lead of the
>> other projects
>> that have been doing portable linux binaries (holy build box, traveling
>> ruby, portable-pypy,
>> firefox, enthought, continuum) for some time, all based on CentOS 5.
>
>
> That's the point -- they have been doing it for some time -- some time ago,
> you really wouls want a version that old.
>
> The question is -- how many systems are there in teh wild now that are older
> than CentOS6? I have no idea how to even find out. But if we're starting
> somethign new, why start with what was appropriate 2+ years ago?

Well, the people who know what they're doing are still recommending
CentOS 5 today, and we don't know what we're doing :-).

Transitioning to a CentOS6-based manylinux2 shouldn't be a huge
problem -- basically it just requires a pip release + a tweak to pypi
to allow them, and then projects will probably want to provide
manylinux1 and manylinux2 wheels alongside each other for 6 months or
a year to give people a chance to upgrade their pip.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From chris.barker at noaa.gov  Tue Jan 26 19:02:14 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 26 Jan 2016 16:02:14 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
Message-ID: <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>

On Tue, Jan 26, 2016 at 3:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
> Well, the people who know what they're doing are still recommending
> CentOS 5 today, and we don't know what we're doing :-).
>

well, yes, there is that. I sure don't. But the threshold for changing is
higher than for starting fresh.

Transitioning to a CentOS6-based manylinux2 shouldn't be a huge
> problem --


would CentOS5-based wheels run just fine on a centOS-6 based system ?
Alongside CEntOS6-based wheels?

If so, then I guess it's no biggie -- the safe bet is better.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/4c1d41b7/attachment.html>

From njs at pobox.com  Tue Jan 26 19:04:54 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 16:04:54 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
Message-ID: <CAPJVwB=CY1+BFeRgxADKJov6N0gMFvGu5TgLQur5FgybEAcy2Q@mail.gmail.com>

On Tue, Jan 26, 2016 at 4:02 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Tue, Jan 26, 2016 at 3:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Well, the people who know what they're doing are still recommending
>> CentOS 5 today, and we don't know what we're doing :-).
>
>
> well, yes, there is that. I sure don't. But the threshold for changing is
> higher than for starting fresh.
>
>> Transitioning to a CentOS6-based manylinux2 shouldn't be a huge
>> problem --
>
>
> would CentOS5-based wheels run just fine on a centOS-6 based system ?
> Alongside CEntOS6-based wheels?

Yes, the whole idea is that CentOS5-based wheels will run basically
everywhere :-)

> If so, then I guess it's no biggie -- the safe bet is better.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From donald at stufft.io  Tue Jan 26 20:08:40 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 26 Jan 2016 20:08:40 -0500
Subject: [Distutils] PEP: Build system abstraction for pip/conda etc
Message-ID: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>

As many know there has been an effort to get a generalized interface that a build system can implement so that we can break the hard dependency on setuptools that Python packaging currently has. After a lot of deliberation and threads back and forth (as well as video meetings!) I think that the version of the PEP that is currently on the PR in https://github.com/pypa/interoperability-peps/pull/54 looks like it?s generally the right thing to move forward with. I made a few little comments but overall I think it?s there and we?re ready to move forward on trying to get a reference implementation done that can validate the decisions made in that PEP (and then, hopefully finalize the PEP and merge those implementations).

So many thanks to everyone involved in hammering this out thus far :) I?m nervous but excited about the possibility of making setuptools just another build system.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/6fdcec88/attachment.sig>

From robertc at robertcollins.net  Tue Jan 26 22:19:22 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Wed, 27 Jan 2016 16:19:22 +1300
Subject: [Distutils] PEP: Build system abstraction for pip/conda etc
In-Reply-To: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
References: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
Message-ID: <CAJ3HoZ3+=H6yW47PTaYJP1pWo3yB9j-864zfQNvQwaOJ84raPw@mail.gmail.com>

Cool - thank you.

Right now it looks like there is one edit (add PATH as a variable folk
can depend on being set) and one undecided (metadata file format).

I've pushed a shim that permits having a setup.py for non-setuptools
projects using solely the abstract interface to
https://pypi.python.org/pypi/setuptools_shim and
https://github.com/rbtcollins/setuptools_shim

It depends on https://github.com/pypa/packaging/pull/50 - the test
suite is hardcoded to locate that in ../packaging at the moment - but
I believe that that will be merging soon, so it should be generally
usable from there on out.

I'd love it if someone put together a flit implementation of the
abstract build system and checked that the shim does let pip work :).
Theory and practice and all that.

-Rob


On 27 January 2016 at 14:08, Donald Stufft <donald at stufft.io> wrote:
> As many know there has been an effort to get a generalized interface that a build system can implement so that we can break the hard dependency on setuptools that Python packaging currently has. After a lot of deliberation and threads back and forth (as well as video meetings!) I think that the version of the PEP that is currently on the PR in https://github.com/pypa/interoperability-peps/pull/54 looks like it?s generally the right thing to move forward with. I made a few little comments but overall I think it?s there and we?re ready to move forward on trying to get a reference implementation done that can validate the decisions made in that PEP (and then, hopefully finalize the PEP and merge those implementations).
>
> So many thanks to everyone involved in hammering this out thus far :) I?m nervous but excited about the possibility of making setuptools just another build system.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud

From njs at pobox.com  Tue Jan 26 22:37:22 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 19:37:22 -0800
Subject: [Distutils] PEP: Build system abstraction for pip/conda etc
In-Reply-To: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
References: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
Message-ID: <CAPJVwBmi79jsC246q=Cy6Owa-wJhP7g2vAf6pPYBCTsc58N-gw@mail.gmail.com>

On Tue, Jan 26, 2016 at 5:08 PM, Donald Stufft <donald at stufft.io> wrote:
> As many know there has been an effort to get a generalized interface that
a build system can implement so that we can break the hard dependency on
setuptools that Python packaging currently has. After a lot of deliberation
and threads back and forth (as well as video meetings!) I think that the
version of the PEP that is currently on the PR in
https://github.com/pypa/interoperability-peps/pull/54 looks like it?s
generally the right thing to move forward with. I made a few little
comments but overall I think it?s there and we?re ready to move forward on
trying to get a reference implementation done that can validate the
decisions made in that PEP (and then, hopefully finalize the PEP and merge
those implementations).
>
> So many thanks to everyone involved in hammering this out thus far :) I?m
nervous but excited about the possibility of making setuptools just another
build system.

AFAICR at the time the set of public threads petered out (I guess because
Robert and I both got busy with other things), there were several
fundamental disagreements still unresolved:

- You and Paul were strongly in favor of splitting up "working directories"
and "sdists"; Robert and I didn't like the idea. The argument in favor is
that we have to support working directory -> sdist and sdist -> wheel, so
adding working directory -> wheel as an additional pathway creates larger
test matrices and more opportunities for things to go wrong; the argument
against is that this requires a new sdist format definition (currently it's
basically "a zipped up working dir"), and it breaks like incremental
rebuilds, which are a critical feature for developer adoption.

- Robert was in favor of a command-line based interface; I favored a hook
based interface. I won't try to summarize the disagreement here b/c I'm
unable to articulate any advantages of the command-line based interface :-)

- Robert was in favor of preserving the current 'setup.py develop'
semantics as-is, and in particular keeping the current behavior where it's
the build system's job to pick which directory the .egg-link file should be
installed into and then do the actual work; I favored an interface where
the build system says "here's the path to a directory with the package and
metadata", and then the installation tool (pip) picks the location and
installs the .egg-link file (or whatever) pointing to that directory. The
argument in favor of the current semantics is that hey, we need to get this
show on the road, and this will require less code added to pip. The
information is against is that the current design gets the separation of
concerns wrong, impossible to implement without relying on unstandardized
features like egg-link, and also I don't think it actually works right
currently (e.g. if the user supplies --user or --root to the pip install -e
command, then what happens? But I could be wrong -- haven't had a chance to
fully reverse engineer pip/setuptools interaction here).

My personal feeling is that on the first two points my position is
objectively correct ;-), while the third is more confusing and more of a
judgement call.

I haven't looked at the current version of the PEP, so the above may be out
of date, but I think it's accurate WRT what's been discussed on the mailing
list. Have there been some private video meetings where all these issues
got hashed out somehow, or...?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160126/b5723a24/attachment.html>

From robertc at robertcollins.net  Wed Jan 27 00:39:25 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Wed, 27 Jan 2016 18:39:25 +1300
Subject: [Distutils] PEP: Build system abstraction for pip/conda etc
In-Reply-To: <CAPJVwBmi79jsC246q=Cy6Owa-wJhP7g2vAf6pPYBCTsc58N-gw@mail.gmail.com>
References: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
 <CAPJVwBmi79jsC246q=Cy6Owa-wJhP7g2vAf6pPYBCTsc58N-gw@mail.gmail.com>
Message-ID: <CAJ3HoZ2qqtJ_QiYR5OMkU2i8tFx=DteKimbfunT94Cf8+A-BBQ@mail.gmail.com>

On 27 January 2016 at 16:37, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Jan 26, 2016 at 5:08 PM, Donald Stufft <donald at stufft.io> wrote:
>> As many know there has been an effort to get a generalized interface that
>> a build system can implement so that we can break the hard dependency on
>> setuptools that Python packaging currently has. After a lot of deliberation
>> and threads back and forth (as well as video meetings!) I think that the
>> version of the PEP that is currently on the PR in
>> https://github.com/pypa/interoperability-peps/pull/54 looks like it?s
>> generally the right thing to move forward with. I made a few little comments
>> but overall I think it?s there and we?re ready to move forward on trying to
>> get a reference implementation done that can validate the decisions made in
>> that PEP (and then, hopefully finalize the PEP and merge those
>> implementations).
>>
>> So many thanks to everyone involved in hammering this out thus far :) I?m
>> nervous but excited about the possibility of making setuptools just another
>> build system.
>
> AFAICR at the time the set of public threads petered out (I guess because
> Robert and I both got busy with other things), there were several
> fundamental disagreements still unresolved:

There were two pep drafts in flight, but I would hesitate to call it a
fundamental disagreement :).

> - You and Paul were strongly in favor of splitting up "working directories"
> and "sdists"; Robert and I didn't like the idea. The argument in favor is
> that we have to support working directory -> sdist and sdist -> wheel, so
> adding working directory -> wheel as an additional pathway creates larger
> test matrices and more opportunities for things to go wrong; the argument
> against is that this requires a new sdist format definition (currently it's
> basically "a zipped up working dir"), and it breaks like incremental
> rebuilds, which are a critical feature for developer adoption.

That was something that occurred in the rabbit-hole discussion of your
first draft; The PR Donald is proposing to accept is I think the
fourth major iteration? Two from you, two from me - building on
feedback each time. While I don't think Donalds position here has
changed, the draft in question doesn't alter any semantics of anything
- it captures a subset of the semantics such that the existing
behaviour of pip can be modelled on the resulting interface. In
particular, for this question, it retains 'develop', which is the
primary developer mode in use today: it has no implications for or
against incremental builds - it doesn't alter pips copy-before-build
behaviour, which pip already has for non-develop installs. E.g. its
moot; we can debate that further - and i'm sure we shall - but it has
no impact on the interface.

> - Robert was in favor of a command-line based interface; I favored a hook
> based interface. I won't try to summarize the disagreement here b/c I'm
> unable to articulate any advantages of the command-line based interface :-)

it has the fairly large advantage that the developers of pip are in
favour of it - as the folk that will have to deal with and debug what
goes on, thats more than sufficient for me.

> - Robert was in favor of preserving the current 'setup.py develop' semantics
> as-is, and in particular keeping the current behavior where it's the build
> system's job to pick which directory the .egg-link file should be installed
> into and then do the actual work; I favored an interface where the build
> system says "here's the path to a directory with the package and metadata",
> and then the installation tool (pip) picks the location and installs the
> .egg-link file (or whatever) pointing to that directory. The argument in
> favor of the current semantics is that hey, we need to get this show on the
> road, and this will require less code added to pip. The information is
> against is that the current design gets the separation of concerns wrong,
> impossible to implement without relying on unstandardized features like
> egg-link, and also I don't think it actually works right currently (e.g. if
> the user supplies --user or --root to the pip install -e command, then what
> happens? But I could be wrong -- haven't had a chance to fully reverse
> engineer pip/setuptools interaction here).

That feels like an exaggeration of my position. I'd love to evolve
those semantics. But since such an evolution is a hard API break
across everything, I don't want to make fixing the much more
significant problems of setup_requires and having to use setuptools
conditional on that other thing: its a massive adoption hurdle I
suspect.

> My personal feeling is that on the first two points my position is
> objectively correct ;-), while the third is more confusing and more of a
> judgement call.
>
> I haven't looked at the current version of the PEP, so the above may be out
> of date, but I think it's accurate WRT what's been discussed on the mailing
> list. Have there been some private video meetings where all these issues got
> hashed out somehow, or...?

No; there was the one video call, which was advertised here, and then
the subsequent drafts. The PR hasn't been changed in a couple of
months: in that time we've had Xmas, various work things, hacking
older pip version compat stuff into pbr, and me nagging Donald a lot
to get a review of the PR :).

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud

From njs at pobox.com  Wed Jan 27 02:30:18 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 26 Jan 2016 23:30:18 -0800
Subject: [Distutils] PEP: Build system abstraction for pip/conda etc
In-Reply-To: <CAJ3HoZ2qqtJ_QiYR5OMkU2i8tFx=DteKimbfunT94Cf8+A-BBQ@mail.gmail.com>
References: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
 <CAPJVwBmi79jsC246q=Cy6Owa-wJhP7g2vAf6pPYBCTsc58N-gw@mail.gmail.com>
 <CAJ3HoZ2qqtJ_QiYR5OMkU2i8tFx=DteKimbfunT94Cf8+A-BBQ@mail.gmail.com>
Message-ID: <CAPJVwBmru_Pr9mm6j1WU02aVgS8U14WS4uP1PwgRJTM4JwHyYg@mail.gmail.com>

On Tue, Jan 26, 2016 at 9:39 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On 27 January 2016 at 16:37, Nathaniel Smith <njs at pobox.com> wrote:
>> On Tue, Jan 26, 2016 at 5:08 PM, Donald Stufft <donald at stufft.io> wrote:
>>> As many know there has been an effort to get a generalized interface that
>>> a build system can implement so that we can break the hard dependency on
>>> setuptools that Python packaging currently has. After a lot of deliberation
>>> and threads back and forth (as well as video meetings!) I think that the
>>> version of the PEP that is currently on the PR in
>>> https://github.com/pypa/interoperability-peps/pull/54 looks like it?s
>>> generally the right thing to move forward with. I made a few little comments
>>> but overall I think it?s there and we?re ready to move forward on trying to
>>> get a reference implementation done that can validate the decisions made in
>>> that PEP (and then, hopefully finalize the PEP and merge those
>>> implementations).
>>>
>>> So many thanks to everyone involved in hammering this out thus far :) I?m
>>> nervous but excited about the possibility of making setuptools just another
>>> build system.
>>
>> AFAICR at the time the set of public threads petered out (I guess because
>> Robert and I both got busy with other things), there were several
>> fundamental disagreements still unresolved:
>
> There were two pep drafts in flight, but I would hesitate to call it a
> fundamental disagreement :).

Yeah, that may be overstating it :-). I guess I mean something like
"basic design issues that people were still arguing about". (As
compared to stuff like "what file format should we use for the config
file" where it's worth talking about but fundamentally it just doesn't
really matter.)

>> - You and Paul were strongly in favor of splitting up "working directories"
>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>> that we have to support working directory -> sdist and sdist -> wheel, so
>> adding working directory -> wheel as an additional pathway creates larger
>> test matrices and more opportunities for things to go wrong; the argument
>> against is that this requires a new sdist format definition (currently it's
>> basically "a zipped up working dir"), and it breaks like incremental
>> rebuilds, which are a critical feature for developer adoption.
>
> That was something that occurred in the rabbit-hole discussion of your
> first draft; The PR Donald is proposing to accept is I think the
> fourth major iteration? Two from you, two from me - building on
> feedback each time. While I don't think Donalds position here has
> changed, the draft in question doesn't alter any semantics of anything
> - it captures a subset of the semantics such that the existing
> behaviour of pip can be modelled on the resulting interface. In
> particular, for this question, it retains 'develop', which is the
> primary developer mode in use today: it has no implications for or
> against incremental builds - it doesn't alter pips copy-before-build
> behaviour, which pip already has for non-develop installs. E.g. its
> moot; we can debate that further - and i'm sure we shall - but it has
> no impact on the interface.

I'm pretty sure there was another big subthread about this just before
things petered out, but yeah, fair enough. And as far as I'm concerned
there's only one possible answer here since if pip insists on
copy-before-build then all the projects I'm involved in will simply
refuse to use pip for building, so I certainly don't object to
anything in your draft in this regard :-).

>> - Robert was in favor of a command-line based interface; I favored a hook
>> based interface. I won't try to summarize the disagreement here b/c I'm
>> unable to articulate any advantages of the command-line based interface :-)
>
> it has the fairly large advantage that the developers of pip are in
> favour of it - as the folk that will have to deal with and debug what
> goes on, thats more than sufficient for me.

Can you link to where they said this?

>> - Robert was in favor of preserving the current 'setup.py develop' semantics
>> as-is, and in particular keeping the current behavior where it's the build
>> system's job to pick which directory the .egg-link file should be installed
>> into and then do the actual work; I favored an interface where the build
>> system says "here's the path to a directory with the package and metadata",
>> and then the installation tool (pip) picks the location and installs the
>> .egg-link file (or whatever) pointing to that directory. The argument in
>> favor of the current semantics is that hey, we need to get this show on the
>> road, and this will require less code added to pip. The information is
>> against is that the current design gets the separation of concerns wrong,
>> impossible to implement without relying on unstandardized features like
>> egg-link, and also I don't think it actually works right currently (e.g. if
>> the user supplies --user or --root to the pip install -e command, then what
>> happens? But I could be wrong -- haven't had a chance to fully reverse
>> engineer pip/setuptools interaction here).
>
> That feels like an exaggeration of my position. I'd love to evolve
> those semantics. But since such an evolution is a hard API break
> across everything, I don't want to make fixing the much more
> significant problems of setup_requires and having to use setuptools
> conditional on that other thing: its a massive adoption hurdle I
> suspect.

Are you concerned that this would make it impossible to do a simple
shim to run old setuptools projects under the new interface? I think
this part is fine: basically for the hook that I propose, a
PEPXX->setup.py shim would implement it by running

  setup.py build_ext -i && setup.py egg_info

which sets up the current directory to be (a) importable, and (b) have
metadata, and then pip would be responsible for installing a .egg-link
pointing to this directory. IIUC this is exactly what 'develop'
currently does, except that it creates the .egg-link itself.

Doh, and of course that was the other substantive disagreement:
whether to mandate egg_info.

Well, at least now it looks like we can forget about that whole debate
about the need to quickly get metadata from scipy sdists, since
apparently scipy will have wheels on all supported platforms before
pip implements either the new build system interface or the resolver
:-). Of course there will still be an issue for more obscure platforms
and for projects whose developers can't be arsed to provide wheels.

[...]
> No; there was the one video call, which was advertised here, and then
> the subsequent drafts. The PR hasn't been changed in a couple of
> months: in that time we've had Xmas, various work things, hacking
> older pip version compat stuff into pbr, and me nagging Donald a lot
> to get a review of the PR :).

Plus all your great work on getting PEP 508 sorted out so it wouldn't
block this build system stuff :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From robertc at robertcollins.net  Wed Jan 27 02:53:32 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Wed, 27 Jan 2016 20:53:32 +1300
Subject: [Distutils] PEP: Build system abstraction for pip/conda etc
In-Reply-To: <CAPJVwBmru_Pr9mm6j1WU02aVgS8U14WS4uP1PwgRJTM4JwHyYg@mail.gmail.com>
References: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
 <CAPJVwBmi79jsC246q=Cy6Owa-wJhP7g2vAf6pPYBCTsc58N-gw@mail.gmail.com>
 <CAJ3HoZ2qqtJ_QiYR5OMkU2i8tFx=DteKimbfunT94Cf8+A-BBQ@mail.gmail.com>
 <CAPJVwBmru_Pr9mm6j1WU02aVgS8U14WS4uP1PwgRJTM4JwHyYg@mail.gmail.com>
Message-ID: <CAJ3HoZ05KvViCZRNmwOk-1yjV4XGpLPDJcFfc0cPiePoksJSag@mail.gmail.com>

On 27 January 2016 at 20:30, Nathaniel Smith <njs at pobox.com> wrote:

>>> - Robert was in favor of a command-line based interface; I favored a hook
>>> based interface. I won't try to summarize the disagreement here b/c I'm
>>> unable to articulate any advantages of the command-line based interface :-)
>>
>> it has the fairly large advantage that the developers of pip are in
>> favour of it - as the folk that will have to deal with and debug what
>> goes on, thats more than sufficient for me.
>
> Can you link to where they said this?

Donald specifically:

https://mail.python.org/pipermail/distutils-sig/2015-November/027727.html
https://mail.python.org/pipermail/distutils-sig/2015-November/027722.html


>>> - Robert was in favor of preserving the current 'setup.py develop' semantics
>>> as-is, and in particular keeping the current behavior where it's the build
>>> system's job to pick which directory the .egg-link file should be installed
>>> into and then do the actual work; I favored an interface where the build
>>> system says "here's the path to a directory with the package and metadata",
>>> and then the installation tool (pip) picks the location and installs the
>>> .egg-link file (or whatever) pointing to that directory. The argument in
>>> favor of the current semantics is that hey, we need to get this show on the
>>> road, and this will require less code added to pip. The information is
>>> against is that the current design gets the separation of concerns wrong,
>>> impossible to implement without relying on unstandardized features like
>>> egg-link, and also I don't think it actually works right currently (e.g. if
>>> the user supplies --user or --root to the pip install -e command, then what
>>> happens? But I could be wrong -- haven't had a chance to fully reverse
>>> engineer pip/setuptools interaction here).
>>
>> That feels like an exaggeration of my position. I'd love to evolve
>> those semantics. But since such an evolution is a hard API break
>> across everything, I don't want to make fixing the much more
>> significant problems of setup_requires and having to use setuptools
>> conditional on that other thing: its a massive adoption hurdle I
>> suspect.
>
> Are you concerned that this would make it impossible to do a simple
> shim to run old setuptools projects under the new interface? I think
> this part is fine: basically for the hook that I propose, a
> PEPXX->setup.py shim would implement it by running
>
>   setup.py build_ext -i && setup.py egg_info


> which sets up the current directory to be (a) importable, and (b) have
> metadata, and then pip would be responsible for installing a .egg-link
> pointing to this directory. IIUC this is exactly what 'develop'
> currently does, except that it creates the .egg-link itself.

And it needs to do command line entry point processing, not that that
is super hard.

My concern is mainly about unknown unknowns. We've been blocked on
lots of incremental improvement because we've been taking really big
bites. I firmly believe we should take small bites: do one thing, do
it well. E.g. PEP 508, which really did just one thing - fixed the
python 2.7.10 < 2.7.9 issue - but even it took weeks of drafting,
testing, and the patches for it are still wending through review into
setuptools, pip and packaging, let along projects further out on the
periphery like pbr.


The one thing I want to accomplish here is enabling alternative build
systems, because it seem(ed) like there was lots of pent up demand for
that; I specifically don't want to reengineer what pip needs from the
build systems, nor what they need to deliver *in this iteration*: the
tech debt we have here is enormous: there is /no/ standard that we can
point at and say 'implement that', because its literally 'whatever pip
calls setup.py with', and that is subject to change whenever someone
puts a clever patch into pip.

My view is that we won't get it right, no matter what. We have so many
things we're still working on getting agreement on in the long view -
like the copy-before-build question; should progress bars be a thing?
What about incremental builds? What about Wheel format 2? ... once we
have a standard, that standard can be evolved both compatibly, and
incompatibly, with a reference baseline we can compare too, rather
than a defacto thing. We probably want to batch up the changes, and
not dribble them out once every few months - but I definitely see more
iterations happening.

> Doh, and of course that was the other substantive disagreement:
> whether to mandate egg_info.

Yeah - note that my PR punts on this - it says PEP 427 METADATA file,
but as we know the contents of that are a snapshot of earlier drafts
of PEP-426, so its really not all that well defined, and before folk
can iterate on that further we'll need to do something about that -
either by a rewrite of PEP 426 into a much more pithy thing now,
without the enterprise feel and the moon-shot aspects, or by issuing a
PEP that specifies exactly what is in METADATA.

> Well, at least now it looks like we can forget about that whole debate
> about the need to quickly get metadata from scipy sdists, since
> apparently scipy will have wheels on all supported platforms before
> pip implements either the new build system interface or the resolver
> :-). Of course there will still be an issue for more obscure platforms
> and for projects whose developers can't be arsed to provide wheels.

There is no install command in the interface; if projects can't
provide wheels, they can't migrate to the new abstract system - it may
be that some projects need an updated wheel spec to be able to
migrate, and it may also be that pip retains the ability to run
setup.py install for unmigrated projects forever (since pypi has deep
history), but from a design perspective, we are explicitly assuming
'can make wheels'. - Thats reasonable. 'making wheels is cheap' is not
so reasonable :).

> [...]
>> No; there was the one video call, which was advertised here, and then
>> the subsequent drafts. The PR hasn't been changed in a couple of
>> months: in that time we've had Xmas, various work things, hacking
>> older pip version compat stuff into pbr, and me nagging Donald a lot
>> to get a review of the PR :).
>
> Plus all your great work on getting PEP 508 sorted out so it wouldn't
> block this build system stuff :-).

Yah - that was a group effort though, I was just the ring leader :0

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud

From p.f.moore at gmail.com  Wed Jan 27 04:24:35 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Jan 2016 09:24:35 +0000
Subject: [Distutils] PEP: Build system abstraction for pip/conda etc
In-Reply-To: <CAJ3HoZ2qqtJ_QiYR5OMkU2i8tFx=DteKimbfunT94Cf8+A-BBQ@mail.gmail.com>
References: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
 <CAPJVwBmi79jsC246q=Cy6Owa-wJhP7g2vAf6pPYBCTsc58N-gw@mail.gmail.com>
 <CAJ3HoZ2qqtJ_QiYR5OMkU2i8tFx=DteKimbfunT94Cf8+A-BBQ@mail.gmail.com>
Message-ID: <CACac1F_2beN_Q_tYN9BAsinyWe2Dg7hvBFaaAOUdkWze8EypuA@mail.gmail.com>

On 27 January 2016 at 05:39, Robert Collins <robertc at robertcollins.net> wrote:
>> - You and Paul were strongly in favor of splitting up "working directories"
>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>> that we have to support working directory -> sdist and sdist -> wheel, so
>> adding working directory -> wheel as an additional pathway creates larger
>> test matrices and more opportunities for things to go wrong; the argument
>> against is that this requires a new sdist format definition (currently it's
>> basically "a zipped up working dir"), and it breaks like incremental
>> rebuilds, which are a critical feature for developer adoption.
>
> That was something that occurred in the rabbit-hole discussion of your
> first draft; The PR Donald is proposing to accept is I think the
> fourth major iteration? Two from you, two from me - building on
> feedback each time. While I don't think Donalds position here has
> changed, the draft in question doesn't alter any semantics of anything
> - it captures a subset of the semantics such that the existing
> behaviour of pip can be modelled on the resulting interface. In
> particular, for this question, it retains 'develop', which is the
> primary developer mode in use today: it has no implications for or
> against incremental builds - it doesn't alter pips copy-before-build
> behaviour, which pip already has for non-develop installs. E.g. its
> moot; we can debate that further - and i'm sure we shall - but it has
> no impact on the interface.

I'm still not clear on what the position is here. The PEP as I read it
still offers no means to ask the build system to produce a sdist, so
I'm concerned as to what will happen here.

In the absence of a "pip sdist" command I can see that you're saying
that we can implement pip's functionality without caring about this.
But fundamentally, people upload sdists and wheels to PyPI. A build
system that doesn't support sdists (which this PEP allows) will
therefore have to publish wheel-only builds to PyPI, and I am very
much against the idea of PyPI hosting "binary only" distributions.

If project foo has no sdist, "pip wheel foo" for a platform where
there's no compatible wheel available will fail, as there's no way for
pip to locate the source.

So can we please revisit the question of whether build systems will be
permitted to refuse to generate sdists? Note that I don't care whether
we formally define a new sdist format, or go with something adhoc, or
whatever. All I care about is that the PEP states that build systems
must support generating a file that can be uploaded to PyPI and used
by pip to build a wheel as described above (not "git clone this
location and do 'pip wheel .'"). I think that making build systems
expose how you make such a file by requiring a "sdist" subcommand is a
reasonable approach, but I'm OK with naming the subcommand
differently.

Thanks,
Paul

PS I agree with Nathaniel, insofar as there were definitely a number
of unresolved points remaining after the various public discussions,
and I don't think it's reasonable to say that any proposal is "ready
for implementation" without a public confirmation that those issues
have been resolved.

From cournape at gmail.com  Wed Jan 27 04:37:52 2016
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 27 Jan 2016 09:37:52 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
Message-ID: <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>

On Wed, Jan 27, 2016 at 12:02 AM, Chris Barker <chris.barker at noaa.gov>
wrote:

> On Tue, Jan 26, 2016 at 3:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Well, the people who know what they're doing are still recommending
>> CentOS 5 today, and we don't know what we're doing :-).
>>
>
> well, yes, there is that. I sure don't. But the threshold for changing is
> higher than for starting fresh.
>
> Transitioning to a CentOS6-based manylinux2 shouldn't be a huge
>> problem --
>
>
> would CentOS5-based wheels run just fine on a centOS-6 based system ?
> Alongside CEntOS6-based wheels?
>
> If so, then I guess it's no biggie -- the safe bet is better.
>

I will make sure to let the manylinux effort know when we decide to move to
Centos6 as the base system.

David


>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/ac362226/attachment.html>

From ncoghlan at gmail.com  Wed Jan 27 06:47:26 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Jan 2016 21:47:26 +1000
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
Message-ID: <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>

On 27 January 2016 at 08:36, Nathaniel Smith <njs at pobox.com> wrote:
> On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
>> I agree that this is an important detail. I generally use machines that have
>> many different Python interpreters installed (some distro-provided and
>> others in my home directory), and can easily imagine wanting them to have
>> different behavior w.r.t. manylinux1 wheels.
>>
>> Perhaps the option could be put in site.py, or somewhere in
>> lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution here
>> is.
>
> On further thought, the site.py solution has sorta grown on me...
> basically the PEP text would look something like:
>
> "If there exists an attribute sys._manylinux1_compatible, then
> bool(sys._manylinux1_compatible) determines whether the given
> interpreter should be considered manylinux1-compatible. By default,
> upstream Python builds will not provide any variable with this name,
> but a distributor may create it if they choose (e.g. from
> sitecustomize.py)."
>
> It's not exactly pretty, but it's kinda elegant: it neatly solves the
> problem of binding the configuration to an individual python
> environment, allows it to be set from site.py or sitecustomize.py or
> from a user's PYTHONSTARTUP or usercustomize.py or even by a local
> patch to the interpreter, it's naturally inherited across
> virtualenv/venvs, can be checked in very little code, and can be
> specified in very few words.
>
> I guess we can bikeshed about whether 'sys' is the appropriate place :-)

I'd prefer to leave standard library modules out of this if we can -
they're a rabbit warren of political complexity, even when upstream
and redistributors agree on what needs to be done (since
redistributors have customers, and having people pay you tends to make
them even *less* forgiving if you change something in a way that
breaks their systems).

However, I do like the idea of defining a *Python* API for accessing
the configuration data as the initial solution. Since nothing in the
standard library needs it, it can just be an ordinary module called
"manylinux".

If "import manylinux" fails, then we're in the "compatibility not
specified" case (or we're not on a Linux system).

If "import manylinux" works, then the module can give details (perhaps
pulled from a config file, perhaps figured out on the fly by querying
the running system, perhaps hardcoded by the OS vendor) on the
compatibility level.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Wed Jan 27 06:38:37 2016
From: donald at stufft.io (Donald Stufft)
Date: Wed, 27 Jan 2016 06:38:37 -0500
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBmh5S5i7rhtNaLBAskgKeWYD36Rih=U7NZPYduVmGdMYA@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CAA4OSDD9WtGU-nJCJtkwyephWaQJ8L9FZuxRu5nv6YvkjMo5-Q@mail.gmail.com>
 <CAPJVwBmh5S5i7rhtNaLBAskgKeWYD36Rih=U7NZPYduVmGdMYA@mail.gmail.com>
Message-ID: <2763BD67-3196-4168-B497-63328B520D61@stufft.io>


> On Jan 26, 2016, at 6:41 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
>    sys.supported_platform_tags = ["manylinux2_x86_64"]
>    sys.unsupported_platform_tags = set(["manylinux1_x86_64"])

Personally, I like this a lot more than a configuration file.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/080c1a71/attachment.sig>

From ncoghlan at gmail.com  Wed Jan 27 06:49:03 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Jan 2016 21:49:03 +1000
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8Hxn7tZ-3wL1gs6riYDKBf-CHGbaiYbfQ3+wRP7mRDfyQ@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAN4+E8Hxn7tZ-3wL1gs6riYDKBf-CHGbaiYbfQ3+wRP7mRDfyQ@mail.gmail.com>
Message-ID: <CADiSq7dHsFcgx_XhFh8=RMnjL0h3smpLKi1UAScB0FzApTG+-g@mail.gmail.com>

On 27 January 2016 at 04:59, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> On Mon, Jan 25, 2016 at 1:51 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
> wrote:
>>
>>
>> HTML version:
>> https://github.com/manylinux/manylinux/blob/master/pep-513.rst
>
> Nick, at some point would it be possible to put update the version in the
> PEPs repo to
> this version?

Done: https://www.python.org/dev/peps/pep-0513/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Jan 27 07:32:57 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Jan 2016 22:32:57 +1000
Subject: [Distutils] Changing the way we use the PEP process (redux)
In-Reply-To: <CADiSq7es0tR_hijUFhOKxq_+8CVRETzUg92qKUKopVH3SP5W9Q@mail.gmail.com>
References: <CADiSq7dTAW3VDn2G2q0HJmHQW65EVqQU42kG137qHtM8dcjWQQ@mail.gmail.com>
 <CADiSq7fPtrQ3nmO5SC0uwgOuf_N_5Po9uJr6W8_Q=A8WC1choQ@mail.gmail.com>
 <B4D1F6AD-EA2A-44DA-A668-6604DB7A5778@stufft.io>
 <CADiSq7es0tR_hijUFhOKxq_+8CVRETzUg92qKUKopVH3SP5W9Q@mail.gmail.com>
Message-ID: <CADiSq7dMYUgk8Opn+RkbJJ1HUnah3zsV5_wT+C+3qky8Xv5V=Q@mail.gmail.com>

On 25 January 2016 at 14:54, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 25 January 2016 at 13:30, Donald Stufft <donald at stufft.io> wrote:
>> Will we start moving the actual specifications into packaging.python.org, or will they stay in the PEP repository? I?m not sure I can tell from your two PRs currently.
>
> The initial PRs are aimed more at documenting the status quo, so the
> current crop of specifications won't move, but may gain
> packaging.python.org annotations explaining how reality differs from
> the nominal description in the PEP (I already did this for the PEP 345
> metadata spec, and to a lesser degree for the PEP 376 installation
> database spec). I figure that's useful even if we *don't* change the
> process, so it's a good place to start.

I've pushed the big green Merge button on these changes now.

Process PR: https://github.com/pypa/pypa.io/pull/14
Live at: https://www.pypa.io/en/latest/specifications/

Specifications PR: https://github.com/pypa/python-packaging-user-guide/pull/213
Live at: https://packaging.python.org/en/latest/specifications/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Jan 27 07:41:02 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Jan 2016 22:41:02 +1000
Subject: [Distutils] PEP: Build system abstraction for pip/conda etc
In-Reply-To: <CAJ3HoZ05KvViCZRNmwOk-1yjV4XGpLPDJcFfc0cPiePoksJSag@mail.gmail.com>
References: <816A15ED-814C-4587-8A56-BED1C96365DD@stufft.io>
 <CAPJVwBmi79jsC246q=Cy6Owa-wJhP7g2vAf6pPYBCTsc58N-gw@mail.gmail.com>
 <CAJ3HoZ2qqtJ_QiYR5OMkU2i8tFx=DteKimbfunT94Cf8+A-BBQ@mail.gmail.com>
 <CAPJVwBmru_Pr9mm6j1WU02aVgS8U14WS4uP1PwgRJTM4JwHyYg@mail.gmail.com>
 <CAJ3HoZ05KvViCZRNmwOk-1yjV4XGpLPDJcFfc0cPiePoksJSag@mail.gmail.com>
Message-ID: <CADiSq7eMDpYKCShJjj552q8MV-_-xWej=BzvEWfDPeafsWcgWQ@mail.gmail.com>

On 27 January 2016 at 17:53, Robert Collins <robertc at robertcollins.net> wrote:
> Yeah - note that my PR punts on this - it says PEP 427 METADATA file,
> but as we know the contents of that are a snapshot of earlier drafts
> of PEP-426, so its really not all that well defined, and before folk
> can iterate on that further we'll need to do something about that -
> either by a rewrite of PEP 426 into a much more pithy thing now,
> without the enterprise feel and the moon-shot aspects, or by issuing a
> PEP that specifies exactly what is in METADATA.

As of about a minute ago, there's a slightly better reference for that
now: https://packaging.python.org/en/latest/specifications/#core-metadata

It gives us something to point to that says that in practice, the
current metadata spec is the file format and field definitions in PEP
345 + Provides-Extra + PEP 440 + PEP 508.

My thread about it didn't garner a lot of attention, so I figured it
made more sense to just go ahead and flip the switch, rather than
trying to encourage more interest.

More generally, it means any "the PEPs don't always match reality"
clarifications can now just go into the PyPUG specs section until such
time as we get around to updating the relevant spec.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Jan 27 07:54:34 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Jan 2016 22:54:34 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7c6LNmaKqFdnDy545xcxaYBtDQk_kEzRv97a_3apgCU_w@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CADiSq7eS0=JpA2a0nQ-sP21abZ7HtoraV5Kf-DjnLKkJsm=hMQ@mail.gmail.com>
 <20160126124411.222ba104@fsol>
 <CADiSq7c6LNmaKqFdnDy545xcxaYBtDQk_kEzRv97a_3apgCU_w@mail.gmail.com>
Message-ID: <CADiSq7eGErYV9CXVOneqS8NvwEbEkjSPuj7_iYppa4+hYjz0_Q@mail.gmail.com>

On 26 January 2016 at 23:17, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 26 January 2016 at 21:44, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Tue, 26 Jan 2016 20:36:26 +1000
>> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>> If I understand the problem correctly, the CentOS 5 gcc toolchain is
>>> old enough that it simply doesn't emit the info libabigail needs in
>>> order to work.
>>
>> If you build on CentOS 5, you certainly want to use the RH developer
>> toolset 2 which gives you a modern-ish toolchain (gcc 4.8.2, IIRC).
>
> Yeah, that's the part I haven't clarified yet - whether it was just
> the base CentOS toolchain that was too old, or devtoolset-2.
>
> If abicompat et al all work with devtoolset-2, then it definitely
> makes sense to stick with the CentOS 5 baseline for now.

I followed this up with the ABI folks, and the problem is that the
elfutils in even DTS 2 is too old to support building libabigail, and
later versions of the developer toolset (3 & 4) don't support being
run on CentOS 5.

However, even if the build system is based on CentOS 5, *compatibility
scanners* like auditwheel can potentially be run on something newer,
so I've asked if it would work to use the older toolchain to build the
binaries, but then run the relevant ABI compatibility checks on the
almost-certainly-newer target distro.

If that's the case, then folks would be able to run a *static*
abicompat check over a virtualenv including pre-built extensions from
PyPI and be alerted to ABI compatibility problems, rather than getting
hard-to-debug segfaults at runtime.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris.barker at noaa.gov  Wed Jan 27 11:37:16 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 27 Jan 2016 08:37:16 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
Message-ID: <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>

On Wed, Jan 27, 2016 at 1:37 AM, David Cournapeau <cournape at gmail.com>
wrote:

>
> I will make sure to let the manylinux effort know when we decide to move
> to Centos6 as the base system.
>

Thanks -- do you have any idea how many of your customers are running
systems that old?

i.e. have you stuck with CentOS5 because of actual customer demand as
opposed to  uncertainty or inertia?


-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/303eb921/attachment.html>

From cournape at gmail.com  Wed Jan 27 12:00:01 2016
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 27 Jan 2016 17:00:01 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
Message-ID: <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>

On Wed, Jan 27, 2016 at 4:37 PM, Chris Barker <chris.barker at noaa.gov> wrote:

>
>
> On Wed, Jan 27, 2016 at 1:37 AM, David Cournapeau <cournape at gmail.com>
> wrote:
>
>>
>> I will make sure to let the manylinux effort know when we decide to move
>> to Centos6 as the base system.
>>
>
> Thanks -- do you have any idea how many of your customers are running
> systems that old?
>
> i.e. have you stuck with CentOS5 because of actual customer demand as
> opposed to  uncertainty or inertia?
>

As mentioned by others before, Centos5 is a good way to ensure we link
against an old glibc (and few other key libraries, mostly X11-related).
That's really the main thing, as in general, you want to depend on the
system as little as possible when deploying binaries on Linux. Centos 6
uses glibc 2.12, which is newer than debian 6 and ubuntu 10.04 versions.
Even if debian 6 is still old, we see it on systems, and ubuntu 10.04 LTS
is definitely still out there in companies, even if officially unsupported.
And unsupported old versions of OS are used much more often than you may
think in enterprise (can't give names but companies anyone has heard of
still rely a lot on windows XP).

So now, one could argue that it is not the community's job to tackle old
OS, and they would be right, but:

1. Updating to e.g. 6 does not help that much, as the basic components
(compiler toolchain) are still old.
2. Updating the toolchain even on centos 5 is quite easy thanks to the
devtoolset effort.

The main argument against using centos 5 is GUI-related components, as the
old fontconfig/glib (the GTK one, not Gnu libc) are a problem. But those
are a tiny minority of what people do with python nowadays, and they
require a lot of work to get right.

David


>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/246a8c38/attachment.html>

From donald at stufft.io  Wed Jan 27 12:18:27 2016
From: donald at stufft.io (Donald Stufft)
Date: Wed, 27 Jan 2016 12:18:27 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
 <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
Message-ID: <435764D0-DA18-44A8-A42E-DB61C9BF11C8@stufft.io>


> On Jan 27, 2016, at 12:00 PM, David Cournapeau <cournape at gmail.com> wrote:
> 
> So now, one could argue that it is not the community's job to tackle old OS, and they would be right, but:

We can make a data driven decision here.

Here is the top 100 *nix OSs that are downloading files from PyPI using a version of pip >= 6.0. Any version of pip older than that and we don?t have insight into what they are (we can get kernel version if they?re using pip >= 1.4) and any installer other than that we don?t get any insight into either.

One problem of course is deciding how representative only people who are using pip >= 6.0 is, though since we can?t get manylinux support into already released versions of pip it may be pretty representative of people who will use this feature (unless this feature causes people to upgrade their pip when they wouldn?t otherwise).

Anyways, here?s the data: https://gist.github.com/dstufft/e1b1fbebb3482362198f <https://gist.github.com/dstufft/e1b1fbebb3482362198f>

It doesn?t matter to me if we use CentOS5 or CentOS6 as our base, but having some information to inform our choices is never a bad thing!

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/bfc80e9f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/bfc80e9f/attachment.sig>

From cournape at gmail.com  Wed Jan 27 12:29:36 2016
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 27 Jan 2016 17:29:36 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <435764D0-DA18-44A8-A42E-DB61C9BF11C8@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
 <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
 <435764D0-DA18-44A8-A42E-DB61C9BF11C8@stufft.io>
Message-ID: <CAGY4rcX8PAVsi16h8TBz21p5RMDSGfFQffK2YeuZnGpr1zs6oA@mail.gmail.com>

On Wed, Jan 27, 2016 at 5:18 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Jan 27, 2016, at 12:00 PM, David Cournapeau <cournape at gmail.com> wrote:
>
> So now, one could argue that it is not the community's job to tackle old
> OS, and they would be right, but:
>
>
> We can make a data driven decision here.
>

I like data-driven decision as well :)
Can you give an approximate of total download to convert this in % ?

David

>
> Here is the top 100 *nix OSs that are downloading files from PyPI using a
> version of pip >= 6.0. Any version of pip older than that and we don?t have
> insight into what they are (we can get kernel version if they?re using pip
> >= 1.4) and any installer other than that we don?t get any insight into
> either.
>
> One problem of course is deciding how representative only people who are
> using pip >= 6.0 is, though since we can?t get manylinux support into
> already released versions of pip it may be pretty representative of people
> who will use this feature (unless this feature causes people to upgrade
> their pip when they wouldn?t otherwise).
>
> Anyways, here?s the data:
> https://gist.github.com/dstufft/e1b1fbebb3482362198f
>
> It doesn?t matter to me if we use CentOS5 or CentOS6 as our base, but
> having some information to inform our choices is never a bad thing!
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/00b08106/attachment-0001.html>

From donald at stufft.io  Wed Jan 27 12:32:17 2016
From: donald at stufft.io (Donald Stufft)
Date: Wed, 27 Jan 2016 12:32:17 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAGY4rcX8PAVsi16h8TBz21p5RMDSGfFQffK2YeuZnGpr1zs6oA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
 <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
 <435764D0-DA18-44A8-A42E-DB61C9BF11C8@stufft.io>
 <CAGY4rcX8PAVsi16h8TBz21p5RMDSGfFQffK2YeuZnGpr1zs6oA@mail.gmail.com>
Message-ID: <6FC24E2C-264F-4366-AA95-0DAEEA771CE5@stufft.io>


> On Jan 27, 2016, at 12:29 PM, David Cournapeau <cournape at gmail.com> wrote:
> 
> 
> 
> On Wed, Jan 27, 2016 at 5:18 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> 
>> On Jan 27, 2016, at 12:00 PM, David Cournapeau <cournape at gmail.com <mailto:cournape at gmail.com>> wrote:
>> 
>> So now, one could argue that it is not the community's job to tackle old OS, and they would be right, but:
> 
> We can make a data driven decision here.
> 
> I like data-driven decision as well :)
> Can you give an approximate of total download to convert this in % ?
> 
> David
> 
> Here is the top 100 *nix OSs that are downloading files from PyPI using a version of pip >= 6.0. Any version of pip older than that and we don?t have insight into what they are (we can get kernel version if they?re using pip >= 1.4) and any installer other than that we don?t get any insight into either.
> 
> One problem of course is deciding how representative only people who are using pip >= 6.0 is, though since we can?t get manylinux support into already released versions of pip it may be pretty representative of people who will use this feature (unless this feature causes people to upgrade their pip when they wouldn?t otherwise).
> 
> Anyways, here?s the data: https://gist.github.com/dstufft/e1b1fbebb3482362198f <https://gist.github.com/dstufft/e1b1fbebb3482362198f>
> 
> It doesn?t matter to me if we use CentOS5 or CentOS6 as our base, but having some information to inform our choices is never a bad thing!
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 


Across all of PyPI? Total Downloads are 119890375 for roughly the same time period (data is continuously streaming in), for only Linux on pip >= 6.0 it?s 40210457.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/f69602aa/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/f69602aa/attachment.sig>

From njs at vorpus.org  Wed Jan 27 12:37:29 2016
From: njs at vorpus.org (Nathaniel Smith)
Date: Wed, 27 Jan 2016 09:37:29 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <435764D0-DA18-44A8-A42E-DB61C9BF11C8@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
 <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
 <435764D0-DA18-44A8-A42E-DB61C9BF11C8@stufft.io>
Message-ID: <CAPJVwBmMmP8DiAVYLAW+t60Mkuk+pxQJxXF9YdkeP-A7CcAf6Q@mail.gmail.com>

On Jan 27, 2016 09:18, "Donald Stufft" <donald at stufft.io> wrote:
>
>
>> On Jan 27, 2016, at 12:00 PM, David Cournapeau <cournape at gmail.com>
wrote:
>>
>> So now, one could argue that it is not the community's job to tackle old
OS, and they would be right, but:
>
>
> We can make a data driven decision here.
>
> Here is the top 100 *nix OSs that are downloading files from PyPI using a
version of pip >= 6.0. Any version of pip older than that and we don?t have
insight into what they are (we can get kernel version if they?re using pip
>= 1.4) and any installer other than that we don?t get any insight into
either.

Is the kernel version table you mentioned trivial to get? I bet it's very
closely correlated with glibc version.

> One problem of course is deciding how representative only people who are
using pip >= 6.0 is, though since we can?t get manylinux support into
already released versions of pip it may be pretty representative of people
who will use this feature (unless this feature causes people to upgrade
their pip when they wouldn?t otherwise).
>
> Anyways, here?s the data:
https://gist.github.com/dstufft/e1b1fbebb3482362198f

One short summary: "wow, there are a ton of Debian 6 users out there". Esp.
considering that even Debian unstable isn't shipping pip 6.0 yet. (Or who
knows, I guess that could just be one very heavy user.)

> It doesn?t matter to me if we use CentOS5 or CentOS6 as our base, but
having some information to inform our choices is never a bad thing!

Indeed!

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/44dcae90/attachment.html>

From njs at pobox.com  Wed Jan 27 12:43:08 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 27 Jan 2016 09:43:08 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
 <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
Message-ID: <CAPJVwB=6_ZsaUM3-WbDBcJBjRsR-wn+8OECh5QR3Q_RDzuC2Dg@mail.gmail.com>

On Jan 27, 2016 09:00, "David Cournapeau" <cournape at gmail.com> wrote:
>
[...]
> The main argument against using centos 5 is GUI-related components, as
the old fontconfig/glib (the GTK one, not Gnu libc) are a problem. But
those are a tiny minority of what people do with python nowadays, and they
require a lot of work to get right.

This is the part that intrigued me :-). Can you elaborate at all on what
kind of problems you've encountered with fontconfig and glib?

(Well, I can guess at one piece of excitement maybe: that glib is not
really vendorable because a process can have only one main loop, and that
lives in glib these days for both gtk and Qt, so any packages doing any GUI
stuff are required to agree to use the same glib version.)

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/ebeef800/attachment-0001.html>

From donald at stufft.io  Wed Jan 27 12:50:41 2016
From: donald at stufft.io (Donald Stufft)
Date: Wed, 27 Jan 2016 12:50:41 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwBmMmP8DiAVYLAW+t60Mkuk+pxQJxXF9YdkeP-A7CcAf6Q@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
 <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
 <435764D0-DA18-44A8-A42E-DB61C9BF11C8@stufft.io>
 <CAPJVwBmMmP8DiAVYLAW+t60Mkuk+pxQJxXF9YdkeP-A7CcAf6Q@mail.gmail.com>
Message-ID: <A8E7BEB8-A6E7-446E-8870-F5F0F48CE251@stufft.io>


> On Jan 27, 2016, at 12:37 PM, Nathaniel Smith <njs at vorpus.org> wrote:
> 
> Is the kernel version table you mentioned trivial to get? I bet it's very closely correlated with glibc version.



https://gist.github.com/dstufft/1eaa826361ac5b755f17 <https://gist.github.com/dstufft/1eaa826361ac5b755f17>

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/667178b3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/667178b3/attachment.sig>

From cournape at gmail.com  Wed Jan 27 13:29:58 2016
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 27 Jan 2016 18:29:58 +0000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwB=6_ZsaUM3-WbDBcJBjRsR-wn+8OECh5QR3Q_RDzuC2Dg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
 <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
 <CAPJVwB=6_ZsaUM3-WbDBcJBjRsR-wn+8OECh5QR3Q_RDzuC2Dg@mail.gmail.com>
Message-ID: <CAGY4rcU5syf=6uLOH6sQseP-m5zokM0G6DKey-hYCMW5Jt0nhw@mail.gmail.com>

On Wed, Jan 27, 2016 at 5:43 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Jan 27, 2016 09:00, "David Cournapeau" <cournape at gmail.com> wrote:
> >
> [...]
> > The main argument against using centos 5 is GUI-related components, as
> the old fontconfig/glib (the GTK one, not Gnu libc) are a problem. But
> those are a tiny minority of what people do with python nowadays, and they
> require a lot of work to get right.
>
> This is the part that intrigued me :-). Can you elaborate at all on what
> kind of problems you've encountered with fontconfig and glib?
>
> (Well, I can guess at one piece of excitement maybe: that glib is not
> really vendorable because a process can have only one main loop, and that
> lives in glib these days for both gtk and Qt, so any packages doing any GUI
> stuff are required to agree to use the same glib version.)
>

So vendoring glib is possible (we actually do it ATM, though we may revert
that). The problem is that now when you load say PyQt w/ Qt linked against
your vendored glib, you get into issues if that glib is higher than the
glib used in the system (that happens through pango IIRC). So if you want
to stay compatible, you need to build an old glib, which is what you were
trying to avoid in the first place. There is no good solution, really

David

> -n
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/55a98422/attachment.html>

From njs at vorpus.org  Wed Jan 27 14:28:34 2016
From: njs at vorpus.org (Nathaniel Smith)
Date: Wed, 27 Jan 2016 11:28:34 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <A8E7BEB8-A6E7-446E-8870-F5F0F48CE251@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CALGmxEKwFL5x8xQr+1iJJdHeK0knQMMcnZvG061369aGu_t9hA@mail.gmail.com>
 <CAPJVwBkjY3n_26siishZ0PH+KwjANMW7CSeMd-Dp5YAspDcrRQ@mail.gmail.com>
 <CALGmxEKm8cd7=N4SSe7LOkBM9sZc8BWoeJ020Mos8aMDLd0Y=w@mail.gmail.com>
 <CAGY4rcX55658HwVGtZUWKc3suS_ZP0HC_Pg16Sgc_936eb1gBg@mail.gmail.com>
 <CALGmxE+w64r4NNrdZntbX2bW2-m+CvOWhpx9i+xgSuzjzngNKQ@mail.gmail.com>
 <CAGY4rcUMSCHjShdPrK-3u677KcG-6cfkxvd7VZhKUEJV6o8x8Q@mail.gmail.com>
 <435764D0-DA18-44A8-A42E-DB61C9BF11C8@stufft.io>
 <CAPJVwBmMmP8DiAVYLAW+t60Mkuk+pxQJxXF9YdkeP-A7CcAf6Q@mail.gmail.com>
 <A8E7BEB8-A6E7-446E-8870-F5F0F48CE251@stufft.io>
Message-ID: <CAPJVwBkLdWrv53-Cvn2oKCqyHW2ETurM8vtGQR-w2gt0qxZxSg@mail.gmail.com>

Tl;dr: Looks like the oldest kernel that makes the top 100 list is 2.6.32,
which is used in both RHEL6 and Debian 6.
On Jan 27, 2016 9:50 AM, "Donald Stufft" <donald at stufft.io> wrote:

>
> On Jan 27, 2016, at 12:37 PM, Nathaniel Smith <njs at vorpus.org> wrote:
>
> Is the kernel version table you mentioned trivial to get? I bet it's very
> closely correlated with glibc version.
>
>
>
> https://gist.github.com/dstufft/1eaa826361ac5b755f17
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/12738709/attachment.html>

From brett at python.org  Wed Jan 27 15:26:18 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Jan 2016 20:26:18 +0000
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAN4+E8Hxn7tZ-3wL1gs6riYDKBf-CHGbaiYbfQ3+wRP7mRDfyQ@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAN4+E8Hxn7tZ-3wL1gs6riYDKBf-CHGbaiYbfQ3+wRP7mRDfyQ@mail.gmail.com>
Message-ID: <CAP1=2W794+4ZaO8FsW1XQRA0z90Ty2xkUw21+XCe6idCh_8PmQ@mail.gmail.com>

On Tue, 26 Jan 2016 at 10:59 Robert T. McGibbon <rmcgibbo at gmail.com> wrote:

> On Mon, Jan 25, 2016 at 1:51 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
> wrote:
>
>>
>> HTML version:
>> https://github.com/manylinux/manylinux/blob/master/pep-513.rst
>>
>
> Nick, at some point would it be possible to put update the version in the
> PEPs repo to
> this version?
>

The easiest way to get a PEP updated for python.org/dev/peps is to email
peps at python.org with a patch containing the update.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/a003209b/attachment-0001.html>

From brett at python.org  Wed Jan 27 15:30:21 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Jan 2016 20:30:21 +0000
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
Message-ID: <CAP1=2W5awJfXDkRFUWupHR28h+YE_+zX75itsQg-3CzO+NJYyA@mail.gmail.com>

On Tue, 26 Jan 2016 at 14:36 Nathaniel Smith <njs at pobox.com> wrote:

> On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
> wrote:
> > I agree that this is an important detail. I generally use machines that
> have
> > many different Python interpreters installed (some distro-provided and
> > others in my home directory), and can easily imagine wanting them to have
> > different behavior w.r.t. manylinux1 wheels.
> >
> > Perhaps the option could be put in site.py, or somewhere in
> > lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution here
> > is.
>
> On further thought, the site.py solution has sorta grown on me...
>

What if someone runs Python with `-S`?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/33af8c8d/attachment.html>

From njs at pobox.com  Wed Jan 27 15:47:52 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 27 Jan 2016 12:47:52 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAP1=2W5awJfXDkRFUWupHR28h+YE_+zX75itsQg-3CzO+NJYyA@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CAP1=2W5awJfXDkRFUWupHR28h+YE_+zX75itsQg-3CzO+NJYyA@mail.gmail.com>
Message-ID: <CAPJVwBn8GMgJ1P0qd9uT1MN_pwJUuMs=6YToD1MQW02Sm9TnAQ@mail.gmail.com>

On Wed, Jan 27, 2016 at 12:30 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Tue, 26 Jan 2016 at 14:36 Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
>> wrote:
>> > I agree that this is an important detail. I generally use machines that
>> have
>> > many different Python interpreters installed (some distro-provided and
>> > others in my home directory), and can easily imagine wanting them to
>> have
>> > different behavior w.r.t. manylinux1 wheels.
>> >
>> > Perhaps the option could be put in site.py, or somewhere in
>> > lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution
>> here
>> > is.
>>
>> On further thought, the site.py solution has sorta grown on me...
>>
>
> What if someone runs Python with `-S`?
>

This would only matter if someone ran 'python -S -m pip', since pip is the
only thing whose behavior would be affected by being able to see/not-see
the special flag. Except of course 'python -S -m pip' doesn't work, because
-S disables access to all non-stdlib packages, including pip (you just get
"No module named pip"). Someone could make it work with sufficient brute
force (e.g. manually manipulating $PYTHONPATH to point to pip), but I think
at that point it can be assumed that they don't want our help anyway...

Also if a distributor is really worried, then they could set the flag via
some mechanism other than modifying site.py/sitecustomize.py -- in the
email you're replying to, site.py was just a metonym for the general idea
:-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org <http://vorpus.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/8a948c61/attachment.html>

From njs at pobox.com  Wed Jan 27 16:36:52 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 27 Jan 2016 13:36:52 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
Message-ID: <CAPJVwBnY_6Wwkj3euT_MJF+OJnHR9h_USpGE7ZuVW4B0eyHC1Q@mail.gmail.com>

On Wed, Jan 27, 2016 at 3:47 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 27 January 2016 at 08:36, Nathaniel Smith <njs at pobox.com> wrote:
> > On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
> wrote:
> >> I agree that this is an important detail. I generally use machines that
> have
> >> many different Python interpreters installed (some distro-provided and
> >> others in my home directory), and can easily imagine wanting them to
> have
> >> different behavior w.r.t. manylinux1 wheels.
> >>
> >> Perhaps the option could be put in site.py, or somewhere in
> >> lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution
> here
> >> is.
> >
> > On further thought, the site.py solution has sorta grown on me...
> > basically the PEP text would look something like:
> >
> > "If there exists an attribute sys._manylinux1_compatible, then
> > bool(sys._manylinux1_compatible) determines whether the given
> > interpreter should be considered manylinux1-compatible. By default,
> > upstream Python builds will not provide any variable with this name,
> > but a distributor may create it if they choose (e.g. from
> > sitecustomize.py)."
> >
> > It's not exactly pretty, but it's kinda elegant: it neatly solves the
> > problem of binding the configuration to an individual python
> > environment, allows it to be set from site.py or sitecustomize.py or
> > from a user's PYTHONSTARTUP or usercustomize.py or even by a local
> > patch to the interpreter, it's naturally inherited across
> > virtualenv/venvs, can be checked in very little code, and can be
> > specified in very few words.
> >
> > I guess we can bikeshed about whether 'sys' is the appropriate place :-)
>
> I'd prefer to leave standard library modules out of this if we can -
> they're a rabbit warren of political complexity, even when upstream
> and redistributors agree on what needs to be done (since
> redistributors have customers, and having people pay you tends to make
> them even *less* forgiving if you change something in a way that
> breaks their systems).
>
> However, I do like the idea of defining a *Python* API for accessing
> the configuration data as the initial solution. Since nothing in the
> standard library needs it, it can just be an ordinary module called
> "manylinux".
>
> If "import manylinux" fails, then we're in the "compatibility not
> specified" case (or we're not on a Linux system).
>
> If "import manylinux" works, then the module can give details (perhaps
> pulled from a config file, perhaps figured out on the fly by querying
> the running system, perhaps hardcoded by the OS vendor) on the
> compatibility level.
>

Fair enough -- how about we use "_manylinux" to emphasize that this is an
internal module (as opposed to e.g. some package you'd get on pypi that
provides manylinux-related functionality)? It doesn't have as many features
as Xavier's suggestion of exporting a generic preference-ordered list
combining manylinux tags, distro-specific tags, etc., but that seems a bit
premature right now and could always be added on top later (_wheel_tags.py
can import _manylinux).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org <http://vorpus.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/bfb09034/attachment-0001.html>

From xav.fernandez at gmail.com  Wed Jan 27 17:47:26 2016
From: xav.fernandez at gmail.com (Xavier Fernandez)
Date: Wed, 27 Jan 2016 23:47:26 +0100
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBnY_6Wwkj3euT_MJF+OJnHR9h_USpGE7ZuVW4B0eyHC1Q@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
 <CAPJVwBnY_6Wwkj3euT_MJF+OJnHR9h_USpGE7ZuVW4B0eyHC1Q@mail.gmail.com>
Message-ID: <CAA4OSDB9SUr51zpt+c4_ZFkyB5eE7i=_imnQa5SqeCGV0BgF_Q@mail.gmail.com>

Well, the way I read the pep is that manylinux1 is a policy and also the
name of a wheel platform tag.
The policy has nothing special except the fact that it is the first one. I
think everyone agrees that if this successfully enables linux wheels, there
is no reason we won't have any other tags, maybe more focused on some
subset of libraries.
I don't think we want to have a _tag_module for each new type of tag.

I personally like:
sys.supported_platform_tags = ["manylinux2_x86_64"]
sys.unsupported_platform_tags = set(["manylinux1_x86_64"])
but the platform module seems even more appropriate than the sys one.

But this still needs to touch the standard library.

So maybe the simplest solution would be to leave this to pip config.
I personally liked the idea of this PR:
https://github.com/pypa/pip/pull/3092 that would gives the possibility from
linux to "pip download lxml --platform=win-amd64" in order to download
win-amd64 compatible wheels.
We could imagine a similar option for "pip install
--platforms=manylinux2,manylinux1 lxml".

pip already looks for its options in several places so maybe the OS
distribution could drop the compatibility information in one of these
places (like /etc/pip.conf...).

-- 
Xavier

On Wed, Jan 27, 2016 at 10:36 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Jan 27, 2016 at 3:47 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 27 January 2016 at 08:36, Nathaniel Smith <njs at pobox.com> wrote:
>> > On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com>
>> wrote:
>> >> I agree that this is an important detail. I generally use machines
>> that have
>> >> many different Python interpreters installed (some distro-provided and
>> >> others in my home directory), and can easily imagine wanting them to
>> have
>> >> different behavior w.r.t. manylinux1 wheels.
>> >>
>> >> Perhaps the option could be put in site.py, or somewhere in
>> >> lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution
>> here
>> >> is.
>> >
>> > On further thought, the site.py solution has sorta grown on me...
>> > basically the PEP text would look something like:
>> >
>> > "If there exists an attribute sys._manylinux1_compatible, then
>> > bool(sys._manylinux1_compatible) determines whether the given
>> > interpreter should be considered manylinux1-compatible. By default,
>> > upstream Python builds will not provide any variable with this name,
>> > but a distributor may create it if they choose (e.g. from
>> > sitecustomize.py)."
>> >
>> > It's not exactly pretty, but it's kinda elegant: it neatly solves the
>> > problem of binding the configuration to an individual python
>> > environment, allows it to be set from site.py or sitecustomize.py or
>> > from a user's PYTHONSTARTUP or usercustomize.py or even by a local
>> > patch to the interpreter, it's naturally inherited across
>> > virtualenv/venvs, can be checked in very little code, and can be
>> > specified in very few words.
>> >
>> > I guess we can bikeshed about whether 'sys' is the appropriate place :-)
>>
>> I'd prefer to leave standard library modules out of this if we can -
>> they're a rabbit warren of political complexity, even when upstream
>> and redistributors agree on what needs to be done (since
>> redistributors have customers, and having people pay you tends to make
>> them even *less* forgiving if you change something in a way that
>> breaks their systems).
>>
>> However, I do like the idea of defining a *Python* API for accessing
>> the configuration data as the initial solution. Since nothing in the
>> standard library needs it, it can just be an ordinary module called
>> "manylinux".
>>
>> If "import manylinux" fails, then we're in the "compatibility not
>> specified" case (or we're not on a Linux system).
>>
>> If "import manylinux" works, then the module can give details (perhaps
>> pulled from a config file, perhaps figured out on the fly by querying
>> the running system, perhaps hardcoded by the OS vendor) on the
>> compatibility level.
>>
>
> Fair enough -- how about we use "_manylinux" to emphasize that this is an
> internal module (as opposed to e.g. some package you'd get on pypi that
> provides manylinux-related functionality)? It doesn't have as many features
> as Xavier's suggestion of exporting a generic preference-ordered list
> combining manylinux tags, distro-specific tags, etc., but that seems a bit
> premature right now and could always be added on top later (_wheel_tags.py
> can import _manylinux).
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org <http://vorpus.org>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160127/e291234f/attachment.html>

From njs at pobox.com  Wed Jan 27 18:19:43 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 27 Jan 2016 15:19:43 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAA4OSDB9SUr51zpt+c4_ZFkyB5eE7i=_imnQa5SqeCGV0BgF_Q@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
 <CAPJVwBnY_6Wwkj3euT_MJF+OJnHR9h_USpGE7ZuVW4B0eyHC1Q@mail.gmail.com>
 <CAA4OSDB9SUr51zpt+c4_ZFkyB5eE7i=_imnQa5SqeCGV0BgF_Q@mail.gmail.com>
Message-ID: <CAPJVwBndL7Ag3sPaNagn1a81cgg6OVpmu+f4EKX0jHuiCn432Q@mail.gmail.com>

On Wed, Jan 27, 2016 at 2:47 PM, Xavier Fernandez
<xav.fernandez at gmail.com> wrote:
>
> Well, the way I read the pep is that manylinux1 is a policy and also the name of a wheel platform tag.
> The policy has nothing special except the fact that it is the first one. I think everyone agrees that if this successfully enables linux wheels, there is no reason we won't have any other tags, maybe more focused on some subset of libraries.
> I don't think we want to have a _tag_module for each new type of tag.
>
> I personally like:
> sys.supported_platform_tags = ["manylinux2_x86_64"]
> sys.unsupported_platform_tags = set(["manylinux1_x86_64"])
> but the platform module seems even more appropriate than the sys one.
>
> But this still needs to touch the standard library.
>
> So maybe the simplest solution would be to leave this to pip config.
> I personally liked the idea of this PR: https://github.com/pypa/pip/pull/3092 that would gives the possibility from linux to "pip download lxml --platform=win-amd64" in order to download win-amd64 compatible wheels.
> We could imagine a similar option for "pip install --platforms=manylinux2,manylinux1 lxml".
>
> pip already looks for its options in several places so maybe the OS distribution could drop the compatibility information in one of these places (like /etc/pip.conf...).

I think we should distinguish between the *information* that a given
python environment is marked as being manylinux1
compatible/incompatible (which should be scoped to an individual
environment and should be controllable by distributions and in
principle is useful to other package managers besides pip), versus the
*policy* that pip applies (which can take this information into
account along with other factors including ad hoc configuration,
user-supplied override options, whatever). Here I'm just trying to
focus on the first part :-).

I also like the idea of having a more central place for configuring
multiple platform tags for distro-specific tags or
library-subset-focused platform tags, defining preference orders, on
them, etc., but I really feel like this is a complex enough problem
that it needs to be its own PEP. And, well, personally I would guess
that there's at least a 50/50 chance that these new and more complex
platform tags will never actually exist. I don't have anything against
them, but given that no-one's done the work yet and that manylinux
will solve the most urgent part of the problem... I'd rather not
over-engineer now for a future that might never come :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Thu Jan 28 02:23:27 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 28 Jan 2016 17:23:27 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7eGErYV9CXVOneqS8NvwEbEkjSPuj7_iYppa4+hYjz0_Q@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALGmxEJuOa+sW40W_58UsH68Of2Hy8eFqw5BcNSQuNS879bGcw@mail.gmail.com>
 <CAGY4rcVMubUKwqxnCKGm6pa6NqGeHW9WnbdTi34p6YpRGWUJHw@mail.gmail.com>
 <-4166805547148809032@unknownmsgid>
 <CAFvE7K65WBAX6-Q9k6XmZPhK5KLq94ceQ28cofxqDC0pVq2+AA@mail.gmail.com>
 <CALGmxE+ijmeiLRzyg1b-R7EEa8rO=9K0CfDA18b-1k5g1NOZ=w@mail.gmail.com>
 <CAN4+E8H2Pnc8WC6mWY=0TrQEOCHWUM5kFaE1WXOZPPqahsWXfg@mail.gmail.com>
 <CADiSq7d_mbyXHyhDHiH3vUD_QETK+AVkgRUHhJBxTqJ1sdsaCw@mail.gmail.com>
 <CAPJVwBk5qWn_d2iD7PNQmD6L9gts-Wm=9pW5dQo9cTCRSJ4sHg@mail.gmail.com>
 <CADiSq7fpBxN6C=w9PaiFu1qiMYM0XOOd7FD4gnLkmG5PSdWwHg@mail.gmail.com>
 <CAPJVwBkm_vmWZ-sBmjXCt1nsCfsLR5Ephifq72DGs5ayFnNRzQ@mail.gmail.com>
 <-4219403055735535576@unknownmsgid>
 <CAN4+E8HXg==B9uR_zVSDZMyohXgbnt0CyRZJ=uqEpUwzHAATPg@mail.gmail.com>
 <CADiSq7eS0=JpA2a0nQ-sP21abZ7HtoraV5Kf-DjnLKkJsm=hMQ@mail.gmail.com>
 <20160126124411.222ba104@fsol>
 <CADiSq7c6LNmaKqFdnDy545xcxaYBtDQk_kEzRv97a_3apgCU_w@mail.gmail.com>
 <CADiSq7eGErYV9CXVOneqS8NvwEbEkjSPuj7_iYppa4+hYjz0_Q@mail.gmail.com>
Message-ID: <CADiSq7ebpUf794RWLzMj5dv71jdcmx5xGKawgWMa9UXuTwK6_A@mail.gmail.com>

On 27 January 2016 at 22:54, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I followed this up with the ABI folks, and the problem is that the
> elfutils in even DTS 2 is too old to support building libabigail, and
> later versions of the developer toolset (3 & 4) don't support being
> run on CentOS 5.
>
> However, even if the build system is based on CentOS 5, *compatibility
> scanners* like auditwheel can potentially be run on something newer,
> so I've asked if it would work to use the older toolchain to build the
> binaries, but then run the relevant ABI compatibility checks on the
> almost-certainly-newer target distro.
>
> If that's the case, then folks would be able to run a *static*
> abicompat check over a virtualenv including pre-built extensions from
> PyPI and be alerted to ABI compatibility problems, rather than getting
> hard-to-debug segfaults at runtime.

Good news! The toolchain that matters for libabigail based
compatibility scans is the one used to run the scan, *not* the one
used to build the binaries.

This means that even though libabigail itself can't be used on CentOS
5, it *can* be used to check if specific binaries built on CentOS 5
are ABI compatible with your current distro, which is what we actually
want people to be able to do.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From njs at pobox.com  Thu Jan 28 02:46:04 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 27 Jan 2016 23:46:04 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
Message-ID: <CAPJVwBnMAPuDQS5adoixPBFVmsPUsHKQnykTB=miZ1XjHNBb8Q@mail.gmail.com>

On Jan 27, 2016 3:47 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
> On 27 January 2016 at 08:36, Nathaniel Smith <njs at pobox.com> wrote:
> > On Mon, Jan 25, 2016 at 8:37 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> >> I agree that this is an important detail. I generally use machines that have
> >> many different Python interpreters installed (some distro-provided and
> >> others in my home directory), and can easily imagine wanting them to have
> >> different behavior w.r.t. manylinux1 wheels.
> >>
> >> Perhaps the option could be put in site.py, or somewhere in
> >> lib/pythonX.Y/configX.Y/? I'm not sure what the appropriate solution here
> >> is.
> >
> > On further thought, the site.py solution has sorta grown on me...
> > basically the PEP text would look something like:
> >
> > "If there exists an attribute sys._manylinux1_compatible, then
> > bool(sys._manylinux1_compatible) determines whether the given
> > interpreter should be considered manylinux1-compatible. By default,
> > upstream Python builds will not provide any variable with this name,
> > but a distributor may create it if they choose (e.g. from
> > sitecustomize.py)."
> >
> > It's not exactly pretty, but it's kinda elegant: it neatly solves the
> > problem of binding the configuration to an individual python
> > environment, allows it to be set from site.py or sitecustomize.py or
> > from a user's PYTHONSTARTUP or usercustomize.py or even by a local
> > patch to the interpreter, it's naturally inherited across
> > virtualenv/venvs, can be checked in very little code, and can be
> > specified in very few words.
> >
> > I guess we can bikeshed about whether 'sys' is the appropriate place :-)
>
> I'd prefer to leave standard library modules out of this if we can -
> they're a rabbit warren of political complexity, even when upstream
> and redistributors agree on what needs to be done (since
> redistributors have customers, and having people pay you tends to make
> them even *less* forgiving if you change something in a way that
> breaks their systems).
>
> However, I do like the idea of defining a *Python* API for accessing
> the configuration data as the initial solution. Since nothing in the
> standard library needs it, it can just be an ordinary module called
> "manylinux".

On further thought, I realized that it actually has to be in the
standard library directory / namespace, and can't live in
site-packages: for the correct semantics it needs to be inherited by
virtualenvs; if it isn't then we'll see confusing rare problems. And
virtualenvs inherit the stdlib, but not the site-packages (in
general).

So... here's some new text with the _manylinux module in it:
https://github.com/manylinux/manylinux/pull/14/files

and for now I've just put in the text that if you're a distributor,
then you should add this to the stdlib instead of treating it like a
regular package. I'm not sure if this is actually better than just
sticking an attribute in sys or something though.

Nick: I think this is the last issue before we're ready for
pronouncement, and you're the one who wants to keep the stdlib out of
it, so... what do you want to do? :-)

-n

From oscar.j.benjamin at gmail.com  Thu Jan 28 05:35:41 2016
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Thu, 28 Jan 2016 10:35:41 +0000
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBnMAPuDQS5adoixPBFVmsPUsHKQnykTB=miZ1XjHNBb8Q@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
 <CAPJVwBnMAPuDQS5adoixPBFVmsPUsHKQnykTB=miZ1XjHNBb8Q@mail.gmail.com>
Message-ID: <CAHVvXxS0ir4TLX7_g1U19uCQo7Rxkg1_vMcmDLbnw7uhgzRtaw@mail.gmail.com>

On 28 January 2016 at 07:46, Nathaniel Smith <njs at pobox.com> wrote:
>
> On further thought, I realized that it actually has to be in the
> standard library directory / namespace, and can't live in
> site-packages: for the correct semantics it needs to be inherited by
> virtualenvs; if it isn't then we'll see confusing rare problems. And
> virtualenvs inherit the stdlib, but not the site-packages (in
> general).

Surely virtualenv can be made to import this information from the
parent environment. It's already virtualenv's job to set up pip etc.
so that you're ready to install things from PyPI.

--
Oscar

From chris.barker at noaa.gov  Thu Jan 28 19:28:20 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 28 Jan 2016 16:28:20 -0800
Subject: [Distutils] How to get pip to really, really,
 I mean it -- rebuild this damn package!
Message-ID: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>

Context:

I'm maintaining a number of conda packages of various packages, some  of
which are mine, some others, some pure python, some extensions, etc.

The way conda build works is you  specify some meta data, and a build
script(s), and conda:

sets up an isolated environment in which to build.
installs the build dependencies
runs teh build script
see's what got installed, and makes a package of it.
(there are complications, but that's the idea)


so what to do in the build script for a python package? the simple anser is:

$PYTHON setup.py install

But then you get those god- awful eggs, or if it's not a setuptools built
package, you don't get the right meta data for pip, etc. to resolve
dependencies.

[NOTE: I do want all the pip compatible meta data, otherwise, you have pip
trying to re-instll stuff, etc if someone does install something with pip,
or pip in editable mode, or...]

so some of us have started doing:

$PYTHON setup.py install --single-version-externally-managed  --record
record.txt

Which mostly seems to work -- though that is a God-awful command line to
remember....

And it fails if the package has a plain old distuitls-based setup.py

so I started going with:

$PYTHON -m pip install ./

and that seemed to work for awhile for me. However, I've been having
problems lately with pip not bulding and re-installing the package. This is
really weird, as the conda build environment is a clean environment, there
really isn't a package already installed.

here is the log:

+ /Users/chris.barker/miniconda2/envs/_build/bin/python -m pip install -v ./

Processing /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3

  Running setup.py (path:/tmp/pip-umxsOD-build/setup.py) egg_info for
package from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3

    Running command python setup.py egg_info

  Source in /tmp/pip-umxsOD-build has version 3.0.3, which satisfies
requirement gsw==3.0.3 from
file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3

  Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from
file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
/Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3

Requirement already satisfied (use --upgrade to upgrade): numpy in
/Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages
(from gsw==3.0.3)

Requirement already satisfied (use --upgrade to upgrade): nose in
/Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages
(from gsw==3.0.3)

Building wheels for collected packages: gsw

  Running setup.py bdist_wheel for gsw ...   Destination directory:
/tmp/tmprPhOYkpip-wheel-

  Running command /Users/chris.barker/miniconda2/envs/_build/bin/python -u
-c "import setuptools,
tokenize;__file__='/tmp/pip-umxsOD-build/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
bdist_wheel -d /tmp/tmprPhOYkpip-wheel- --python-tag cp27

done

  Stored in directory:
/Users/chris.barker/Library/Caches/pip/wheels/51/4e/d7/b4cfa75866df9da00f4e4f8a9c5c35cfacfa9e92c4885ec5c4

  Removing source in /tmp/pip-umxsOD-build

Successfully built gsw

Cleaning up...

You are using pip version 8.0.1, however version 8.0.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

So it seems to think it's already installed -- huh? what? IN any case, it
doesn't install anything. It looks like it's referencing some cache, or
manifest or something outside of the python environment itself. So if I
installed it in a different Anaconda environment, it gets confused here.

(BTW, I can replicate this behavior outside of conda build by creating a
new conda environment by hand, and trying to ues pip to build a package
locally)

So I tried various command-line options:

$PYTHON -m pip install -I -v --upgrade --no-deps ./
but no dice.

I also tried --no-cache-dir -- no change.

So how can I tell pip that I really do want it to bulid and install this
dran package from source, damn it!

Other option -- go back to:

$PYTHON setup.py install --single-version-externally-managed  --record
record.txt
And have to fight with pip only for the non-setuptools packages. Does the
--single-version-externally-managedcommand do ayting different than pip?

Thanks,

-Chris











-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160128/3d02dd30/attachment.html>

From rmcgibbo at gmail.com  Thu Jan 28 20:00:00 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Thu, 28 Jan 2016 17:00:00 -0800
Subject: [Distutils] How to get pip to really, really,
 I mean it -- rebuild this damn package!
In-Reply-To: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
References: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
Message-ID: <CAN4+E8FqOKSJCg=+h=sP5SZQ04FGZOiWK_vuzB6TsOA9DvaY4w@mail.gmail.com>

The pip wheel cache is in ~/Library/Caches/pip/wheels (OS X) or
~/.cache/pip/wheels (Linux). I'm not sure about Windows. You might have
some luck deleting files from there.

-Robert

On Thu, Jan 28, 2016 at 4:28 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> Context:
>
> I'm maintaining a number of conda packages of various packages, some  of
> which are mine, some others, some pure python, some extensions, etc.
>
> The way conda build works is you  specify some meta data, and a build
> script(s), and conda:
>
> sets up an isolated environment in which to build.
> installs the build dependencies
> runs teh build script
> see's what got installed, and makes a package of it.
> (there are complications, but that's the idea)
>
>
> so what to do in the build script for a python package? the simple anser
> is:
>
> $PYTHON setup.py install
>
> But then you get those god- awful eggs, or if it's not a setuptools built
> package, you don't get the right meta data for pip, etc. to resolve
> dependencies.
>
> [NOTE: I do want all the pip compatible meta data, otherwise, you have pip
> trying to re-instll stuff, etc if someone does install something with pip,
> or pip in editable mode, or...]
>
> so some of us have started doing:
>
> $PYTHON setup.py install --single-version-externally-managed  --record
> record.txt
>
> Which mostly seems to work -- though that is a God-awful command line to
> remember....
>
> And it fails if the package has a plain old distuitls-based setup.py
>
> so I started going with:
>
> $PYTHON -m pip install ./
>
> and that seemed to work for awhile for me. However, I've been having
> problems lately with pip not bulding and re-installing the package. This is
> really weird, as the conda build environment is a clean environment, there
> really isn't a package already installed.
>
> here is the log:
>
> + /Users/chris.barker/miniconda2/envs/_build/bin/python -m pip install -v
> ./
>
> Processing /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
>   Running setup.py (path:/tmp/pip-umxsOD-build/setup.py) egg_info for
> package from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
>     Running command python setup.py egg_info
>
>   Source in /tmp/pip-umxsOD-build has version 3.0.3, which satisfies
> requirement gsw==3.0.3 from
> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
>   Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3
> from file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
> Requirement already satisfied (use --upgrade to upgrade): numpy in
> /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages
> (from gsw==3.0.3)
>
> Requirement already satisfied (use --upgrade to upgrade): nose in
> /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages
> (from gsw==3.0.3)
>
> Building wheels for collected packages: gsw
>
>   Running setup.py bdist_wheel for gsw ...   Destination directory:
> /tmp/tmprPhOYkpip-wheel-
>
>   Running command /Users/chris.barker/miniconda2/envs/_build/bin/python -u
> -c "import setuptools,
> tokenize;__file__='/tmp/pip-umxsOD-build/setup.py';exec(compile(getattr(tokenize,
> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
> bdist_wheel -d /tmp/tmprPhOYkpip-wheel- --python-tag cp27
>
> done
>
>   Stored in directory:
> /Users/chris.barker/Library/Caches/pip/wheels/51/4e/d7/b4cfa75866df9da00f4e4f8a9c5c35cfacfa9e92c4885ec5c4
>
>   Removing source in /tmp/pip-umxsOD-build
>
> Successfully built gsw
>
> Cleaning up...
>
> You are using pip version 8.0.1, however version 8.0.2 is available.
>
> You should consider upgrading via the 'pip install --upgrade pip' command.
>
> So it seems to think it's already installed -- huh? what? IN any case, it
> doesn't install anything. It looks like it's referencing some cache, or
> manifest or something outside of the python environment itself. So if I
> installed it in a different Anaconda environment, it gets confused here.
>
> (BTW, I can replicate this behavior outside of conda build by creating a
> new conda environment by hand, and trying to ues pip to build a package
> locally)
>
> So I tried various command-line options:
>
> $PYTHON -m pip install -I -v --upgrade --no-deps ./
> but no dice.
>
> I also tried --no-cache-dir -- no change.
>
> So how can I tell pip that I really do want it to bulid and install this
> dran package from source, damn it!
>
> Other option -- go back to:
>
> $PYTHON setup.py install --single-version-externally-managed  --record
> record.txt
> And have to fight with pip only for the non-setuptools packages. Does the
> --single-version-externally-managedcommand do ayting different than pip?
>
> Thanks,
>
> -Chris
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160128/bb77ddea/attachment-0001.html>

From njs at pobox.com  Thu Jan 28 21:15:27 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 28 Jan 2016 18:15:27 -0800
Subject: [Distutils] How to get pip to really, really,
 I mean it -- rebuild this damn package!
In-Reply-To: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
References: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
Message-ID: <CAPJVwBnRVO17xoxg0iCt5HuZ4vrzh5RjzbizYXDu1pb1WyRkaw@mail.gmail.com>

On Thu, Jan 28, 2016 at 4:28 PM, Chris Barker <chris.barker at noaa.gov> wrote:
[...]
>   Source in /tmp/pip-umxsOD-build has version 3.0.3, which satisfies
> requirement gsw==3.0.3 from
> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
>   Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from
> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3

I think this is saying that pip thinks it has found an
already-installed version of gsw 3.0.3 in sys.path, and that the
directory in your sys.path where it's already installed is

/Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3

I think this means that that directory is (a) in sys.path, and (b)
contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
seems weird and broken.

Do you have "." in your PYTHONPATH or anything like that?

> Requirement already satisfied (use --upgrade to upgrade): numpy in
> /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages (from
> gsw==3.0.3)
>
> Requirement already satisfied (use --upgrade to upgrade): nose in
> /Users/chris.barker/miniconda2/envs/_build/lib/python2.7/site-packages (from
> gsw==3.0.3)
>
> Building wheels for collected packages: gsw
>
>   Running setup.py bdist_wheel for gsw ...   Destination directory:
> /tmp/tmprPhOYkpip-wheel-
>
>   Running command /Users/chris.barker/miniconda2/envs/_build/bin/python -u
> -c "import setuptools,
> tokenize;__file__='/tmp/pip-umxsOD-build/setup.py';exec(compile(getattr(tokenize,
> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
> bdist_wheel -d /tmp/tmprPhOYkpip-wheel- --python-tag cp27

Don't know why it seems to be building a wheel for it, if it already
thinks that it's installed... this is also odd.

> (BTW, I can replicate this behavior outside of conda build by creating a new
> conda environment by hand, and trying to ues pip to build a package locally)
>
> So I tried various command-line options:
>
> $PYTHON -m pip install -I -v --upgrade --no-deps ./
>
> but no dice.
>
> I also tried --no-cache-dir -- no change.

Maybe

$PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./

? Though I'd think that -I would have the same affect as --force-reinstall...

(It doesn't look like the cache dir is your problem here, but you do
probably want to use --no-cache-dir anyway just as good practice, just
because you don't want to accidentally package up a stale version of
the software that got pulled out of your cache instead of the version
you thought you were packaging in the tree in front of you.

Also, I think it's a bug in pip that it caches builds of source trees
-- PyPI can enforce the rule that each (package name, version number)
sdist is unique, but for a work-in-progress VCS checkout it's just not
true that (package name, version number) uniquely identifies a
snapshot of the whole tree. So in something like 'pip install .', then
requirement resolution code should treat this as a special requirement
that it wants *this exact tree*, not just any package that has the
same (package name, version number) as this tree; and the resulting
wheel should not be cached. I don't know if there are any bugs filed
in pip on this...)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Thu Jan 28 22:28:27 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 28 Jan 2016 19:28:27 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAHVvXxS0ir4TLX7_g1U19uCQo7Rxkg1_vMcmDLbnw7uhgzRtaw@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
 <CAPJVwBnMAPuDQS5adoixPBFVmsPUsHKQnykTB=miZ1XjHNBb8Q@mail.gmail.com>
 <CAHVvXxS0ir4TLX7_g1U19uCQo7Rxkg1_vMcmDLbnw7uhgzRtaw@mail.gmail.com>
Message-ID: <CAPJVwBmwvXrX-zJEFgxQSpqLaUyW3aXH-gXy3kZUgrHtZSpv=Q@mail.gmail.com>

On Thu, Jan 28, 2016 at 2:35 AM, Oscar Benjamin
<oscar.j.benjamin at gmail.com> wrote:
> On 28 January 2016 at 07:46, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On further thought, I realized that it actually has to be in the
>> standard library directory / namespace, and can't live in
>> site-packages: for the correct semantics it needs to be inherited by
>> virtualenvs; if it isn't then we'll see confusing rare problems. And
>> virtualenvs inherit the stdlib, but not the site-packages (in
>> general).
>
> Surely virtualenv can be made to import this information from the
> parent environment. It's already virtualenv's job to set up pip etc.
> so that you're ready to install things from PyPI.

This doesn't seem like it would be attractive to distributors to me.
For them, it's just as easy either way to drop a file into
/usr/lib/python3.5/ versus /usr/lib/python3.5/site-packages. But if
you use site-packages/, you have to make sure you also have a patched
virtualenv and pyvenv, and worry about whether your distribution will
ever get used with old versions of virtualenv, etc., whereas if you
drop it into the stdlib directory then it just works robustly
everywhere. I guess we could put some finger-wagging in a PEP telling
them to use site-packages/, patch virtualenv, and eat their
vegetables, but I don't know why they'd listen to us :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From holger at merlinux.eu  Fri Jan 29 06:02:06 2016
From: holger at merlinux.eu (holger krekel)
Date: Fri, 29 Jan 2016 11:02:06 +0000
Subject: [Distutils] new devpi releases (2.6.0) with pip-search/offline mode
 support
Message-ID: <20160129110206.GG3793@merlinux.eu>

This trinity release of devpi, the private packaging and workflow
system, is drop-in compatible to earlier releases and comes with these
improvements:

- support for pip search on the server side which is also configured
  when "devpi use" writes to pip configuration files.

- explicit --offline-mode for devpi-server to avoid trying
  unneccessary and potentially laggy network requests and to
  streamline simple pages to only contain releases that are locally
  cached. thanks Daniel Panteleit for the PR.

- push from root/pypi to other indexes works now.

Docs are to be found as usual at:

    http://doc.devpi.net

This release brought to you mainly by Florian Schulze and me
and a few still unnamed sponsoring companies.  Speaking of which,
if you need support, training, adjustments wrt packaging and
professional testing you may contact us through http://merlinux.eu.

You can also expect devpi-server-3.0 soon, a major new release which
is to bring improvements like generalized mirroring, storage 
backends, speed and internal code cleanups.

best,
holger 


devpi-server-2.6.0 (2016-1-29)
------------------------------

- fix issue262: new experimental option --offline-mode will prevent
  devpi-server from even trying to perform network requests and it
  also strip all non-local release files from the simple index.
  Thanks Daniel Panteleit for the PR.

- fix issue304: mark devpi-server versions older than 2.2.x as incompatible
  and requiring an import/export cycle.

- fix issue296: try to fetch files from master again when requested, if there
  were checksum errors during replication.

- if a user can't be found during authentication (with ``setup.py upload`` for
  example), then the http return code is now 401 instead of 404.

- fix issue293: push from root/pypi to another index is now supported

- fix issue265: ignore HTTP(S) proxies when checking if the server is
                already running.

- Add ``content_type`` route predicate for use by plugins.


devpi-web-2.6.0 (2016-1-29)
---------------------------

- fix issue305: read documentation html files in binary and let BeautifulSoup
                detect the encoding.

- require devpi-server >= 2.6.0

- support for ``pip search`` command on indexes

devpi-client-2.4.0 (2016-1-29)
------------------------------

- fix issue291: transfer file modes with vcs exports.  Thanks Sergey
  Vasilyev for the report.

- new option "--index" for "install", "list", "push", "remove", "upload" and
  "test" which allows to use a different than the current index without using
  "devpi use" before

- set ``index`` in ``[search]`` section of ``pip.cfg`` when writing cfgs, to
  support ``pip search``

From ncoghlan at gmail.com  Fri Jan 29 09:57:15 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 30 Jan 2016 00:57:15 +1000
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CAPJVwBmwvXrX-zJEFgxQSpqLaUyW3aXH-gXy3kZUgrHtZSpv=Q@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
 <CAPJVwBnMAPuDQS5adoixPBFVmsPUsHKQnykTB=miZ1XjHNBb8Q@mail.gmail.com>
 <CAHVvXxS0ir4TLX7_g1U19uCQo7Rxkg1_vMcmDLbnw7uhgzRtaw@mail.gmail.com>
 <CAPJVwBmwvXrX-zJEFgxQSpqLaUyW3aXH-gXy3kZUgrHtZSpv=Q@mail.gmail.com>
Message-ID: <CADiSq7fNGa29aRPtQmfP0L60GWhLhGgzDSVEbbfyahV_+fzwnQ@mail.gmail.com>

On 29 January 2016 at 13:28, Nathaniel Smith <njs at pobox.com> wrote:
> On Thu, Jan 28, 2016 at 2:35 AM, Oscar Benjamin
> <oscar.j.benjamin at gmail.com> wrote:
>> On 28 January 2016 at 07:46, Nathaniel Smith <njs at pobox.com> wrote:
>>>
>>> On further thought, I realized that it actually has to be in the
>>> standard library directory / namespace, and can't live in
>>> site-packages: for the correct semantics it needs to be inherited by
>>> virtualenvs; if it isn't then we'll see confusing rare problems. And
>>> virtualenvs inherit the stdlib, but not the site-packages (in
>>> general).
>>
>> Surely virtualenv can be made to import this information from the
>> parent environment. It's already virtualenv's job to set up pip etc.
>> so that you're ready to install things from PyPI.
>
> This doesn't seem like it would be attractive to distributors to me.
> For them, it's just as easy either way to drop a file into
> /usr/lib/python3.5/ versus /usr/lib/python3.5/site-packages. But if
> you use site-packages/, you have to make sure you also have a patched
> virtualenv and pyvenv, and worry about whether your distribution will
> ever get used with old versions of virtualenv, etc., whereas if you
> drop it into the stdlib directory then it just works robustly
> everywhere. I guess we could put some finger-wagging in a PEP telling
> them to use site-packages/, patch virtualenv, and eat their
> vegetables, but I don't know why they'd listen to us :-).

We'd also be back in "patching the standard library" territory again,
since we'd also have to modify venv.

I like the simple approach Nathaniel now has in the PEP, as that seems
like it should be relatively easy to handle as either modifications
directly to a python package, or as an add-on module (ala installing
LSB support, only much lighter weight). It doesn't scale to lots of
platform tags, but we don't really intend it to - we'd like to avoid
platform tag proliferation if we can, and if we're successful in that,
then we won't need to solve the problem with a lack of scalability.

Cheers,
Nick.

P.S. To provide a bit more context on "Why is a separate file
easier?", the main benefit is that adding a new file simply *cannot*
break the sys or platform modules, while patching them can. That's
still not a guarantee that any given distro will actually provide an
importable "_manylinux" module to indicate compatibility, but given
the fallback glibc check, _manylinux is more a hedge against future
*in*compatibility than it is anything else.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris.barker at noaa.gov  Fri Jan 29 10:48:22 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Fri, 29 Jan 2016 07:48:22 -0800
Subject: [Distutils] How to get pip to really, really,
 I mean it -- rebuild this damn package!
In-Reply-To: <CAPJVwBnRVO17xoxg0iCt5HuZ4vrzh5RjzbizYXDu1pb1WyRkaw@mail.gmail.com>
References: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
 <CAPJVwBnRVO17xoxg0iCt5HuZ4vrzh5RjzbizYXDu1pb1WyRkaw@mail.gmail.com>
Message-ID: <1425275544231181589@unknownmsgid>

>>  Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from
>> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
> I think this is saying that pip thinks it has found an
> already-installed version of gsw 3.0.3 in sys.path, and that the
> directory in your sys.path where it's already installed is
>
> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3

That is the temp dir conda sets up to unpack downloaded files, and do
its work in -- hence the name. I'll look and see what's there. I'm
pretty sure conda build starts out with an empty dir, however. And
that dir should not be on sys.path.

> I think this means that that directory is (a) in sys.path, and (b)
> contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
> seems weird and broken.

Indeed. And I get the same symptoms with a clean environment that I've
set up outside conda build. Though with the same source dir. But with
conda build, it's a fresh unpack of the tarball.

> Do you have "." in your PYTHONPATH or anything like that?

God no!

> Don't know why it seems to be building a wheel for it, if it already
> thinks that it's installed... this is also odd.

Yes it is. But it doesn't install it :-(

>
> $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
>
> ? Though I'd think that -I would have the same affect as --force-reinstall...
>
So did I, and I think I tried --force-reinstall already, but I will again.

> (It doesn't look like the cache dir is your problem here, but you do
> probably want to use --no-cache-dir anyway just as good practice, just
> because you don't want to accidentally package up a stale version of
> the software that got pulled out of your cache instead of the version
> you thought you were packaging in the tree in front of you.

Exactly. Doesn't seem to make a difference, though.

> Also, I think it's a bug in pip that it caches builds of source trees
> -- PyPI can enforce the rule that each (package name, version number)
> sdist is unique, but for a work-in-progress VCS checkout it's just not
> true that (package name, version number) uniquely identifies a
> snapshot of the whole tree. So in something like 'pip install .', then
> requirement resolution code should treat this as a special requirement
> that it wants *this exact tree*, not just any package that has the
> same (package name, version number) as this tree; and the resulting
> wheel should not be cached.

Absolutely! In fact, I'll bet that approach is the source of the
problem here. If not automagically, there should be a flag, at least.

However, what seems to be happening is that pip is looking outside the
current Python environment somewhere to see if this package needs to
be installed. It may be something that works with virtualenv, but
doesn't with conda environments for some reason.

I guess on some level pip simply isn't designed to build and install
from local source :-(

In the end, I'm still confused: does pip install give me anything that:

setup.py install single-version-externally-managed

Doesn't? Other that support for non-setuptools installs, anyway.

CHB


> I don't know if there are any bugs filed
> in pip on this...)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org

From nate at bx.psu.edu  Fri Jan 29 14:30:16 2016
From: nate at bx.psu.edu (Nate Coraor)
Date: Fri, 29 Jan 2016 14:30:16 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7d-4pr-j2bSu=x+yBgNZvVWWdGUYs0TbgY6Xd4qQMvjiA@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CADiSq7d-4pr-j2bSu=x+yBgNZvVWWdGUYs0TbgY6Xd4qQMvjiA@mail.gmail.com>
Message-ID: <CALT861FVLUWiopyCweuGU=E=FgNY3POeJE1paVvyVPYE_FjPmQ@mail.gmail.com>

On Fri, Jan 22, 2016 at 5:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 22 January 2016 at 19:33, M.-A. Lemburg <mal at egenix.com> wrote:
> > For example, if a package needs a specific version of libpng,
> > the package author can document this and the user can then make
> > sure to install that particular version.
>
> The assumption that any given Python user will know how to do this is
> not a reasonable assumption in 2016.
>
> If a publisher wants to bundle a particular version of libpng, they
> can. If (as is also entirely reasonable) they don't want to assume the
> associated responsibilities for responding to CVEs, then they can
> stick with source distributions, or target more specific Linux
> versions (as previously discussed in the context of Nate Coraor's
> Starforge work)
>

I wonder if, in relation to this, it may be best to have two separate tags:
one to indicate that the wheel includes external libraries rolled in to it,
and one to indicate that it doesn't. That way, a user can make a conscious
decision as to whether they want to install any wheels that could include
libraries that won't be maintained by the distribution package manager.
That way if we end up in a future world where manylinux wheels and
distro-specific wheels (that may depend on non-default distro packages)
live in PyPI together, there'd be a way to indicate a preference.

--nate


>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/c21a2d95/attachment.html>

From nate at bx.psu.edu  Fri Jan 29 14:31:11 2016
From: nate at bx.psu.edu (Nate Coraor)
Date: Fri, 29 Jan 2016 14:31:11 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7fvqng6DBFz1D3j6iF9TvKMZp-QVYBxtH_Zd67qrA3AeQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com> <4FBA4669-A5D9-4B65-8F2D-CE835A016660@stufft.io>
 <56A21B7D.7010008@egenix.com>
 <CADiSq7fvqng6DBFz1D3j6iF9TvKMZp-QVYBxtH_Zd67qrA3AeQ@mail.gmail.com>
Message-ID: <CALT861GBB-VDG_wmN+m6-6kjTS71Ak+YXSXg4XD=KvQJMGJr+g@mail.gmail.com>

On Fri, Jan 22, 2016 at 10:26 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 22 January 2016 at 22:07, M.-A. Lemburg <mal at egenix.com> wrote:
> > However, system vendors will often be a lot faster with updates
> > than package authors, simply because it's their business model,
> > so as user you will want to benefit from those updates and
> > not have to rely on the package author to ship new wheel files.
>
> This is true for the subset of packages monitored by distro security
> response teams, but there's a *lot* of software not currently packaged
> for Linux distros that never will be as more attention is given to the
> "rebuild the world on demand" model that elastic cloud computing and
> fast internet connections enable.
>
> My fundamental concern is that if a package author publishes a distro
> dependent wheel file, pip attempts to install it, and it doesn't work,
> the reaction for many users is going to be "Python packaging is
> broken", not "the packaging of this particular package is broken".
>

This is already broken with source dists if you don't have the appropriate
-dev packages (or a compiler) installed. Some package authors provide more
useful feedback explaining what the problem is and how one might resolve
it, rather than dying on a compiler error due to a missing header, but many
do not.

One solution to this for both source and binary distributions is package
manager awareness in the build/install tools, and to have packages declare
their dependencies in structured metadata. A translational layer would make
this easier on package authors: If they only had to say they depend on
"OpenSSL headers" and that was translated to the correct package for the
OS, this could be relayed to the user at build time ("install these
packages using this command") or the package manager could be directly
invoked, if the user has chosen to allow the build/install tool to do that.

--nate


> However, moving the "generic linux wheels are ignored by default"
> behaviour to pip-the-client, rather than enforcing it as a restriction
> on PyPI uploads could definitely be a reasonable alternative way of
> addressing that concern.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/914a811a/attachment.html>

From nate at bx.psu.edu  Fri Jan 29 14:35:12 2016
From: nate at bx.psu.edu (Nate Coraor)
Date: Fri, 29 Jan 2016 14:35:12 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
Message-ID: <CALT861GOm+1MAgefpg76Js=uAMLq1MNzjM=DsQPOssmEuRHpQg@mail.gmail.com>

On Fri, Jan 22, 2016 at 6:29 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 22 January 2016 at 20:48, M.-A. Lemburg <mal at egenix.com> wrote:
> > People who rely on Linux distributions want to continue
> > to do so and get regular updates for system packages from
> > their system vendor. Having wheel files override these
> > system packages by including libs directly in the wheel
> > silently breaks this expectation, potentially opening
> > up the installations for security holes, difficult to
> > track bugs and possible version conflicts with already
> > loaded versions of the shared libs.
>
> For the time being, these users should either pass the "--no-binary"
> option to pip, ask their distro to provide an index of pre-built wheel
> files for that distro (once we have the distro-specific wheel tagging
> PEP sorted out), or else ask their distro to update system Python
> packages in a more timely fashion (or all of the above).
>

Is there a distro-specific wheel tagging PEP in development somewhere that
I missed? If not, I will get the ball rolling on it.

--nate
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/344bafd6/attachment.html>

From nate at bx.psu.edu  Fri Jan 29 14:28:50 2016
From: nate at bx.psu.edu (Nate Coraor)
Date: Fri, 29 Jan 2016 14:28:50 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CAPJVwB=kVbh+o-bHqbCookKLtfFLNKpfbPyWVC2tyzgSHVK-Tw@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <CALT861Hiped2pqtx9wGqweL0jXe9f_zMs7_4_bAJfOZc4giTxw@mail.gmail.com>
 <CAPJVwB=kVbh+o-bHqbCookKLtfFLNKpfbPyWVC2tyzgSHVK-Tw@mail.gmail.com>
Message-ID: <CALT861H+W+yKnBJzYLZAyov52Xuh9D8ed_bfCP10Fz=k+Q=aOA@mail.gmail.com>

On Thu, Jan 21, 2016 at 6:31 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Thu, Jan 21, 2016 at 2:22 PM, Nate Coraor <nate at bx.psu.edu> wrote:
> > Could this instead use the more powerful json-based syntax proposed by
> Nick
> > here:
> >
> >
> https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html
> >
> > I have already implemented support for this in pip and wheel.
>
> Totally happy to change the compatibility.cfg stuff -- the version in
> the PEP was written in about 5 minutes in hopes of sparking discussion
> :-).
>
> Some other questions:
> 1) is this going to work for multi-arch (binaries for multiple cpu
> architectures sharing a single /etc)? Multiple interpreters? I guess
> the advantage of Nick's design is that it's scoped by the value of
> distutils.util.get_platform(), so multi-arch installs could have
> different values -- a distro could declare that their x86-64 python
> builds are manylinux1 compatible but their i386 python builds aren't.
> Maybe it would be even better if the files were
> /etc/python/binary-compatibility/linux_x86_64.cfg etc., so that the
> different .cfg files could be shipped in each architecture's package
> without colliding. OTOH I don't know if any of this is very useful in
> practice.
>

I don't think the proposed syntax would have any trouble with multiarch
other than that it's contained in one file and so would need to live in a
single package, or be dynamically generated based on which arches you
installed. If that was a problem we could support a
/etc/python/binary-compatibility.d type of thing.


> 2) in theory one could imaging overriding this on a per-venv,
> per-user, or per-system level; which of these are useful to support?
> (Per-system seems like the most obvious, since an important use case
> will be distros setting this flag on behalf of their users.)
>

Per-venv overriding was part of the original proposal and my
implementation, per-user could be useful too.


> There is one feature that I do think is important in the PEP 513
> draft, and that Nick's suggestion from July doesn't have: in the PEP
> 513 design, the manylinux1 flag can be true, false, or unspecified,
> and this is independent of other compatibility settings; in Nick's
> proposal there's an exhaustive list of all compatible tags, and
> everything not on that list is assumed to be incompatible. Where this
> matters is when we want to release manylinux2. At this point we'll
> want pip to use some autodetection logic on old distros that were
> released before manylinux2, while respecting the compatibility flag on
> newer distros that do know about manylinux2. This requires a tri-state
> setting with "not specified" as a valid value.
>

One solution would be `compatible` and `incompatible` keys rather than
`install`?

--nate
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/408eb04b/attachment-0001.html>

From robertc at robertcollins.net  Fri Jan 29 15:25:23 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Sat, 30 Jan 2016 09:25:23 +1300
Subject: [Distutils] Fwd:  How to get pip to really, really,
 I mean it -- rebuild this damn package!
In-Reply-To: <CAJ3HoZ1S8px24WGogutcYmMWqiNBsfBZUWooLQnVonEbJ0cBxw@mail.gmail.com>
References: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
 <CAPJVwBnRVO17xoxg0iCt5HuZ4vrzh5RjzbizYXDu1pb1WyRkaw@mail.gmail.com>
 <1425275544231181589@unknownmsgid>
 <CAJ3HoZ1S8px24WGogutcYmMWqiNBsfBZUWooLQnVonEbJ0cBxw@mail.gmail.com>
Message-ID: <CAJ3HoZ10vUf7hRBS2CsoaxMyGJ2SQK-29mG8rJ-FU2bNdHtzXQ@mail.gmail.com>

Bah, offlist by mistake.


---------- Forwarded message ----------
From: Robert Collins <robertc at robertcollins.net>
Date: 30 January 2016 at 09:25
Subject: Re: [Distutils] How to get pip to really, really, I mean it
-- rebuild this damn package!
To: Chris Barker - NOAA Federal <chris.barker at noaa.gov>


Please try pip 7.1? Latest before 8; we're not meant to be caching
wheels of by-location things from my memory, but it may have
regressed/changed with the cache changes made during the 8 development
cycle.

-Rob

On 30 January 2016 at 04:48, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
>>>  Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from
>>> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
>>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>>
>> I think this is saying that pip thinks it has found an
>> already-installed version of gsw 3.0.3 in sys.path, and that the
>> directory in your sys.path where it's already installed is
>>
>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
> That is the temp dir conda sets up to unpack downloaded files, and do
> its work in -- hence the name. I'll look and see what's there. I'm
> pretty sure conda build starts out with an empty dir, however. And
> that dir should not be on sys.path.
>
>> I think this means that that directory is (a) in sys.path, and (b)
>> contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
>> seems weird and broken.
>
> Indeed. And I get the same symptoms with a clean environment that I've
> set up outside conda build. Though with the same source dir. But with
> conda build, it's a fresh unpack of the tarball.
>
>> Do you have "." in your PYTHONPATH or anything like that?
>
> God no!
>
>> Don't know why it seems to be building a wheel for it, if it already
>> thinks that it's installed... this is also odd.
>
> Yes it is. But it doesn't install it :-(
>
>>
>> $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
>>
>> ? Though I'd think that -I would have the same affect as --force-reinstall...
>>
> So did I, and I think I tried --force-reinstall already, but I will again.
>
>> (It doesn't look like the cache dir is your problem here, but you do
>> probably want to use --no-cache-dir anyway just as good practice, just
>> because you don't want to accidentally package up a stale version of
>> the software that got pulled out of your cache instead of the version
>> you thought you were packaging in the tree in front of you.
>
> Exactly. Doesn't seem to make a difference, though.
>
>> Also, I think it's a bug in pip that it caches builds of source trees
>> -- PyPI can enforce the rule that each (package name, version number)
>> sdist is unique, but for a work-in-progress VCS checkout it's just not
>> true that (package name, version number) uniquely identifies a
>> snapshot of the whole tree. So in something like 'pip install .', then
>> requirement resolution code should treat this as a special requirement
>> that it wants *this exact tree*, not just any package that has the
>> same (package name, version number) as this tree; and the resulting
>> wheel should not be cached.
>
> Absolutely! In fact, I'll bet that approach is the source of the
> problem here. If not automagically, there should be a flag, at least.
>
> However, what seems to be happening is that pip is looking outside the
> current Python environment somewhere to see if this package needs to
> be installed. It may be something that works with virtualenv, but
> doesn't with conda environments for some reason.
>
> I guess on some level pip simply isn't designed to build and install
> from local source :-(
>
> In the end, I'm still confused: does pip install give me anything that:
>
> setup.py install single-version-externally-managed
>
> Doesn't? Other that support for non-setuptools installs, anyway.
>
> CHB
>
>
>> I don't know if there are any bugs filed
>> in pip on this...)
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



--
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud

From chris.barker at noaa.gov  Fri Jan 29 16:01:37 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 29 Jan 2016 13:01:37 -0800
Subject: [Distutils] Fwd: How to get pip to really, really,
 I mean it -- rebuild this damn package!
In-Reply-To: <CAJ3HoZ10vUf7hRBS2CsoaxMyGJ2SQK-29mG8rJ-FU2bNdHtzXQ@mail.gmail.com>
References: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
 <CAPJVwBnRVO17xoxg0iCt5HuZ4vrzh5RjzbizYXDu1pb1WyRkaw@mail.gmail.com>
 <1425275544231181589@unknownmsgid>
 <CAJ3HoZ1S8px24WGogutcYmMWqiNBsfBZUWooLQnVonEbJ0cBxw@mail.gmail.com>
 <CAJ3HoZ10vUf7hRBS2CsoaxMyGJ2SQK-29mG8rJ-FU2bNdHtzXQ@mail.gmail.com>
Message-ID: <CALGmxEJy-Gfbc6Gxsvgseqfh41SV-Lx6ji7Uc3EqXR=ZrXrmeg@mail.gmail.com>

On Fri, Jan 29, 2016 at 12:25 PM, Robert Collins <robertc at robertcollins.net>
wrote
>
>
> Please try pip 7.1? Latest before 8; we're not meant to be caching
> wheels of by-location things from my memory, but it may have
> regressed/changed with the cache changes made during the 8 development
>

will do -- in fact, I was pretty sure this was working earlier....

stay tuned.

-CHB



> cycle.
>
> -Rob
>
> On 30 January 2016 at 04:48, Chris Barker - NOAA Federal
> <chris.barker at noaa.gov> wrote:
> >>>  Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3
> from
> >>> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
> >>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
> >>
> >> I think this is saying that pip thinks it has found an
> >> already-installed version of gsw 3.0.3 in sys.path, and that the
> >> directory in your sys.path where it's already installed is
> >>
> >> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
> >
> > That is the temp dir conda sets up to unpack downloaded files, and do
> > its work in -- hence the name. I'll look and see what's there. I'm
> > pretty sure conda build starts out with an empty dir, however. And
> > that dir should not be on sys.path.
> >
> >> I think this means that that directory is (a) in sys.path, and (b)
> >> contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
> >> seems weird and broken.
> >
> > Indeed. And I get the same symptoms with a clean environment that I've
> > set up outside conda build. Though with the same source dir. But with
> > conda build, it's a fresh unpack of the tarball.
> >
> >> Do you have "." in your PYTHONPATH or anything like that?
> >
> > God no!
> >
> >> Don't know why it seems to be building a wheel for it, if it already
> >> thinks that it's installed... this is also odd.
> >
> > Yes it is. But it doesn't install it :-(
> >
> >>
> >> $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
> >>
> >> ? Though I'd think that -I would have the same affect as
> --force-reinstall...
> >>
> > So did I, and I think I tried --force-reinstall already, but I will
> again.
> >
> >> (It doesn't look like the cache dir is your problem here, but you do
> >> probably want to use --no-cache-dir anyway just as good practice, just
> >> because you don't want to accidentally package up a stale version of
> >> the software that got pulled out of your cache instead of the version
> >> you thought you were packaging in the tree in front of you.
> >
> > Exactly. Doesn't seem to make a difference, though.
> >
> >> Also, I think it's a bug in pip that it caches builds of source trees
> >> -- PyPI can enforce the rule that each (package name, version number)
> >> sdist is unique, but for a work-in-progress VCS checkout it's just not
> >> true that (package name, version number) uniquely identifies a
> >> snapshot of the whole tree. So in something like 'pip install .', then
> >> requirement resolution code should treat this as a special requirement
> >> that it wants *this exact tree*, not just any package that has the
> >> same (package name, version number) as this tree; and the resulting
> >> wheel should not be cached.
> >
> > Absolutely! In fact, I'll bet that approach is the source of the
> > problem here. If not automagically, there should be a flag, at least.
> >
> > However, what seems to be happening is that pip is looking outside the
> > current Python environment somewhere to see if this package needs to
> > be installed. It may be something that works with virtualenv, but
> > doesn't with conda environments for some reason.
> >
> > I guess on some level pip simply isn't designed to build and install
> > from local source :-(
> >
> > In the end, I'm still confused: does pip install give me anything that:
> >
> > setup.py install single-version-externally-managed
> >
> > Doesn't? Other that support for non-setuptools installs, anyway.
> >
> > CHB
> >
> >
> >> I don't know if there are any bugs filed
> >> in pip on this...)
> >>
> >> -n
> >>
> >> --
> >> Nathaniel J. Smith -- https://vorpus.org
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> --
> Robert Collins <rbtcollins at hpe.com>
> Distinguished Technologist
> HP Converged Cloud
>
>
> --
> Robert Collins <rbtcollins at hpe.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/745d0978/attachment.html>

From njs at pobox.com  Fri Jan 29 17:43:06 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 29 Jan 2016 14:43:06 -0800
Subject: [Distutils] How to get pip to really, really,
 I mean it -- rebuild this damn package!
In-Reply-To: <1425275544231181589@unknownmsgid>
References: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
 <CAPJVwBnRVO17xoxg0iCt5HuZ4vrzh5RjzbizYXDu1pb1WyRkaw@mail.gmail.com>
 <1425275544231181589@unknownmsgid>
Message-ID: <CAPJVwBmFxsHzhNfP4R7RneEMBdERfH5nKd=KF0g4+2xsARFCPw@mail.gmail.com>

On Jan 29, 2016 7:48 AM, "Chris Barker - NOAA Federal" <
chris.barker at noaa.gov> wrote:
>
> >>  Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3
from
> >> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
> >> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
> >
> > I think this is saying that pip thinks it has found an
> > already-installed version of gsw 3.0.3 in sys.path, and that the
> > directory in your sys.path where it's already installed is
> >
> > /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
> That is the temp dir conda sets up to unpack downloaded files, and do
> its work in -- hence the name. I'll look and see what's there. I'm
> pretty sure conda build starts out with an empty dir, however. And
> that dir should not be on sys.path.

This is the point where I'd probably start adding print statements to my
local copy of pip. E.g. grep for that "requirement already satisfied"
message and then have it print out sys.path as well. To me this still
smells like some simple bug in one of conda, your environment, or pip, so
it's worth figuring out...

> > I think this means that that directory is (a) in sys.path, and (b)
> > contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
> > seems weird and broken.
>
> Indeed. And I get the same symptoms with a clean environment that I've
> set up outside conda build. Though with the same source dir. But with
> conda build, it's a fresh unpack of the tarball.
>
> > Do you have "." in your PYTHONPATH or anything like that?
>
> God no!

Too bad, it would have been an easy explanation ;-)

> > Don't know why it seems to be building a wheel for it, if it already
> > thinks that it's installed... this is also odd.
>
> Yes it is. But it doesn't install it :-(
>
> >
> > $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
> >
> > ? Though I'd think that -I would have the same affect as
--force-reinstall...
> >
> So did I, and I think I tried --force-reinstall already, but I will again.
>
> > (It doesn't look like the cache dir is your problem here, but you do
> > probably want to use --no-cache-dir anyway just as good practice, just
> > because you don't want to accidentally package up a stale version of
> > the software that got pulled out of your cache instead of the version
> > you thought you were packaging in the tree in front of you.
>
> Exactly. Doesn't seem to make a difference, though.
>
> > Also, I think it's a bug in pip that it caches builds of source trees
> > -- PyPI can enforce the rule that each (package name, version number)
> > sdist is unique, but for a work-in-progress VCS checkout it's just not
> > true that (package name, version number) uniquely identifies a
> > snapshot of the whole tree. So in something like 'pip install .', then
> > requirement resolution code should treat this as a special requirement
> > that it wants *this exact tree*, not just any package that has the
> > same (package name, version number) as this tree; and the resulting
> > wheel should not be cached.
>
> Absolutely! In fact, I'll bet that approach is the source of the
> problem here. If not automagically, there should be a flag, at least.

Maybe... it looks like at least @dstufft agrees that the current behavior
is a bug:

  https://github.com/pypa/pip/issues/536

and if this were fixed then it would at least hide your problem. But the
underlying bug here is that pip seems to think that a package is installed
when it wasn't. Hacking it to ignore whether a package is installed would
just be papering over this.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/d151a473/attachment-0001.html>

From chris.barker at noaa.gov  Fri Jan 29 17:58:44 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 29 Jan 2016 14:58:44 -0800
Subject: [Distutils] How to get pip to really, really,
 I mean it -- rebuild this damn package!
In-Reply-To: <CAPJVwBmFxsHzhNfP4R7RneEMBdERfH5nKd=KF0g4+2xsARFCPw@mail.gmail.com>
References: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
 <CAPJVwBnRVO17xoxg0iCt5HuZ4vrzh5RjzbizYXDu1pb1WyRkaw@mail.gmail.com>
 <1425275544231181589@unknownmsgid>
 <CAPJVwBmFxsHzhNfP4R7RneEMBdERfH5nKd=KF0g4+2xsARFCPw@mail.gmail.com>
Message-ID: <CALGmxEJEBDPLmA2TQAmGqeXhXMucm5B1Fu0TaRZUKRPOgZfN6A@mail.gmail.com>

On Fri, Jan 29, 2016 at 2:43 PM, Nathaniel Smith <njs at pobox.com> wrote:

> This is the point where I'd probably start adding print statements to my
> local copy of pip. E.g. grep for that "requirement already satisfied"
> message and then have it print out sys.path as well. To me this still
> smells like some simple bug in one of conda, your environment, or pip, so
> it's worth figuring out...
>

yeah, I was hoping not to have to debug pip :-) But Robert indicate that
this may well be a pipi bug -- I'll try an older version first, then maybe
dig into the pip code...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/735997af/attachment.html>

From njs at pobox.com  Fri Jan 29 18:24:25 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 29 Jan 2016 15:24:25 -0800
Subject: [Distutils] PEP 513: A Platform Tag for Portable Linux Built
 Distributions Version
In-Reply-To: <CADiSq7fNGa29aRPtQmfP0L60GWhLhGgzDSVEbbfyahV_+fzwnQ@mail.gmail.com>
References: <CAN4+E8FPh2kk2OE1mOuUa0Kj+T6qwcUB6CM6RG1Tv+=N18yyiQ@mail.gmail.com>
 <CAPJVwBks44dnprxrnR9xpDOiZo9yQjco5DCMiXpGnKeA_iv5ag@mail.gmail.com>
 <CAN4+E8Fkn_qg2XPC5KgddvAXf7J1ocOUVi6f_Z9gA7S97=SaMw@mail.gmail.com>
 <CAPJVwBn4TNeO+5mycySs7dPKxRzEn2pYAXJ6BcwtYir8xEZOqw@mail.gmail.com>
 <CADiSq7cKQ417uA7d=Sp8Y+MG+NH95e9Dy1FRPDqZJkL0-pWSbA@mail.gmail.com>
 <CAPJVwBnMAPuDQS5adoixPBFVmsPUsHKQnykTB=miZ1XjHNBb8Q@mail.gmail.com>
 <CAHVvXxS0ir4TLX7_g1U19uCQo7Rxkg1_vMcmDLbnw7uhgzRtaw@mail.gmail.com>
 <CAPJVwBmwvXrX-zJEFgxQSpqLaUyW3aXH-gXy3kZUgrHtZSpv=Q@mail.gmail.com>
 <CADiSq7fNGa29aRPtQmfP0L60GWhLhGgzDSVEbbfyahV_+fzwnQ@mail.gmail.com>
Message-ID: <CAPJVwBkw9BUzLkDGpmqAvxLa11ymg-66kGWGbVcdb7=3tb-eDQ@mail.gmail.com>

On Fri, Jan 29, 2016 at 6:57 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I like the simple approach Nathaniel now has in the PEP, as that seems
> like it should be relatively easy to handle as either modifications
> directly to a python package, or as an add-on module (ala installing
> LSB support, only much lighter weight). It doesn't scale to lots of
> platform tags, but we don't really intend it to - we'd like to avoid
> platform tag proliferation if we can, and if we're successful in that,
> then we won't need to solve the problem with a lack of scalability.
>
> Cheers,
> Nick.
>
> P.S. To provide a bit more context on "Why is a separate file
> easier?", the main benefit is that adding a new file simply *cannot*
> break the sys or platform modules, while patching them can. That's
> still not a guarantee that any given distro will actually provide an
> importable "_manylinux" module to indicate compatibility, but given
> the fallback glibc check, _manylinux is more a hedge against future
> *in*compatibility than it is anything else.

Great! I think we're ready for pronouncement then... I'll resend the
(hopefully final) version in a moment.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Fri Jan 29 18:29:00 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 29 Jan 2016 15:29:00 -0800
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable
 Linux Built Distributions
Message-ID: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>

Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!

The only change relative to the last posting is that we rewrote the
section on "Platform detection for installers", to switch to letting
distributors explicitly control manylinux1 compatibility by means of a
_manylinux module.

-n

---

PEP: 513
Title: A Platform Tag for Portable Linux Built Distributions
Version: $Revision$
Last-Modified: $Date$
Author: Robert T. McGibbon <rmcgibbo at gmail.com>, Nathaniel J. Smith
<njs at pobox.com>
BDFL-Delegate: Nick Coghlan <ncoghlan at gmail.com>
Discussions-To: Distutils SIG <distutils-sig at python.org>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 19-Jan-2016
Post-History: 19-Jan-2016, 25-Jan-2016, 29-Jan-2016


Abstract
========

This PEP proposes the creation of a new platform tag for Python package built
distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
external dependencies limited to a standardized, restricted subset of
the Linux kernel and core userspace ABI. It proposes that PyPI support
uploading and distributing wheels with this platform tag, and that ``pip``
support downloading and installing these packages on compatible platforms.


Rationale
=========

Currently, distribution of binary Python extensions for Windows and OS X is
straightforward. Developers and packagers build wheels [1]_ [2]_, which are
assigned platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload
these wheels to PyPI. Users can download and install these wheels using tools
such as ``pip``.

For Linux, the situation is much more delicate. In general, compiled Python
extension modules built on one Linux distribution will not work on other Linux
distributions, or even on different machines running the same Linux
distribution with different system libraries installed.

Build tools using PEP 425 platform tags [3]_ do not track information about the
particular Linux distribution or installed system libraries, and instead assign
all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
this ambiguity, there is no expectation that ``linux``-tagged built
distributions compiled on one machine will work properly on another, and for
this reason, PyPI has not permitted the uploading of wheels for Linux.

It would be ideal if wheel packages could be compiled that would work on *any*
linux system. But, because of the incredible diversity of Linux systems -- from
PCs to Android to embedded systems with custom libcs -- this cannot
be guaranteed in general.

Instead, we define a standard subset of the kernel+core userspace ABI that,
in practice, is compatible enough that packages conforming to this standard
will work on *many* linux systems, including essentially all of the desktop
and server distributions in common use. We know this because there are
companies who have been distributing such widely-portable pre-compiled Python
extension modules for Linux -- e.g. Enthought with Canopy [4]_ and Continuum
Analytics with Anaconda [5]_.

Building on the compability lessons learned from these companies, we thus
define a baseline ``manylinux1`` platform tag for use by binary Python
wheels, and introduce the implementation of preliminary tools to aid in the
construction of these ``manylinux1`` wheels.


Key Causes of Inter-Linux Binary Incompatibility
================================================

To properly define a standard that will guarantee that wheel packages meeting
this specification will operate on *many* linux platforms, it is necessary to
understand the root causes which often prevent portability of pre-compiled
binaries on Linux. The two key causes are dependencies on shared libraries
which are not present on users' systems, and dependencies on particular
versions of certain core libraries like ``glibc``.


External Shared Libraries
-------------------------

Most desktop and server linux distributions come with a system package manager
(examples include ``APT`` on Debian-based systems, ``yum`` on
``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
responsibilities, the installation of shared libraries installed to system
directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
on one or more of these shared libraries, and thus function properly only on
systems where the user has the proper libraries (and the proper
versions thereof), either installed using their package manager, or installed
manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
to notify the runtime linker of the location of the depended-upon shared
libraries.


Versioning of Core Shared Libraries
-----------------------------------

Even if the developers a Python extension module wish to use no
external shared libraries, the modules will generally have a dynamic runtime
dependency on the GNU C library, ``glibc``. While it is possible, statically
linking ``glibc`` is usually a bad idea because certain important C functions
like ``dlopen()`` cannot be called from code that statically links ``glibc``. A
runtime shared library dependency on a system-provided ``glibc`` is unavoidable
in practice.

The maintainers of the GNU C library follow a strict symbol versioning scheme
for backward compatibility. This ensures that binaries compiled against an older
version of ``glibc`` can run on systems that have a newer ``glibc``. The
opposite is generally not true -- binaries compiled on newer Linux
distributions tend to rely upon versioned functions in ``glibc`` that are not
available on older systems.

This generally prevents wheels compiled on the latest Linux distributions
from being portable.


The ``manylinux1`` policy
=========================

For these reasons, to achieve broad portability, Python wheels

* should depend only on an extremely limited set of external shared
  libraries; and
* should depend only on "old" symbol versions in those external shared
  libraries; and
* should depend only on a widely-compatible kernel ABI.

To be eligible for the ``manylinux1`` platform tag, a Python wheel must
therefore both (a) contain binary executables and compiled code that links
*only* to libraries (other than the appropriate ``libpython`` library, which is
always a permitted dependency consistent with the PEP 425 ABI tag) with SONAMEs
included in the following list: ::

    libpanelw.so.5
    libncursesw.so.5
    libgcc_s.so.1
    libstdc++.so.6
    libm.so.6
    libdl.so.2
    librt.so.1
    libcrypt.so.1
    libc.so.6
    libnsl.so.1
    libutil.so.1
    libpthread.so.0
    libX11.so.6
    libXext.so.6
    libXrender.so.1
    libICE.so.6
    libSM.so.6
    libGL.so.1
    libgobject-2.0.so.0
    libgthread-2.0.so.0
    libglib-2.0.so.0

and (b), work on a stock CentOS 5.11 [6]_ system that contains the system
package manager's provided versions of these libraries.

Because CentOS 5 is only available for x86_64 and i386 architectures,
these are the only architectures currently supported by the ``manylinux1``
policy.

On Debian-based systems, these libraries are provided by the packages ::

    libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
    libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0

On RPM-based systems, these libraries are provided by the packages ::

    ncurses libgcc libstdc++ glibc libXext libXrender
    libICE libSM mesa-libGL glib2

This list was compiled by checking the external shared library dependencies of
the Canopy [4]_ and Anaconda [5]_ distributions, which both include a wide array
of the most popular Python modules and have been confirmed in practice to work
across a wide swath of Linux systems in the wild.

Many of the permitted system libraries listed above use symbol versioning
schemes for backward compatibility. The latest symbol versions provided with
the CentOS 5.11 versions of these libraries are: ::

    GLIBC_2.5
    CXXABI_3.4.8
    GLIBCXX_3.4.9
    GCC_4.2.0

Therefore, as a consequence of requirement (b), any wheel that depends on
versioned symbols from the above shared libraries may depend only on symbols
with the following versions: ::

    GLIBC <= 2.5
    CXXABI <= 3.4.8
    GLIBCXX <= 3.4.9
    GCC <= 4.2.0

These recommendations are the outcome of the relevant discussions in January
2016 [7]_, [8]_.

Note that in our recommendations below, we do not suggest that ``pip``
or PyPI should attempt to check for and enforce the details of this
policy (just as they don't check for and enforce the details of
existing platform tags like ``win32``). The text above is provided (a)
as advice to package builders, and (b) as a method for allocating
blame if a given wheel doesn't work on some system: if it satisfies
the policy above, then this is a bug in the spec or the installation
tool; if it does not satisfy the policy above, then it's a bug in the
wheel. One useful consequence of this approach is that it leaves open
the possibility of further updates and tweaks as we gain more
experience, e.g., we could have a "manylinux 1.1" policy which targets
the same systems and uses the same ``manylinux1`` platform tag (and
thus requires no further changes to ``pip`` or PyPI), but that adjusts
the list above to remove libraries that have turned out to be
problematic or add libraries that have turned out to be safe.


Compilation of Compliant Wheels
===============================

The way glibc, libgcc, and libstdc++ manage their symbol versioning
means that in practice, the compiler toolchains that most developers
use to do their daily work are incapable of building
``manylinux1``-compliant wheels. Therefore we do not attempt to change
the default behavior of ``pip wheel`` / ``bdist_wheel``: they will
continue to generate regular ``linux_*`` platform tags, and developers
who wish to use them to generate ``manylinux1``-tagged wheels will
have to change the tag as a second post-processing step.

To support the compilation of wheels meeting the ``manylinux1`` standard, we
provide initial drafts of two tools.


Docker Image
------------

The first tool is a Docker image based on CentOS 5.11, which is recommended as
an easy to use self-contained build box for compiling ``manylinux1`` wheels
[9]_. Compiling on a more recently-released linux distribution will generally
introduce dependencies on too-new versioned symbols. The image comes with a
full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
well as the latest releases of Python and ``pip``.

Auditwheel
----------

The second tool is a command line executable called ``auditwheel`` [10]_ that
may aid in package maintainers in dealing with third-party external
dependencies.

There are at least three methods for building wheels that use third-party
external libraries in a way that meets the above policy.

1. The third-party libraries can be statically linked.
2. The third-party shared libraries can be distributed in
   separate packages on PyPI which are depended upon by the wheel.
3. The third-party shared libraries can be bundled inside the wheel
   libraries, linked with a relative path.

All of these are valid option which may be effectively used by different
packages and communities. Statically linking generally requires
package-specific modifications to the build system, and distributing
third-party dependencies on PyPI may require some coordination of the
community of users of the package.

As an often-automatic alternative to these options, we introduce ``auditwheel``.
The tool inspects all of the ELF files  inside a wheel to check for
dependencies on versioned symbols or external  shared libraries, and verifies
conformance with the ``manylinux1`` policy. This  includes the ability to add
the new platform tag to conforming wheels. More importantly, ``auditwheel`` has
the ability to automatically modify wheels that depend on external shared
libraries by copying those shared libraries from the system into the wheel
itself, and modifying the appropriate ``RPATH`` entries such that these
libraries will be picked up at runtime. This accomplishes a similar result as
if the libraries had been statically linked without requiring changes to the
build system. Packagers are advised that bundling, like static linking, may
implicate copyright concerns.


Bundled Wheels on Linux
=======================

While we acknowledge many approaches for dealing with third-party library
dependencies within ``manylinux1`` wheels, we recognize that the ``manylinux1``
policy encourages bundling external dependencies, a practice
which runs counter to  the package management policies of many linux
distributions' system package  managers [11]_, [12]_. The primary purpose of
this is cross-distro compatibility.  Furthermore, ``manylinux1`` wheels on PyPI
occupy a different  niche than the Python packages available through the
system package manager.

The decision in this PEP to encourage departure from general Linux distribution
unbundling policies is informed by the following concerns:

1. In these days of automated continuous integration and deployment
   pipelines, publishing new versions and updating dependencies is easier
   than it was when those policies were defined.
2. ``pip`` users remain free to use the ``"--no-binary"`` option if they want
   to force local builds rather than using pre-built wheel files.
3. The popularity of modern container based deployment and "immutable
   infrastructure" models involve substantial bundling at the application
   layer anyway.
4. Distribution of bundled wheels through PyPI is currently the norm for
   Windows and OS X.
5. This PEP doesn't rule out the idea of offering more targeted binaries for
   particular Linux distributions in the future.

The model described in this PEP is most ideally suited for cross-platform
Python packages, because it means they can reuse much of the
work that they're already doing to make static Windows and OS X wheels. We
recognize that it is less optimal for Linux-specific packages that might
prefer to interact more closely with Linux's unique package management
functionality and only care about targeting a small set of particular distos.


Security Implications
---------------------

One of the advantages of dependencies on centralized libraries in Linux is
that bugfixes and security updates can be deployed system-wide, and
applications which depend on these libraries will automatically feel the
effects of these patches when the underlying libraries are updated. This can
be particularly important for security updates in packages engaged in
communication across the network or cryptography.

``manylinux1`` wheels distributed through PyPI that bundle security-critical
libraries like OpenSSL will thus assume responsibility for prompt updates in
response disclosed vulnerabilities and patches. This closely parallels the
security implications of the distribution of binary wheels on Windows that,
because the platform lacks a system package manager, generally bundle their
dependencies. In particular, because it lacks a stable ABI, OpenSSL cannot be
included in the ``manylinux1`` profile.



Platform Detection for Installers
=================================

Above, we defined what it means for a *wheel* to be
``manylinux1``-compatible. Here we discuss what it means for a *Python
installation* to be ``manylinux1``-compatible. In particular, this is
important for tools like ``pip`` to know when deciding whether or not
they should consider ``manylinux1``-tagged wheels for installation.

Because the ``manylinux1`` profile is already known to work for the
many thousands of users of popular commercial Python distributions, we
suggest that installation tools should error on the side of assuming
that a system *is* compatible, unless there is specific reason to
think otherwise.

We know of three main sources of potential incompatibility that are likely to
arise in practice:

* Eventually, in the future, there may exist distributions that break
  compatibility with this profile (e.g., if one of the libraries in
  the profile changes its ABI in a backwards-incompatible way)
* A linux distribution that is too old (e.g. RHEL 4)
* A linux distribution that does not use ``glibc`` (e.g. Alpine Linux, which is
  based on musl ``libc``, or Android)

Therefore, we propose a two-pronged approach. To catch the first
case, we standardize a mechanism for a Python distributor to signal
that a particular Python install definitely is or is not compatible
with ``manylinux1``: this is done by installing a module named
``_manylinux``, and setting its ``manylinux1_compatible``
attribute. We do not propose adding any such module to the standard
library -- this is merely a well-known name by which distributors and
installation tools can rendezvous. However, if a distributor does add
this module, *they should add it to the standard library* rather than
to a ``site-packages/`` directory, because the standard library is
inherited by virtualenvs (which we want), and ``site-packages/`` in
general is not.

Then, to handle the latter two cases for existing Python
distributions, we suggest a simple and reliable method to check for
the presence and version of ``glibc`` (basically using it as a "clock"
for the overall age of the distribution).

Specifically, the algorithm we propose is::

    def is_manylinux1_compatible():
        # Only Linux, and only x86-64 / i386
        from distutils.util import get_platform
        if get_platform() not in ["linux_x86_64", "linux_i386"]:
            return False

        # Check for presence of _manylinux module
        try:
            import _manylinux
            return bool(_manylinux.manylinux1_compatible)
        except (ImportError, AttributeError):
            # Fall through to heuristic check below
            pass

        # Check glibc version. CentOS 5 uses glibc 2.5.
        return have_compatible_glibc(2, 5)

    def have_compatible_glibc(major, minimum_minor):
        import ctypes

        process_namespace = ctypes.CDLL(None)
        try:
            gnu_get_libc_version = process_namespace.gnu_get_libc_version
        except AttributeError:
            # Symbol doesn't exist -> therefore, we are not linked to
            # glibc.
            return False

        # Call gnu_get_libc_version, which returns a string like "2.5".
        gnu_get_libc_version.restype = ctypes.c_char_p
        version_str = gnu_get_libc_version()
        # py2 / py3 compatibility:
        if not isinstance(version_str, str):
            version_str = version_str.decode("ascii")

        # Parse string and check against requested version.
        version = [int(piece) for piece in version_str.split(".")]
        assert len(version) == 2
        if major != version[0]:
            return False
        if minimum_minor > version[1]:
            return False
        return True

**Rejected alternatives:** We also considered using a configuration
file, e.g. ``/etc/python/compatibility.cfg``. The problem with this is
that a single filesystem might contain many different interpreter
environments, each with their own ABI profile -- the ``manylinux1``
compatibility of a system-installed x86_64 CPython might not tell us
much about the ``manylinux1`` compatibility of a user-installed i386
PyPy. Locating this configuration information within the Python
environment itself ensures that it remains attached to the correct
binary, and dramatically simplifies lookup code.

We also considered using a more elaborate structure, like a list of
all platform tags that should be considered compatible, together with
their preference ordering, for example: ``_binary_compat.compatible =
["manylinux1_x86_64", "centos5_x86_64", "linux_x86_64"]``. However,
this introduces several complications. For example, we want to be able
to distinguish between the state of "doesn't support ``manylinux1``"
(or eventually ``manylinux2``, etc.) versus "doesn't specify either
way whether it supports ``manylinux1``", which is not entirely obvious
in the above representation; and, it's not at all clear what features
are really needed vis a vis preference ordering given that right now
the only possible platform tags are ``manylinux1`` and ``linux``.  So
we're deferring a more complete solution here for a separate PEP, when
/ if Linux gets more platform tags.

For the library compatibility check, we also considered much more
elaborate checks (e.g. checking the kernel version, searching for and
checking the versions of all the individual libraries listed in the
``manylinux1`` profile, etc.), but ultimately decided that this would
be more likely to introduce confusing bugs than actually help the
user. (For example: different distributions vary in where they
actually put these libraries, and if our checking code failed to use
the correct path search then it could easily return incorrect
answers.)



PyPI Support
============

PyPI should permit wheels containing the ``manylinux1`` platform tag to be
uploaded. PyPI should not attempt to formally verify that wheels containing
the ``manylinux1`` platform tag adhere to the ``manylinux1`` policy described
in this document. This verification tasks should be left to other tools, like
``auditwheel``, that are developed separately.


Rejected Alternatives
=====================

One alternative would be to provide separate platform tags for each Linux
distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
``debian_jessie``, etc. Nothing in this proposal rules out the possibility of
adding such platform tags in the future, or of further extensions to wheel
metadata that would allow wheels to declare dependencies on external
system-installed packages. However, such extensions would require substantially
more work than this proposal, and still might not be appreciated by package
developers who would prefer not to have to maintain multiple build environments
and build multiple wheels in order to cover all the common Linux distributions.
Therefore we consider such proposals to be out-of-scope for this PEP.


Future updates
==============

We anticipate that at some point in the future there will be a
``manylinux2`` specifying a more modern baseline environment (perhaps
based on CentOS 6), and someday a ``manylinux3`` and so forth, but we
defer specifying these until we have more experience with the initial
``manylinux1`` proposal.


References
==========

.. [1] PEP 0427 -- The Wheel Binary Package Format 1.0
   (https://www.python.org/dev/peps/pep-0427/)
.. [2] PEP 0491 -- The Wheel Binary Package Format 1.9
   (https://www.python.org/dev/peps/pep-0491/)
.. [3] PEP 425 -- Compatibility Tags for Built Distributions
   (https://www.python.org/dev/peps/pep-0425/)
.. [4] Enthought Canopy Python Distribution
   (https://store.enthought.com/downloads/)
.. [5] Continuum Analytics Anaconda Python Distribution
   (https://www.continuum.io/downloads)
.. [6] CentOS 5.11 Release Notes
   (https://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.11)
.. [7] manylinux-discuss mailing list discussion
   (https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U)
.. [8] distutils-sig discussion
   (https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html)
.. [9] manylinux1 docker image
   (https://quay.io/repository/manylinux/manylinux)
.. [10] auditwheel tool
   (https://pypi.python.org/pypi/auditwheel)
.. [11] Fedora Bundled Software Policy
   (https://fedoraproject.org/wiki/Bundled_Software_policy)
.. [12] Debian Policy Manual -- 4.13: Convenience copies of code
    (https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles)


Copyright
=========

This document has been placed into the public domain.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:


-- 
Nathaniel J. Smith -- https://vorpus.org

From chris.barker at noaa.gov  Fri Jan 29 18:56:01 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 29 Jan 2016 15:56:01 -0800
Subject: [Distutils] Fwd: How to get pip to really, really,
 I mean it -- rebuild this damn package!
In-Reply-To: <CAJ3HoZ10vUf7hRBS2CsoaxMyGJ2SQK-29mG8rJ-FU2bNdHtzXQ@mail.gmail.com>
References: <CALGmxEJW42EzzASo6RrpLTvxXp7x1jnZOa9gkWfWXCGahHTyyw@mail.gmail.com>
 <CAPJVwBnRVO17xoxg0iCt5HuZ4vrzh5RjzbizYXDu1pb1WyRkaw@mail.gmail.com>
 <1425275544231181589@unknownmsgid>
 <CAJ3HoZ1S8px24WGogutcYmMWqiNBsfBZUWooLQnVonEbJ0cBxw@mail.gmail.com>
 <CAJ3HoZ10vUf7hRBS2CsoaxMyGJ2SQK-29mG8rJ-FU2bNdHtzXQ@mail.gmail.com>
Message-ID: <CALGmxE+d8Ln_SQWbu13xUxhdYEAfJtLXc8QWOrtNjGLxh3GMCQ@mail.gmail.com>

On Fri, Jan 29, 2016 at 12:25 PM, Robert Collins <robertc at robertcollins.net>
wrote:

> Please try pip 7.1? Latest before 8; we're not meant to be caching
> wheels of by-location things from my memory, but it may have
> regressed/changed with the cache changes made during the 8 development
> cycle.
>

indeed this is looking like a regression. I have done:



* create 2 fresh conda environments

* downgrade pip to 7.1.2 in both (conda seems to put the latest pip in a
new environment by default)

$ pip --version

pip 7.1.2 from
/Users/chris.barker/miniconda2/envs/test2/lib/python2.7/site-packages
(python 2.7)

* build and install the package in one of them:

$ pip install ./

Processing /Users/chris.barker/Temp/geojson-1.3.2

Requirement already satisfied (use --upgrade to upgrade): setuptools in
/Users/chris.barker/miniconda2/envs/test1/lib/python2.7/site-packages/setuptools-19.4-py2.7.egg
(from geojson==1.3.2)

Building wheels for collected packages: geojson

  Running setup.py bdist_wheel for geojson

  Stored in directory:
/Users/chris.barker/Library/Caches/pip/wheels/67/47/d8/01cf2332293b60900ec87ff03bbe9fff92bc85f194e0eb0e74

Successfully built geojson

Installing collected packages: geojson

Successfully installed geojson-1.3.2

You are using pip version 7.1.2, however version 8.0.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.
It built and installed fine:

(test1)orrw-m-4179497:geojson-1.3.2 chris.barker$ python -c "import geojson"

(test1)orrw-m-4179497:geojson-1.3.2 chris.barker$
Now I go to install in the other environment:

first, check if geojson is there:

(test2)orrw-m-4179497:Temp chris.barker$ python -c "import geojson"

Traceback (most recent call last):

  File "<string>", line 1, in <module>

ImportError: No module named geojson

nope.

so use pip to install from source

(test2)orrw-m-4179497:geojson-1.3.2 chris.barker$ pip install ./

Processing /Users/chris.barker/Temp/geojson-1.3.2

Requirement already satisfied (use --upgrade to upgrade): setuptools in
/Users/chris.barker/miniconda2/envs/test2/lib/python2.7/site-packages/setuptools-19.4-py2.7.egg
(from geojson==1.3.2)

Building wheels for collected packages: geojson

  Running setup.py bdist_wheel for geojson

  Stored in directory:
/Users/chris.barker/Library/Caches/pip/wheels/67/47/d8/01cf2332293b60900ec87ff03bbe9fff92bc85f194e0eb0e74

Successfully built geojson

Installing collected packages: geojson

Successfully installed geojson-1.3.2

You are using pip version 7.1.2, however version 8.0.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

All good. See if it works:

(test2)orrw-m-4179497:Temp chris.barker$ python -c "import geojson; print
geojson.__version__"

1.3.2

(test2)orrw-m-4179497:Temp chris.barker$
yup -- all good.

However -- AARRGG! I then repeated that same exercise with pip 8.0.1, which
was failing for me before (multiple times) but it jsut worked!

This is really, really frustrating...did the downgrade, then re-upgrade of
pip somehow fix this? really, really weird.

OK -- well, I'm done for now with pip for building conda packages....

-CHB














>
> -Rob
>
> On 30 January 2016 at 04:48, Chris Barker - NOAA Federal
> <chris.barker at noaa.gov> wrote:
> >>>  Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3
> from
> >>> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
> >>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
> >>
> >> I think this is saying that pip thinks it has found an
> >> already-installed version of gsw 3.0.3 in sys.path, and that the
> >> directory in your sys.path where it's already installed is
> >>
> >> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
> >
> > That is the temp dir conda sets up to unpack downloaded files, and do
> > its work in -- hence the name. I'll look and see what's there. I'm
> > pretty sure conda build starts out with an empty dir, however. And
> > that dir should not be on sys.path.
> >
> >> I think this means that that directory is (a) in sys.path, and (b)
> >> contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
> >> seems weird and broken.
> >
> > Indeed. And I get the same symptoms with a clean environment that I've
> > set up outside conda build. Though with the same source dir. But with
> > conda build, it's a fresh unpack of the tarball.
> >
> >> Do you have "." in your PYTHONPATH or anything like that?
> >
> > God no!
> >
> >> Don't know why it seems to be building a wheel for it, if it already
> >> thinks that it's installed... this is also odd.
> >
> > Yes it is. But it doesn't install it :-(
> >
> >>
> >> $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
> >>
> >> ? Though I'd think that -I would have the same affect as
> --force-reinstall...
> >>
> > So did I, and I think I tried --force-reinstall already, but I will
> again.
> >
> >> (It doesn't look like the cache dir is your problem here, but you do
> >> probably want to use --no-cache-dir anyway just as good practice, just
> >> because you don't want to accidentally package up a stale version of
> >> the software that got pulled out of your cache instead of the version
> >> you thought you were packaging in the tree in front of you.
> >
> > Exactly. Doesn't seem to make a difference, though.
> >
> >> Also, I think it's a bug in pip that it caches builds of source trees
> >> -- PyPI can enforce the rule that each (package name, version number)
> >> sdist is unique, but for a work-in-progress VCS checkout it's just not
> >> true that (package name, version number) uniquely identifies a
> >> snapshot of the whole tree. So in something like 'pip install .', then
> >> requirement resolution code should treat this as a special requirement
> >> that it wants *this exact tree*, not just any package that has the
> >> same (package name, version number) as this tree; and the resulting
> >> wheel should not be cached.
> >
> > Absolutely! In fact, I'll bet that approach is the source of the
> > problem here. If not automagically, there should be a flag, at least.
> >
> > However, what seems to be happening is that pip is looking outside the
> > current Python environment somewhere to see if this package needs to
> > be installed. It may be something that works with virtualenv, but
> > doesn't with conda environments for some reason.
> >
> > I guess on some level pip simply isn't designed to build and install
> > from local source :-(
> >
> > In the end, I'm still confused: does pip install give me anything that:
> >
> > setup.py install single-version-externally-managed
> >
> > Doesn't? Other that support for non-setuptools installs, anyway.
> >
> > CHB
> >
> >
> >> I don't know if there are any bugs filed
> >> in pip on this...)
> >>
> >> -n
> >>
> >> --
> >> Nathaniel J. Smith -- https://vorpus.org
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> --
> Robert Collins <rbtcollins at hpe.com>
> Distinguished Technologist
> HP Converged Cloud
>
>
> --
> Robert Collins <rbtcollins at hpe.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/f8e9e0d3/attachment.html>

From njs at pobox.com  Fri Jan 29 20:11:51 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 29 Jan 2016 17:11:51 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALT861GOm+1MAgefpg76Js=uAMLq1MNzjM=DsQPOssmEuRHpQg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
 <CALT861GOm+1MAgefpg76Js=uAMLq1MNzjM=DsQPOssmEuRHpQg@mail.gmail.com>
Message-ID: <CAPJVwBk_ftsEgytzuASHKDm=Y2TNpyBWOO4W=XPi66Xz9Tx8mw@mail.gmail.com>

On Fri, Jan 29, 2016 at 11:35 AM, Nate Coraor <nate at bx.psu.edu> wrote:
> On Fri, Jan 22, 2016 at 6:29 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 22 January 2016 at 20:48, M.-A. Lemburg <mal at egenix.com> wrote:
>> > People who rely on Linux distributions want to continue
>> > to do so and get regular updates for system packages from
>> > their system vendor. Having wheel files override these
>> > system packages by including libs directly in the wheel
>> > silently breaks this expectation, potentially opening
>> > up the installations for security holes, difficult to
>> > track bugs and possible version conflicts with already
>> > loaded versions of the shared libs.
>>
>> For the time being, these users should either pass the "--no-binary"
>> option to pip, ask their distro to provide an index of pre-built wheel
>> files for that distro (once we have the distro-specific wheel tagging
>> PEP sorted out), or else ask their distro to update system Python
>> packages in a more timely fashion (or all of the above).
>
>
> Is there a distro-specific wheel tagging PEP in development somewhere that I
> missed? If not, I will get the ball rolling on it.

It's all yours, I think :-).

Some unsolicited advice that you can take or leave...:

I think there are two separable questions that are easy to conflate
here: (1) the use of a distro tag to specify an ABI ("when I say
libssl.so.1, I mean one that exports the same symbols and semantics as
the one that Fedora shipped"), and (2) the use of a distro tag for
packages that want to depend on more system-supplied libraries and let
the distro worry about updating them.

I also think there are two compelling use cases for these: (a) folks
who would be happy with manylinux, except for whatever reason they
can't use it, e.g. because they're on ARM. I bet a platform tag for
Raspbian wheels would be quite popular, even if it still required
people to vendor dependencies. (b) folks who really want integration
between pip and the distro package manager.

So my suggestion would be to start with one PEP that just tries to
define distro-specific platform tags to answer question (1) and
targeting use case (a), and then a second PEP that adds new metadata
for specifying external system requirements to answer question (1) and
target use case (b). The advantage of doing things in this order is
that once you have a platform tag saying "this is a Debian wheel" or
"this is a Fedora wheel" scoping you to a particular distribution,
then your external package metadata can say "I need the package
'libssl1.0.2' version 1.0.2e-1 or better" or "I need the package
'openssl' version 1.0.2e-5.fc24 or better", and avoid the tarpit of
trying to define some cross-distro standard for package naming.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Fri Jan 29 21:14:41 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 30 Jan 2016 12:14:41 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALT861FVLUWiopyCweuGU=E=FgNY3POeJE1paVvyVPYE_FjPmQ@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CADiSq7d-4pr-j2bSu=x+yBgNZvVWWdGUYs0TbgY6Xd4qQMvjiA@mail.gmail.com>
 <CALT861FVLUWiopyCweuGU=E=FgNY3POeJE1paVvyVPYE_FjPmQ@mail.gmail.com>
Message-ID: <CADiSq7fpd3bHb0a=-Ugwn8VO3nMfvGu8Mi7653Km5Bt1PZZfOQ@mail.gmail.com>

On 30 January 2016 at 05:30, Nate Coraor <nate at bx.psu.edu> wrote:
> I wonder if, in relation to this, it may be best to have two separate tags:
> one to indicate that the wheel includes external libraries rolled in to it,
> and one to indicate that it doesn't. That way, a user can make a conscious
> decision as to whether they want to install any wheels that could include
> libraries that won't be maintained by the distribution package manager. That
> way if we end up in a future world where manylinux wheels and
> distro-specific wheels (that may depend on non-default distro packages) live
> in PyPI together, there'd be a way to indicate a preference.

I don't think we want to go into that level of detail in the platform
tag, but metadata for bundled pre-built binaries in wheels and
vendored dependencies in sdists is worth considering as an enhancement
in its own right.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Fri Jan 29 23:44:01 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 29 Jan 2016 23:44:01 -0500
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALT861GOm+1MAgefpg76Js=uAMLq1MNzjM=DsQPOssmEuRHpQg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
 <CALT861GOm+1MAgefpg76Js=uAMLq1MNzjM=DsQPOssmEuRHpQg@mail.gmail.com>
Message-ID: <0B446D67-F066-4215-8117-4C55C74C4C68@stufft.io>


> On Jan 29, 2016, at 2:35 PM, Nate Coraor <nate at bx.psu.edu> wrote:
> 
> Is there a distro-specific wheel tagging PEP in development somewhere that I missed? If not, I will get the ball rolling on it.


I think this a great idea, and I think it actually pairs nicely with the manylinux proposal. It should be pretty easy to cover the vast bulk of users with a handful of platform specific wheels (1-3ish) and then a manylinux wheel to cover the rest. It would let a project use newer toolchains/libraries in the common case, but still fall back to the older ones on more unusual platforms.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/c093cb8e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/c093cb8e/attachment.sig>

From glyph at twistedmatrix.com  Sat Jan 30 01:46:44 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Fri, 29 Jan 2016 22:46:44 -0800
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <0B446D67-F066-4215-8117-4C55C74C4C68@stufft.io>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
 <CALT861GOm+1MAgefpg76Js=uAMLq1MNzjM=DsQPOssmEuRHpQg@mail.gmail.com>
 <0B446D67-F066-4215-8117-4C55C74C4C68@stufft.io>
Message-ID: <DADA5767-955F-460F-96ED-75AA55FD83AE@twistedmatrix.com>


> On Jan 29, 2016, at 8:44 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Jan 29, 2016, at 2:35 PM, Nate Coraor <nate at bx.psu.edu <mailto:nate at bx.psu.edu>> wrote:
>> 
>> Is there a distro-specific wheel tagging PEP in development somewhere that I missed? If not, I will get the ball rolling on it.
> 
> 
> I think this a great idea, and I think it actually pairs nicely with the manylinux proposal. It should be pretty easy to cover the vast bulk of users with a handful of platform specific wheels (1-3ish) and then a manylinux wheel to cover the rest. It would let a project use newer toolchains/libraries in the common case, but still fall back to the older ones on more unusual platforms.

Yes!  This would be fantastic.  There are some libraries you actually want to dynamically link against from the platform, especially if you're writing desktop apps.  On OS X you can do this because /System/*/ is more or less fixed when you are >= some version; on linux less so but it would be very nice to build artifacts for specific versions when possible.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160129/7c177441/attachment.html>

From ncoghlan at gmail.com  Sat Jan 30 02:20:06 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 30 Jan 2016 17:20:06 +1000
Subject: [Distutils] draft PEP: manylinux1
In-Reply-To: <CALT861GOm+1MAgefpg76Js=uAMLq1MNzjM=DsQPOssmEuRHpQg@mail.gmail.com>
References: <CAPJVwBk3Mg2W4DJhK9ivMbJvtcL6RyAbVkQ7Uyvo2oQgB9+kdg@mail.gmail.com>
 <56A09EDF.3000500@egenix.com>
 <CADiSq7eUAez6FFmooMuw6Low0emCEAh8=KSmqZNr8Q0wvtckcQ@mail.gmail.com>
 <56A0AD76.7090107@egenix.com>
 <CAH6Pt5otpLs_NOmPVTzy_EoZZvGstKPBzBHu37E3WxWQtzw8Tw@mail.gmail.com>
 <56A1F757.4050107@egenix.com>
 <CAPJVwBn46kURWfgvL-Cg6SLJm22Jc2zgPXXefn6vNyH9g69Gcw@mail.gmail.com>
 <56A20901.1050300@egenix.com>
 <CADiSq7egerOXJtqYq0zS4QG-k7sziumVZdioY5DQ2vJOw16rdQ@mail.gmail.com>
 <CALT861GOm+1MAgefpg76Js=uAMLq1MNzjM=DsQPOssmEuRHpQg@mail.gmail.com>
Message-ID: <CADiSq7ffXCt-EaC=j+jOMvH_F=Mq+VUOrZEwoRjCt7Kya=x3GQ@mail.gmail.com>

On 30 January 2016 at 05:35, Nate Coraor <nate at bx.psu.edu> wrote:
> On Fri, Jan 22, 2016 at 6:29 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> For the time being, these users should either pass the "--no-binary"
>> option to pip, ask their distro to provide an index of pre-built wheel
>> files for that distro (once we have the distro-specific wheel tagging
>> PEP sorted out), or else ask their distro to update system Python
>> packages in a more timely fashion (or all of the above).
>
> Is there a distro-specific wheel tagging PEP in development somewhere that I
> missed? If not, I will get the ball rolling on it.

Yeah, the "we" there was actually "Nate", since you recently mentioned
having made progress on this for Galaxy :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Jan 30 02:52:27 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 30 Jan 2016 17:52:27 +1000
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
Message-ID: <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>

On 30 January 2016 at 09:29, Nathaniel Smith <njs at pobox.com> wrote:
> Hi all,
>
> I think this is ready for pronouncement now -- thanks to everyone for
> all their feedback over the last few weeks!
>
> The only change relative to the last posting is that we rewrote the
> section on "Platform detection for installers", to switch to letting
> distributors explicitly control manylinux1 compatibility by means of a
> _manylinux module.

In terms of the proposal itself, I think this version is excellent :)

However, I realised that there's an implicit assumption we've been
making that really should be spelled out explicitly: manylinux1 wheels
targeting CPython 3.2 and earlier need to be compiled against a
CPython built in wide Unicode mode, and in those cases, the detection
of manylinux1 compatibility at the platform level should include
checking for "sys.maxunicode > 0xFFFF".

The main reason we need to spell this out explicitly is that while
distros (and I believe other redistributors) build CPython-for-Linux
in wide mode as a matter of course, a Linux checkout of CPython 2.7
will build in narrow mode by default.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rmcgibbo at gmail.com  Sat Jan 30 03:04:06 2016
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Sat, 30 Jan 2016 00:04:06 -0800
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
Message-ID: <CAN4+E8F66wwwWH3aeZ5VHiYn_NYxgU5J1ZrWb7OHG39hrSTuvA@mail.gmail.com>

Oh yes, this is very important! I will have to put a check in auditwheel as
well to verify this in the extensions too.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160130/b5f558b2/attachment.html>

From njs at pobox.com  Sat Jan 30 03:37:14 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 30 Jan 2016 00:37:14 -0800
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
Message-ID: <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>

On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 30 January 2016 at 09:29, Nathaniel Smith <njs at pobox.com> wrote:
>> Hi all,
>>
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
>>
>> The only change relative to the last posting is that we rewrote the
>> section on "Platform detection for installers", to switch to letting
>> distributors explicitly control manylinux1 compatibility by means of a
>> _manylinux module.
>
> In terms of the proposal itself, I think this version is excellent :)
>
> However, I realised that there's an implicit assumption we've been
> making that really should be spelled out explicitly: manylinux1 wheels
> targeting CPython 3.2 and earlier need to be compiled against a
> CPython built in wide Unicode mode, and in those cases, the detection
> of manylinux1 compatibility at the platform level should include
> checking for "sys.maxunicode > 0xFFFF".

Doh, excellent catch!

I've just pushed the obvious update to handle this directly to the
copy of the PEP in the manylinux repository.

Diff: https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc

New text in full:
https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst

I haven't sent to the PEP editors, because they already have another
diff from me sitting in their inboxes and I'm not sure how to do this
in a way that doesn't confuse things :-)

> The main reason we need to spell this out explicitly is that while
> distros (and I believe other redistributors) build CPython-for-Linux
> in wide mode as a matter of course, a Linux checkout of CPython 2.7
> will build in narrow mode by default.

I can confirm that Debian and Anaconda builds of CPython 2.7 both have
sys.maxunicode == 0x10ffff, but Enthought Canopy has sys.maxunicode ==
0xffff. Hmm. I guess they should fix that.

Also the manylinux docker image currently has sys.maxunicode ==
0xffff, so we should definitely fix that :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Sat Jan 30 03:58:08 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 30 Jan 2016 18:58:08 +1000
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
 <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
Message-ID: <CADiSq7ef9z_PQqaXNS=-A=c_7LyM9B4-RXVoMMKwxfThoQ0brA@mail.gmail.com>

On 30 January 2016 at 18:37, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 30 January 2016 at 09:29, Nathaniel Smith <njs at pobox.com> wrote:
>>> Hi all,
>>>
>>> I think this is ready for pronouncement now -- thanks to everyone for
>>> all their feedback over the last few weeks!
>>>
>>> The only change relative to the last posting is that we rewrote the
>>> section on "Platform detection for installers", to switch to letting
>>> distributors explicitly control manylinux1 compatibility by means of a
>>> _manylinux module.
>>
>> In terms of the proposal itself, I think this version is excellent :)
>>
>> However, I realised that there's an implicit assumption we've been
>> making that really should be spelled out explicitly: manylinux1 wheels
>> targeting CPython 3.2 and earlier need to be compiled against a
>> CPython built in wide Unicode mode, and in those cases, the detection
>> of manylinux1 compatibility at the platform level should include
>> checking for "sys.maxunicode > 0xFFFF".
>
> Doh, excellent catch!
>
> I've just pushed the obvious update to handle this directly to the
> copy of the PEP in the manylinux repository.
>
> Diff: https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc
>
> New text in full:
> https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst
>
> I haven't sent to the PEP editors, because they already have another
> diff from me sitting in their inboxes and I'm not sure how to do this
> in a way that doesn't confuse things :-)

I applied both this iteration and the previous one to the PEPs repo in
order to review it, so modulo caching issues, this latest draft is
live now.

I also think this version covers everything we need it to cover, so
I'm going to mark it as Active and point to this post as the
resolution :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From oscar.j.benjamin at gmail.com  Sat Jan 30 08:45:54 2016
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sat, 30 Jan 2016 13:45:54 +0000
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CADiSq7ef9z_PQqaXNS=-A=c_7LyM9B4-RXVoMMKwxfThoQ0brA@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
 <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
 <CADiSq7ef9z_PQqaXNS=-A=c_7LyM9B4-RXVoMMKwxfThoQ0brA@mail.gmail.com>
Message-ID: <CAHVvXxQk4EpMUOYYwnu7Tx6sfsYhNSECF=-zbAGW13FXzycAVw@mail.gmail.com>

On 30 January 2016 at 08:58, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> I applied both this iteration and the previous one to the PEPs repo in
> order to review it, so modulo caching issues, this latest draft is
> live now.
>
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

I had to see PEP 1 to understand what "Active" means but now I see
that it means that this PEP is approved but subject to indefinite
tinkering:
https://www.python.org/dev/peps/pep-0001/#pep-review-resolution

Brilliant, good work everyone! I'm looking forward to this.

So AFAICT the actions are:
1) PyPI allows uploading wheels with manylinux tag.
2) pip updated to recognise these wheels on the appropriate Linux systems.
3) Packagers make and upload wheels

Is there also a policy of informing python-dev once a PEP is approved
here? PEP 1 is ambiguous on that point but something like this really
needs to be known about more widely than this list.

--
Oscar

From vinay_sajip at yahoo.co.uk  Sat Jan 30 11:19:46 2016
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 30 Jan 2016 16:19:46 +0000 (UTC)
Subject: [Distutils] ANN: distlib 0.2.2 released on PyPI
References: <321345900.4034462.1454170786049.JavaMail.yahoo.ref@mail.yahoo.com>
Message-ID: <321345900.4034462.1454170786049.JavaMail.yahoo@mail.yahoo.com>

I've just released version 0.2.2 of distlib on PyPI [1]. For newcomers,
distlib is a library of packaging functionality which is intended to be
usable as the basis for third-party packaging tools.

The main changes in this release are as follows:

* Fixed issue #81: Added support for detecting distributions installed by
  wheel versions >= 0.23 (which use metadata.json rather than pydist.json).
* Updated default PyPI URL to https://pypi.python.org/pypi
* Updated to use different formatting for description field for V1.1
  metadata.
* Corrected ?classifier? to ?classifiers? in the mapping for V1.0 metadata.
* Improved support for Jython when quoting executables in output scripts.
* Fixed issue #77: Made the internal URL used for extended metadata fetches
  configurable via a module attribute.
* Fixed issue #78: Improved entry point parsing to handle leading spaces in
  ini-format files.

A more detailed change log is available at [2].

Please try it out, and if you find any problems or have any suggestions for
improvements, please give some feedback using the issue tracker! [3]

Regards,

Vinay Sajip

[1] https://pypi.python.org/pypi/distlib/0.2.2
[2] https://goo.gl/M3kQzR
[3] https://bitbucket.org/pypa/distlib/issues/new

From donald at stufft.io  Sat Jan 30 11:49:54 2016
From: donald at stufft.io (Donald Stufft)
Date: Sat, 30 Jan 2016 11:49:54 -0500
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CADiSq7ef9z_PQqaXNS=-A=c_7LyM9B4-RXVoMMKwxfThoQ0brA@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
 <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
 <CADiSq7ef9z_PQqaXNS=-A=c_7LyM9B4-RXVoMMKwxfThoQ0brA@mail.gmail.com>
Message-ID: <14911ACD-028D-4121-ABC1-73629B5B09AA@stufft.io>


> On Jan 30, 2016, at 3:58 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

Hilariously I just read on Hacker news:

"Incidentay, we have hired Natanael Copa, the awesome creator of Alpine Linux and are in the process of switching the Docker official image library from ubuntu to Alpine.? [1]

So we might end up needing a MUSL based platform tag in the near future ;)


[1] https://news.ycombinator.com/item?id=11000827

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160130/7f77b66c/attachment.sig>

From njs at pobox.com  Sat Jan 30 14:50:02 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 30 Jan 2016 11:50:02 -0800
Subject: [Distutils] Incorrect detection of 32/64-bit arch on Linux for old
 32-bit VMs migrated to 64-bit kernels
Message-ID: <CAPJVwBnn6Xw=Jswytm1SfHeN3nRbbaYrSfCZK5=8tTZvSu2FuA@mail.gmail.com>

Here's an interesting bug I just discovered on the old and poorly
maintained VM that I keep for running random junk:

# 64-bit kernel
$ uname -a
Linux roberts 4.1.5-x86_64-linode61 #7 SMP Mon Aug 24 13:46:31 EDT
2015 x86_64 x86_64 x86_64 GNU/Linux

# 32-bit userland
$ file /usr/bin/python2.7
/usr/bin/python2.7: ELF 32-bit LSB executable, Intel 80386, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
BuildID[sha1]=0xf481c2b1f8b4328b2f56b642022cc35b4ad91b61, stripped

$ /usr/bin/python2.7

# Yep, definitely 32-bit
>>> import sys; sys.maxint
2147483647

# Uh oh.
>>> import distutils.util; distutils.util.get_platform()
'linux-x86_64'

# Yeah, this is bad
>>> import distlib.wheel; distlib.wheel.COMPATIBLE_TAGS
set([(u'cp21', u'none', u'any'), (u'py21', u'none', u'any'), (u'py23',
u'none', u'any'), (u'cp24', u'none', u'any'), (u'py2', u'none',
u'any'), (u'cp27', u'none', u'any'), (u'cp20', u'none', u'any'),
(u'py22', u'none', u'any'), (u'cp26', u'none', u'any'), (u'cp27',
u'cp27mu', u'linux_x86_64'), (u'py25', u'none', u'any'), (u'cp23',
u'none', u'any'), (u'py24', u'none', u'any'), (u'cp25', u'none',
u'any'), (u'py27', u'none', u'any'), (u'cp22', u'none', u'any'),
(u'cp2', u'none', u'any'), (u'cp27', u'none', u'linux_x86_64'),
(u'py26', u'none', u'any'), (u'py20', u'none', u'any')])

In the past this has never mattered, because there were no linux
wheels on pypi, so even if pip was using the wrong platform tag it
still wouldn't download the wrong thing. But once manylinux1 goes
live, any systems configured like this will start downloading and
installing totally wheels that will crash on import.

The problem seems to be that distutils.util.get_platform() assumes
that the architecture is whatever uname's "machine" field says
(equivalent to uname -m). Unfortunately, this gives the architecture
of the kernel, not of the Python interpreter.

I think the fix is that we should add some check like

if osname == "linux" and machine == "x86_64" and sys.maxsize == 2147483647:
    machine = "i686"

to distutils (in the branches where it's still maintained), and also
need to add a similar workaround to distlib? (With the later needing
to land before or together with manylinux1 enablement.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From matthew.brett at gmail.com  Sat Jan 30 15:17:02 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 30 Jan 2016 12:17:02 -0800
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
 <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
Message-ID: <CAH6Pt5pSOhXwsEDpuxBB1FDEsyMkJT=fHmJ5nayxwOABTP5=iQ@mail.gmail.com>

Hi,

> I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> sys.maxunicode == 0x10ffff, but Enthought Canopy has sys.maxunicode ==
> 0xffff. Hmm. I guess they should fix that.
>
> Also the manylinux docker image currently has sys.maxunicode ==
> 0xffff, so we should definitely fix that :-).

A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
unicode by default.  Are there any known distributions providing UCS2
unicode Pythons?

Cheers,

Matthew

From njs at pobox.com  Sat Jan 30 15:40:53 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 30 Jan 2016 12:40:53 -0800
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
 <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
Message-ID: <CAPJVwB=ZKPec2d-+_ALyhPaMvSt7U0GANrhPpAEQ2VT7NJ8_kQ@mail.gmail.com>

On Sat, Jan 30, 2016 at 12:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
> Also the manylinux docker image currently has sys.maxunicode ==
> 0xffff, so we should definitely fix that :-).

This is fixed now (or will be in a few minutes when this build
finishes: https://quay.io/repository/manylinux/manylinux/build/40961d45-aff6-4ed2-a778-f73413340800)

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Sat Jan 30 15:48:47 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 30 Jan 2016 12:48:47 -0800
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CADiSq7ef9z_PQqaXNS=-A=c_7LyM9B4-RXVoMMKwxfThoQ0brA@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
 <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
 <CADiSq7ef9z_PQqaXNS=-A=c_7LyM9B4-RXVoMMKwxfThoQ0brA@mail.gmail.com>
Message-ID: <CAPJVwBnTHDzr+8u4gCieuW9ZEN_Dk4m+hRQBtzY4hY6UGaWuRw@mail.gmail.com>

On Sat, Jan 30, 2016 at 12:58 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

Awesome! Thanks everyone :-)

Of course, as soon as this happened I discovered two small but
substantive bugs in the text...

- the text refers to 32-bit linux as "i386" (because I foolishly
trusted PEP 425); it turns out that on any modern system the platform
tag is "linux_i686".
- there was a small bug in the is_manylinux1_compatible sample code,
because I missed that distutils.util.get_platform() uses a hyphen
separator instead of an underscore. (But now the sample code is
running as part of the test suite for the docker image, so I know it
works.)

Both are fixed in the diff that I just sent to the pep editors:
  https://github.com/manylinux/manylinux/pull/20/files

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From greg.ewing at canterbury.ac.nz  Sat Jan 30 16:46:32 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 31 Jan 2016 10:46:32 +1300
Subject: [Distutils] Incorrect detection of 32/64-bit arch on Linux for
 old 32-bit VMs migrated to 64-bit kernels
In-Reply-To: <CAPJVwBnn6Xw=Jswytm1SfHeN3nRbbaYrSfCZK5=8tTZvSu2FuA@mail.gmail.com>
References: <CAPJVwBnn6Xw=Jswytm1SfHeN3nRbbaYrSfCZK5=8tTZvSu2FuA@mail.gmail.com>
Message-ID: <56AD2F38.5050605@canterbury.ac.nz>

Nathaniel Smith wrote:
> The problem seems to be that distutils.util.get_platform() assumes
> that the architecture is whatever uname's "machine" field says
> (equivalent to uname -m). Unfortunately, this gives the architecture
> of the kernel, not of the Python interpreter.
> 
> I think the fix is that we should add some check like
> 
> if osname == "linux" and machine == "x86_64" and sys.maxsize == 2147483647:
>     machine = "i686"

If you fix this, please do it for other platforms as well.
I encountered the same problem on MacOSX, where for historical
reasons I have installed Python and all my compiled-from-source
libraries as 32 bit. The result is that py2app includes the
wrong version of the stub executable in the apps it builds.

Conceivably a similar thing could happen on Windows, although
I haven't tested it.

-- 
Greg

From ncoghlan at gmail.com  Sun Jan 31 01:48:35 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 31 Jan 2016 16:48:35 +1000
Subject: [Distutils] [final version?] PEP 513 - A Platform Tag for
 Portable Linux Built Distributions
In-Reply-To: <CAHVvXxQk4EpMUOYYwnu7Tx6sfsYhNSECF=-zbAGW13FXzycAVw@mail.gmail.com>
References: <CAPJVwBnrLNPwMWRtR2DnrGbQc3_E2nE1RDihJZy5M0ksiQgezQ@mail.gmail.com>
 <CADiSq7chmPEa+pPxBFyDmF-E0ueZZxm0JPk=c9duvercFYLbmw@mail.gmail.com>
 <CAPJVwBnJ6vYjcFL5kTZQRv7pmu3TETZdUJ-17aoy5GQH=DagJQ@mail.gmail.com>
 <CADiSq7ef9z_PQqaXNS=-A=c_7LyM9B4-RXVoMMKwxfThoQ0brA@mail.gmail.com>
 <CAHVvXxQk4EpMUOYYwnu7Tx6sfsYhNSECF=-zbAGW13FXzycAVw@mail.gmail.com>
Message-ID: <CADiSq7foKUc1YD_+iJUr7-r5HeTOHKZ=afSNnmBi+g5mudou4g@mail.gmail.com>

On 30 January 2016 at 23:45, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
> On 30 January 2016 at 08:58, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> I applied both this iteration and the previous one to the PEPs repo in
>> order to review it, so modulo caching issues, this latest draft is
>> live now.
>>
>> I also think this version covers everything we need it to cover, so
>> I'm going to mark it as Active and point to this post as the
>> resolution :)
>
> I had to see PEP 1 to understand what "Active" means but now I see
> that it means that this PEP is approved but subject to indefinite
> tinkering:
> https://www.python.org/dev/peps/pep-0001/#pep-review-resolution

Yeah, getting us away from Information PEPs to normal Standards Track
ones is one of the reasons I started tinkering with the way we use the
PEP process. We really do want to be using Standards Track, but that
requires reference specs and implementations that provide the basis
for declaring the PEP "FInal".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia