From qwcode at gmail.com  Mon Dec  1 02:41:16 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 30 Nov 2014 17:41:16 -0800
Subject: [Distutils] Proposal: using /etc/os-release in the "platform
 tag" definition for wheel files
In-Reply-To: <CADiSq7eq4Sx05gm_q3u97boEAk_BbvPZZuynQP2aqLKYcBtXSg@mail.gmail.com>
References: <CADiSq7eq4Sx05gm_q3u97boEAk_BbvPZZuynQP2aqLKYcBtXSg@mail.gmail.com>
Message-ID: <CAPYWazq1NEYApJyMK_rsWNkNJ2tbJT7GfG1eJfhZ782PNH1mvA@mail.gmail.com>

+1 to the idea in general.
Would this be an *edit* to PEP425/427/Wheel-1.0  OR new peps, and a new
wheel version?
As someone using cent6 daily (with no os-release file), I'm greedy for
another fallback technique, but the simplicity of just using os-release
makes sense.
Could a published "linux_x86_64_fedora_20" wheel ever become broken just
due to normal "yum update" activity on fedora_20?  When? Why?

On Thu, Nov 27, 2014 at 10:03 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> We've discussed the idea of changing the wheel file naming scheme to
> deal with Linux previously, but never put together a concrete
> proposal.
>
> The closest we've got is the idea of allowing the platform tag to be
> customised in pip and perhaps bdist_wheel, and while that's good from
> an "enabling experimentation" perspective, it may be overkill if the
> primary goal is just to better support handling of Linux distros.
>
> For starters, here's the current definition of the platform tag in PEP 425:
>
> =================
> The platform tag is simply distutils.util.get_platform() with all
> hyphens - and periods . replaced with underscore _ .
>
> * win32
> * linux_i386
> * linux_x86_64
> =================
>
> Here's my proposed change:
>
> =================
> The default platform tag is distutils.util.get_platform() with all
> hyphens - and periods . replaced with underscore _ . If
> /etc/os-release [N] exists on the system, then the values in the 'ID'
> and 'VERSION_ID' fields are read, all hyphens - and periods . replaced
> with underscore _ , and the results appended to the default tag after
> a separating underscore."
>
> Examples:
>
> * win32
> * macosx_10_6_intel
> * linux_x86_64_fedora_20
> * linux_x86_64_rhel_7_0
> * linux_x86_64_debian_7_0
> * linux_x86_64_ubuntu_14_04
> =================
>
> The [N] reference would then be a reference to
> http://www.freedesktop.org/software/systemd/man/os-release.html for
> the definition of the format of os-release. (Note that while the
> format originated with systemd, plenty of distros have also started
> providing it regardless of which init system they use)
>
> Now, this slightly overspecifies on the *consumer* side. A binary
> wheel that works on "rhel_7_0" for example, should almost certainly
> work on "rhel_7_1". However, that can be addressed on the tooling side
> (e.g. permitting the specification of "additional compatible
> platforms" when invoking pip), rather than needing to be in the
> specification.
>
> This also won't help with older Linux distros that don't offer
> /etc/os-release, but I'm OK with that - those can just continue to
> show up as "linux_x86_64", and PyPI can continue to disallow those
> uploads.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141130/f528ff85/attachment.html>

From holger at merlinux.eu  Mon Dec  1 10:25:17 2014
From: holger at merlinux.eu (holger krekel)
Date: Mon, 1 Dec 2014 09:25:17 +0000
Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2
In-Reply-To: <2EC8ACAD-7E01-4B57-BD79-4882368CE04F@stufft.io>
References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io>
 <2EC8ACAD-7E01-4B57-BD79-4882368CE04F@stufft.io>
Message-ID: <20141201092517.GQ25600@merlinux.eu>

Hi Donald,

On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
> > On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald at stufft.io> wrote:
> > 
> > Starting a new thread with more explicit details at Richard?s request.
> > Essentially the tl;dr here is that we'll switch to using sha2 (specifically
> > sha256).
> 
> Ping?
> 
> Are we OK to make this change?

sorry i didn't get back earlier.  Before the minor release of devpi-server
last week i tried for two hours to change devpi-server to accomodate
your planned pypi.python.org checksum changes.

I found the change cannot easily be done without changes to the underlying
database schema and thus needs a major new release of devpi-server because
an export/import cycle is needed.  When doing that i also want to do
some internal cleanup related to name normalization (and also relating
to recent pypi.python.org changes) but i need a week or two i guess to
do that.  However i now think that if you do the pypi.python.org checksum
change it shouldn't directly break devpi-server but it would remove 
checksum checking.  I'd rather like to have a new major devpi-server 
release out when you do the change.  Is it ok for you to wait a bit still?

best,
holger

From wichert at wiggy.net  Mon Dec  1 13:40:19 2014
From: wichert at wiggy.net (Wichert Akkerman)
Date: Mon, 1 Dec 2014 13:40:19 +0100
Subject: [Distutils] SNI support in pip
Message-ID: <DB8367B2-1358-40AC-8DE3-A4101EE8C9FE@wiggy.net>

I just noticed that pip does not support SNI (on Python 2.7.8). This is a bit problematic for us since we use a private index on a server using SNI, and right pip always aborts with a certificate error. I found a year old ticket that seems related (https://github.com/pypa/pip/issues/1511 <https://github.com/pypa/pip/issues/1511> ) which suggested installing pyOpenSSL, pyasn1 and ndg-httpsclient as a workaround. This did not work for me either: I got a link error during installation.

I am wondering: with Python 2.7.9 about to be released with a backport of Python 3?s ssl module, can pip start supporting SNI without any external dependencies? That would be a huge help for people who need to use SNI.

Regards,
Wichert.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141201/cdfa8d4b/attachment.html>

From donald at stufft.io  Mon Dec  1 13:58:31 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 1 Dec 2014 07:58:31 -0500
Subject: [Distutils] SNI support in pip
In-Reply-To: <DB8367B2-1358-40AC-8DE3-A4101EE8C9FE@wiggy.net>
References: <DB8367B2-1358-40AC-8DE3-A4101EE8C9FE@wiggy.net>
Message-ID: <05E82576-33B8-4C7B-AC19-D1284FB7AE98@stufft.io>


> On Dec 1, 2014, at 7:40 AM, Wichert Akkerman <wichert at wiggy.net> wrote:
> 
> I just noticed that pip does not support SNI (on Python 2.7.8). This is a bit problematic for us since we use a private index on a server using SNI, and right pip always aborts with a certificate error. I found a year old ticket that seems related (https://github.com/pypa/pip/issues/1511 <https://github.com/pypa/pip/issues/1511> ) which suggested installing pyOpenSSL, pyasn1 and ndg-httpsclient as a workaround. This did not work for me either: I got a link error during installation.
> 
> I am wondering: with Python 2.7.9 about to be released with a backport of Python 3?s ssl module, can pip start supporting SNI without any external dependencies? That would be a huge help for people who need to use SNI.
> 
> Regards,
> Wichert.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


Yes, pip just uses requests to validate HTTPS. Requests gates the stdlib SNI stuff on whether or not the stdlib has SSLContext and the HAS_SNI variables. This should automatically work with 2.7.9.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141201/11d3363a/attachment.html>

From p at lists-2014.dobrogost.net  Mon Dec  1 14:55:35 2014
From: p at lists-2014.dobrogost.net (Piotr Dobrogost)
Date: Mon, 1 Dec 2014 14:55:35 +0100
Subject: [Distutils] Are there any plans to move to pip/wheels in buildout?
Message-ID: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>

Hi,

Are there any plans to move from easy_install/eggs to pip/wheels in buildout?
I have an impression that buildout project has stagnated which is
unfortunate taking into consideration how much python packaging has
changed recently.


Regards,
Piotr Dobrogost

From jim at zope.com  Mon Dec  1 15:53:37 2014
From: jim at zope.com (Jim Fulton)
Date: Mon, 1 Dec 2014 09:53:37 -0500
Subject: [Distutils] Standard packaging API? (was Re: Are there any plans to
 move to pip/wheels in buildout?)
Message-ID: <CAPDm-FgMXot-KGO=b2eQ4ME=iXicvYe_xVMKLGEpMX_pDRMYxA@mail.gmail.com>

On Mon, Dec 1, 2014 at 8:55 AM, Piotr Dobrogost
<p at lists-2014.dobrogost.net> wrote:
> Are there any plans to move from easy_install/eggs to pip/wheels in buildout?

Buildout doesn't really use easy_install. It uses
setuptools. Originally, I tried to use easy_install directly (and do
in some special cases where I shouldn't), but I needed a real API,
which is why I moved down to setuptools.  My hope is that some new API
will emerge to replace setuptools.

> I have an impression that buildout project has stagnated

I prefer to say it's stable. :)

A great sign is that other folks on the project have driven recent work.

> which is
> unfortunate taking into consideration how much python packaging has
> changed recently.

Buildout as it, is is entirely dependent on setuptools to add wheel
support, at least until a new API emerges.

AFAIK, pip doesn't provide an API for use by other tools. I'd be very
happy to find out I'm wrong.

I hope there's room for more than one command-line tool for working
with packages in the ecosystem.  It would be crazy for each tool to
implement the low-level packaging machinery separately.  We need a
library to replace setuptools that pip uses and that other tools can
use.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton

From leorochael at gmail.com  Mon Dec  1 17:23:50 2014
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Mon, 1 Dec 2014 14:23:50 -0200
Subject: [Distutils] Standard packaging API? (was Re: Are there any
 plans to move to pip/wheels in buildout?)
In-Reply-To: <CAPDm-FgMXot-KGO=b2eQ4ME=iXicvYe_xVMKLGEpMX_pDRMYxA@mail.gmail.com>
References: <CAPDm-FgMXot-KGO=b2eQ4ME=iXicvYe_xVMKLGEpMX_pDRMYxA@mail.gmail.com>
Message-ID: <CAMxX_oNEJVtjnDMEC6E=DKWjV_TSCSfRe27eb1FhD7Br_kyVYA@mail.gmail.com>

I thought distlib was supposed to be that API... Even though pip doesn't
use it.

Though that would mean a new major version of buildout that worked on
wheels exclusively instead of eggs.

Pip itself has an internal API in the `pip.commands` package. From a casual
glance it seems usable from other programs. I.e. you could import
`pip.commands.install:Install`, instantiate and call `.run()`.



On 1 December 2014 at 12:53, Jim Fulton <jim at zope.com> wrote:

> On Mon, Dec 1, 2014 at 8:55 AM, Piotr Dobrogost
> <p at lists-2014.dobrogost.net> wrote:
> > Are there any plans to move from easy_install/eggs to pip/wheels in
> buildout?
>
> Buildout doesn't really use easy_install. It uses
> setuptools. Originally, I tried to use easy_install directly (and do
> in some special cases where I shouldn't), but I needed a real API,
> which is why I moved down to setuptools.  My hope is that some new API
> will emerge to replace setuptools.
>
> > I have an impression that buildout project has stagnated
>
> I prefer to say it's stable. :)
>
> A great sign is that other folks on the project have driven recent work.
>
> > which is
> > unfortunate taking into consideration how much python packaging has
> > changed recently.
>
> Buildout as it, is is entirely dependent on setuptools to add wheel
> support, at least until a new API emerges.
>
> AFAIK, pip doesn't provide an API for use by other tools. I'd be very
> happy to find out I'm wrong.
>
> I hope there's room for more than one command-line tool for working
> with packages in the ecosystem.  It would be crazy for each tool to
> implement the low-level packaging machinery separately.  We need a
> library to replace setuptools that pip uses and that other tools can
> use.
>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141201/a8ae9607/attachment-0001.html>

From donald at stufft.io  Mon Dec  1 19:35:27 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 1 Dec 2014 13:35:27 -0500
Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2
In-Reply-To: <20141201092517.GQ25600@merlinux.eu>
References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io>
 <2EC8ACAD-7E01-4B57-BD79-4882368CE04F@stufft.io>
 <20141201092517.GQ25600@merlinux.eu>
Message-ID: <696CC3BA-4269-4CDB-94F0-6C44031356A5@stufft.io>


> On Dec 1, 2014, at 4:25 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> Hi Donald,
> 
> On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
>>> On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald at stufft.io> wrote:
>>> 
>>> Starting a new thread with more explicit details at Richard?s request.
>>> Essentially the tl;dr here is that we'll switch to using sha2 (specifically
>>> sha256).
>> 
>> Ping?
>> 
>> Are we OK to make this change?
> 
> sorry i didn't get back earlier.  Before the minor release of devpi-server
> last week i tried for two hours to change devpi-server to accomodate
> your planned pypi.python.org checksum changes.
> 
> I found the change cannot easily be done without changes to the underlying
> database schema and thus needs a major new release of devpi-server because
> an export/import cycle is needed.  When doing that i also want to do
> some internal cleanup related to name normalization (and also relating
> to recent pypi.python.org changes) but i need a week or two i guess to
> do that.  However i now think that if you do the pypi.python.org checksum
> change it shouldn't directly break devpi-server but it would remove 
> checksum checking.  I'd rather like to have a new major devpi-server 
> release out when you do the change.  Is it ok for you to wait a bit still?
> 
> best,
> holger

Yes, we can wait a bit. I was just going over my TODO list and making sure
things weren?t getting lost in the shuffle.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From graffatcolmingov at gmail.com  Mon Dec  1 19:45:23 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 1 Dec 2014 12:45:23 -0600
Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2
In-Reply-To: <696CC3BA-4269-4CDB-94F0-6C44031356A5@stufft.io>
References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io>
 <2EC8ACAD-7E01-4B57-BD79-4882368CE04F@stufft.io>
 <20141201092517.GQ25600@merlinux.eu>
 <696CC3BA-4269-4CDB-94F0-6C44031356A5@stufft.io>
Message-ID: <CAN-Kwu2sFpqHsFNMaPq0_uUqRs6bSNq04nNJ4MKJ05VqiZtwMw@mail.gmail.com>

On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 1, 2014, at 4:25 AM, holger krekel <holger at merlinux.eu> wrote:
>>
>> Hi Donald,
>>
>> On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
>>>> On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald at stufft.io> wrote:
>>>>
>>>> Starting a new thread with more explicit details at Richard?s request.
>>>> Essentially the tl;dr here is that we'll switch to using sha2 (specifically
>>>> sha256).
>>>
>>> Ping?
>>>
>>> Are we OK to make this change?
>>
>> sorry i didn't get back earlier.  Before the minor release of devpi-server
>> last week i tried for two hours to change devpi-server to accomodate
>> your planned pypi.python.org checksum changes.
>>
>> I found the change cannot easily be done without changes to the underlying
>> database schema and thus needs a major new release of devpi-server because
>> an export/import cycle is needed.  When doing that i also want to do
>> some internal cleanup related to name normalization (and also relating
>> to recent pypi.python.org changes) but i need a week or two i guess to
>> do that.  However i now think that if you do the pypi.python.org checksum
>> change it shouldn't directly break devpi-server but it would remove
>> checksum checking.  I'd rather like to have a new major devpi-server
>> release out when you do the change.  Is it ok for you to wait a bit still?
>>
>> best,
>> holger
>
> Yes, we can wait a bit. I was just going over my TODO list and making sure
> things weren?t getting lost in the shuffle.
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Holger,

Is there anyway people on this list can help with the updates to devpi
so that we can get this out sooner?

Cheers,
Ian

From vinay_sajip at yahoo.co.uk  Mon Dec  1 20:54:18 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 1 Dec 2014 19:54:18 +0000 (UTC)
Subject: [Distutils] Standard packaging API? (was Re: Are there any
 plans to move to pip/wheels in buildout?)
In-Reply-To: <CAMxX_oNEJVtjnDMEC6E=DKWjV_TSCSfRe27eb1FhD7Br_kyVYA@mail.gmail.com>
References: <CAMxX_oNEJVtjnDMEC6E=DKWjV_TSCSfRe27eb1FhD7Br_kyVYA@mail.gmail.com>
Message-ID: <138848078.3905590.1417463658265.JavaMail.yahoo@jws11159.mail.ir2.yahoo.com>

AFAIK pip does use distlib (it is vendored by pip), but only for some ancillary functions such as pre-release version checks.
I'm not sure it's a good idea to use pip's internal API (as it's internal, and I don't believe it's been designed for use as a library by external code).
Regards,
Vinay Sajip      From: Leonardo Rochael Almeida <leorochael at gmail.com>
 To: distutils sig <distutils-sig at python.org> 
 Sent: Monday, 1 December 2014, 16:23
 Subject: Re: [Distutils] Standard packaging API? (was Re: Are there any plans to move to pip/wheels in buildout?)
   
I thought distlib was supposed to be that API... Even though pip doesn't use it.
Though that would mean a new major version of buildout that worked on wheels exclusively instead of eggs.
Pip itself has an internal API in the `pip.commands` package. From a casual glance it seems usable from other programs. I.e. you could import `pip.commands.install:Install`, instantiate and call `.run()`.




On 1 December 2014 at 12:53, Jim Fulton <jim at zope.com> wrote:

On Mon, Dec 1, 2014 at 8:55 AM, Piotr Dobrogost
<p at lists-2014.dobrogost.net> wrote:
> Are there any plans to move from easy_install/eggs to pip/wheels in buildout?

Buildout doesn't really use easy_install. It uses
setuptools. Originally, I tried to use easy_install directly (and do
in some special cases where I shouldn't), but I needed a real API,
which is why I moved down to setuptools.? My hope is that some new API
will emerge to replace setuptools.

> I have an impression that buildout project has stagnated

I prefer to say it's stable. :)

A great sign is that other folks on the project have driven recent work.

> which is
> unfortunate taking into consideration how much python packaging has
> changed recently.

Buildout as it, is is entirely dependent on setuptools to add wheel
support, at least until a new API emerges.

AFAIK, pip doesn't provide an API for use by other tools. I'd be very
happy to find out I'm wrong.

I hope there's room for more than one command-line tool for working
with packages in the ecosystem.? It would be crazy for each tool to
implement the low-level packaging machinery separately.? We need a
library to replace setuptools that pip uses and that other tools can
use.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist? -? Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig



_______________________________________________
Distutils-SIG maillist? -? Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig


  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141201/a9ab4de3/attachment.html>

From holger at merlinux.eu  Mon Dec  1 22:23:44 2014
From: holger at merlinux.eu (holger krekel)
Date: Mon, 1 Dec 2014 21:23:44 +0000
Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2
In-Reply-To: <CAN-Kwu2sFpqHsFNMaPq0_uUqRs6bSNq04nNJ4MKJ05VqiZtwMw@mail.gmail.com>
References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io>
 <2EC8ACAD-7E01-4B57-BD79-4882368CE04F@stufft.io>
 <20141201092517.GQ25600@merlinux.eu>
 <696CC3BA-4269-4CDB-94F0-6C44031356A5@stufft.io>
 <CAN-Kwu2sFpqHsFNMaPq0_uUqRs6bSNq04nNJ4MKJ05VqiZtwMw@mail.gmail.com>
Message-ID: <20141201212344.GD5241@merlinux.eu>

On Mon, Dec 01, 2014 at 12:45 -0600, Ian Cordasco wrote:
> On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald at stufft.io> wrote:
> >
> >> On Dec 1, 2014, at 4:25 AM, holger krekel <holger at merlinux.eu> wrote:
> >>
> >> Hi Donald,
> >>
> >> On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
> >>>> On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald at stufft.io> wrote:
> >>>>
> >>>> Starting a new thread with more explicit details at Richard?s request.
> >>>> Essentially the tl;dr here is that we'll switch to using sha2 (specifically
> >>>> sha256).
> >>>
> >>> Ping?
> >>>
> >>> Are we OK to make this change?
> >>
> >> sorry i didn't get back earlier.  Before the minor release of devpi-server
> >> last week i tried for two hours to change devpi-server to accomodate
> >> your planned pypi.python.org checksum changes.
> >>
> >> I found the change cannot easily be done without changes to the underlying
> >> database schema and thus needs a major new release of devpi-server because
> >> an export/import cycle is needed.  When doing that i also want to do
> >> some internal cleanup related to name normalization (and also relating
> >> to recent pypi.python.org changes) but i need a week or two i guess to
> >> do that.  However i now think that if you do the pypi.python.org checksum
> >> change it shouldn't directly break devpi-server but it would remove
> >> checksum checking.  I'd rather like to have a new major devpi-server
> >> release out when you do the change.  Is it ok for you to wait a bit still?
> >>
> >> best,
> >> holger
> >
> > Yes, we can wait a bit. I was just going over my TODO list and making sure
> > things weren?t getting lost in the shuffle.
> 
> Holger,
> 
> Is there anyway people on this list can help with the updates to devpi
> so that we can get this out sooner?

Looking at devpi/server/devpi_server/extpypi.py and 
devpi/server/devpi_server/model.py mainly and changing most places
where "md5" is found in the source and adapting related tests.

Is there a specific reason you are in a hurry if i may ask?

best,
holger

From graffatcolmingov at gmail.com  Mon Dec  1 22:29:47 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 1 Dec 2014 15:29:47 -0600
Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2
In-Reply-To: <20141201212344.GD5241@merlinux.eu>
References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io>
 <2EC8ACAD-7E01-4B57-BD79-4882368CE04F@stufft.io>
 <20141201092517.GQ25600@merlinux.eu>
 <696CC3BA-4269-4CDB-94F0-6C44031356A5@stufft.io>
 <CAN-Kwu2sFpqHsFNMaPq0_uUqRs6bSNq04nNJ4MKJ05VqiZtwMw@mail.gmail.com>
 <20141201212344.GD5241@merlinux.eu>
Message-ID: <CAN-Kwu1at_VMFTTjrkUbVJLMm1_cdAqeA1_+UbjZQrKjc40cxA@mail.gmail.com>

On Mon, Dec 1, 2014 at 3:23 PM, holger krekel <holger at merlinux.eu> wrote:
> On Mon, Dec 01, 2014 at 12:45 -0600, Ian Cordasco wrote:
>> On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald at stufft.io> wrote:
>> >
>> >> On Dec 1, 2014, at 4:25 AM, holger krekel <holger at merlinux.eu> wrote:
>> >>
>> >> Hi Donald,
>> >>
>> >> On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
>> >>>> On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald at stufft.io> wrote:
>> >>>>
>> >>>> Starting a new thread with more explicit details at Richard?s request.
>> >>>> Essentially the tl;dr here is that we'll switch to using sha2 (specifically
>> >>>> sha256).
>> >>>
>> >>> Ping?
>> >>>
>> >>> Are we OK to make this change?
>> >>
>> >> sorry i didn't get back earlier.  Before the minor release of devpi-server
>> >> last week i tried for two hours to change devpi-server to accomodate
>> >> your planned pypi.python.org checksum changes.
>> >>
>> >> I found the change cannot easily be done without changes to the underlying
>> >> database schema and thus needs a major new release of devpi-server because
>> >> an export/import cycle is needed.  When doing that i also want to do
>> >> some internal cleanup related to name normalization (and also relating
>> >> to recent pypi.python.org changes) but i need a week or two i guess to
>> >> do that.  However i now think that if you do the pypi.python.org checksum
>> >> change it shouldn't directly break devpi-server but it would remove
>> >> checksum checking.  I'd rather like to have a new major devpi-server
>> >> release out when you do the change.  Is it ok for you to wait a bit still?
>> >>
>> >> best,
>> >> holger
>> >
>> > Yes, we can wait a bit. I was just going over my TODO list and making sure
>> > things weren?t getting lost in the shuffle.
>>
>> Holger,
>>
>> Is there anyway people on this list can help with the updates to devpi
>> so that we can get this out sooner?
>
> Looking at devpi/server/devpi_server/extpypi.py and
> devpi/server/devpi_server/model.py mainly and changing most places
> where "md5" is found in the source and adapting related tests.
>
> Is there a specific reason you are in a hurry if i may ask?
>
> best,
> holger

No real hurry. I just like helping out when there's an opening and
this thread has been around for a short while already. Given the topic
is related to the security of PyPI and its users, I'd like to help
move that forward if possible. That's all. (It's mostly me being
selfish.)

From holger at merlinux.eu  Mon Dec  1 22:41:14 2014
From: holger at merlinux.eu (holger krekel)
Date: Mon, 1 Dec 2014 21:41:14 +0000
Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2
In-Reply-To: <CAN-Kwu1at_VMFTTjrkUbVJLMm1_cdAqeA1_+UbjZQrKjc40cxA@mail.gmail.com>
References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io>
 <2EC8ACAD-7E01-4B57-BD79-4882368CE04F@stufft.io>
 <20141201092517.GQ25600@merlinux.eu>
 <696CC3BA-4269-4CDB-94F0-6C44031356A5@stufft.io>
 <CAN-Kwu2sFpqHsFNMaPq0_uUqRs6bSNq04nNJ4MKJ05VqiZtwMw@mail.gmail.com>
 <20141201212344.GD5241@merlinux.eu>
 <CAN-Kwu1at_VMFTTjrkUbVJLMm1_cdAqeA1_+UbjZQrKjc40cxA@mail.gmail.com>
Message-ID: <20141201214114.GF5241@merlinux.eu>

On Mon, Dec 01, 2014 at 15:29 -0600, Ian Cordasco wrote:
> On Mon, Dec 1, 2014 at 3:23 PM, holger krekel <holger at merlinux.eu> wrote:
> > On Mon, Dec 01, 2014 at 12:45 -0600, Ian Cordasco wrote:
> >> On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald at stufft.io> wrote:
> >> >
> >> >> On Dec 1, 2014, at 4:25 AM, holger krekel <holger at merlinux.eu> wrote:
> >> >>
> >> >> Hi Donald,
> >> >>
> >> >> On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
> >> >>>> On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald at stufft.io> wrote:
> >> >>>>
> >> >>>> Starting a new thread with more explicit details at Richard?s request.
> >> >>>> Essentially the tl;dr here is that we'll switch to using sha2 (specifically
> >> >>>> sha256).
> >> >>>
> >> >>> Ping?
> >> >>>
> >> >>> Are we OK to make this change?
> >> >>
> >> >> sorry i didn't get back earlier.  Before the minor release of devpi-server
> >> >> last week i tried for two hours to change devpi-server to accomodate
> >> >> your planned pypi.python.org checksum changes.
> >> >>
> >> >> I found the change cannot easily be done without changes to the underlying
> >> >> database schema and thus needs a major new release of devpi-server because
> >> >> an export/import cycle is needed.  When doing that i also want to do
> >> >> some internal cleanup related to name normalization (and also relating
> >> >> to recent pypi.python.org changes) but i need a week or two i guess to
> >> >> do that.  However i now think that if you do the pypi.python.org checksum
> >> >> change it shouldn't directly break devpi-server but it would remove
> >> >> checksum checking.  I'd rather like to have a new major devpi-server
> >> >> release out when you do the change.  Is it ok for you to wait a bit still?
> >> >>
> >> >> best,
> >> >> holger
> >> >
> >> > Yes, we can wait a bit. I was just going over my TODO list and making sure
> >> > things weren?t getting lost in the shuffle.
> >>
> >> Holger,
> >>
> >> Is there anyway people on this list can help with the updates to devpi
> >> so that we can get this out sooner?
> >
> > Looking at devpi/server/devpi_server/extpypi.py and
> > devpi/server/devpi_server/model.py mainly and changing most places
> > where "md5" is found in the source and adapting related tests.
> >
> > Is there a specific reason you are in a hurry if i may ask?
> >
> > best,
> > holger
> 
> No real hurry. I just like helping out when there's an opening and
> this thread has been around for a short while already. Given the topic
> is related to the security of PyPI and its users, I'd like to help
> move that forward if possible. That's all. (It's mostly me being
> selfish.)

Quite an empathic form of selfishness.  If you want to check things out
and have questions please feel free to ask maybe privately.

holger

From ben+python at benfinney.id.au  Tue Dec  2 01:21:49 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 02 Dec 2014 11:21:49 +1100
Subject: [Distutils] Package's declared latest version ignored by Warehouse
Message-ID: <85k32avr36.fsf@benfinney.id.au>

Howdy all,

The Warehouse is ignoring the feature of PyPI which sets particular
versions of a package visible or not visible. It makes all versions
visible regardless.

This is a problem when, for example, a package has been uploaded but
should not be shown by default.

An example is the ?python-daemon? package. At PyPI, the latest visible
version is 1.5.5, as requested by the package maintainer. That is
reflected as the default version shown at
<URL:https://pypi.python.org/pypi/python-daemon/>. Also, when viewing
other versions, the ?Latest version? link appears, and correctly shows
that the latest version is 1.5.5.

But at Warehouse, the settings for which versions should be hidden are
ignored by the application, and a different version is shown by default
<URL:https://warehouse.python.org/project/python-daemon/>. That page
should show version 1.5.5, as selected by the package maintainer in the
PyPI database.

-- 
 \      ?He who allows oppression, shares the crime.? ?Erasmus Darwin, |
  `\                                     grandfather of Charles Darwin |
_o__)                                                                  |
Ben Finney


From graffatcolmingov at gmail.com  Tue Dec  2 01:31:01 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 1 Dec 2014 18:31:01 -0600
Subject: [Distutils] Package's declared latest version ignored by
	Warehouse
In-Reply-To: <85k32avr36.fsf@benfinney.id.au>
References: <85k32avr36.fsf@benfinney.id.au>
Message-ID: <CAN-Kwu01VBuAdQpTSTqtSnngb1Q7=vaohrBb5KL95rqn84kx=g@mail.gmail.com>

On Dec 1, 2014 6:22 PM, "Ben Finney" <ben+python at benfinney.id.au> wrote:
>
> Howdy all,
>
> The Warehouse is ignoring the feature of PyPI which sets particular
> versions of a package visible or not visible. It makes all versions
> visible regardless.
>
> This is a problem when, for example, a package has been uploaded but
> should not be shown by default.
>
> An example is the ?python-daemon? package. At PyPI, the latest visible
> version is 1.5.5, as requested by the package maintainer. That is
> reflected as the default version shown at
> <URL:https://pypi.python.org/pypi/python-daemon/>. Also, when viewing
> other versions, the ?Latest version? link appears, and correctly shows
> that the latest version is 1.5.5.
>
> But at Warehouse, the settings for which versions should be hidden are
> ignored by the application, and a different version is shown by default
> <URL:https://warehouse.python.org/project/python-daemon/>. That page
> should show version 1.5.5, as selected by the package maintainer in the
> PyPI database.
>
> --
>  \      ?He who allows oppression, shares the crime.? ?Erasmus Darwin, |
>   `\                                     grandfather of Charles Darwin |
> _o__)                                                                  |
> Ben Finney
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

I haven't verified this but could you file a bug report at the warehouse
project with this information?

Thanks,
Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141201/b2063bd7/attachment.html>

From donald at stufft.io  Tue Dec  2 01:36:15 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 1 Dec 2014 19:36:15 -0500
Subject: [Distutils] Package's declared latest version ignored by
	Warehouse
In-Reply-To: <85k32avr36.fsf@benfinney.id.au>
References: <85k32avr36.fsf@benfinney.id.au>
Message-ID: <385C9FA9-BB01-46C4-940F-2BC66F48E94A@stufft.io>


> On Dec 1, 2014, at 7:21 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> 
> Howdy all,
> 
> The Warehouse is ignoring the feature of PyPI which sets particular
> versions of a package visible or not visible. It makes all versions
> visible regardless.
> 
> This is a problem when, for example, a package has been uploaded but
> should not be shown by default.
> 
> An example is the ?python-daemon? package. At PyPI, the latest visible
> version is 1.5.5, as requested by the package maintainer. That is
> reflected as the default version shown at
> <URL:https://pypi.python.org/pypi/python-daemon/>. Also, when viewing
> other versions, the ?Latest version? link appears, and correctly shows
> that the latest version is 1.5.5.
> 
> But at Warehouse, the settings for which versions should be hidden are
> ignored by the application, and a different version is shown by default
> <URL:https://warehouse.python.org/project/python-daemon/>. That page
> should show version 1.5.5, as selected by the package maintainer in the
> PyPI database.

Warehouse drops support for the ?hidden? setting of PyPI. For a few
reasons:

- It?s confusing
  - Package authors often think that if they hide a version it means
    it won?t be installable by pip, however the flag is UI only.
  - Package installers looking at the page expect that ``pip install foo``
    will get the version shown by default, however that might not be
    the case.
- It was somewhat of a workaround for the way the legacy UI was structured
  that no longer exists anymore. Legacy UI if you have more than one version
  unhidden instead of showing you the latest release information on /pypi/foo/
  it shows you a list of versions which you can click to get to each unhidden
  release. On Warehouse /project/foo/ always shows the latest release and there
  is also always a list to view the details for each individual release.

Can you go into some detail about what the use case is for having something other
than the latest version be the default version shown?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From ben+python at benfinney.id.au  Tue Dec  2 01:40:03 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 02 Dec 2014 11:40:03 +1100
Subject: [Distutils] Package's declared latest version ignored by
	Warehouse
References: <85k32avr36.fsf@benfinney.id.au>
 <385C9FA9-BB01-46C4-940F-2BC66F48E94A@stufft.io>
Message-ID: <85fvcyvq8s.fsf@benfinney.id.au>

Donald Stufft <donald at stufft.io> writes:

> Can you go into some detail about what the use case is for having
> something other than the latest version be the default version shown?

Not much detail needed: The package version was released before it was
realised that it breaks many people's systems on upgrade.

To address this, without re-writing history, it is desirable that the
default version be an earlier, known-working version.

I didn't realise that package fetch-and-install tools fail to honour
that setting. So now I don't have a good solution.

-- 
 \        ?All opinions are not equal. Some are a very great deal more |
  `\   robust, sophisticated, and well supported in logic and argument |
_o__)                                     than others.? ?Douglas Adams |
Ben Finney


From donald at stufft.io  Tue Dec  2 01:44:50 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 1 Dec 2014 19:44:50 -0500
Subject: [Distutils] Package's declared latest version ignored by
	Warehouse
In-Reply-To: <85fvcyvq8s.fsf@benfinney.id.au>
References: <85k32avr36.fsf@benfinney.id.au>
 <385C9FA9-BB01-46C4-940F-2BC66F48E94A@stufft.io>
 <85fvcyvq8s.fsf@benfinney.id.au>
Message-ID: <AEE9E077-7DC1-4F80-8143-7843EAB9676E@stufft.io>


> On Dec 1, 2014, at 7:40 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> 
> Donald Stufft <donald at stufft.io> writes:
> 
>> Can you go into some detail about what the use case is for having
>> something other than the latest version be the default version shown?
> 
> Not much detail needed: The package version was released before it was
> realised that it breaks many people's systems on upgrade.
> 
> To address this, without re-writing history, it is desirable that the
> default version be an earlier, known-working version.
> 
> I didn't realise that package fetch-and-install tools fail to honour
> that setting. So now I don't have a good solution.

You can always remove a version from PyPI if it?s breaking things for
people. That?ll stop *new* people from being broken. You can also of
course revert whatever changes people are being broken by and upload
that as a new version.

Generally it?s hard to go ?backwards? and once you release a bit of
software assume that it?s released and just increment versions again.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From ncoghlan at gmail.com  Tue Dec  2 02:17:49 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 2 Dec 2014 11:17:49 +1000
Subject: [Distutils] SNI support in pip
In-Reply-To: <05E82576-33B8-4C7B-AC19-D1284FB7AE98@stufft.io>
References: <DB8367B2-1358-40AC-8DE3-A4101EE8C9FE@wiggy.net>
 <05E82576-33B8-4C7B-AC19-D1284FB7AE98@stufft.io>
Message-ID: <CADiSq7dGSsxtzLT-fCEiMOc2WB8rWCq_13qdEey28qVCdE0mFw@mail.gmail.com>

On 1 December 2014 at 22:58, Donald Stufft <donald at stufft.io> wrote:
> On Dec 1, 2014, at 7:40 AM, Wichert Akkerman <wichert at wiggy.net> wrote:
> I am wondering: with Python 2.7.9 about to be released with a backport of
> Python 3?s ssl module, can pip start supporting SNI without any external
> dependencies? That would be a huge help for people who need to use SNI.
>
> Regards,
> Wichert.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> Yes, pip just uses requests to validate HTTPS. Requests gates the stdlib SNI
> stuff on whether or not the stdlib has SSLContext and the HAS_SNI variables.
> This should automatically work with 2.7.9.

For the benefit of anyone that isn't already aware, the transport
security work in the packaging ecosystem was actually one of the
factors feeding into the eventual decision to backport the Python 3
transport security improvements to Python 2.7.

ensurepip + the SSL/TLS upgrades + defaulting to verified HTTPS all
ending up in the same Python 2.7 release arose out of the process of
figuring out how to address an inter-connected set of significant
usability issues (which become much harder to ignore once you're
working on secure package distribution infrastructure).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Dec  2 02:44:23 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 2 Dec 2014 11:44:23 +1000
Subject: [Distutils] Standard packaging API? (was Re: Are there any
 plans to move to pip/wheels in buildout?)
In-Reply-To: <138848078.3905590.1417463658265.JavaMail.yahoo@jws11159.mail.ir2.yahoo.com>
References: <CAMxX_oNEJVtjnDMEC6E=DKWjV_TSCSfRe27eb1FhD7Br_kyVYA@mail.gmail.com>
 <138848078.3905590.1417463658265.JavaMail.yahoo@jws11159.mail.ir2.yahoo.com>
Message-ID: <CADiSq7fynj_M407opkeZEDTJswuFRhvJCX9fhgzsgPBiA=ZNEw@mail.gmail.com>

On 2 December 2014 at 05:54, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> AFAIK pip does use distlib (it is vendored by pip), but only for some
> ancillary functions such as pre-release version checks.

The current direction things appear to be going is:

* use distlib to get a "batteries included" approach to packaging
infrastructure (which may include some distlib specific capabilities)
* use the reborn "packaging" if you want a minimalist "just the
approved PEPs, thanks" API

The latter is at https://pypi.python.org/pypi/packaging, and
maintained by Donald & Marcus as a way of factoring out a supported
pip independent library API for the interoperable PEP based behaviour.

The current focus is on ensuring it provides a suitable API to
implement the PEP 440 support in both pip and setuptools, so I'm not
sure if it can be considered stable just yet.

Longer term, it may potentially make sense to have distlib also depend
on the lower level packaging API, but there's also value in having two
somewhat independent implementations to help reduce the chance of the
overall ecosystem coming to rely on implementation defined behaviour
again.

> I'm not sure it's a good idea to use pip's internal API (as it's internal,
> and I don't believe it's been designed for use as a library by external
> code).

Agreed - the components intended for external use are the ones being
factored out into the packaging.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ben+python at benfinney.id.au  Tue Dec  2 02:44:31 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 02 Dec 2014 12:44:31 +1100
Subject: [Distutils] Package's declared latest version ignored by
	Warehouse
References: <85k32avr36.fsf@benfinney.id.au>
 <385C9FA9-BB01-46C4-940F-2BC66F48E94A@stufft.io>
 <85fvcyvq8s.fsf@benfinney.id.au>
 <AEE9E077-7DC1-4F80-8143-7843EAB9676E@stufft.io>
Message-ID: <85bnnmvn9c.fsf@benfinney.id.au>

Donald Stufft <donald at stufft.io> writes:

> You can always remove a version from PyPI if it?s breaking things for
> people.

I ond't consider that a good option, since I do want some testers to get
it from the published location.

> You can also of course revert whatever changes people are being broken
> by and upload that as a new version.

Obviously, yes. That leaves the undesirable version online until such a
fix is available.

But before a fixed version is available, a quick ?this version is not
for public consumption (now)? is what I looked for, and believed I had
found with the ?hide this version? feature.

-- 
 \       ?Faith is the determination to remain ignorant in the face of |
  `\                 all evidence that you are ignorant.? ?Shaun Mason |
_o__)                                                                  |
Ben Finney


From donald at stufft.io  Tue Dec  2 02:56:38 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 1 Dec 2014 20:56:38 -0500
Subject: [Distutils] Standard packaging API? (was Re: Are there any
	plans to move to pip/wheels in buildout?)
In-Reply-To: <CADiSq7fynj_M407opkeZEDTJswuFRhvJCX9fhgzsgPBiA=ZNEw@mail.gmail.com>
References: <CAMxX_oNEJVtjnDMEC6E=DKWjV_TSCSfRe27eb1FhD7Br_kyVYA@mail.gmail.com>
 <138848078.3905590.1417463658265.JavaMail.yahoo@jws11159.mail.ir2.yahoo.com>
 <CADiSq7fynj_M407opkeZEDTJswuFRhvJCX9fhgzsgPBiA=ZNEw@mail.gmail.com>
Message-ID: <735ED8CB-D7B5-4FCD-B054-6A793C05B8EF@stufft.io>


> On Dec 1, 2014, at 8:44 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 2 December 2014 at 05:54, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> AFAIK pip does use distlib (it is vendored by pip), but only for some
>> ancillary functions such as pre-release version checks.
> 
> The current direction things appear to be going is:
> 
> * use distlib to get a "batteries included" approach to packaging
> infrastructure (which may include some distlib specific capabilities)
> * use the reborn "packaging" if you want a minimalist "just the
> approved PEPs, thanks? API

This is how I view the two projects yes. The only things I view as
acceptable for the packaging lib that isn?t directly from a PEP are
?supporting? things.

An example would be we have a Version class which implements PEP 440
style versions, that comes straight from the PEP. However we also have
a LegacyVersion class which implements the setuptools style version
numbers, this doesn?t come from a PEP but it?s needed as a companion
to Version in order to sanely handle versions in the real world. An
additional example of what wouldn?t be allowed is a SemanticVersion
class which implements https://semver.org/ (unless of course a PEP was
written that detailed how to use that within the python packaging
ecosystem).

At some point it may make sense to move the packaging lib into the
stdlib. However I don?t really feel a compelling need to do this
since both 2.x and 3.x have ensurepip now and the state of the ecosystem
is changing rapidly enough that I don?t suspect the stdlib version to
be very useful to many people. This is particularly true since it?s
a pure Python lib which is super easy to install.

> 
> The latter is at https://pypi.python.org/pypi/packaging, and
> maintained by Donald & Marcus as a way of factoring out a supported
> pip independent library API for the interoperable PEP based behaviour.
> 
> The current focus is on ensuring it provides a suitable API to
> implement the PEP 440 support in both pip and setuptools, so I'm not
> sure if it can be considered stable just yet.

Right. The status of packaging right now is more or less we?ll change
things as we need to. Probably we should mark specific APIs as provisional
or not. For instance we have releases of setuptools/pip that support
PEP 440 using the packaging lib I consider those particular APIs to require
a fairly strong level of backwards compatibility.

> 
> Longer term, it may potentially make sense to have distlib also depend
> on the lower level packaging API, but there's also value in having two
> somewhat independent implementations to help reduce the chance of the
> overall ecosystem coming to rely on implementation defined behaviour
> again.
> 
>> I'm not sure it's a good idea to use pip's internal API (as it's internal,
>> and I don't believe it's been designed for use as a library by external
>> code).
> 
> Agreed - the components intended for external use are the ones being
> factored out into the packaging.

Historically our answer has been that pip?s API is internal and subject
to change at any time. However we?re willing to listen to specific
requests for particular APIs to be made public and given real backwards
compatibility promises. At this point though I think you?d need to make
a pretty compelling argument about why it needed to be done immediately
and couldn?t wait for a standards based implementation to be created
that pip itself would switch to.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From marius at pov.lt  Tue Dec  2 11:26:21 2014
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 2 Dec 2014 12:26:21 +0200
Subject: [Distutils] Are there any plans to move to pip/wheels in
 buildout?
In-Reply-To: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
Message-ID: <20141202102621.GB17115@fridge.pov.lt>

On Mon, Dec 01, 2014 at 02:55:35PM +0100, Piotr Dobrogost wrote:
> Are there any plans to move from easy_install/eggs to pip/wheels in buildout?

No plans, AFAIK.  But if setuptools/easy_install gained native support
for wheels, buildout would get it too.  Are there any plans to add
native wheel support to easy_install?

> I have an impression that buildout project has stagnated which is
> unfortunate taking into consideration how much python packaging has
> changed recently.

Very unfortunate.

(I'm trying to avoid buildout for new projects, but scripting around
Virtualenv with Makefiles is sometimes rather clumsy.)

Marius Gedminas
-- 
HOST SYSTEM RESPONDING, PROBABLY UP...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141202/f397e029/attachment.sig>

From olivier.grisel at ensta.org  Tue Dec  2 11:57:35 2014
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Tue, 2 Dec 2014 11:57:35 +0100
Subject: [Distutils] Proposal: using /etc/os-release in the "platform
 tag" definition for wheel files
In-Reply-To: <CAPYWazq1NEYApJyMK_rsWNkNJ2tbJT7GfG1eJfhZ782PNH1mvA@mail.gmail.com>
References: <CADiSq7eq4Sx05gm_q3u97boEAk_BbvPZZuynQP2aqLKYcBtXSg@mail.gmail.com>
 <CAPYWazq1NEYApJyMK_rsWNkNJ2tbJT7GfG1eJfhZ782PNH1mvA@mail.gmail.com>
Message-ID: <CAFvE7K7iXY2A9bTLaEBmngx8zeX2kAHdTOQ4Dt5rOwUXrWpe0g@mail.gmail.com>

+1 to the idea in general. as well.

Here is another use case to motivate distro-aware platform tags for
linux wheels: speeding up continuous integration.

Many Python projects related to the SciPy ecosystem by sharing use the
following rackspace cloud storage container to mutualize build
artifacts:

http://wheels.scikit-image.org/

This makes it really easy to setup build chains to test libraries with
against recent (or even development) version of they dependencies that
cannot be apt-get installed on travis hosts for instance. Building
from source is often not an option for a project that have a
dependency on scipy for instance, the scipy build time itself would
eat most of the travis job allowed time and would not leave enough
time. And it would be a complete waste of resource to rebuild scipy
over and over again.

Unfortunately, all the wheels you find on
http://wheels.scikit-image.org with the "linux_x86_64" are only
expected to work on the Ubuntu Precise platform (which is used by
travis).

This ambiguous platform tag is really annoying:

- it prevents people who would like to setup CI for other linux
platforms (e.g. docker based CI engines that would test wheels on
redhat or centos) to share the same distribution infrastructure,
- the day travis changes its base distro we will have no way to detect
which wheels were built on the old distro and which wheels are built
on the newer version of the distro.

Note that those wheels are not meant to be used by our end-users, only
by CI tools for testing or by developers who would like to quickly
reproduce without having to rebuild everything from scratch.

If the experiment show that such distro tagged wheels are actually as
stable as their are supposed too (which can be mostly tested via CI),
then we could discuss later about the opportunity to distribute Linux
wheels on PyPI for the end users but to me this is not a priority at
all.

-- 
Olivier

From ncoghlan at gmail.com  Tue Dec  2 12:56:13 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 2 Dec 2014 21:56:13 +1000
Subject: [Distutils] Proposal: using /etc/os-release in the "platform
 tag" definition for wheel files
In-Reply-To: <CAPYWazq1NEYApJyMK_rsWNkNJ2tbJT7GfG1eJfhZ782PNH1mvA@mail.gmail.com>
References: <CADiSq7eq4Sx05gm_q3u97boEAk_BbvPZZuynQP2aqLKYcBtXSg@mail.gmail.com>
 <CAPYWazq1NEYApJyMK_rsWNkNJ2tbJT7GfG1eJfhZ782PNH1mvA@mail.gmail.com>
Message-ID: <CADiSq7ehhiX6jjDA26SLLazMA8=aRSKCKGbZJi4bgQYuMNy2SQ@mail.gmail.com>

On 1 December 2014 at 11:41, Marcus Smith <qwcode at gmail.com> wrote:
> +1 to the idea in general.
> Would this be an *edit* to PEP425/427/Wheel-1.0  OR new peps, and a new
> wheel version?

I'm currently thinking no change to the wheel spec, but potentially a
PEP to define a standard way to override and/or supplement the default
compatibility tags.

> As someone using cent6 daily (with no os-release file), I'm greedy for
> another fallback technique, but the simplicity of just using os-release
> makes sense.
> Could a published "linux_x86_64_fedora_20" wheel ever become broken just due
> to normal "yum update" activity on fedora_20?  When? Why?

It's technically possible to get an ABI break mid-cycle on Fedora, as
it doesn't have the same level of restrictions against rebasing
components that RHEL/CentOS do. Actually encountering an ABI break
would still be pretty unlucky though.

However, I realised my original idea doesn't quite work, since
derivative distros may strive to be ABI compatible with their
upstreams. While /etc/os-release sort of supports that concept via
ID_LIKE, it's entirely vague about what that actually means, and the
"ID_LIKE" distro references aren't versioned at all. There's also the
fact that at least RHEL/CentOS aim to remain ABI compatible for the
duration of an entire release series, so including the full version of
/etc/os-release would overspecify things.

That got me back to the idea of being able to customise the platform
tag which various folks have brought up in the past. At that point,
the definition in PEP 425 would become the specification for the
*default* platform tag, and we'd look at adding ways to override it
both when building wheels and when installing them.

The main possibility I thought of there was to come up with a
convention for overriding the platform tag at the *virtual
environment* level. Then bdist_wheel, pip and other projects could
check for the override before falling back to the default definition
from PEP 425. If we designed the mechanisms correctly, it could
potentially also be used to address the "no SOABI details" problem on
Python 2.7.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From reinout at vanrees.org  Tue Dec  2 13:43:45 2014
From: reinout at vanrees.org (Reinout van Rees)
Date: Tue, 02 Dec 2014 13:43:45 +0100
Subject: [Distutils] SNI support in pip
In-Reply-To: <DB8367B2-1358-40AC-8DE3-A4101EE8C9FE@wiggy.net>
References: <DB8367B2-1358-40AC-8DE3-A4101EE8C9FE@wiggy.net>
Message-ID: <m5kc62$ded$1@ger.gmane.org>

On 01-12-14 13:40, Wichert Akkerman wrote:
> I just noticed that pip does not support SNI (on Python 2.7.8). This is
> a bit problematic for us since we use a private index on a server using
> SNI, and right pip always aborts with a certificate error.

Same problem here (though with buildout). We did manage to get it 
working, though.

a) Using the latest and greatest setuptools/buildout. Not sure it helped 
in the end, because I thought setuptool contained requests, but 
apparently that's pip :-)

b) Making sure the server that you use for your packaging is the first 
one in the SNI list. This was the biggest part of the fix.

c) We still had errors afterwards, but that was due to a misconfigured 
certificate chain. All the parent certificates between your certificate 
and a well-known root certificate need to be inside the certificate 
file. A browser will happily fetch the intermediate certificates, but 
curl, wget, setuptools, older apt-get and so on won't.




Reinout

-- 
Reinout van Rees                          http://reinout.vanrees.org/
reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"


From reinout at vanrees.org  Tue Dec  2 13:53:19 2014
From: reinout at vanrees.org (Reinout van Rees)
Date: Tue, 02 Dec 2014 13:53:19 +0100
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
Message-ID: <m5kco1$n7g$1@ger.gmane.org>

On 01-12-14 14:55, Piotr Dobrogost wrote:
> Are there any plans to move from easy_install/eggs to pip/wheels in buildout?

I asked that too, once. I think Jim Fulton said that there was a 
mismatch between pip's and buildout's goal. If I remember correctly the 
problem is this:

- Pip installs everything into (normally) a virtualenv. So your have ONE 
project with ONE single set of packages.

- Buildout allows different combinations of packages per part. So your 
sphinx documentation buildout doesn't have to have your webserver 
packages in its sys.path.

Jim, is that a fair summary? (Of course, this only looks at the python 
package part of what buildout does, not at the whole recipe mechanism).


Something I don't understand here is that there *is* a single version 
list. And so it feels like it doesn't really matter that much if you 
just lump all the packages together (like pip does).

I could imagine a buildout that just uses pip every time it has to 
install a python package. Feeding pip the version list, of course. And 
then puts the buildout.cfg mechanism on top of it.


The alternative is, like Marius said in his reply, to use virtualenv + 
pip + makefiles or so. And you'd have to add something that replaces the 
recipes. For that, I really like buildout :-)


Reinout

-- 
Reinout van Rees                          http://reinout.vanrees.org/
reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"


From leorochael at gmail.com  Tue Dec  2 14:10:43 2014
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Tue, 2 Dec 2014 11:10:43 -0200
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <m5kco1$n7g$1@ger.gmane.org>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
Message-ID: <CAMxX_oPgpu95HLj0AbPsMm0y+MdRrVr-F+MyVW70FKK8njQcow@mail.gmail.com>

On 2 December 2014 at 10:53, Reinout van Rees <reinout at vanrees.org> wrote:

> [...]
>
>
> Something I don't understand here is that there *is* a single version
> list. And so it feels like it doesn't really matter that much if you just
> lump all the packages together (like pip does).
>

FWIW, In the past I did mention that it would be nice if there was a way to
specify a different `[versions]` section per part that would layered on top
of the original `[versions]`. The APIs used could easily accommodate that,
last I checked. I'm still waiting for the round tuits to implement it in a
fork for a pull-request.


> I could imagine a buildout that just uses pip every time it has to install
> a python package. Feeding pip the version list, of course. And then puts
> the buildout.cfg mechanism on top of it.

[...]


I could imagine a buildout that uses distlib to install wheels in a
per-distribution directory (i.e. allowing multiple versions of the same
wheel to coexist). These directories would be then added to sys.path in the
script wrappers like buildout usually does.

Regards,

Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141202/1c9212c3/attachment.html>

From jim at zope.com  Tue Dec  2 16:07:48 2014
From: jim at zope.com (Jim Fulton)
Date: Tue, 2 Dec 2014 10:07:48 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAMxX_oPgpu95HLj0AbPsMm0y+MdRrVr-F+MyVW70FKK8njQcow@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAMxX_oPgpu95HLj0AbPsMm0y+MdRrVr-F+MyVW70FKK8njQcow@mail.gmail.com>
Message-ID: <CAPDm-FhWo97dECWDBBNBMY_TOxMFwcfcOjSFO0_cmkf8XAo1hA@mail.gmail.com>

On Tue, Dec 2, 2014 at 8:10 AM, Leonardo Rochael Almeida
<leorochael at gmail.com> wrote:
>
> On 2 December 2014 at 10:53, Reinout van Rees <reinout at vanrees.org> wrote:
>>
>> [...]
>>
>>
>> Something I don't understand here is that there *is* a single version
>> list. And so it feels like it doesn't really matter that much if you just
>> lump all the packages together (like pip does).
>
>
> FWIW, In the past I did mention that it would be nice if there was a way to
> specify a different `[versions]` section per part that would layered on top
> of the original `[versions]`. The APIs used could easily accommodate that,
> last I checked. I'm still waiting for the round tuits to implement it in a
> fork for a pull-request.

This works now.  If you specify specific versions in a part,
they override the versions list.


>
>>
>> I could imagine a buildout that just uses pip every time it has to install
>> a python package. Feeding pip the version list, of course. And then puts the
>> buildout.cfg mechanism on top of it.
>>
>> [...]
>
>
> I could imagine a buildout that uses distlib to install wheels in a
> per-distribution directory (i.e. allowing multiple versions of the same
> wheel to coexist).

Yes.  Effectively eggs.

> These directories would be then added to sys.path in the
> script wrappers like buildout usually does.

Yup.  This is certainly an option.

My concern is that no one seems to be using distlib.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From jim at zope.com  Tue Dec  2 16:04:44 2014
From: jim at zope.com (Jim Fulton)
Date: Tue, 2 Dec 2014 10:04:44 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <m5kco1$n7g$1@ger.gmane.org>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
Message-ID: <CAPDm-FgokDXecAGJHh_i=wZd9xG7f6ds8CpMjFLod=20GXMsjQ@mail.gmail.com>

On Tue, Dec 2, 2014 at 7:53 AM, Reinout van Rees <reinout at vanrees.org> wrote:
> On 01-12-14 14:55, Piotr Dobrogost wrote:
>>
>> Are there any plans to move from easy_install/eggs to pip/wheels in
>> buildout?
>
>
> I asked that too, once. I think Jim Fulton said that there was a mismatch
> between pip's and buildout's goal. If I remember correctly the problem is
> this:
>
> - Pip installs everything into (normally) a virtualenv. So your have ONE
> project with ONE single set of packages.
>
> - Buildout allows different combinations of packages per part. So your
> sphinx documentation buildout doesn't have to have your webserver packages
> in its sys.path.
>
> Jim, is that a fair summary? (Of course, this only looks at the python
> package part of what buildout does, not at the whole recipe mechanism).

Yes, but ...

> Something I don't understand here is that there *is* a single version list.

Right, although explicit requirements in a part can override the
versions list.  In practice, most projects use the same versions for
everything.

In the past, this has been an important requirement for us (ZC). We've
even had projects that used different versions of Python for different
parts, although ultimately, this "requirement" didn't justify the
cost.

Something to keep in mind (I know you know this) is that buildout is
used to assemble applications with many parts that may be separate
applications and may not all be written in Python.

> And so it feels like it doesn't really matter that much if you just lump all
> the packages together (like pip does).

In practice, for most projects this is true.

> I could imagine a buildout that just uses pip every time it has to install a
> python package. Feeding pip the version list, of course. And then puts the
> buildout.cfg mechanism on top of it.

I only use versions lists for applications. I almost never pin
versions for libraries.

Another important goal of buildout is that, given a buildout
configuration, two buildouts run at the same time get equivalent
results regardless of their ages and histories.

For example, I create a buildout on Monday and it gets version 42 of
package A. On a different machine, I checkout and update the buildout
to add package B. It gets version 9 of package B and version 43 of
package A. If I go back to the other machine, and rerun the buildout,
I'll get versions 43 and 9 for package A and B.  This was impossible
to achieve with easy_install, which is what drove me down to
setuptools.  I don't know how difficult this would be with pip. Given
that pip had largely the same goals as easy_install, I'm not
optimistic.  It's worth looking into.

There's also the fact that pip doesn't provide an API other than the
command line or the "command" interface which is essentially the same
things plus ... distutils.  I'm not opposed to trying to create a
version of buildout on top of pip+virtualenv, but I don't think that's
a very good architecture and I don't really expect it to turn out
well. If no standard API emerges, we may have no choice.  Maybe we
should bet on distlib, but with pip not using it...

> The alternative is, like Marius said in his reply, to use virtualenv + pip +
> makefiles or so.

That sounds like a terrible idea to me. On multiple levels.

> And you'd have to add something that replaces the recipes.

Well, people use chef and puppet and similar tools to configure docker
containers.  Then there are Java-based tools, like ant, maven, gradle,
sbt...  The recipe pattern is well established.

> For that, I really like buildout :-)

Me too. As long as I'm using Python, I'll maintain buildout if only
for my own use.  I think recipe-based automation systems make a lot of
sense, and I prefer to write recipes, like most other things, in
Python.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton

From p at lists-2014.dobrogost.net  Tue Dec  2 14:09:59 2014
From: p at lists-2014.dobrogost.net (Piotr Dobrogost)
Date: Tue, 2 Dec 2014 14:09:59 +0100
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <m5kco1$n7g$1@ger.gmane.org>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
Message-ID: <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>

On Tue, Dec 2, 2014 at 1:53 PM, Reinout van Rees <reinout at vanrees.org> wrote:
>
> The alternative is, like Marius said in his reply, to use virtualenv + pip +
> makefiles or so. And you'd have to add something that replaces the recipes.
> For that, I really like buildout :-)

It seems like building/installing python packages should be moved from
buildout's core into plain recipe so that different tools for this
task could be used. Even if this is possible I guess it's really major
change (and paradigm shift to some extent).


Regards,
Piotr

From jim at zope.com  Wed Dec  3 15:01:32 2014
From: jim at zope.com (Jim Fulton)
Date: Wed, 3 Dec 2014 09:01:32 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
Message-ID: <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>

On Tue, Dec 2, 2014 at 8:09 AM, Piotr Dobrogost
<p at lists-2014.dobrogost.net> wrote:
> On Tue, Dec 2, 2014 at 1:53 PM, Reinout van Rees <reinout at vanrees.org> wrote:
>>
>> The alternative is, like Marius said in his reply, to use virtualenv + pip +
>> makefiles or so. And you'd have to add something that replaces the recipes.
>> For that, I really like buildout :-)
>
> It seems like building/installing python packages should be moved from
> buildout's core into plain recipe so that different tools for this
> task could be used.

That's how buildout works today. There's nothing stopping someone from
building a recipe today that uses pip + virtualenv.

You might lose some features with this, like caching of built packages
but that may not be important to you. <shrug>  It looks like pip has
no way to cache builds as buildout does, possibly because it's
allergic to eggs. But it does support download caches. Yay!  And it
should be possible to "cache" builds in wheels, though I think this
would still be slower than what buildout does with eggs.  But you'd
probably gain some things, like a much shorter path to search at run
time.

Buildout still wants to download packages for it's own use.  I suppose
buildout could run in it's own virtualenv with recipes installed by
pip.

It remains that the API to pip would be via subprocess calls or
through a hideous distutils command interface.

> Even if this is possible I guess it's really major
> change (and paradigm shift to some extent).

There would be be no change needed to buildout to provide a recipe that
built using virtualenv and pip.

At some point, somethings going to have to change:

- As you noted, Python packaging is changing, and eventually
  the changes will break buildout.

  Many users, like myself, download source releases exclusively, and
  to us, wheels are irrelevent.  However, users on Windows or who use
  systems without build tools for some reason, will experience
  breakage as soon as people stop creating traditional binary releases
  in favor of wheels.

- Implementing and maintaining packaging code isn't really all that
  interesting to me and, I'd hazard to guess, to anyone else on the
  buildout team.  Buildout has more packaging code than I ever wanted
  to write.  It does largely because it needed policies and features
  not provided by easy_install. (This is not meant to disparage
  easy_install, which was a ground-breaking tool.)

  Buildout needs to build on something else. For the time being, it
  makes sense to continue to use setuptools, as this requires very
  little effort. Eventually, this will be something else and the
  transition is bound to be bumpy.

In the long run, I think keeping buildout alive with a manageble
effort is worth transition pain.  I think buildout offers a lot of
benefit for a lot of projects, and I'd hate to see people use
something worse.

Jim

P.S. I think it's a fallacy to assume that lack of activity or commits
     (buildout has had both recently) necessarily indicates that a
     project is dead. Lack of activity often just means that something
     doesn't require change.  I prefer to spread my work over many
     small projects and it's not uncommon for projects I use every day
     to go years without change.

     I realize that activity is an easy metric, especially when
     looking at projects with which you aren't familiar. I often use
     it myself, but it's important to keep it's limitations in mind.

--
Jim Fulton
http://www.linkedin.com/in/jimfulton

From donald at stufft.io  Wed Dec  3 15:12:13 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 3 Dec 2014 09:12:13 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
Message-ID: <1AA0B3AC-F84C-45DC-9DD2-950ACA59E060@stufft.io>


> On Dec 3, 2014, at 9:01 AM, Jim Fulton <jim at zope.com> wrote:
> 
> - Implementing and maintaining packaging code isn't really all that
>  interesting to me and, I'd hazard to guess, to anyone else on the
>  buildout team.  Buildout has more packaging code than I ever wanted
>  to write.  It does largely because it needed policies and features
>  not provided by easy_install. (This is not meant to disparage
>  easy_install, which was a ground-breaking tool.)

If you can enumerate what buildout needs or wants from a packaging tool that would be useful to keep in mind going forward. 

From jim at zope.com  Wed Dec  3 15:14:39 2014
From: jim at zope.com (Jim Fulton)
Date: Wed, 3 Dec 2014 09:14:39 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <1AA0B3AC-F84C-45DC-9DD2-950ACA59E060@stufft.io>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <1AA0B3AC-F84C-45DC-9DD2-950ACA59E060@stufft.io>
Message-ID: <CAPDm-FgaYbe86+Fnq+7m=StOnY7KRLBNo7sY87g_qZRB4=NVGg@mail.gmail.com>

On Wed, Dec 3, 2014 at 9:12 AM, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 3, 2014, at 9:01 AM, Jim Fulton <jim at zope.com> wrote:
>>
>> - Implementing and maintaining packaging code isn't really all that
>>  interesting to me and, I'd hazard to guess, to anyone else on the
>>  buildout team.  Buildout has more packaging code than I ever wanted
>>  to write.  It does largely because it needed policies and features
>>  not provided by easy_install. (This is not meant to disparage
>>  easy_install, which was a ground-breaking tool.)
>
> If you can enumerate what buildout needs or wants from a packaging tool that would be useful to keep in mind going forward.

Great idea. I'll do that.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From jim at zope.com  Wed Dec  3 15:47:14 2014
From: jim at zope.com (Jim Fulton)
Date: Wed, 3 Dec 2014 09:47:14 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <547F1F03.9060206@thesyrings.us>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
Message-ID: <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>

On Wed, Dec 3, 2014 at 9:32 AM, Randy Syring <randy at thesyrings.us> wrote:
>
> On 12/03/2014 09:01 AM, Jim Fulton wrote:
>
>   Many users, like myself, download source releases exclusively, and
>   to us, wheels are irrelevent.  However, users on Windows or who use
>   systems without build tools for some reason, will experience
>   breakage as soon as people stop creating traditional binary releases
>   in favor of wheels.
>
>
> Does that have to be the case?

Yes, users of buildout on systems without build tools will experience
breakage if packagers stop distributing any of the older binary
formats.

This will remain true until either setuptools grows wheel support, or
until buildout uses something else (other than setuptools).

> As one of the maintainers of pymssql, we are
> in the process of figuring out what type of packages we want to ship as
> maintainers.  Wheel's can be binary releases and, in fact, is what we are
> going to build for Windows support going forward.  That may certainly
> frustrate some automated processes that expect the old package types we used
> to ship, but we absolutely want to support binary releases with wheels
> specifically to support users that don't have build tools.

I understand.  I assume you'll continue to supply source distributions.

I don't want to dictate, but the situation is what it is. If you've
followed this thread, I hope you can appreciate the quandary we're in.

> Am I missing something about binary support with wheels?

I don't think so.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton

From p.f.moore at gmail.com  Wed Dec  3 16:12:37 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 3 Dec 2014 15:12:37 +0000
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
Message-ID: <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>

On 3 December 2014 at 14:47, Jim Fulton <jim at zope.com> wrote:
>> Am I missing something about binary support with wheels?
>
> I don't think so.

Agreed.

It seems to me that the key issue currently is that for binary
distribution, pip only supports wheels, whereas buildout only supports
eggs. There are other tools which also tend to only support one or the
other (e.g., easy_install -> egg, distil -> wheel) but I think they
are the minority. I don't know relative popularity between pip and
buildout - I suspect it depends on the community involved.

For a project, decisions on which binary formats to provide probably
need to be based on the expected numbers of users of each tool.

The wheel and egg formats are similar enough to allow auto-conversion
(wheel convert does egg->wheel, I don't know if there's a tool to do
wheel->egg, but it wouldn't be hard to write) but expecting users to
maintain local converted repositories is a burden in itself, so the
question remains what binaries to upload to PyPI.

Paul

From jim at zope.com  Wed Dec  3 16:30:26 2014
From: jim at zope.com (Jim Fulton)
Date: Wed, 3 Dec 2014 10:30:26 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
 <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>
Message-ID: <CAPDm-FhJD5rCR81NaLy7kNne5SBdrROSSP+i8O95b_gCeTUv6w@mail.gmail.com>

On Wed, Dec 3, 2014 at 10:12 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 3 December 2014 at 14:47, Jim Fulton <jim at zope.com> wrote:
>>> Am I missing something about binary support with wheels?
>>
>> I don't think so.
>
> Agreed.
>
> It seems to me that the key issue currently is that for binary
> distribution, pip only supports wheels, whereas buildout only supports
> eggs.

(And .exe)

> There are other tools which also tend to only support one or the
> other (e.g., easy_install -> egg, distil -> wheel) but I think they
> are the minority. I don't know relative popularity between pip and
> buildout - I suspect it depends on the community involved.
>
> For a project, decisions on which binary formats to provide probably
> need to be based on the expected numbers of users of each tool.
>
> The wheel and egg formats are similar enough to allow auto-conversion
> (wheel convert does egg->wheel, I don't know if there's a tool to do
> wheel->egg, but it wouldn't be hard to write) but expecting users to
> maintain local converted repositories is a burden in itself, so the
> question remains what binaries to upload to PyPI.

For now, it would be humane to upload both wheel and egg. :)

We're happy to move buildout to using wheel.  It will make me
very sad if we have to do that by invoking pip as a subprocess,
but I'll cope if I have to.

I think the next step is for me to enumerate buildout's packaging api
requirements and desires. I'll try to get that done this weekend.
(I'll need to spend some time refreshing my memory on some things.)

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From randy at thesyrings.us  Wed Dec  3 15:32:35 2014
From: randy at thesyrings.us (Randy Syring)
Date: Wed, 03 Dec 2014 09:32:35 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
Message-ID: <547F1F03.9060206@thesyrings.us>


On 12/03/2014 09:01 AM, Jim Fulton wrote:
>    Many users, like myself, download source releases exclusively, and
>    to us, wheels are irrelevent.  However, users on Windows or who use
>    systems without build tools for some reason, will experience
>    breakage as soon as people stop creating traditional binary releases
>    in favor of wheels.
>

Does that have to be the case?  As one of the maintainers of pymssql, we 
are in the process of figuring out what type of packages we want to ship 
as maintainers.  Wheel's can be binary releases and, in fact, is what we 
are going to build for Windows support going forward.  That may 
certainly frustrate some automated processes that expect the old package 
types we used to ship, but we absolutely want to support binary releases 
with wheels specifically to support users that don't have build tools.

Am I missing something about binary support with wheels?

Thanks.


*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141203/c9446a0e/attachment.html>

From randy at thesyrings.us  Wed Dec  3 15:54:09 2014
From: randy at thesyrings.us (Randy Syring)
Date: Wed, 03 Dec 2014 09:54:09 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
Message-ID: <547F2411.9040709@thesyrings.us>


On 12/03/2014 09:47 AM, Jim Fulton wrote:
> On Wed, Dec 3, 2014 at 9:32 AM, Randy Syring <randy at thesyrings.us> wrote:
>> On 12/03/2014 09:01 AM, Jim Fulton wrote:
>>
>>    Many users, like myself, download source releases exclusively, and
>>    to us, wheels are irrelevent.  However, users on Windows or who use
>>    systems without build tools for some reason, will experience
>>    breakage as soon as people stop creating traditional binary releases
>>    in favor of wheels.
>>
>>
>> Does that have to be the case?
> Yes, users of buildout on systems without build tools will experience
> breakage if packagers stop distributing any of the older binary
> formats.
>
> This will remain true until either setuptools grows wheel support, or
> until buildout uses something else (other than setuptools).

Ic.  I didn't catch that you were talking specifically about buildout 
support, but it makes sense now.

>
>> As one of the maintainers of pymssql, we are
>> in the process of figuring out what type of packages we want to ship as
>> maintainers.  Wheel's can be binary releases and, in fact, is what we are
>> going to build for Windows support going forward.  That may certainly
>> frustrate some automated processes that expect the old package types we used
>> to ship, but we absolutely want to support binary releases with wheels
>> specifically to support users that don't have build tools.
> I understand.  I assume you'll continue to supply source distributions.
Yes, absolutely.

>
> I don't want to dictate, but the situation is what it is. If you've
> followed this thread, I hope you can appreciate the quandary we're in.

No problem.  It's actually been a helpful discussion.  I'm glad I joined 
the mailing list and appreciate your feedback.


*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141203/b08e74e0/attachment.html>

From dholth at gmail.com  Wed Dec  3 17:05:16 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 3 Dec 2014 11:05:16 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAPDm-FhJD5rCR81NaLy7kNne5SBdrROSSP+i8O95b_gCeTUv6w@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
 <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>
 <CAPDm-FhJD5rCR81NaLy7kNne5SBdrROSSP+i8O95b_gCeTUv6w@mail.gmail.com>
Message-ID: <CAG8k2+7CrxZJ_9wW92-gcaddvCpO-y_BYhkem9SSPko5CAn78Q@mail.gmail.com>

Wheel also has its own installer. It hasn't received as much attention
since pip gained support.

It would be good to pay some additional attention to the "[unzipped]
wheel as sys.path entry" case. It will usually work; most packages do
not use the .data/ directories for anything but scripts; these wheels
are very similar to eggs.

https://bitbucket.org/pypa/wheel/src/tip/wheel/tool/__init__.py?at=default#cl-141

On Wed, Dec 3, 2014 at 10:30 AM, Jim Fulton <jim at zope.com> wrote:
> On Wed, Dec 3, 2014 at 10:12 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 3 December 2014 at 14:47, Jim Fulton <jim at zope.com> wrote:
>>>> Am I missing something about binary support with wheels?
>>>
>>> I don't think so.
>>
>> Agreed.
>>
>> It seems to me that the key issue currently is that for binary
>> distribution, pip only supports wheels, whereas buildout only supports
>> eggs.
>
> (And .exe)
>
>> There are other tools which also tend to only support one or the
>> other (e.g., easy_install -> egg, distil -> wheel) but I think they
>> are the minority. I don't know relative popularity between pip and
>> buildout - I suspect it depends on the community involved.
>>
>> For a project, decisions on which binary formats to provide probably
>> need to be based on the expected numbers of users of each tool.
>>
>> The wheel and egg formats are similar enough to allow auto-conversion
>> (wheel convert does egg->wheel, I don't know if there's a tool to do
>> wheel->egg, but it wouldn't be hard to write) but expecting users to
>> maintain local converted repositories is a burden in itself, so the
>> question remains what binaries to upload to PyPI.
>
> For now, it would be humane to upload both wheel and egg. :)
>
> We're happy to move buildout to using wheel.  It will make me
> very sad if we have to do that by invoking pip as a subprocess,
> but I'll cope if I have to.
>
> I think the next step is for me to enumerate buildout's packaging api
> requirements and desires. I'll try to get that done this weekend.
> (I'll need to spend some time refreshing my memory on some things.)
>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From p.f.moore at gmail.com  Wed Dec  3 17:14:55 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 3 Dec 2014 16:14:55 +0000
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAG8k2+7CrxZJ_9wW92-gcaddvCpO-y_BYhkem9SSPko5CAn78Q@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
 <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>
 <CAPDm-FhJD5rCR81NaLy7kNne5SBdrROSSP+i8O95b_gCeTUv6w@mail.gmail.com>
 <CAG8k2+7CrxZJ_9wW92-gcaddvCpO-y_BYhkem9SSPko5CAn78Q@mail.gmail.com>
Message-ID: <CACac1F_vZJ5C8s+kXkYuGqGKnyquS9saA3svHFmaimnX3jt3oA@mail.gmail.com>

On 3 December 2014 at 16:05, Daniel Holth <dholth at gmail.com> wrote:
> Wheel also has its own installer. It hasn't received as much attention
> since pip gained support.

That's actually a very good point - the wheel project has its own
"install a wheel" API, not just the command line tool. If that's all
buildout needs, it may be a simple way of adding wheel support. If
buildout needs "the other stuff" that pip/setuptools provides, around
discovery and downloading from PyPI, version/requirements checking,
script wrappers, etc, then that's a bigger issue of course.

Paul

From jim at zope.com  Wed Dec  3 17:19:30 2014
From: jim at zope.com (Jim Fulton)
Date: Wed, 3 Dec 2014 11:19:30 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CACac1F_vZJ5C8s+kXkYuGqGKnyquS9saA3svHFmaimnX3jt3oA@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
 <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>
 <CAPDm-FhJD5rCR81NaLy7kNne5SBdrROSSP+i8O95b_gCeTUv6w@mail.gmail.com>
 <CAG8k2+7CrxZJ_9wW92-gcaddvCpO-y_BYhkem9SSPko5CAn78Q@mail.gmail.com>
 <CACac1F_vZJ5C8s+kXkYuGqGKnyquS9saA3svHFmaimnX3jt3oA@mail.gmail.com>
Message-ID: <CAPDm-FgPzDgNsJU4JgGnRSDKvL331W73+06iJxa69=qM+WUemw@mail.gmail.com>

On Wed, Dec 3, 2014 at 11:14 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 3 December 2014 at 16:05, Daniel Holth <dholth at gmail.com> wrote:
>> Wheel also has its own installer. It hasn't received as much attention
>> since pip gained support.
>
> That's actually a very good point - the wheel project has its own
> "install a wheel" API, not just the command line tool. If that's all
> buildout needs, it may be a simple way of adding wheel support. If
> buildout needs "the other stuff" that pip/setuptools provides, around
> discovery and downloading from PyPI, version/requirements checking,
> script wrappers, etc, then that's a bigger issue of course.

Yup. Buildout uses setuptools to find distributions.

I suppose the wheel installer library could facilitate adding wheel
support to setuptools.  Maybe someone on the buildout team should
look at that.  My guess is that adding discovery of wheels in indexes and
such probably isn't that hard.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From donald at stufft.io  Wed Dec  3 17:21:52 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 3 Dec 2014 11:21:52 -0500
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAPDm-FgPzDgNsJU4JgGnRSDKvL331W73+06iJxa69=qM+WUemw@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
 <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>
 <CAPDm-FhJD5rCR81NaLy7kNne5SBdrROSSP+i8O95b_gCeTUv6w@mail.gmail.com>
 <CAG8k2+7CrxZJ_9wW92-gcaddvCpO-y_BYhkem9SSPko5CAn78Q@mail.gmail.com>
 <CACac1F_vZJ5C8s+kXkYuGqGKnyquS9saA3svHFmaimnX3jt3oA@mail.gmail.com>
 <CAPDm-FgPzDgNsJU4JgGnRSDKvL331W73+06iJxa69=qM+WUemw@mail.gmail.com>
Message-ID: <D4E25281-090C-471E-BCC8-91EF07E4CCA3@stufft.io>


> On Dec 3, 2014, at 11:19 AM, Jim Fulton <jim at zope.com> wrote:
> 
> On Wed, Dec 3, 2014 at 11:14 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 3 December 2014 at 16:05, Daniel Holth <dholth at gmail.com> wrote:
>>> Wheel also has its own installer. It hasn't received as much attention
>>> since pip gained support.
>> 
>> That's actually a very good point - the wheel project has its own
>> "install a wheel" API, not just the command line tool. If that's all
>> buildout needs, it may be a simple way of adding wheel support. If
>> buildout needs "the other stuff" that pip/setuptools provides, around
>> discovery and downloading from PyPI, version/requirements checking,
>> script wrappers, etc, then that's a bigger issue of course.
> 
> Yup. Buildout uses setuptools to find distributions.
> 
> I suppose the wheel installer library could facilitate adding wheel
> support to setuptools.  Maybe someone on the buildout team should
> look at that.  My guess is that adding discovery of wheels in indexes and
> such probably isn't that hard.
> 

Installing Wheels in general isn?t that hard either TBH. It probably
wouldn?t be a ton of work to add support for that to setuptools.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From matt at notevencode.com  Sun Dec  7 10:28:02 2014
From: matt at notevencode.com (Matthew Iversen)
Date: Sun, 7 Dec 2014 20:28:02 +1100
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <D4E25281-090C-471E-BCC8-91EF07E4CCA3@stufft.io>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
 <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>
 <CAPDm-FhJD5rCR81NaLy7kNne5SBdrROSSP+i8O95b_gCeTUv6w@mail.gmail.com>
 <CAG8k2+7CrxZJ_9wW92-gcaddvCpO-y_BYhkem9SSPko5CAn78Q@mail.gmail.com>
 <CACac1F_vZJ5C8s+kXkYuGqGKnyquS9saA3svHFmaimnX3jt3oA@mail.gmail.com>
 <CAPDm-FgPzDgNsJU4JgGnRSDKvL331W73+06iJxa69=qM+WUemw@mail.gmail.com>
 <D4E25281-090C-471E-BCC8-91EF07E4CCA3@stufft.io>
Message-ID: <CAMAgpEQ9ikv-gmpjgzxSUpqoPQhDgVosA9zc_KUCbr5cdZDuqw@mail.gmail.com>

Between setuptools / wheel / pip / virtualenv there is actually a good case
for a shared common code project to vendor from. The easiest example of
functionality contained within this would be version parsing, although
there are a number of other features as well (eg installing a wheel).

Currently the case is that this is half done by pip vendoring setuptools'
pkg_resources.py although I think it would be of great benefit to all
projects to formalise this situation into a new discrete codebase in the
future. Distlib is a candidate project to fulfill this role but it would
need a good bit of vetting beforehand AFAIK.

Pip doesn't offer any api mainly because is too messy / large / would take
away too much time for the developers to spend time creating a good api
that we could stick to. Sort of "We'd love to, but have you seen the issue
list... :(".

In that respect that is actually the use case that distlib aims to provide.
Pip currently uses distlib's code to install wheels.

That said I'd reiterate Donald's comment that we'd love to see a
wanted-functionality list from projects like buildout for such a
hypothetical official future utility project.
On 4 Dec 2014 03:22, "Donald Stufft" <donald at stufft.io> wrote:

>
> > On Dec 3, 2014, at 11:19 AM, Jim Fulton <jim at zope.com> wrote:
> >
> > On Wed, Dec 3, 2014 at 11:14 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> >> On 3 December 2014 at 16:05, Daniel Holth <dholth at gmail.com> wrote:
> >>> Wheel also has its own installer. It hasn't received as much attention
> >>> since pip gained support.
> >>
> >> That's actually a very good point - the wheel project has its own
> >> "install a wheel" API, not just the command line tool. If that's all
> >> buildout needs, it may be a simple way of adding wheel support. If
> >> buildout needs "the other stuff" that pip/setuptools provides, around
> >> discovery and downloading from PyPI, version/requirements checking,
> >> script wrappers, etc, then that's a bigger issue of course.
> >
> > Yup. Buildout uses setuptools to find distributions.
> >
> > I suppose the wheel installer library could facilitate adding wheel
> > support to setuptools.  Maybe someone on the buildout team should
> > look at that.  My guess is that adding discovery of wheels in indexes and
> > such probably isn't that hard.
> >
>
> Installing Wheels in general isn?t that hard either TBH. It probably
> wouldn?t be a ton of work to add support for that to setuptools.
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141207/04b30e35/attachment.html>

From matt at notevencode.com  Sun Dec  7 10:44:15 2014
From: matt at notevencode.com (Matthew Iversen)
Date: Sun, 7 Dec 2014 20:44:15 +1100
Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible?
In-Reply-To: <CALGmxEK2J_jmBUW7QGyT7dx8kt3pKz3BVPxJFQTmA2f-RNp-UA@mail.gmail.com>
References: <CAH3X-tkG==PjQKC+gp5LsjKMGaq8nYiumM=terTzfqPFbKQ=sg@mail.gmail.com>
 <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net>
 <CACac1F_g2GnE4S=skwb02763ofdjFDLsx4RmXkuYxbdOZENcEA@mail.gmail.com>
 <CAG8k2+4zb72m030sr7CmSe4B8BcrEs_MEYzw7fAHKCD--k0m4Q@mail.gmail.com>
 <CAH3X-tnrCP-MTYrfdKxo4x0O16BWeGPh92wL41M7-q+=HWQALg@mail.gmail.com>
 <CALGmxEK2J_jmBUW7QGyT7dx8kt3pKz3BVPxJFQTmA2f-RNp-UA@mail.gmail.com>
Message-ID: <CAMAgpETfMazw3Q5-835JUJDVH9JQVPk4EZ84N7CD1DgS0X9j8g@mail.gmail.com>

Wheels can't be required because they can be platform/python version
specific, being binary distributions. We can't demand that independent
developers support some combination of platforms and versions when their
computer might only be running one of them, simply because their wheel
might be platform dependant rather than universal. Pypi doesn't even allow
Linux binary wheels at the moment because compatibility between different
things and distros that all call themselves "Linux" running on some
instruction set is a big wild problem for the moment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141207/8c2d8265/attachment.html>

From matt at notevencode.com  Sun Dec  7 12:42:44 2014
From: matt at notevencode.com (Matthew Iversen)
Date: Sun, 7 Dec 2014 22:42:44 +1100
Subject: [Distutils] Are there any plans to move to pip/wheels in
	buildout?
In-Reply-To: <CAMAgpEQ9ikv-gmpjgzxSUpqoPQhDgVosA9zc_KUCbr5cdZDuqw@mail.gmail.com>
References: <CAA6tFMtOoK0UUqM=atZSVcdiHD2LOTZbDGufKiBPHSLORzXH9w@mail.gmail.com>
 <m5kco1$n7g$1@ger.gmane.org>
 <CAA6tFMukWudG8bhddu0d9ySTJfoLVr8hVYrRNRUL7N1qpVcCHg@mail.gmail.com>
 <CAPDm-Fio+R6-ZZxEuSNzdPamQ4P9Uo+4VqA0nifWNPQ9Daq3VA@mail.gmail.com>
 <547F1F03.9060206@thesyrings.us>
 <CAPDm-FhXP9UDhE77a2ydMC==p4QcqWd8AKije5L8K5m9wT1fDw@mail.gmail.com>
 <CACac1F_2xvZO+3gt5qbZH5HBxpGxBhWJbSAzfKk7UycO9hANOQ@mail.gmail.com>
 <CAPDm-FhJD5rCR81NaLy7kNne5SBdrROSSP+i8O95b_gCeTUv6w@mail.gmail.com>
 <CAG8k2+7CrxZJ_9wW92-gcaddvCpO-y_BYhkem9SSPko5CAn78Q@mail.gmail.com>
 <CACac1F_vZJ5C8s+kXkYuGqGKnyquS9saA3svHFmaimnX3jt3oA@mail.gmail.com>
 <CAPDm-FgPzDgNsJU4JgGnRSDKvL331W73+06iJxa69=qM+WUemw@mail.gmail.com>
 <D4E25281-090C-471E-BCC8-91EF07E4CCA3@stufft.io>
 <CAMAgpEQ9ikv-gmpjgzxSUpqoPQhDgVosA9zc_KUCbr5cdZDuqw@mail.gmail.com>
Message-ID: <CAMAgpEQ=Bwn3nnT9uZ41yYqzwfUBGCPGYw2XAbLRywKixTb56Q@mail.gmail.com>

Err, slipped my mind that Donald is working on
https://github.com/pypa/packaging which also might end up being what I was
describing in my previous message

On 7 December 2014 at 20:28, Matthew Iversen <matt at notevencode.com> wrote:

> Between setuptools / wheel / pip / virtualenv there is actually a good
> case for a shared common code project to vendor from. The easiest example
> of functionality contained within this would be version parsing, although
> there are a number of other features as well (eg installing a wheel).
>
> Currently the case is that this is half done by pip vendoring setuptools'
> pkg_resources.py although I think it would be of great benefit to all
> projects to formalise this situation into a new discrete codebase in the
> future. Distlib is a candidate project to fulfill this role but it would
> need a good bit of vetting beforehand AFAIK.
>
> Pip doesn't offer any api mainly because is too messy / large / would take
> away too much time for the developers to spend time creating a good api
> that we could stick to. Sort of "We'd love to, but have you seen the issue
> list... :(".
>
> In that respect that is actually the use case that distlib aims to
> provide. Pip currently uses distlib's code to install wheels.
>
> That said I'd reiterate Donald's comment that we'd love to see a
> wanted-functionality list from projects like buildout for such a
> hypothetical official future utility project.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141207/7ae3965d/attachment.html>

From vladimir.v.diaz at gmail.com  Thu Dec 11 04:16:48 2014
From: vladimir.v.diaz at gmail.com (Vladimir Diaz)
Date: Wed, 10 Dec 2014 22:16:48 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
Message-ID: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>

Hello everyone,

I am a research programmer at the NYU School of Engineering.  My colleagues
(Trishank Kuppusamy and Justin Cappos) and I are requesting community
feedback on our proposal, "Surviving a Compromise of PyPI."  The two-stage
proposal can be reviewed online at:

PEP 458
http://legacy.python.org/dev/peps/pep-0458/

PEP 480
http://legacy.python.org/dev/peps/pep-0480/


Summary of the Proposal:

"Surviving a Compromise of PyPI" proposes how the Python Package Index
(PyPI) can be amended to better protect end users from altered or malicious
packages, and to minimize the extent of PyPI compromises against affected
users.  The proposed integration allows package managers such as pip to be
more secure against various types of security attacks on PyPI and defend
end users from attackers responding to package requests. Specifically,
these PEPs describe how PyPI processes should be adapted to generate and
incorporate repository metadata, which are signed text files that describe
the packages and metadata available on PyPI.  Package managers request
(along with the packages) the metadata on PyPI to verify the authenticity
of packages before they are installed.  The changes to PyPI and tools will
be minimal by leveraging a library, The Update Framework
<https://github.com/theupdateframework/tuf>, that generates and
transparently validates the relevant metadata.

The first stage of the proposal (PEP 458
<http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model
that supports verification of PyPI packages signed with cryptographic keys
stored on PyPI, requires no action from developers and end users, and
protects against malicious CDNs and public mirrors. To support continuous
delivery of uploaded packages, PyPI administrators sign for uploaded
packages with an online key stored on PyPI infrastructure. This level of
security prevents packages from being accidentally or deliberately tampered
with by a mirror or a CDN because the mirror or CDN will not have any of
the keys required to sign for projects.

The second stage of the proposal (PEP 480
<http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic
security model (discussed in PEP 458) that supports end-to-end verification
of signed packages. End-to-end signing allows both PyPI and developers to
sign for the packages that are downloaded by end users.  If the PyPI
infrastructure were to be compromised, attackers would be unable to serve
malicious versions of these packages without access to the project's
developer key.  As in PEP 458, no additional action is required by end
users.  However, PyPI administrators will need to periodically (perhaps
every few months) sign metadata with an offline key.  PEP 480 also proposes
an easy-to-use key management solution for developers, how to interface
with a potential build farm on PyPI infrastructure, and discusses the
security benefits of end-to-end signing.  The second stage of the proposal
simultaneously supports real-time project registration and developer
signatures, and when configured to maximize security on PyPI, less than 1%
of end users will be at risk even if an attacker controls PyPI and goes
undetected for a month.

We thank Nick Coghlan and Donald Stufft for their valuable contributions,
and Giovanni Bajo and Anatoly Techtonik for their feedback.

Thanks,
PEP 458 & 480 authors.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141210/5505a7f4/attachment.html>

From ben at bendarnell.com  Fri Dec 12 17:09:26 2014
From: ben at bendarnell.com (Ben Darnell)
Date: Fri, 12 Dec 2014 11:09:26 -0500
Subject: [Distutils] Varying dependencies by python minor version (2.7.9)
Message-ID: <CAFkYKJ5q7OFJ8RUhz++4P_1rF_YWfjyxByu-Cs-y=4VYfqgoiA@mail.gmail.com>

Hi distutils-sig,

Tornado depends on backports.ssl_match_hostname on python 2, but this is no
longer needed with (cpython) 2.7.9. We have a PR to make this dependency
conditional on the minor version of python (
https://github.com/tornadoweb/tornado/pull/1276/files). We could also make
the test something like `hasattr(ssl, 'SSLContext')`, which would be more
compatible with pypy and other implementations, but the fundamental issues
remain. Is it a good idea to distribute packages that will install
different dependencies on different minor versions of python? Is it even
possible with binary distributions? I'm thinking it's probably best to just
live with the unused extra dependency (until we can completely drop support
for python versions without modern ssl libraries), but I wanted to check
and see if there is a clean way to deal with this.

Thanks,
-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141212/4ebc129d/attachment.html>

From dholth at gmail.com  Fri Dec 12 21:52:54 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 12 Dec 2014 15:52:54 -0500
Subject: [Distutils] Varying dependencies by python minor version (2.7.9)
In-Reply-To: <CAFkYKJ5q7OFJ8RUhz++4P_1rF_YWfjyxByu-Cs-y=4VYfqgoiA@mail.gmail.com>
References: <CAFkYKJ5q7OFJ8RUhz++4P_1rF_YWfjyxByu-Cs-y=4VYfqgoiA@mail.gmail.com>
Message-ID: <CAG8k2+6KAj6Os_MxnFAm9W84MDm2Uj0jr-tFV4RyNjnu7Dj6Qg@mail.gmail.com>

It is possible and supported but not very elegant since the marker
syntax can only compare versions as strings (you wouldn't want to use
< and have the wrong dependencies under Python 2.7.10). One solution
would be to check that you are not running on any of the older
releases of Python 2.7, like so:

In setup.py:

      extras_require={
          ':python_version == "2.7" and (python_full_version!="2.7.8"
and python_full_version!="2.7.7" and ...)':
['backports.ssl_match_hostname'],
      }

This means the extra named '' a.k.a. the default dependencies includes
your package, but only if the expression after the colon : evaluates
to True. bdist_wheel preserves the expression in .whl binary packages.


On Fri, Dec 12, 2014 at 11:09 AM, Ben Darnell <ben at bendarnell.com> wrote:
> Hi distutils-sig,
>
> Tornado depends on backports.ssl_match_hostname on python 2, but this is no
> longer needed with (cpython) 2.7.9. We have a PR to make this dependency
> conditional on the minor version of python
> (https://github.com/tornadoweb/tornado/pull/1276/files). We could also make
> the test something like `hasattr(ssl, 'SSLContext')`, which would be more
> compatible with pypy and other implementations, but the fundamental issues
> remain. Is it a good idea to distribute packages that will install different
> dependencies on different minor versions of python? Is it even possible with
> binary distributions? I'm thinking it's probably best to just live with the
> unused extra dependency (until we can completely drop support for python
> versions without modern ssl libraries), but I wanted to check and see if
> there is a clean way to deal with this.
>
> Thanks,
> -Ben
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From Steve.Dower at microsoft.com  Sat Dec 13 00:23:03 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Fri, 12 Dec 2014 23:23:03 +0000
Subject: [Distutils] PYD/SO platform tags (#22980)
Message-ID: <DM2PR0301MB073458ABED6F46FA106DEE08F5600@DM2PR0301MB0734.namprd03.prod.outlook.com>

Hi distutils-sig,

There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows.

There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :)

Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h).

By default, distutils' build_ext will use the first extension, and so builds will get the ".cp35-win32.pyd" tag. You can only get the ".cp3-win32.pyd" tag (for stable ABI) by customizing the build process - I don't think that's an unreasonable expectation, especially on Windows.

I can see potential for making more generic installable packages with this change, though I'm not pushing for that. The big win is when you're building and testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how the output directories and search paths are organized.

Thoughts/concerns?

Cheers,
Steve

From donald at stufft.io  Sat Dec 13 00:37:23 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 12 Dec 2014 18:37:23 -0500
Subject: [Distutils] PYD/SO platform tags (#22980)
In-Reply-To: <DM2PR0301MB073458ABED6F46FA106DEE08F5600@DM2PR0301MB0734.namprd03.prod.outlook.com>
References: <DM2PR0301MB073458ABED6F46FA106DEE08F5600@DM2PR0301MB0734.namprd03.prod.outlook.com>
Message-ID: <36A08FD8-E78E-4130-B23E-529939A482DF@stufft.io>


> On Dec 12, 2014, at 6:23 PM, Steve Dower <Steve.Dower at microsoft.com> wrote:
> 
> Hi distutils-sig,
> 
> There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows.
> 
> There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :)
> 
> Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h).
> 
> By default, distutils' build_ext will use the first extension, and so builds will get the ".cp35-win32.pyd" tag. You can only get the ".cp3-win32.pyd" tag (for stable ABI) by customizing the build process - I don't think that's an unreasonable expectation, especially on Windows.
> 
> I can see potential for making more generic installable packages with this change, though I'm not pushing for that. The big win is when you're building and testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how the output directories and search paths are organized.
> 
> Thoughts/concerns?

Well here?s the code that computes what tags pip will accept: https://github.com/pypa/pip/blob/develop/pip/pep425tags.py

Obviously any additional information put into the tag will increase the number of builds someone has to do in order to get full coverage, however if all of the data in that are things that genuinely make things binary incompatible then that?s the right thing to do.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From solipsis at pitrou.net  Sat Dec 13 00:51:20 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 13 Dec 2014 00:51:20 +0100
Subject: [Distutils] PYD/SO platform tags (#22980)
References: <DM2PR0301MB073458ABED6F46FA106DEE08F5600@DM2PR0301MB0734.namprd03.prod.outlook.com>
Message-ID: <20141213005120.38c6ed04@fsol>

On Fri, 12 Dec 2014 23:23:03 +0000
Steve Dower <Steve.Dower at microsoft.com> wrote:

> Hi distutils-sig,
> 
> There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows.
> 
> There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :)
> 
> Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h).

For the record, https://www.python.org/dev/peps/pep-3149/#pep-384
suggests "abi3" as the ABI tag for the Python 3 stable ABI.

Regards

Antoine.



From Steve.Dower at microsoft.com  Sat Dec 13 01:01:34 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 13 Dec 2014 00:01:34 +0000
Subject: [Distutils] PYD/SO platform tags (#22980)
In-Reply-To: <20141213005120.38c6ed04@fsol>
References: <DM2PR0301MB073458ABED6F46FA106DEE08F5600@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <20141213005120.38c6ed04@fsol>
Message-ID: <DM2PR0301MB07341314CA32E5F1D8E24C89F5610@DM2PR0301MB0734.namprd03.prod.outlook.com>

Antoine Pitrou wrote:
> Sent: Friday, December 12, 2014 1551
> To: Distutils-Sig at Python.Org
> Subject: Re: [Distutils] PYD/SO platform tags (#22980)
> 
> On Fri, 12 Dec 2014 23:23:03 +0000
> Steve Dower <Steve.Dower at microsoft.com> wrote:
> 
>> Hi distutils-sig,
>>
>> There's a bit of discussion going on at http://bugs.python.org/issue22980
> about extending SO tags to include bitness values, and I took the opportunity to
> look into adding platform tags for .pyd files on Windows.
>>
>> There's a patch up there, but I'm interested in any thoughts or
>> concerns people may have that I haven't considered. Distribution is
>> likely to be most affected, and I figured distutils-sig is most likely
>> to have people with strong opinions on the subject anyway, so I'm
>> posting here :)
>>
>> Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll
> go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32"
> may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that
> have #ifdefs in pyconfig.h).
> 
> For the record, https://www.python.org/dev/peps/pep-3149/#pep-384
> suggests "abi3" as the ABI tag for the Python 3 stable ABI.

My legalistic rationale for using cp3 is that it's actually the version tag, not the ABI tag. It seemed from my reading that you'd get tags like "cp35-abi3-win32", which is not helpful because you've still specified the minor version in the tag. Either it'll work in any CPython 3.x ("cp3") or it needs a specific minor version ("cp35"), so I didn't see the value in keeping the ABI tag.

Of course, this only works because Windows doesn't use the ABI tag for anything else. Debug builds are encoded differently, and AFAICT the malloc and Unicode options are currently unnecessary and I'm not making any change to make them necessary.

I could be swayed on this point fairly easily though, if say there are other interpreters out there not compatible enough with CPython to use the cp tag but that can still use stable ABI extensions that were build assuming CPython. Or I could just drop the second tag and only have "cp35-win32.pyd" or ".pyd". We've made it this long without a specifier, so there's no real need to go overboard :)

Cheers,
Steve

> Regards
> 
> Antoine.

From Steve.Dower at microsoft.com  Sat Dec 13 01:08:12 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 13 Dec 2014 00:08:12 +0000
Subject: [Distutils] PYD/SO platform tags (#22980)
In-Reply-To: <36A08FD8-E78E-4130-B23E-529939A482DF@stufft.io>
References: <DM2PR0301MB073458ABED6F46FA106DEE08F5600@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <36A08FD8-E78E-4130-B23E-529939A482DF@stufft.io>
Message-ID: <DM2PR0301MB07340142166E3FA76098AA6BF5610@DM2PR0301MB0734.namprd03.prod.outlook.com>

> Donald Stufft wrote:
>> On Dec 12, 2014, at 6:23 PM, Steve Dower <Steve.Dower at microsoft.com> wrote:
>>
>> Hi distutils-sig,
>>
>> There's a bit of discussion going on at http://bugs.python.org/issue22980
>> about extending SO tags to include bitness values, and I took the opportunity to
>> look into adding platform tags for .pyd files on Windows.
>>
>> There's a patch up there, but I'm interested in any thoughts or concerns
>> people may have that I haven't considered. Distribution is likely to be most
>> affected, and I figured distutils-sig is most likely to have people with strong
>> opinions on the subject anyway, so I'm posting here :)
>>
>> Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll
>> go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32"
>> may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that
>> have #ifdefs in pyconfig.h).
>>
>> By default, distutils' build_ext will use the first extension, and so builds
>> will get the ".cp35-win32.pyd" tag. You can only get the ".cp3-win32.pyd" tag
>> (for stable ABI) by customizing the build process - I don't think that's an
>> unreasonable expectation, especially on Windows.
>>
>> I can see potential for making more generic installable packages with this
>> change, though I'm not pushing for that. The big win is when you're building and
>> testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how the
>> output directories and search paths are organized.
>>
>> Thoughts/concerns?
> 
> Well here?s the code that computes what tags pip will accept:
> https://github.com/pypa/pip/blob/develop/pip/pep425tags.py
> 
> Obviously any additional information put into the tag will increase the number
> of builds someone has to do in order to get full coverage, however if all of the
> data in that are things that genuinely make things binary incompatible then
> that?s the right thing to do.

It could reduce the number of builds that someone has to do, as you could have a single wheel (for Windows) that includes all the platform-specific extensions rather than needing separate wheels. Whether this is an advantage or not deserves to be debated, but there are other installers that could benefit from having incompatible pyd's side-by-side in a way that Python will load the correct one.

Oh, it's also essential for user site-packages on Windows, since the same directory is used for both 32-bit and 64-bit versions of Python. That's actually the most compelling use-case for me, since I'm still keen to see about moving to a secure install for Python 3.5 and in that case we'll have more people using it.

Cheers,
Steve

From Steve.Dower at microsoft.com  Sat Dec 13 01:11:09 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 13 Dec 2014 00:11:09 +0000
Subject: [Distutils] PYD/SO platform tags (#22980)
In-Reply-To: <DM2PR0301MB07341314CA32E5F1D8E24C89F5610@DM2PR0301MB0734.namprd03.prod.outlook.com>
References: <DM2PR0301MB073458ABED6F46FA106DEE08F5600@DM2PR0301MB0734.namprd03.prod.outlook.com>
 <20141213005120.38c6ed04@fsol>
 <DM2PR0301MB07341314CA32E5F1D8E24C89F5610@DM2PR0301MB0734.namprd03.prod.outlook.com>
Message-ID: <DM2PR0301MB0734463AAD24A9D79D7E3B35F5610@DM2PR0301MB0734.namprd03.prod.outlook.com>

Steve Dower wrote:
> Antoine Pitrou wrote:
> 
> My legalistic rationale for using cp3 is that it's actually the version tag, not
> the ABI tag. It seemed from my reading that you'd get tags like
> "cp35-abi3-win32", which is not helpful because you've still specified the minor
> version in the tag. Either it'll work in any CPython 3.x ("cp3") or it needs a
> specific minor version ("cp35"), so I didn't see the value in keeping the ABI
> tag.
> 

Apologies, I got this understanding from PEP 425, not 3149/384. Using "abi3-win32" rather than "cp3-win32" would also make sense. I'm not strongly for or against either variant.

Cheers,
Steve

From jim at zope.com  Sun Dec 14 22:16:48 2014
From: jim at zope.com (Jim Fulton)
Date: Sun, 14 Dec 2014 16:16:48 -0500
Subject: [Distutils] zc.buildout 2.3.0 released
Message-ID: <CAPDm-Fg0onPFHdnN6ENkFLerONPZmMyZRDuo3HDehn4yq=tR4g@mail.gmail.com>

This fixes some compatibility issues with (and requires) setuptools 8.

The bootstrap file has been fixed and also moved to a more secure location:

  https://bootstrap.pypa.io/bootstrap-buildout.py

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From ben+python at benfinney.id.au  Mon Dec 15 07:17:25 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 15 Dec 2014 17:17:25 +1100
Subject: [Distutils] Distribution metadata: build timestamp of the
	distribution
Message-ID: <85d27lcu8a.fsf@benfinney.id.au>

Howdy all,

Where can I find the build timestamp when a distribution was built by
?distutils.setup??

The application I'm writing has a need to get a distribution and use the
timestamp when that distribution was built::

    import pkg_resources

    distribution_name = "foo"
    distribution = pkg_resources.get_distribution(distribution_name)
    distribution_timestamp = distribution.build_timestamp

That doesn't work because the attribute doesn't exist. Where can I find
the timestamp the specified distribution was built?

-- 
 \        ?The right to search for truth implies also a duty; one must |
  `\      not conceal any part of what one has recognized to be true.? |
_o__)                                                 ?Albert Einstein |
Ben Finney


From marius at pov.lt  Mon Dec 15 09:00:46 2014
From: marius at pov.lt (Marius Gedminas)
Date: Mon, 15 Dec 2014 10:00:46 +0200
Subject: [Distutils] Cookie-related PyPI 503 errors
Message-ID: <20141215080046.GA10932@fridge.pov.lt>

I keep getting 503 errors from PyPI that go away after I clear my
cookies.

The entire cycle is as follows:

1. Visit PyPI, log in (because I need to do some maintenance or
   something of my packages).
2. Things continue to work for about a week.
3. Visit PyPI, get a 503 error from Varnish after a noticeable timeout:

    Error 503 backend read error

    backend read error

    Guru Mediation:

    Details: cache-iad2132-IAD 1418630124 4121731453

    Varnish cache server

4. Clear the PyPI cookie, reload page, things work again.
5. Go back to step 1.

Having to clear my cookies about once a week is getting annoying.

Can this be fixed please?

Marius Gedminas
-- 
Beware of bugs in the above code; I have only proved it correct, not tried it.
                -- Donald Knuth
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141215/5f07d723/attachment.sig>

From robin at reportlab.com  Mon Dec 15 12:03:13 2014
From: robin at reportlab.com (Robin Becker)
Date: Mon, 15 Dec 2014 11:03:13 +0000
Subject: [Distutils] change in setuptools 8.0
Message-ID: <548EBFF1.5070500@chamonix.reportlab.co.uk>

A bitbucket user informs me angrily that he cannot use the version of reportlab 
that's latest on pypi because it has a dependency

pillow==2.0.0,>=2.4.0

which is now treated as an 'and' condition by setuptools 8.0 so can not be 
satisfied.

In our latest code we have removed the '==2.0.0,', but presumably there's 
nothing I can do to make the situation less broken for older versions other than 
remove those from pypi.

Unfortunately we had this as part of the install_requires as

install_requires=['pillow ==2.0.0, >=2.4.0','pip>=1.4.1', 'setuptools>=2.2']

so it's our fault for being too lax in describing the requirement. Presumably 
the , in the list was always an 'and' and now the ',' in the elements is also :(
-- 
Robin Becker

From donald at stufft.io  Mon Dec 15 13:20:13 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 07:20:13 -0500
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <548EBFF1.5070500@chamonix.reportlab.co.uk>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
Message-ID: <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io>


> On Dec 15, 2014, at 6:03 AM, Robin Becker <robin at reportlab.com> wrote:
> 
> A bitbucket user informs me angrily that he cannot use the version of reportlab that's latest on pypi because it has a dependency
> 
> pillow==2.0.0,>=2.4.0
> 
> which is now treated as an 'and' condition by setuptools 8.0 so can not be satisfied.
> 
> In our latest code we have removed the '==2.0.0,', but presumably there's nothing I can do to make the situation less broken for older versions other than remove those from pypi.
> 
> Unfortunately we had this as part of the install_requires as
> 
> install_requires=['pillow ==2.0.0, >=2.4.0','pip>=1.4.1', 'setuptools>=2.2']
> 
> so it's our fault for being too lax in describing the requirement. Presumably the , in the list was always an 'and' and now the ',' in the elements is also :(

Historically the meaning of a comma inside of a version specifier is? well complicated. Honestly I have a hard time even putting into words what a comma means at all in a historical context. Sometimes it acts as an OR, sometimes it acts as and AND, and sometimes it acts as something else that I can?t quite explain.

This was part of how setuptools was designed, it valued giving an answer, any answer, over saying ?Sorry this doesn?t make sense?. You can see this most clearly in the version parsing code which would allow versions such as ?dog? or ?this isn?t a version but setuptools will parse it as one?. In PEP 440 we attempted to standardize what a version and what a specifier means, and as part of that we made the decision that we are going to be stricter in what we accept. This means that some things that used to be valid versions are no longer valid versions and in your case, relying on the old, complicated behavior, of a comma that sometimes means different things.

So yea, in a PEP 440 world the comma is AND.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From m.van.rees at zestsoftware.nl  Mon Dec 15 15:05:56 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Mon, 15 Dec 2014 15:05:56 +0100
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io>
Message-ID: <m6mps4$m0m$1@ger.gmane.org>

Donald Stufft schreef op 15-12-14 13:20:
>
>> On Dec 15, 2014, at 6:03 AM, Robin Becker <robin at reportlab.com> wrote:
>>
>> A bitbucket user informs me angrily that he cannot use the version of reportlab that's latest on pypi because it has a dependency
>>
>> pillow==2.0.0,>=2.4.0
>>
>> which is now treated as an 'and' condition by setuptools 8.0 so can not be satisfied.
>>
>> In our latest code we have removed the '==2.0.0,', but presumably there's nothing I can do to make the situation less broken for older versions other than remove those from pypi.
>>
>> Unfortunately we had this as part of the install_requires as
>>
>> install_requires=['pillow ==2.0.0, >=2.4.0','pip>=1.4.1', 'setuptools>=2.2']
>>
>> so it's our fault for being too lax in describing the requirement. Presumably the , in the list was always an 'and' and now the ',' in the elements is also :(
>
> Historically the meaning of a comma inside of a version specifier is? well complicated. Honestly I have a hard time even putting into words what a comma means at all in a historical context. Sometimes it acts as an OR, sometimes it acts as and AND, and sometimes it acts as something else that I can?t quite explain.
>
> This was part of how setuptools was designed, it valued giving an answer, any answer, over saying ?Sorry this doesn?t make sense?. You can see this most clearly in the version parsing code which would allow versions such as ?dog? or ?this isn?t a version but setuptools will parse it as one?. In PEP 440 we attempted to standardize what a version and what a specifier means, and as part of that we made the decision that we are going to be stricter in what we accept. This means that some things that used to be valid versions are no longer valid versions and in your case, relying on the old, complicated behavior, of a comma that sometimes means different things.
>
> So yea, in a PEP 440 world the comma is AND.

Sounds sane.

But I now run into unexpected behaviour when two packages have a 
constraint on the same third package.  For example one has 
'zest.releaser==3.50' and another has 'zest.releaser>=3.40'.  Wanted and 
expected behaviour is to get 3.50, as that satisfies both constraints.

You can test this in a virtualenv with setuptools 8.0.2:

$ pip install 'zest.releaser==3.50,>=3.40'
Downloading/unpacking zest.releaser>=3.40,==3.50
   Downloading zest.releaser-3.53.2.zip
...

So expected is 3.50, but you get the latest version, currently 3.53.2.
Sound like a bug?


Where I am seeing this error in practice is in a buildout.  I have not 
managed to reproduce my error in a small enough buildout that is sane to 
share.  But for the idea, it goes like this.  Latest 
buildout-bootstrap.py gives me zc.buildout 2.3.0 and setuptools 0.8.2. 
The buildout config has pinned zc.buildout to version 2.2.5 and 
setuptools to 7.0 and allow-picked-versions to false.  Then I run 
bin/buildout.   It fails with:

While:
   Installing.
   Getting section _mr.developer.
   Initializing section _mr.developer.
   Installing recipe zc.recipe.egg.
   Getting distribution for 'zc.buildout==2.2.5,>=1.5.0'.
Error: Picked: zc.buildout = 2.2.5

So one package (zc.recipe egg 1.3.2, but similar with latest 2.0.1) has 
a dependency on zc.buildout>=1.5.0 and the buildout config pins 
zc.buildout to 2.2.5 and this somehow fails.

Oddly enough, it goed alright when I set allow-picked-versions to true...


For the record, it then goes wrong later with an error that indicates a 
casualty of the more strict version checking:

The constraint, 2.0.5, is not consistent with the requirement, 
'five.localsitemanager>2.0dev'.
While:
   Installing zeoclient.
Error: Bad constraint 2.0.5 five.localsitemanager>2.0dev

The bad constraint '>2.0dev' is in the five.grok package.  I guess it 
should have been '>2.0.dev0' (or by now simply '>=2.0').  I'll pick it 
up for that package.




-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From donald at stufft.io  Mon Dec 15 18:26:49 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 12:26:49 -0500
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <m6mps4$m0m$1@ger.gmane.org>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io> <m6mps4$m0m$1@ger.gmane.org>
Message-ID: <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>


> On Dec 15, 2014, at 9:05 AM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
> 
> Donald Stufft schreef op 15-12-14 13:20:
>> 
>>> On Dec 15, 2014, at 6:03 AM, Robin Becker <robin at reportlab.com> wrote:
>>> 
>>> A bitbucket user informs me angrily that he cannot use the version of reportlab that's latest on pypi because it has a dependency
>>> 
>>> pillow==2.0.0,>=2.4.0
>>> 
>>> which is now treated as an 'and' condition by setuptools 8.0 so can not be satisfied.
>>> 
>>> In our latest code we have removed the '==2.0.0,', but presumably there's nothing I can do to make the situation less broken for older versions other than remove those from pypi.
>>> 
>>> Unfortunately we had this as part of the install_requires as
>>> 
>>> install_requires=['pillow ==2.0.0, >=2.4.0','pip>=1.4.1', 'setuptools>=2.2']
>>> 
>>> so it's our fault for being too lax in describing the requirement. Presumably the , in the list was always an 'and' and now the ',' in the elements is also :(
>> 
>> Historically the meaning of a comma inside of a version specifier is? well complicated. Honestly I have a hard time even putting into words what a comma means at all in a historical context. Sometimes it acts as an OR, sometimes it acts as and AND, and sometimes it acts as something else that I can?t quite explain.
>> 
>> This was part of how setuptools was designed, it valued giving an answer, any answer, over saying ?Sorry this doesn?t make sense?. You can see this most clearly in the version parsing code which would allow versions such as ?dog? or ?this isn?t a version but setuptools will parse it as one?. In PEP 440 we attempted to standardize what a version and what a specifier means, and as part of that we made the decision that we are going to be stricter in what we accept. This means that some things that used to be valid versions are no longer valid versions and in your case, relying on the old, complicated behavior, of a comma that sometimes means different things.
>> 
>> So yea, in a PEP 440 world the comma is AND.
> 
> Sounds sane.
> 
> But I now run into unexpected behaviour when two packages have a constraint on the same third package.  For example one has 'zest.releaser==3.50' and another has 'zest.releaser>=3.40'. Wanted and expected behaviour is to get 3.50, as that satisfies both constraints.
> 
> You can test this in a virtualenv with setuptools 8.0.2:
> 
> $ pip install 'zest.releaser==3.50,>=3.40'
> Downloading/unpacking zest.releaser>=3.40,==3.50
>  Downloading zest.releaser-3.53.2.zip
> ...
> 
> So expected is 3.50, but you get the latest version, currently 3.53.2.
> Sound like a bug?

Try with the develop version of pip. pip bundles setuptools internally in order to prevent issues from setuptools accidentally getting uninstalled breaking pip and such,

> 
> 
> Where I am seeing this error in practice is in a buildout.  I have not managed to reproduce my error in a small enough buildout that is sane to share.  But for the idea, it goes like this. Latest buildout-bootstrap.py gives me zc.buildout 2.3.0 and setuptools 0.8.2. The buildout config has pinned zc.buildout to version 2.2.5 and setuptools to 7.0 and allow-picked-versions to false.  Then I run bin/buildout.   It fails with:
> 
> While:
>  Installing.
>  Getting section _mr.developer.
>  Initializing section _mr.developer.
>  Installing recipe zc.recipe.egg.
>  Getting distribution for 'zc.buildout==2.2.5,>=1.5.0'.
> Error: Picked: zc.buildout = 2.2.5
> 
> So one package (zc.recipe egg 1.3.2, but similar with latest 2.0.1) has a dependency on zc.buildout>=1.5.0 and the buildout config pins zc.buildout to 2.2.5 and this somehow fails.
> 
> Oddly enough, it goed alright when I set allow-picked-versions to true...
> 
> 
> For the record, it then goes wrong later with an error that indicates a casualty of the more strict version checking:
> 
> The constraint, 2.0.5, is not consistent with the requirement, 'five.localsitemanager>2.0dev'.
> While:
>  Installing zeoclient.
> Error: Bad constraint 2.0.5 five.localsitemanager>2.0dev
> 
> The bad constraint '>2.0dev' is in the five.grok package.  I guess it should have been '>2.0.dev0' (or by now simply '>=2.0').  I'll pick it up for that package.
> 

I don?t know much (or anything really) about buildout or what it does with that.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From jim at zope.com  Mon Dec 15 19:15:06 2014
From: jim at zope.com (Jim Fulton)
Date: Mon, 15 Dec 2014 13:15:06 -0500
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io>
 <m6mps4$m0m$1@ger.gmane.org> <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
Message-ID: <CAPDm-Fgiuf+xBCrE9=34=aiTcSf2XM0BUgk88ewF17K8K7hPrA@mail.gmail.com>

On Mon, Dec 15, 2014 at 12:26 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 15, 2014, at 9:05 AM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>
>> Donald Stufft schreef op 15-12-14 13:20:
>>>
>>>> On Dec 15, 2014, at 6:03 AM, Robin Becker <robin at reportlab.com> wrote:
>>>>
>>>> A bitbucket user informs me angrily that he cannot use the version of reportlab that's latest on pypi because it has a dependency
>>>>
>>>> pillow==2.0.0,>=2.4.0
>>>>
>>>> which is now treated as an 'and' condition by setuptools 8.0 so can not be satisfied.
>>>>
>>>> In our latest code we have removed the '==2.0.0,', but presumably there's nothing I can do to make the situation less broken for older versions other than remove those from pypi.
>>>>
>>>> Unfortunately we had this as part of the install_requires as
>>>>
>>>> install_requires=['pillow ==2.0.0, >=2.4.0','pip>=1.4.1', 'setuptools>=2.2']
>>>>
>>>> so it's our fault for being too lax in describing the requirement. Presumably the , in the list was always an 'and' and now the ',' in the elements is also :(
>>>
>>> Historically the meaning of a comma inside of a version specifier is? well complicated. Honestly I have a hard time even putting into words what a comma means at all in a historical context. Sometimes it acts as an OR, sometimes it acts as and AND, and sometimes it acts as something else that I can?t quite explain.
>>>
>>> This was part of how setuptools was designed, it valued giving an answer, any answer, over saying ?Sorry this doesn?t make sense?. You can see this most clearly in the version parsing code which would allow versions such as ?dog? or ?this isn?t a version but setuptools will parse it as one?. In PEP 440 we attempted to standardize what a version and what a specifier means, and as part of that we made the decision that we are going to be stricter in what we accept. This means that some things that used to be valid versions are no longer valid versions and in your case, relying on the old, complicated behavior, of a comma that sometimes means different things.
>>>
>>> So yea, in a PEP 440 world the comma is AND.
>>
>> Sounds sane.
>>
>> But I now run into unexpected behaviour when two packages have a constraint on the same third package.  For example one has 'zest.releaser==3.50' and another has 'zest.releaser>=3.40'. Wanted and expected behaviour is to get 3.50, as that satisfies both constraints.
>>
>> You can test this in a virtualenv with setuptools 8.0.2:
>>
>> $ pip install 'zest.releaser==3.50,>=3.40'
>> Downloading/unpacking zest.releaser>=3.40,==3.50
>>  Downloading zest.releaser-3.53.2.zip
>> ...
>>
>> So expected is 3.50, but you get the latest version, currently 3.53.2.
>> Sound like a bug?
>
> Try with the develop version of pip. pip bundles setuptools internally in order to prevent issues from setuptools accidentally getting uninstalled breaking pip and such,
>
>>
>>
>> Where I am seeing this error in practice is in a buildout.  I have not managed to reproduce my error in a small enough buildout that is sane to share.  But for the idea, it goes like this. Latest buildout-bootstrap.py gives me zc.buildout 2.3.0 and setuptools 0.8.2. The buildout config has pinned zc.buildout to version 2.2.5 and setuptools to 7.0 and allow-picked-versions to false.  Then I run bin/buildout.   It fails with:
>>
>> While:
>>  Installing.
>>  Getting section _mr.developer.
>>  Initializing section _mr.developer.
>>  Installing recipe zc.recipe.egg.
>>  Getting distribution for 'zc.buildout==2.2.5,>=1.5.0'.
>> Error: Picked: zc.buildout = 2.2.5
>>
>> So one package (zc.recipe egg 1.3.2, but similar with latest 2.0.1) has a dependency on zc.buildout>=1.5.0 and the buildout config pins zc.buildout to 2.2.5 and this somehow fails.
>>
>> Oddly enough, it goed alright when I set allow-picked-versions to true...
>>
>>
>> For the record, it then goes wrong later with an error that indicates a casualty of the more strict version checking:
>>
>> The constraint, 2.0.5, is not consistent with the requirement, 'five.localsitemanager>2.0dev'.
>> While:
>>  Installing zeoclient.
>> Error: Bad constraint 2.0.5 five.localsitemanager>2.0dev
>>
>> The bad constraint '>2.0dev' is in the five.grok package.  I guess it should have been '>2.0.dev0' (or by now simply '>=2.0').  I'll pick it up for that package.
>>
>
> I don?t know much (or anything really) about buildout or what it does with that.

Buildout doesn't try to merge dependency requirements from different packages.
It just installs packages and their dependencies and builds up a working set.

It does merge constraints from the versions section (if any) with
whatever requirement
it's working on at any point in time.

I'm guessing that the requirement: zc.buildout==2.2.5,>=1.5.0
is coming from a setup spec or from some buildout section.  It was a
valid spec once
but isn't any more.

Buildout 2 before 2.3 did a lot of strange gymnastics to try to merge
version constraints
with requirement specs given the strange old rules.

Now, with the new sane rules, it just adds the constraint to the
whatever existing spec is
in the requirement.

There error "Bad constraint 2.0.5 five.localsitemanager>2.0dev" looks
to me like a bug. :)
When buildout gets a single-version constraint, like 2.0.5, it tests
whether that version
is "in" (__contains__) a requirement.

>>> import pkg_resources
>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0dev")
>>> '2.0.5' in r
False
>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0.dev0")
>>> '2.0.5' in r
True

I have a feeling the definition of "dev" has changed too.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From donald at stufft.io  Mon Dec 15 19:26:56 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 13:26:56 -0500
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <CAPDm-Fgiuf+xBCrE9=34=aiTcSf2XM0BUgk88ewF17K8K7hPrA@mail.gmail.com>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io> <m6mps4$m0m$1@ger.gmane.org>
 <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
 <CAPDm-Fgiuf+xBCrE9=34=aiTcSf2XM0BUgk88ewF17K8K7hPrA@mail.gmail.com>
Message-ID: <8767D699-5519-475B-9C4A-68C602222215@stufft.io>


> On Dec 15, 2014, at 1:15 PM, Jim Fulton <jim at zope.com> wrote:
> 
>>>> import pkg_resources
>>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0dev")
>>>> '2.0.5' in r
> False
>>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0.dev0")
>>>> '2.0.5' in r
> True
> 
> I have a feeling the definition of "dev" has changed too.

I think this is a bug with the packaging lib that has to do with the fact that
the ?>2.0dev? doesn?t have a . separating it. Taking a look at it now.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From jim at zope.com  Mon Dec 15 19:27:26 2014
From: jim at zope.com (Jim Fulton)
Date: Mon, 15 Dec 2014 13:27:26 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
Message-ID: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>

I think the changes in version management in setuptools 8 are a great
step forward,
but I think the transition is going to hurt a lot.

For buildout, I'm thinking of of releasing 2.3.1 that reverts the
changes in 2.3 and
adds a requirement for setuptools <8 to give more time to respond to
these  changes.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From donald at stufft.io  Mon Dec 15 19:44:04 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 13:44:04 -0500
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <8767D699-5519-475B-9C4A-68C602222215@stufft.io>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io> <m6mps4$m0m$1@ger.gmane.org>
 <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
 <CAPDm-Fgiuf+xBCrE9=34=aiTcSf2XM0BUgk88ewF17K8K7hPrA@mail.gmail.com>
 <8767D699-5519-475B-9C4A-68C602222215@stufft.io>
Message-ID: <363DF675-CE54-4E13-BF2C-CF216C15CECF@stufft.io>


> On Dec 15, 2014, at 1:26 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Dec 15, 2014, at 1:15 PM, Jim Fulton <jim at zope.com> wrote:
>> 
>>>>> import pkg_resources
>>>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0dev")
>>>>> '2.0.5' in r
>> False
>>>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0.dev0")
>>>>> '2.0.5' in r
>> True
>> 
>> I have a feeling the definition of "dev" has changed too.
> 
> I think this is a bug with the packaging lib that has to do with the fact that
> the ?>2.0dev? doesn?t have a . separating it. Taking a look at it now.
> 

Found the bug, it?s fixed in the packaging lib here: https://github.com/pypa/packaging/pull/18

Will get that merged once tests passed and a new release out and a PR to setuptools.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From donald at stufft.io  Mon Dec 15 19:45:28 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 13:45:28 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
Message-ID: <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>


> On Dec 15, 2014, at 1:27 PM, Jim Fulton <jim at zope.com> wrote:
> 
> I think the changes in version management in setuptools 8 are a great
> step forward,
> but I think the transition is going to hurt a lot.
> 
> For buildout, I'm thinking of of releasing 2.3.1 that reverts the
> changes in 2.3 and
> adds a requirement for setuptools <8 to give more time to respond to
> these  changes.

Is there something I?m not aware that is broken currently? I thought the
transition was going pretty smoothly overall considering that a core piece
of code inside of setuptools was touched.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From robertc at robertcollins.net  Mon Dec 15 19:41:50 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Tue, 16 Dec 2014 07:41:50 +1300
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
Message-ID: <CAJ3HoZ1z8+E+ra04RPOkOHgRzmSaf2+SG6zUrKwPhdmjuFu-fQ@mail.gmail.com>

+1 - I think thats a great idea. We've had to pin < setuptools 8 in
the OpenStack ecosystem too.

It would have been nice to offer some period of time where the cases
which were going to change would have warned. I realise thats
non-trivial, but this is systemic infrastructure at the core of our
dependency graph - everything that expresses requirements is affected.

-Rob

On 16 December 2014 at 07:27, Jim Fulton <jim at zope.com> wrote:
> I think the changes in version management in setuptools 8 are a great
> step forward,
> but I think the transition is going to hurt a lot.
>
> For buildout, I'm thinking of of releasing 2.3.1 that reverts the
> changes in 2.3 and
> adds a requirement for setuptools <8 to give more time to respond to
> these  changes.
>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From donald at stufft.io  Mon Dec 15 19:50:49 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 13:50:49 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <CAJ3HoZ1z8+E+ra04RPOkOHgRzmSaf2+SG6zUrKwPhdmjuFu-fQ@mail.gmail.com>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <CAJ3HoZ1z8+E+ra04RPOkOHgRzmSaf2+SG6zUrKwPhdmjuFu-fQ@mail.gmail.com>
Message-ID: <124627F2-EB27-45A4-B3F8-3EC883F691D7@stufft.io>


> On Dec 15, 2014, at 1:41 PM, Robert Collins <robertc at robertcollins.net> wrote:
> 
> +1 - I think thats a great idea. We've had to pin < setuptools 8 in
> the OpenStack ecosystem too.

As far as I?m aware, Openstack is just about ready to unpin setuptools. I believe
the only thing holding it back is a pbr change.

> 
> It would have been nice to offer some period of time where the cases
> which were going to change would have warned. I realise thats
> non-trivial, but this is systemic infrastructure at the core of our
> dependency graph - everything that expresses requirements is affected.
> 
> -Rob
> 
> On 16 December 2014 at 07:27, Jim Fulton <jim at zope.com> wrote:
>> I think the changes in version management in setuptools 8 are a great
>> step forward,
>> but I think the transition is going to hurt a lot.
>> 
>> For buildout, I'm thinking of of releasing 2.3.1 that reverts the
>> changes in 2.3 and
>> adds a requirement for setuptools <8 to give more time to respond to
>> these  changes.
>> 
>> Jim
>> 
>> --
>> Jim Fulton
>> http://www.linkedin.com/in/jimfulton
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> 
> 
> 
> -- 
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From jim at zope.com  Mon Dec 15 19:51:31 2014
From: jim at zope.com (Jim Fulton)
Date: Mon, 15 Dec 2014 13:51:31 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
Message-ID: <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>

On Mon, Dec 15, 2014 at 1:45 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 15, 2014, at 1:27 PM, Jim Fulton <jim at zope.com> wrote:
>>
>> I think the changes in version management in setuptools 8 are a great
>> step forward,
>> but I think the transition is going to hurt a lot.
>>
>> For buildout, I'm thinking of of releasing 2.3.1 that reverts the
>> changes in 2.3 and
>> adds a requirement for setuptools <8 to give more time to respond to
>> these  changes.
>
> Is there something I?m not aware that is broken currently? I thought the
> transition was going pretty smoothly overall considering that a core piece
> of code inside of setuptools was touched.

The problem is that version numbers and requirements aren't interpreted
the same way. I think that's going to take some time to sort out.  In
the mean time,
I'd like to try to stop the pain.

I think you've been very responsive and that's much appreciated.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From robertc at robertcollins.net  Mon Dec 15 20:10:14 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Tue, 16 Dec 2014 08:10:14 +1300
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <124627F2-EB27-45A4-B3F8-3EC883F691D7@stufft.io>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <CAJ3HoZ1z8+E+ra04RPOkOHgRzmSaf2+SG6zUrKwPhdmjuFu-fQ@mail.gmail.com>
 <124627F2-EB27-45A4-B3F8-3EC883F691D7@stufft.io>
Message-ID: <CAJ3HoZ2_7xZ1n6OOwrT6OWbpgHBS0rk4cOeONa0MJHfDRyU8KQ@mail.gmail.com>

We're currently pinned -
http://lists.openstack.org/pipermail/openstack-dev/2014-December/052985.html
- we tried a fix, it broke things more, so we rolled it back and are
now working on a new one - https://review.openstack.org/#/c/141667/ -
which will simply *stop including version data* in our version
strings. As setuptools 7 and 8 are mutually incompatible, we can
either support this feature in 8 and break setuptools 7 use, or we can
support it in 7, and break installs on 8.

I haven't tracked *all* the details of this (I'm currently away from
home dealing with more estate cleanup :( ) - see the #openstack-infra
channel logs over the last 48hr or so for more details.

-Rob

On 16 December 2014 at 07:50, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 15, 2014, at 1:41 PM, Robert Collins <robertc at robertcollins.net> wrote:
>>
>> +1 - I think thats a great idea. We've had to pin < setuptools 8 in
>> the OpenStack ecosystem too.
>
> As far as I?m aware, Openstack is just about ready to unpin setuptools. I believe
> the only thing holding it back is a pbr change.
>
>>
>> It would have been nice to offer some period of time where the cases
>> which were going to change would have warned. I realise thats
>> non-trivial, but this is systemic infrastructure at the core of our
>> dependency graph - everything that expresses requirements is affected.
>>
>> -Rob
>>
>> On 16 December 2014 at 07:27, Jim Fulton <jim at zope.com> wrote:
>>> I think the changes in version management in setuptools 8 are a great
>>> step forward,
>>> but I think the transition is going to hurt a lot.
>>>
>>> For buildout, I'm thinking of of releasing 2.3.1 that reverts the
>>> changes in 2.3 and
>>> adds a requirement for setuptools <8 to give more time to respond to
>>> these  changes.
>>>
>>> Jim
>>>
>>> --
>>> Jim Fulton
>>> http://www.linkedin.com/in/jimfulton
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>>
>> --
>> Robert Collins <rbtcollins at hp.com>
>> Distinguished Technologist
>> HP Converged Cloud
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>



-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From donald at stufft.io  Mon Dec 15 22:02:57 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 16:02:57 -0500
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <363DF675-CE54-4E13-BF2C-CF216C15CECF@stufft.io>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io> <m6mps4$m0m$1@ger.gmane.org>
 <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
 <CAPDm-Fgiuf+xBCrE9=34=aiTcSf2XM0BUgk88ewF17K8K7hPrA@mail.gmail.com>
 <8767D699-5519-475B-9C4A-68C602222215@stufft.io>
 <363DF675-CE54-4E13-BF2C-CF216C15CECF@stufft.io>
Message-ID: <BC26AA7F-42CA-434D-86D8-AE29B6F2A823@stufft.io>

A new setuptools release is out that should fix the 2.0.5 in dev bug. 


> On Dec 15, 2014, at 1:44 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Dec 15, 2014, at 1:26 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>> 
>>> On Dec 15, 2014, at 1:15 PM, Jim Fulton <jim at zope.com> wrote:
>>> 
>>>>>> import pkg_resources
>>>>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0dev")
>>>>>> '2.0.5' in r
>>> False
>>>>>> r = pkg_resources.Requirement.parse("five.localsitemanager>2.0.dev0")
>>>>>> '2.0.5' in r
>>> True
>>> 
>>> I have a feeling the definition of "dev" has changed too.
>> 
>> I think this is a bug with the packaging lib that has to do with the fact that
>> the ?>2.0dev? doesn?t have a . separating it. Taking a look at it now.
> 
> Found the bug, it?s fixed in the packaging lib here: https://github.com/pypa/packaging/pull/18
> 
> Will get that merged once tests passed and a new release out and a PR to setuptools.
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From m.van.rees at zestsoftware.nl  Mon Dec 15 23:27:17 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Mon, 15 Dec 2014 23:27:17 +0100
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io> <m6mps4$m0m$1@ger.gmane.org>
 <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
Message-ID: <m6nn86$3s0$1@ger.gmane.org>

Donald Stufft schreef op 15-12-14 18:26:
>
>> On Dec 15, 2014, at 9:05 AM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>
>> But I now run into unexpected behaviour when two packages have a constraint on the same third package.  For example one has 'zest.releaser==3.50' and another has 'zest.releaser>=3.40'. Wanted and expected behaviour is to get 3.50, as that satisfies both constraints.
>>
>> You can test this in a virtualenv with setuptools 8.0.2:
>>
>> $ pip install 'zest.releaser==3.50,>=3.40'
>> Downloading/unpacking zest.releaser>=3.40,==3.50
>>   Downloading zest.releaser-3.53.2.zip
>> ...
>>
>> So expected is 3.50, but you get the latest version, currently 3.53.2.
>> Sound like a bug?
>
> Try with the develop version of pip. pip bundles setuptools internally in order to prevent issues from setuptools accidentally getting uninstalled breaking pip and such,

Ah, that works.  Thanks.


(venv)mauritsvanrees at procyon:venv $ pip install 
https://github.com/pypa/pip/tarball/develop#egg=pip-dev
Downloading/unpacking pip-dev from 
https://github.com/pypa/pip/tarball/develop
...
Successfully installed pip-dev
Cleaning up...
(venv)mauritsvanrees at procyon:venv $ pip --version
pip 6.0.dev1 from 
/Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages (python 2.7)
(venv)mauritsvanrees at procyon:venv $ pip install 'zest.releaser==3.50,>=3.40'
Collecting zest.releaser==3.50,>=3.40
   Downloading zest.releaser-3.50.zip (124kB)
     100% |################################| 126kB 1.1MB/s
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./lib/python2.7/site-packages (from zest.releaser==3.50,>=3.40)
Installing collected packages: zest.releaser
   Found existing installation: zest.releaser 3.53.2
     Uninstalling zest.releaser:
       Successfully uninstalled zest.releaser
   Running setup.py install for zest.releaser
     Skipping installation of 
/Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/zest/__init__.py 
(namespace package)
     Installing 
/Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/zest.releaser-3.50-py2.7-nspkg.pth
...
Successfully installed zest.releaser


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From donald at stufft.io  Mon Dec 15 23:47:09 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 17:47:09 -0500
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <m6nn86$3s0$1@ger.gmane.org>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io> <m6mps4$m0m$1@ger.gmane.org>
 <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io> <m6nn86$3s0$1@ger.gmane.org>
Message-ID: <0A61790D-A73F-4582-9088-1755CF05209A@stufft.io>

Great. We're gonna hopefully release pip 6 this week. 


> On Dec 15, 2014, at 5:27 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
> 
> Donald Stufft schreef op 15-12-14 18:26:
>> 
>>> On Dec 15, 2014, at 9:05 AM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>> 
>>> But I now run into unexpected behaviour when two packages have a constraint on the same third package.  For example one has 'zest.releaser==3.50' and another has 'zest.releaser>=3.40'. Wanted and expected behaviour is to get 3.50, as that satisfies both constraints.
>>> 
>>> You can test this in a virtualenv with setuptools 8.0.2:
>>> 
>>> $ pip install 'zest.releaser==3.50,>=3.40'
>>> Downloading/unpacking zest.releaser>=3.40,==3.50
>>>  Downloading zest.releaser-3.53.2.zip
>>> ...
>>> 
>>> So expected is 3.50, but you get the latest version, currently 3.53.2.
>>> Sound like a bug?
>> 
>> Try with the develop version of pip. pip bundles setuptools internally in order to prevent issues from setuptools accidentally getting uninstalled breaking pip and such,
> 
> Ah, that works.  Thanks.
> 
> 
> (venv)mauritsvanrees at procyon:venv $ pip install https://github.com/pypa/pip/tarball/develop#egg=pip-dev
> Downloading/unpacking pip-dev from https://github.com/pypa/pip/tarball/develop
> ...
> Successfully installed pip-dev
> Cleaning up...
> (venv)mauritsvanrees at procyon:venv $ pip --version
> pip 6.0.dev1 from /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages (python 2.7)
> (venv)mauritsvanrees at procyon:venv $ pip install 'zest.releaser==3.50,>=3.40'
> Collecting zest.releaser==3.50,>=3.40
>  Downloading zest.releaser-3.50.zip (124kB)
>    100% |################################| 126kB 1.1MB/s
> Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages (from zest.releaser==3.50,>=3.40)
> Installing collected packages: zest.releaser
>  Found existing installation: zest.releaser 3.53.2
>    Uninstalling zest.releaser:
>      Successfully uninstalled zest.releaser
>  Running setup.py install for zest.releaser
>    Skipping installation of /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/zest/__init__.py (namespace package)
>    Installing /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/zest.releaser-3.50-py2.7-nspkg.pth
> ...
> Successfully installed zest.releaser
> 
> 
> -- 
> Maurits van Rees: http://maurits.vanrees.org/
> Zest Software: http://zestsoftware.nl
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From m.van.rees at zestsoftware.nl  Mon Dec 15 23:50:17 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Mon, 15 Dec 2014 23:50:17 +0100
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
Message-ID: <m6noj9$oiu$1@ger.gmane.org>

Jim Fulton schreef op 15-12-14 19:51:
> On Mon, Dec 15, 2014 at 1:45 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>>> On Dec 15, 2014, at 1:27 PM, Jim Fulton <jim at zope.com> wrote:
>>>
>>> I think the changes in version management in setuptools 8 are a great
>>> step forward,
>>> but I think the transition is going to hurt a lot.
>>>
>>> For buildout, I'm thinking of of releasing 2.3.1 that reverts the
>>> changes in 2.3 and
>>> adds a requirement for setuptools <8 to give more time to respond to
>>> these  changes.

Possibly.
Or maybe a zc.buildout 2.2.x that requires setuptools<8 or a 
bootstrap.py that defaults to zc.buildout 2.2.5 and setuptools 7.0. 
(Just some options, I haven't thought them through.)

>>
>> Is there something I?m not aware that is broken currently? I thought the
>> transition was going pretty smoothly overall considering that a core piece
>> of code inside of setuptools was touched.
>
> The problem is that version numbers and requirements aren't interpreted
> the same way. I think that's going to take some time to sort out.  In
> the mean time,
> I'd like to try to stop the pain.

An example of some pain, although it is only a repeated warning:

$ curl -O https://bootstrap.pypa.io/bootstrap-buildout.py
$ python2.7 bootstrap-buildout.py
Downloading 
https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.4.zip
Extracting in TMP1
Now working in TMP1/setuptools-8.0.4
Building a Setuptools egg in TMP2
TMP2/setuptools-8.0.4-py2.7.egg
TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning: 
'zc.buildout-1.0.0 ()' is being parsed as a legacy, non PEP 440, 
version. You may find odd behavior and sort order. In particular it will 
be sorted as less than 0.0. It is recommend to migrate to PEP 440 
compatible versions.
...
(dozens more warnings, two for every zc.buildout version on PyPI)
...
TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning: 
'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440, 
version. You may find odd behavior and sort order. In particular it will 
be sorted as less than 0.0. It is recommend to migrate to PEP 440 
compatible versions.

I'm not sure if that is setuptools doing funky stuff or zc.buildout 
doing funky stuff.  Probably both.  And probably they both *need* to do 
funky stuff. ;-)

> I think you've been very responsive and that's much appreciated.

+1


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From m.van.rees at zestsoftware.nl  Tue Dec 16 00:46:07 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Tue, 16 Dec 2014 00:46:07 +0100
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <m6noj9$oiu$1@ger.gmane.org>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org>
Message-ID: <m6nrs0$bqq$1@ger.gmane.org>

Maurits van Rees schreef op 15-12-14 23:50:
> TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning:
> 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440,
> version. You may find odd behavior and sort order. In particular it will
> be sorted as less than 0.0. It is recommend to migrate to PEP 440
> compatible versions.

Actually, I have configured zc.buildout to use a download-cache 
directory where it saves downloaded zips/tarballs/eggs from PyPI. 
Buildout then adds this directory as a find-link and setuptools calls 
package_index.scan_url on it.  This prints the warnings, because it sees 
for example a file
'/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip'
This gets parsed as a Distribution with project_name 
'Acquisition-2.13.8' and an empty version.

Ah, a demo with plain setuptools 0.8.4 is easy to setup:

$ mkdir bar
$ touch bar/project-1.0.zip
$ . bin/activate
(venv) $ python
Python 2.7.8 (default, Jul 28 2014, 10:41:45)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> from setuptools import package_index
 >>> pi = package_index.PackageIndex()
 >>> pi.add_find_links(['bar'])
/Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: 
RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 
440, version. You may find odd behavior and sort order. In particular it 
will be sorted as less than 0.0. It is recommend to migrate to PEP 440 
compatible versions.
   RuntimeWarning,
 >>>



-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From donald at stufft.io  Tue Dec 16 00:48:31 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Dec 2014 18:48:31 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <m6nrs0$bqq$1@ger.gmane.org>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
Message-ID: <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>


> On Dec 15, 2014, at 6:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
> 
> Maurits van Rees schreef op 15-12-14 23:50:
>> TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning:
>> 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440,
>> version. You may find odd behavior and sort order. In particular it will
>> be sorted as less than 0.0. It is recommend to migrate to PEP 440
>> compatible versions.
> 
> Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it.  This prints the warnings, because it sees for example a file
> '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip'
> This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.
> 
> Ah, a demo with plain setuptools 0.8.4 is easy to setup:
> 
> $ mkdir bar
> $ touch bar/project-1.0.zip
> $ . bin/activate
> (venv) $ python
> Python 2.7.8 (default, Jul 28 2014, 10:41:45)
> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from setuptools import package_index
> >>> pi = package_index.PackageIndex()
> >>> pi.add_find_links(['bar'])
> /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.
>  RuntimeWarning,
> >>>

Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From m.van.rees at zestsoftware.nl  Tue Dec 16 01:05:53 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Tue, 16 Dec 2014 01:05:53 +0100
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <CAPDm-Fgiuf+xBCrE9=34=aiTcSf2XM0BUgk88ewF17K8K7hPrA@mail.gmail.com>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io> <m6mps4$m0m$1@ger.gmane.org>
 <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
 <CAPDm-Fgiuf+xBCrE9=34=aiTcSf2XM0BUgk88ewF17K8K7hPrA@mail.gmail.com>
Message-ID: <m6nt11$rug$1@ger.gmane.org>

Jim Fulton schreef op 15-12-14 19:15:
> Buildout doesn't try to merge dependency requirements from different packages.
> It just installs packages and their dependencies and builds up a working set.
>
> It does merge constraints from the versions section (if any) with
> whatever requirement
> it's working on at any point in time.
>
> I'm guessing that the requirement: zc.buildout==2.2.5,>=1.5.0
> is coming from a setup spec or from some buildout section.  It was a
> valid spec once
> but isn't any more.

Let's try a buildout with Plone.  This one goes fine (except for 
possible warnings about legacy versions):

$ curl -O https://bootstrap.pypa.io/bootstrap-buildout.py
$ cat buildout.cfg
[buildout]
extends = http://dist.plone.org/release/4.3-latest/versions.cfg
versions = versions
parts = instance

[versions]
# Override version pins that are in the file of the extends line.
zc.buildout = 2.3.0
setuptools = 8.0.4

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
http-address = 8080
eggs =
     Plone

$ python bootstrap-buildout.py
$ bin/buildout

All is well.

Now edit buildout.cfg and add this line in the [buildout] section:

   allow-picked-versions = false

$ bin/buildout
...
While:
   Installing.
   Getting section instance.
   Initializing section instance.
   Installing recipe plone.recipe.zope2instance.
   Getting distribution for 'ZODB3==3.10.5,>=3.9'.
Error: Picked: ZODB3 = 3.10.5

Somehow a problem appears because the buildout config pins ZODB3 to 
3.10.5, and buildout installs plone.recipe.zope2instance which requires 
'ZODB3 >= 3.9'.  (I wonder if this behaviour is specific for recipes, 
but I guess not.)

Now revert the change, and instead add this line in the [buildout] section:

   show-picked-versions = true

$ bin/buildout
...
Versions had to be automatically picked.
The following part definition lists the versions picked:
[versions]
AccessControl = 3.0.8
Markdown = 2.0.3
Products.ATContentTypes = 2.1.14
Products.OFSP = 2.13.2
Products.PlonePAS = 4.1.4
ZODB3 = 3.10.5
Zope2 = 2.13.22
archetypes.schemaextender = 2.1.4
collective.z3cform.datetimewidget = 1.2.6
diazo = 1.0.6
lxml = 2.3.6
mechanize = 0.2.5
plone.app.contentmenu = 2.0.10
plone.app.imaging = 1.0.11
plone.app.jquery = 1.7.2
plone.app.linkintegrity = 1.5.5
plone.app.registry = 1.2.3
plone.app.z3cform = 0.7.7
plone.autoform = 1.6.1
plone.behavior = 1.0.2
plone.browserlayer = 2.1.3
plone.dexterity = 2.2.4
plone.keyring = 2.0.1
plone.namedfile = 2.0.7
plone.registry = 1.0.2
plone.resource = 1.0.3
plone.scale = 1.3.4
plone.schemaeditor = 1.3.8
plone.supermodel = 1.2.6
plone.z3cform = 0.8.0
python-gettext = 1.2
python-openid = 2.2.5
repoze.xmliter = 0.5
transaction = 1.1.1
z3c.form = 3.2.1
zope.app.locales = 3.6.2
zope.tales = 3.5.3

All these versions are definitely pinned in the file we extend (or in 
files that this file extends):
http://dist.plone.org/release/4.3-latest/versions.cfg

So: strange behaviour.

-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From chris.jerdonek at gmail.com  Tue Dec 16 07:08:29 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 15 Dec 2014 22:08:29 -0800
Subject: [Distutils] Cookie-related PyPI 503 errors
In-Reply-To: <20141215080046.GA10932@fridge.pov.lt>
References: <20141215080046.GA10932@fridge.pov.lt>
Message-ID: <CAOTb1we3H79a3czgMSAcuOz5kPfakVt+d2vJEenJypEXTNbg=w@mail.gmail.com>

On Mon, Dec 15, 2014 at 12:00 AM, Marius Gedminas <marius at pov.lt> wrote:
>
> I keep getting 503 errors from PyPI that go away after I clear my
> cookies.
>
> The entire cycle is as follows:
>
> 1. Visit PyPI, log in (because I need to do some maintenance or
>    something of my packages).
> 2. Things continue to work for about a week.
> 3. Visit PyPI, get a 503 error from Varnish after a noticeable timeout:
>
>     Error 503 backend read error
>
>     backend read error
>
>     Guru Mediation:
>
>     Details: cache-iad2132-IAD 1418630124 4121731453
>
>     Varnish cache server
>


This also happened to me today.  I timed it twice, and both times it took
about 26 seconds for the 503 to occur after reloading the page.

--Chris




>
> 4. Clear the PyPI cookie, reload page, things work again.
> 5. Go back to step 1.
>
> Having to clear my cookies about once a week is getting annoying.
>
> Can this be fixed please?
>
> Marius Gedminas
> --
> Beware of bugs in the above code; I have only proved it correct, not tried
> it.
>                 -- Donald Knuth
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141215/a77bb0fb/attachment.html>

From donald at stufft.io  Tue Dec 16 07:45:05 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 01:45:05 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
Message-ID: <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>


> On Dec 16, 2014, at 1:42 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> 
> On Tue, Dec 16, 2014 at 2:48 AM, Donald Stufft <donald at stufft.io> wrote:
>>> On Dec 15, 2014, at 6:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>> 
>>> Maurits van Rees schreef op 15-12-14 23:50:
>>>> TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning:
>>>> 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440,
>>>> version. You may find odd behavior and sort order. In particular it will
>>>> be sorted as less than 0.0. It is recommend to migrate to PEP 440
>>>> compatible versions.
>>> 
>>> Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it.  This prints the warnings, because it sees for example a file
>>> '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip'
>>> This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.
>>> 
>>> Ah, a demo with plain setuptools 0.8.4 is easy to setup:
>>> 
>>> $ mkdir bar
>>> $ touch bar/project-1.0.zip
>>> $ . bin/activate
>>> (venv) $ python
>>> Python 2.7.8 (default, Jul 28 2014, 10:41:45)
>>> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> from setuptools import package_index
>>>>>> pi = package_index.PackageIndex()
>>>>>> pi.add_find_links(['bar'])
>>> /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.
>>> RuntimeWarning,
>>>>>> 
>> 
>> Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam.
> 
> 
> I'd like to opt-out from PEP440 and use semantic versioning (that is
> incompatible with PEP440) for my Python packages. I think
> pip/setuptools/whatever should not stay in the way and give authors
> ability to choose versioning strategy that is most suitable for their
> workflow.
> 
> To achieve this:
> 1. pip/setuptools needs to be aware of different versioning strategies
> 2. package needs to convey which versioning strategy it uses
> 3. pip/setuptools needs a documented requirement for versions semantic
> (what it understands about versions and how it uses them)
> 
> For example of the above - pip could document that it doesn't install
> development and alpha versions unless explicitly requested with
> command line switch, and instead of taking users to read gory details
> of PEP440 just mention how package may tell pip that it is alpha or
> development.
> 
> Previous vague attempt to request the same thing
> https://github.com/pypa/pip/issues/1557 (just for the history)
> -- 
> anatoly t.

It?s not going to happen, it can?t happen reasonably. Luckily though it?s
quite trivial for an author to adapt semver to work with PEP 440 syntax.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From donald at stufft.io  Tue Dec 16 08:25:33 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 02:25:33 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
 <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
 <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>
Message-ID: <4652E410-7764-4A3C-997F-FA2F6C804567@stufft.io>


> On Dec 16, 2014, at 2:24 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> 
> On Tue, Dec 16, 2014 at 9:45 AM, Donald Stufft <donald at stufft.io> wrote:
>>> On Dec 16, 2014, at 1:42 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>>> 
>>> On Tue, Dec 16, 2014 at 2:48 AM, Donald Stufft <donald at stufft.io> wrote:
>>>>> On Dec 15, 2014, at 6:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>>>> 
>>>>> Maurits van Rees schreef op 15-12-14 23:50:
>>>>>> TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning:
>>>>>> 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440,
>>>>>> version. You may find odd behavior and sort order. In particular it will
>>>>>> be sorted as less than 0.0. It is recommend to migrate to PEP 440
>>>>>> compatible versions.
>>>>> 
>>>>> Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it.  This prints the warnings, because it sees for example a file
>>>>> '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip'
>>>>> This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.
>>>>> 
>>>>> Ah, a demo with plain setuptools 0.8.4 is easy to setup:
>>>>> 
>>>>> $ mkdir bar
>>>>> $ touch bar/project-1.0.zip
>>>>> $ . bin/activate
>>>>> (venv) $ python
>>>>> Python 2.7.8 (default, Jul 28 2014, 10:41:45)
>>>>> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>> from setuptools import package_index
>>>>>>>> pi = package_index.PackageIndex()
>>>>>>>> pi.add_find_links(['bar'])
>>>>> /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.
>>>>> RuntimeWarning,
>>>>>>>> 
>>>> 
>>>> Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam.
>>> 
>>> 
>>> I'd like to opt-out from PEP440 and use semantic versioning (that is
>>> incompatible with PEP440) for my Python packages. I think
>>> pip/setuptools/whatever should not stay in the way and give authors
>>> ability to choose versioning strategy that is most suitable for their
>>> workflow.
>>> 
>>> To achieve this:
>>> 1. pip/setuptools needs to be aware of different versioning strategies
>>> 2. package needs to convey which versioning strategy it uses
>>> 3. pip/setuptools needs a documented requirement for versions semantic
>>> (what it understands about versions and how it uses them)
>>> 
>>> For example of the above - pip could document that it doesn't install
>>> development and alpha versions unless explicitly requested with
>>> command line switch, and instead of taking users to read gory details
>>> of PEP440 just mention how package may tell pip that it is alpha or
>>> development.
>>> 
>>> Previous vague attempt to request the same thing
>>> https://github.com/pypa/pip/issues/1557 (just for the history)
>>> --
>>> anatoly t.
>> 
>> It?s not going to happen, it can?t happen reasonably. Luckily though it?s
>> quite trivial for an author to adapt semver to work with PEP 440 syntax.
> 
> Or move to other distribution method than PyPI FWIW. If OpenStack
> ecosystem could be able to provide an alternative, I'd gladly switch.
> There are already too many packaging bicycles out there for dynamic
> languages - Ruby, npm, PyPI - but in fact the only difference that
> people need for server side of this stuff is distinctive design and
> URL/subdomain.

Openstack won?t be providing one, they are adapting server to PEP 440
syntax.

If you want to go elsewhere be my guest.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From jim at zope.com  Tue Dec 16 11:55:58 2014
From: jim at zope.com (Jim Fulton)
Date: Tue, 16 Dec 2014 05:55:58 -0500
Subject: [Distutils] change in setuptools 8.0
In-Reply-To: <m6nt11$rug$1@ger.gmane.org>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io>
 <m6mps4$m0m$1@ger.gmane.org> <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
 <CAPDm-Fgiuf+xBCrE9=34=aiTcSf2XM0BUgk88ewF17K8K7hPrA@mail.gmail.com>
 <m6nt11$rug$1@ger.gmane.org>
Message-ID: <CAPDm-FiLYFsr38kHu8UmuWtHRAPMVLdmO1F6W5wLPaoDT0LTZQ@mail.gmail.com>

On Mon, Dec 15, 2014 at 7:05 PM, Maurits van Rees
<m.van.rees at zestsoftware.nl> wrote:
> Jim Fulton schreef op 15-12-14 19:15:
>>
>> Buildout doesn't try to merge dependency requirements from different
>> packages.
>> It just installs packages and their dependencies and builds up a working
>> set.
>>
>> It does merge constraints from the versions section (if any) with
>> whatever requirement
>> it's working on at any point in time.
>>
>> I'm guessing that the requirement: zc.buildout==2.2.5,>=1.5.0
>> is coming from a setup spec or from some buildout section.  It was a
>> valid spec once
>> but isn't any more.
>
>
> Let's try a buildout with Plone.  This one goes fine (except for possible
> warnings about legacy versions):
>
> $ curl -O https://bootstrap.pypa.io/bootstrap-buildout.py
> $ cat buildout.cfg
> [buildout]
> extends = http://dist.plone.org/release/4.3-latest/versions.cfg
> versions = versions
> parts = instance
>
> [versions]
> # Override version pins that are in the file of the extends line.
> zc.buildout = 2.3.0
> setuptools = 8.0.4
>
> [instance]
> recipe = plone.recipe.zope2instance
> user = admin:admin
> http-address = 8080
> eggs =
>     Plone
>
> $ python bootstrap-buildout.py
> $ bin/buildout
>
> All is well.
>
> Now edit buildout.cfg and add this line in the [buildout] section:
>
>   allow-picked-versions = false
>
> $ bin/buildout
> ...
> While:
>   Installing.
>   Getting section instance.
>   Initializing section instance.
>   Installing recipe plone.recipe.zope2instance.
>   Getting distribution for 'ZODB3==3.10.5,>=3.9'.
> Error: Picked: ZODB3 = 3.10.5
>
> Somehow a problem appears because the buildout config pins ZODB3 to 3.10.5,
> and buildout installs plone.recipe.zope2instance which requires 'ZODB3 >=
> 3.9'.  (I wonder if this behaviour is specific for recipes, but I guess
> not.)
>
> Now revert the change, and instead add this line in the [buildout] section:
>
>   show-picked-versions = true
>
> $ bin/buildout
> ...
> Versions had to be automatically picked.
> The following part definition lists the versions picked:
> [versions]
> AccessControl = 3.0.8
> Markdown = 2.0.3
> Products.ATContentTypes = 2.1.14
> Products.OFSP = 2.13.2
> Products.PlonePAS = 4.1.4
> ZODB3 = 3.10.5
> Zope2 = 2.13.22
> archetypes.schemaextender = 2.1.4
> collective.z3cform.datetimewidget = 1.2.6
> diazo = 1.0.6
> lxml = 2.3.6
> mechanize = 0.2.5
> plone.app.contentmenu = 2.0.10
> plone.app.imaging = 1.0.11
> plone.app.jquery = 1.7.2
> plone.app.linkintegrity = 1.5.5
> plone.app.registry = 1.2.3
> plone.app.z3cform = 0.7.7
> plone.autoform = 1.6.1
> plone.behavior = 1.0.2
> plone.browserlayer = 2.1.3
> plone.dexterity = 2.2.4
> plone.keyring = 2.0.1
> plone.namedfile = 2.0.7
> plone.registry = 1.0.2
> plone.resource = 1.0.3
> plone.scale = 1.3.4
> plone.schemaeditor = 1.3.8
> plone.supermodel = 1.2.6
> plone.z3cform = 0.8.0
> python-gettext = 1.2
> python-openid = 2.2.5
> repoze.xmliter = 0.5
> transaction = 1.1.1
> z3c.form = 3.2.1
> zope.app.locales = 3.6.2
> zope.tales = 3.5.3
>
> All these versions are definitely pinned in the file we extend (or in files
> that this file extends):
> http://dist.plone.org/release/4.3-latest/versions.cfg
>
> So: strange behaviour.

I'm not sure how this relates to what you reported before, but this is
a known bug that I think I introduced on Sunday. I'm about to work on a fix.

Jim


-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From marius at pov.lt  Tue Dec 16 12:41:38 2014
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 16 Dec 2014 13:41:38 +0200
Subject: [Distutils] pip 6? (was Re:  change in setuptools 8.0)
In-Reply-To: <0A61790D-A73F-4582-9088-1755CF05209A@stufft.io>
References: <548EBFF1.5070500@chamonix.reportlab.co.uk>
 <ED99966B-13A1-4936-B8B0-D193C6D36CFD@stufft.io>
 <m6mps4$m0m$1@ger.gmane.org>
 <CB88BCBD-D4D2-418C-9337-71A0A8799756@stufft.io>
 <m6nn86$3s0$1@ger.gmane.org>
 <0A61790D-A73F-4582-9088-1755CF05209A@stufft.io>
Message-ID: <20141216114138.GA22133@fridge.pov.lt>

On Mon, Dec 15, 2014 at 05:47:09PM -0500, Donald Stufft wrote:
> Great. We're gonna hopefully release pip 6 this week. 

IIRC I saw it said somewhere that virtualenv's releases are tied to the
pip release schedule.  In that case I'd like to bring
https://github.com/pypa/virtualenv/issues/673 to your attention, because
it's a regression that breaks virtualenv -p and therefore tox.

(There are two alternative fixes, PR #674 and PR #672.)

Marius Gedminas
-- 
I spent a lot of time in the Java world. I'm so glad I'm on Planet Python
now :-)
        -- Steve Alexander
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141216/be4927b5/attachment.sig>

From jim at zope.com  Tue Dec 16 12:50:05 2014
From: jim at zope.com (Jim Fulton)
Date: Tue, 16 Dec 2014 06:50:05 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
Message-ID: <CAPDm-FgvcmyhxiTDx3AOPgt3bpH+5rM=hhvgxFriY1kFB_2Xpg@mail.gmail.com>

On Mon, Dec 15, 2014 at 6:48 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 15, 2014, at 6:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>
>> Maurits van Rees schreef op 15-12-14 23:50:
>>> TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning:
>>> 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440,
>>> version. You may find odd behavior and sort order. In particular it will
>>> be sorted as less than 0.0. It is recommend to migrate to PEP 440
>>> compatible versions.
>>
>> Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it.  This prints the warnings, because it sees for example a file
>> '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip'
>> This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.
>>
>> Ah, a demo with plain setuptools 0.8.4 is easy to setup:
>>
>> $ mkdir bar
>> $ touch bar/project-1.0.zip
>> $ . bin/activate
>> (venv) $ python
>> Python 2.7.8 (default, Jul 28 2014, 10:41:45)
>> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> from setuptools import package_index
>> >>> pi = package_index.PackageIndex()
>> >>> pi.add_find_links(['bar'])
>> /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.
>>  RuntimeWarning,
>> >>>
>
> Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam.

That's awesome, because with large download caches, there can be a lot
of spam, as in 2800 lines of spam for me. :)

I guess this is a good a time as any to clean up my cache.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From marius at pov.lt  Tue Dec 16 13:02:15 2014
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 16 Dec 2014 14:02:15 +0200
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
Message-ID: <20141216120215.GB22133@fridge.pov.lt>

On Mon, Dec 15, 2014 at 01:45:28PM -0500, Donald Stufft wrote:
> 
> > On Dec 15, 2014, at 1:27 PM, Jim Fulton <jim at zope.com> wrote:
> > I think the changes in version management in setuptools 8 are a
> > great step forward, but I think the transition is going to hurt a
> > lot.
> > 
> > For buildout, I'm thinking of of releasing 2.3.1 that reverts the
> > changes in 2.3 and adds a requirement for setuptools <8 to give more
> > time to respond to these  changes.
> 
> Is there something I?m not aware that is broken currently? I thought the
> transition was going pretty smoothly overall considering that a core piece
> of code inside of setuptools was touched.

~80 zope.* test builds are currently failing, for mysterious reasons due
to setuptools 8.0.x not interacting well with zc.buildout:
https://mail.zope.org/pipermail/zope-dev/2014-December/046509.html

Here's a summary of the various errors:
https://mail.zope.org/pipermail/zope-dev/2014-December/046508.html

Newer builds also added a bunch of warnings of the form
/var/lib/jenkins/jobs/zopetoolkit_trunk/workspace/lib/python2.7/site-packages/pkg_resources.py:2425:
RuntimeWarning: 'zc.recipe.testrunner-1.0.5 ()' is being parsed as a
legacy, non PEP 440, version. You may find odd behavior and sort order.
In particular it will be sorted as less than 0.0. It is recommend to
migrate to PEP 440 compatible versions.

Recent threads on distutils-sig@ help explain some of what's happening.
E.g. the 'zope.app.wsgi<3.11,<4.0dev,>=3.12' probably used to be
interpreted as an "<3.11 or >=3.12" and just needs to be hunted down and
replaced with "!=3.11.*", or something like that.

(It's painful when you get requirement conflict errors with no
indication about the source of those requirements.)

Marius Gedminas
-- 
The difference between Microsoft and 'Jurassic Parc':
In one, a mad businessman makes a lot of money with beasts that should be
extinct.
The other is a film.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141216/c877fb0b/attachment.sig>

From jim at zope.com  Tue Dec 16 14:13:35 2014
From: jim at zope.com (Jim Fulton)
Date: Tue, 16 Dec 2014 08:13:35 -0500
Subject: [Distutils] Buildout 2.3.1 released
Message-ID: <CAPDm-FiCO79XYGfrchai5Ti=+0yHsxvydfOXK6q92qnCCSz5xw@mail.gmail.com>

This:

- Fixes the picked-version bug

- Suppresses spurious (and not spurious) legacy version warnings.

  (When that setuptools/packaging problem is fixed. I'll release 2.3.2
   that undoes this.)

If we're still in significant pain tomorrow, then I'll release a 2.4
that requires setuptools <8.
This isn't a long-term, because old setuptools won't handle new-style
(better) versions.


Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From techtonik at gmail.com  Tue Dec 16 07:42:06 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 16 Dec 2014 09:42:06 +0300
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
Message-ID: <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>

On Tue, Dec 16, 2014 at 2:48 AM, Donald Stufft <donald at stufft.io> wrote:
>> On Dec 15, 2014, at 6:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>
>> Maurits van Rees schreef op 15-12-14 23:50:
>>> TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning:
>>> 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440,
>>> version. You may find odd behavior and sort order. In particular it will
>>> be sorted as less than 0.0. It is recommend to migrate to PEP 440
>>> compatible versions.
>>
>> Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it.  This prints the warnings, because it sees for example a file
>> '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip'
>> This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.
>>
>> Ah, a demo with plain setuptools 0.8.4 is easy to setup:
>>
>> $ mkdir bar
>> $ touch bar/project-1.0.zip
>> $ . bin/activate
>> (venv) $ python
>> Python 2.7.8 (default, Jul 28 2014, 10:41:45)
>> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> from setuptools import package_index
>> >>> pi = package_index.PackageIndex()
>> >>> pi.add_find_links(['bar'])
>> /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.
>>  RuntimeWarning,
>> >>>
>
> Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam.


I'd like to opt-out from PEP440 and use semantic versioning (that is
incompatible with PEP440) for my Python packages. I think
pip/setuptools/whatever should not stay in the way and give authors
ability to choose versioning strategy that is most suitable for their
workflow.

To achieve this:
1. pip/setuptools needs to be aware of different versioning strategies
2. package needs to convey which versioning strategy it uses
3. pip/setuptools needs a documented requirement for versions semantic
(what it understands about versions and how it uses them)

For example of the above - pip could document that it doesn't install
development and alpha versions unless explicitly requested with
command line switch, and instead of taking users to read gory details
of PEP440 just mention how package may tell pip that it is alpha or
development.

Previous vague attempt to request the same thing
https://github.com/pypa/pip/issues/1557 (just for the history)
-- 
anatoly t.

From techtonik at gmail.com  Tue Dec 16 08:24:16 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 16 Dec 2014 10:24:16 +0300
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
 <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
Message-ID: <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>

On Tue, Dec 16, 2014 at 9:45 AM, Donald Stufft <donald at stufft.io> wrote:
>> On Dec 16, 2014, at 1:42 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>>
>> On Tue, Dec 16, 2014 at 2:48 AM, Donald Stufft <donald at stufft.io> wrote:
>>>> On Dec 15, 2014, at 6:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>>>
>>>> Maurits van Rees schreef op 15-12-14 23:50:
>>>>> TMP2/setuptools-8.0.4-py2.7.egg/pkg_resources.py:2425: RuntimeWarning:
>>>>> 'zc.buildout-2.3.0 ()' is being parsed as a legacy, non PEP 440,
>>>>> version. You may find odd behavior and sort order. In particular it will
>>>>> be sorted as less than 0.0. It is recommend to migrate to PEP 440
>>>>> compatible versions.
>>>>
>>>> Actually, I have configured zc.buildout to use a download-cache directory where it saves downloaded zips/tarballs/eggs from PyPI. Buildout then adds this directory as a find-link and setuptools calls package_index.scan_url on it.  This prints the warnings, because it sees for example a file
>>>> '/Users/mauritsvanrees/cached-downloads/dist/Acquisition-2.13.8.zip'
>>>> This gets parsed as a Distribution with project_name 'Acquisition-2.13.8' and an empty version.
>>>>
>>>> Ah, a demo with plain setuptools 0.8.4 is easy to setup:
>>>>
>>>> $ mkdir bar
>>>> $ touch bar/project-1.0.zip
>>>> $ . bin/activate
>>>> (venv) $ python
>>>> Python 2.7.8 (default, Jul 28 2014, 10:41:45)
>>>> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>> from setuptools import package_index
>>>>>>> pi = package_index.PackageIndex()
>>>>>>> pi.add_find_links(['bar'])
>>>> /Users/mauritsvanrees/tmp/venv/lib/python2.7/site-packages/pkg_resources.py:2425: RuntimeWarning: 'project-1.0 ()' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.
>>>> RuntimeWarning,
>>>>>>>
>>>
>>> Ah, this is probably an issue with how I detected a legacy version and how setuptools parses a filename. I can probably get rid of the warning spam.
>>
>>
>> I'd like to opt-out from PEP440 and use semantic versioning (that is
>> incompatible with PEP440) for my Python packages. I think
>> pip/setuptools/whatever should not stay in the way and give authors
>> ability to choose versioning strategy that is most suitable for their
>> workflow.
>>
>> To achieve this:
>> 1. pip/setuptools needs to be aware of different versioning strategies
>> 2. package needs to convey which versioning strategy it uses
>> 3. pip/setuptools needs a documented requirement for versions semantic
>> (what it understands about versions and how it uses them)
>>
>> For example of the above - pip could document that it doesn't install
>> development and alpha versions unless explicitly requested with
>> command line switch, and instead of taking users to read gory details
>> of PEP440 just mention how package may tell pip that it is alpha or
>> development.
>>
>> Previous vague attempt to request the same thing
>> https://github.com/pypa/pip/issues/1557 (just for the history)
>> --
>> anatoly t.
>
> It?s not going to happen, it can?t happen reasonably. Luckily though it?s
> quite trivial for an author to adapt semver to work with PEP 440 syntax.

Or move to other distribution method than PyPI FWIW. If OpenStack
ecosystem could be able to provide an alternative, I'd gladly switch.
There are already too many packaging bicycles out there for dynamic
languages - Ruby, npm, PyPI - but in fact the only difference that
people need for server side of this stuff is distinctive design and
URL/subdomain.
-- 
anatoly t.

From fungi at yuggoth.org  Tue Dec 16 15:08:17 2014
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Tue, 16 Dec 2014 14:08:17 +0000
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <4652E410-7764-4A3C-997F-FA2F6C804567@stufft.io>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
 <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
 <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>
 <4652E410-7764-4A3C-997F-FA2F6C804567@stufft.io>
Message-ID: <20141216140817.GS2497@yuggoth.org>

On 2014-12-16 02:25:33 -0500 (-0500), Donald Stufft wrote:
> Openstack won?t be providing one, they are adapting server to PEP
> 440 syntax.
[...]

Confirmed, we're very close now and will likely be interoperating
with Setuptools 8 in our test infrastructure for all supported
branches later this week (depending on what direction we end up
going for embedded Git SHAs in unreleased dev sdist/wheel version
strings).
-- 
Jeremy Stanley

From m.van.rees at zestsoftware.nl  Tue Dec 16 15:57:33 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Tue, 16 Dec 2014 15:57:33 +0100
Subject: [Distutils] Buildout 2.3.1 released
In-Reply-To: <CAPDm-FiCO79XYGfrchai5Ti=+0yHsxvydfOXK6q92qnCCSz5xw@mail.gmail.com>
References: <CAPDm-FiCO79XYGfrchai5Ti=+0yHsxvydfOXK6q92qnCCSz5xw@mail.gmail.com>
Message-ID: <m6ph8t$if$1@ger.gmane.org>

Jim Fulton schreef op 16-12-14 14:13:
> This:
>
> - Fixes the picked-version bug
>
> - Suppresses spurious (and not spurious) legacy version warnings.
>
>    (When that setuptools/packaging problem is fixed. I'll release 2.3.2
>     that undoes this.)
>
> If we're still in significant pain tomorrow, then I'll release a 2.4
> that requires setuptools <8.
> This isn't a long-term, because old setuptools won't handle new-style
> (better) versions.

zc.buildout 2.3.1 seems good here, thanks.

For the record, the spurious warnings while bootstrapping are still 
there, but zc.buildout cannot fix it there.  Donald is working on that 
in setuptools/packaging.  During the buildout run the warnings are not 
there.

There is one thing that surprises me, but that may be nothing new.  If 
you pin older versions of zc.buildout or setuptools and have 
newest=false in your buildout (or call 'bin/buildout -N') then these 
older versions are not used.  Bootstrap gives you the most recent ones 
and the bin/buildout run will not upgrade/downgrade the versions.  The 
'_maybe_upgrade' call in buildout returns immediately.

Sample buildout config:

[buildout]
parts = test
# uncomment this as you see fit:
newest = false

[test]
recipe = zc.recipe.egg
eggs = zest.releaser

[versions]
setuptools = 7.0
zc.buildout = 2.2.5
zest.releaser = 3.53.2

Anyway, newest=true is the default in zc.buildout.

-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From techtonik at gmail.com  Tue Dec 16 16:11:12 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 16 Dec 2014 18:11:12 +0300
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <20141216140817.GS2497@yuggoth.org>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
 <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
 <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>
 <4652E410-7764-4A3C-997F-FA2F6C804567@stufft.io>
 <20141216140817.GS2497@yuggoth.org>
Message-ID: <CAPkN8xKx7p5c8pLLcxGoCaBvMAoqzR_hnCjXbar9_u8wPtWq8w@mail.gmail.com>

On Tue, Dec 16, 2014 at 5:08 PM, Jeremy Stanley <fungi at yuggoth.org> wrote:
> On 2014-12-16 02:25:33 -0500 (-0500), Donald Stufft wrote:
>> Openstack won?t be providing one, they are adapting server to PEP
>> 440 syntax.
> [...]
>
> Confirmed, we're very close now and will likely be interoperating
> with Setuptools 8 in our test infrastructure for all supported
> branches later this week (depending on what direction we end up
> going for embedded Git SHAs in unreleased dev sdist/wheel version
> strings).

It would be more useful if the powers behind OpenStack could sponsor
development of proper PEP scheme to untangle its own workflow instead
of relying on imperfect PEP solutions that are being done by
volunteers in their free time. I don't know where Donald finds his
time for all his contributions, but in my world it is unreal even
without full time job. Judging from activity in OpenStack community,
maybe it can teach PSF a thing or two.
-- 
anatoly t.

From fungi at yuggoth.org  Tue Dec 16 16:40:30 2014
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Tue, 16 Dec 2014 15:40:30 +0000
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <CAPkN8xKx7p5c8pLLcxGoCaBvMAoqzR_hnCjXbar9_u8wPtWq8w@mail.gmail.com>
References: <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
 <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
 <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>
 <4652E410-7764-4A3C-997F-FA2F6C804567@stufft.io>
 <20141216140817.GS2497@yuggoth.org>
 <CAPkN8xKx7p5c8pLLcxGoCaBvMAoqzR_hnCjXbar9_u8wPtWq8w@mail.gmail.com>
Message-ID: <20141216154030.GT2497@yuggoth.org>

On 2014-12-16 18:11:12 +0300 (+0300), anatoly techtonik wrote:
> It would be more useful if the powers behind OpenStack could sponsor
> development of proper PEP scheme to untangle its own workflow instead
> of relying on imperfect PEP solutions that are being done by
> volunteers in their free time. I don't know where Donald finds his
> time for all his contributions, but in my world it is unreal even
> without full time job. Judging from activity in OpenStack community,
> maybe it can teach PSF a thing or two.

The way I see it, we're all volunteers. We voluntarily work on free
software and associated support infrastructure. Some of us are
simply lucky enough to find sponsors willing to pay us to do it full
time (and while I won't speak for Donald I get the impression he
feels the same, whether he's working on PSF or OpenStack projects).

As for PEP 440, I believe it's a necessary step forward in
untangling the previously ad-hoc state of versions in the Python
packaging ecosystem. Many of OpenStack's versioning decisions were
made specifically with Python tools and PyPI in mind, in a desire to
be a good Python community citizen, so it has been our (from the
standpoint of the people writing the integration tooling for it)
intent to continue to follow standards set by that community.
-- 
Jeremy Stanley

From david.genest at ubisoft.com  Tue Dec 16 17:21:36 2014
From: david.genest at ubisoft.com (David Genest)
Date: Tue, 16 Dec 2014 16:21:36 +0000
Subject: [Distutils] Support for UNC paths on windows in pip 6?
Message-ID: <ea586310c166407cbfdb42d25be46734@MSR-MAIL-EXCH01.ubisoft.org>

Hi, 

Is there by any chance an opening to include UNC path support in pip 6?

I made a Pull request (https://github.com/pypa/pip/pull/1834) some time ago, which received positive attention from Steve Dower, but nobody else commented on the patch.

This is linked to (https://github.com/pypa/pip/issues/1846)

Thanks, 

D.



From p.f.moore at gmail.com  Tue Dec 16 17:45:01 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 16 Dec 2014 16:45:01 +0000
Subject: [Distutils] Support for UNC paths on windows in pip 6?
In-Reply-To: <ea586310c166407cbfdb42d25be46734@MSR-MAIL-EXCH01.ubisoft.org>
References: <ea586310c166407cbfdb42d25be46734@MSR-MAIL-EXCH01.ubisoft.org>
Message-ID: <CACac1F-0NTJ4jzh_AzG7ZVzj=hpqBkjeuYtJki_w=d8O5JNTHQ@mail.gmail.com>

On 16 December 2014 at 16:21, David Genest <david.genest at ubisoft.com> wrote:
> Hi,
>
> Is there by any chance an opening to include UNC path support in pip 6?
>
> I made a Pull request (https://github.com/pypa/pip/pull/1834) some time ago, which received positive attention from Steve Dower, but nobody else commented on the patch.
>
> This is linked to (https://github.com/pypa/pip/issues/1846)

I've added some comments to the PR.
Paul

From donald at stufft.io  Tue Dec 16 17:48:35 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 11:48:35 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <20141216154030.GT2497@yuggoth.org>
References: <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
 <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
 <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>
 <4652E410-7764-4A3C-997F-FA2F6C804567@stufft.io>
 <20141216140817.GS2497@yuggoth.org>
 <CAPkN8xKx7p5c8pLLcxGoCaBvMAoqzR_hnCjXbar9_u8wPtWq8w@mail.gmail.com>
 <20141216154030.GT2497@yuggoth.org>
Message-ID: <3A660023-5E2F-48CE-A955-61A592DA9AA2@stufft.io>


> On Dec 16, 2014, at 10:40 AM, Jeremy Stanley <fungi at yuggoth.org> wrote:
> 
> The way I see it, we're all volunteers. We voluntarily work on free
> software and associated support infrastructure. Some of us are
> simply lucky enough to find sponsors willing to pay us to do it full
> time (and while I won't speak for Donald I get the impression he
> feels the same, whether he's working on PSF or OpenStack projects).

Yea, I?m incredibly lucky to have found someone willing to pay me for my
hobby. Rackspace pays me to work on Python packaging as well as pays me
to work on Openstack.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From jim at zope.com  Tue Dec 16 18:53:34 2014
From: jim at zope.com (Jim Fulton)
Date: Tue, 16 Dec 2014 12:53:34 -0500
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <3A660023-5E2F-48CE-A955-61A592DA9AA2@stufft.io>
References: <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
 <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
 <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>
 <4652E410-7764-4A3C-997F-FA2F6C804567@stufft.io>
 <20141216140817.GS2497@yuggoth.org>
 <CAPkN8xKx7p5c8pLLcxGoCaBvMAoqzR_hnCjXbar9_u8wPtWq8w@mail.gmail.com>
 <20141216154030.GT2497@yuggoth.org>
 <3A660023-5E2F-48CE-A955-61A592DA9AA2@stufft.io>
Message-ID: <CAPDm-Fjm0UZCVK4w7pV4_v5TYQkXO8auE2FcY5K4qGfbdhcF0g@mail.gmail.com>

On Tue, Dec 16, 2014 at 11:48 AM, Donald Stufft <donald at stufft.io> wrote:
...
> Yea, I?m incredibly lucky to have found someone willing to pay me for my
> hobby. Rackspace pays me to work on Python packaging

Thanks Rackspace!

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From donald at stufft.io  Tue Dec 16 18:56:13 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 12:56:13 -0500
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release Candidates
Message-ID: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>

I?ve had a number of people complain that PEP 440 normalizes a release
candidate as 1.0c1 instead of 1.0rc1. Do we want to ammend the PEP to switch
the normalization around so that it normalizes 1.0c1 to 1.0rc1? This probably
matches what people expect better in general? It wouldn't have any effect on
sort order or the like, it would just change which form is preferred.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From donald at stufft.io  Tue Dec 16 19:14:38 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 13:14:38 -0500
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release
	Candidates
In-Reply-To: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
References: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
Message-ID: <3E469D40-6DC2-45E9-A524-7E44CF7E3CCC@stufft.io>


> On Dec 16, 2014, at 12:56 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> I?ve had a number of people complain that PEP 440 normalizes a release
> candidate as 1.0c1 instead of 1.0rc1. Do we want to ammend the PEP to switch
> the normalization around so that it normalizes 1.0c1 to 1.0rc1? This probably
> matches what people expect better in general? It wouldn't have any effect on
> sort order or the like, it would just change which form is preferred.
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Just so people know, here?s a PR to packaging that implements this
https://github.com/pypa/packaging/pull/19.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From p.f.moore at gmail.com  Tue Dec 16 20:51:35 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 16 Dec 2014 19:51:35 +0000
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release
	Candidates
In-Reply-To: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
References: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
Message-ID: <CACac1F9CZYCEpbxXrJuwEOV4EBLxdNqo+0a6yRcQyZ3vTLi1yQ@mail.gmail.com>

On 16 December 2014 at 17:56, Donald Stufft <donald at stufft.io> wrote:
> I?ve had a number of people complain that PEP 440 normalizes a release
> candidate as 1.0c1 instead of 1.0rc1. Do we want to ammend the PEP to switch
> the normalization around so that it normalizes 1.0c1 to 1.0rc1? This probably
> matches what people expect better in general? It wouldn't have any effect on
> sort order or the like, it would just change which form is preferred.

I'd typically expect to see 1.0rc1 rather than 1.0c1. I don't think
it's a crisis either way, but I would mildly prefer it if this change
was made.
Paul

From ncoghlan at gmail.com  Wed Dec 17 00:01:39 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Dec 2014 09:01:39 +1000
Subject: [Distutils] Changing the status of PEP 440 to Provisional
In-Reply-To: <CADiSq7dE=E-iN83fMgRj81VRB+ZzOoVF+=fWTsiRULD=fB1PSQ@mail.gmail.com>
References: <CADiSq7dE=E-iN83fMgRj81VRB+ZzOoVF+=fWTsiRULD=fB1PSQ@mail.gmail.com>
Message-ID: <CADiSq7cggt1BUwVDRCWX3FZy7HCFLd4abvKCBBWqg3VRyzYMOA@mail.gmail.com>

After an offline discussion with Donald regarding feedback on the
setuptools 8.0 release, I'm proposing we change the status of PEP 440 to be
Provisional (in the PEP 411 sense) until we sort out the additional issues
that were revealed through actual adoption.

I can't actually make that change to the PEP text itself yet (as PEP 1, and
hence the PEP index generator, don't currently include the notion of
provisional acceptance), but this is the basis on which we're proposing to
make changes to the previously accepted PEP text (rather than creating a
new PEP).

I'll also be proposing the relevant changes to PEP 1 and the index
generator to make this an official feature of the PEP process to python-dev.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141217/737a9dce/attachment-0001.html>

From donald at stufft.io  Wed Dec 17 00:42:52 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 18:42:52 -0500
Subject: [Distutils] Changing the status of PEP 440 to Provisional
In-Reply-To: <CADiSq7cggt1BUwVDRCWX3FZy7HCFLd4abvKCBBWqg3VRyzYMOA@mail.gmail.com>
References: <CADiSq7dE=E-iN83fMgRj81VRB+ZzOoVF+=fWTsiRULD=fB1PSQ@mail.gmail.com>
 <CADiSq7cggt1BUwVDRCWX3FZy7HCFLd4abvKCBBWqg3VRyzYMOA@mail.gmail.com>
Message-ID: <F1ED0F6D-493F-4380-BEE3-F6EA82D5A86E@stufft.io>


> On Dec 16, 2014, at 6:01 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> After an offline discussion with Donald regarding feedback on the setuptools 8.0 release, I'm proposing we change the status of PEP 440 to be Provisional (in the PEP 411 sense) until we sort out the additional issues that were revealed through actual adoption.
> 
> I can't actually make that change to the PEP text itself yet (as PEP 1, and hence the PEP index generator, don't currently include the notion of provisional acceptance), but this is the basis on which we're proposing to make changes to the previously accepted PEP text (rather than creating a new PEP).
> 
> I'll also be proposing the relevant changes to PEP 1 and the index generator to make this an official feature of the PEP process to python-dev.
> 
> Cheers,
> Nick.
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


Note that as of now Openstack has decided on a different path and currently does not need any adjustments to PEP 440 as I had talked about with you, so as of right now the only thing I really have to change (unless they change their mind or someone else comes forward) is that I?d like to move c -> rc in the preferred form. I?m not against making PEP 440 Provisional though and I have no idea if more issues will come up in the coming days.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141216/9098e4e3/attachment.html>

From m.van.rees at zestsoftware.nl  Wed Dec 17 00:53:08 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 17 Dec 2014 00:53:08 +0100
Subject: [Distutils] Local version identifiers from PEP 440 in practice
Message-ID: <m6qgl4$bd6$1@ger.gmane.org>

I wanted to try out local version identifiers for packages.  Or
actually, I already used them, but with a wrong notation, and ran into
trouble, also after fixing the notation.  So I did some testing and
made notes on what does or does not work.  See conclusions at the
bottom.


Local version identifiers: general idea
---------------------------------------

PEP440 is here, specifically the local version identifiers:
http://legacy.python.org/dev/peps/pep-0440/#local-version-identifiers

Use case in short, for those who do not know what local version
identifiers are.  Upstream community package largetrout has version
1.0.  As integrator you notice a problem, fix it in the code, and
create a pull request.  While waiting for a new official release you
already want to use the new code in production.  If you release a
version 1.1 yourself, this will conflict with a later official
version.  This is where local version identifiers come in.  You create
largetrout version 1.0+local.1 that you put in a directory on a
webserver.  You add the url of the package as find-link for
pip/zc.buildout and can now use your local version.

That is the general idea.  How does it work in practice?  Two things
need to work: creating a distribution, and installing the distribution
with pip or buildout.


Creating a distribution
-----------------------

I have created a very basic python project called 'myproject'.  It
does nothing.  I have released a few versions here:
http://pypi.zestsoftware.nl/public/packagingtest/

The various versions only differ in their version numbers and in the
way in which the distributions were created.  All were created with
Python 2.7.8. on Mac OSX 10.9.5, with ``python setup.py sdist 
--formats=zip``.

- setup.py version = 1.0
   * Created with: setuptools 2.2
   * Distribution file: myproject-1.0.zip
   * PKG-INFO version: 1.0
   * works everywhere

- setup.py version = 1.1+maurits.1
   * Created with: setuptools 2.2
   * Distribution file: myproject-1.1-maurits.1.zip
   * PKG-INFO version: 1.1-maurits.1
   * NOT installable with zc.buildout 2.3 and setuptools 8 (see below)

- setup.py version = 1.1+maurits.2
   * Created with: setuptools 7.0
   * Distribution file: myproject-1.1-maurits.2.zip
   * PKG-INFO version: 1.1-maurits.2
   * NOT installable with zc.buildout 2.3 and setuptools 8 (see below)

- setup.py version = 1.1+maurits.3
   * Created with: setuptools 8.0.4
   * Distribution file: myproject-1.1+maurits.3.zip
   * PKG-INFO version: 1.1+maurits.3
   * NOT installable with current pip (see below)
   * hard to install with development pip (BUG, see below)
   * NOT installable with zc.buildout 2.2 and setuptools<8

So: with any setuptools version you can specify a local version
identifier (``+something``) in the setup.py, but you need setuptools
8.0+ to correctly handle it.  In earlier versions, the plus sign is
replaced by a dash.  This may lead to problems while installing; we
will see that below.

Note: ``python2.7 setup.py --version`` always correctly returns the
version number as specified in setup.py.


Installing with pip
-------------------

Now let's see how installing goes.  Again we are using Python 2.7.

Below, some warnings or errors are expected (possibly with a
workaround, marked with TIP), some may be surprising
(marked with DANGER), others may be bugs (marked with BUG).

I picked two combinations of pip and setuptools versions.  Other 
combinations may give other results.


current pip, setuptools 7.0
---------------------------

A basic virtualenv:

$ pip list
pip (1.5.6)
setuptools (7.0)
wsgiref (0.1.2)
$ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ 
myproject==1.0
...
$ python -c "import myproject" && echo "importing works"
importing works

We try ``pip install`` for the available versions.

- myproject==1.0
   * importing works

- myproject=1.1-maurits.1
   * importing works

- myproject==1.1-maurits.2
   * importing works

- myproject==1.1-maurits.3
   * pip says:
     Could not find a version that satisfies the requirement
     myproject==1.1-maurits.3 (from versions: 1.0, 1.1, 1.1-maurits.1,
     1.1-maurits.2)

- myproject==1.1+maurits.3
   * pip says:
     Exception:
     Traceback (most recent call last):
       ...
       File ".../pip/_vendor/pkg_resources.py", line 2583, in scan_list
         "Expected ',' or end-of-list in",line,"at",line[p:]
     ValueError: ("Expected ',' or end-of-list in", 
'myproject==1.1+maurits.3', 'at', '+maurits.3')
   * DANGER: older pip versions cannot install packages with local 
version identifiers.

- pip install 
http://pypi.zestsoftware.nl/public/packagingtest/myproject-1.1+maurits.3.zip
   * importing works


development pip, setuptools 8.0
-------------------------------

A basic virtualenv:

$ pip list
pip (6.0.dev1)
setuptools (8.0.4)
$ pip install -U --trusted-host pypi.zestsoftware.nl -f 
http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.0
...
$ python -c "import myproject" && echo "importing works"
importing works

We try ``pip install`` for the available versions.

- myproject==1.0
   * importing works

- myproject==1.1-maurits.1
   * pip warns:
     Requested myproject==1.1-maurits.1, but installing version 
1.1+maurits.1
   * importing works

- myproject==1.1-maurits.2
   * pip warns:
     Requested myproject==1.1-maurits.2, but installing version 
1.1+maurits.2
   * importing works

- myproject==1.1-maurits.3
   * pip says:
     Collecting myproject==1.1+maurits.3
       Could not find a version that satisfies the requirement 
myproject==1.1-maurits.3 (from versions: )
       No distributions matching the version for myproject==1.1-maurits.3

- myproject==1.1+maurits.3
   * pip says:
     Collecting myproject==1.1+maurits.3
       Could not find a version that satisfies the requirement 
myproject==1.1+maurits.3 (from versions: )
       No distributions matching the version for myproject==1.1+maurits.3
   * BUG: this should work, right?

- pip install 
http://pypi.zestsoftware.nl/public/packagingtest/myproject-1.1+maurits.3.zip
   * importing works


Installing with buildout
------------------------

We try the same with buildout.  I use this basic buildout config:

[buildout]
find-links = http://pypi.zestsoftware.nl/public/packagingtest/
eggs-directory = eggs
download-cache = downloads
parts = mypy test
show-picked-versions = true

[mypy]
recipe = zc.recipe.egg
eggs = myproject
interpreter = mypy

[test]
recipe = plone.recipe.command
command = grep myproject bin/mypy && bin/mypy -c "import myproject" && 
echo "importing works"
update-command = ${:command}

[versions]
myproject = 1.0
plone.recipe.command = 1.1
#setuptools = ...
#zc.buildout = ...
zc.recipe.egg = 2.0.1


This buildout creates a bin/mypy python interpreter with the myproject
package installed.  And it greps bin/mypy for 'myproject' so I can
easily see whether this has the version I expect.  And it checks to
see it importing the module works.


Using previous zc.buildout and setuptools
-----------------------------------------

zc.buildout = 2.2.5
setuptools = 7.0

- myproject = 1.0
   * all is well

- myproject = 1.1-maurits.1
   * all is well

- myproject = 1.1-maurits.2
   * all is well

- myproject = 1.1-maurits.3
   * all is well

- myproject = 1.1+maurits.3
   * buildout fails:
     Traceback (most recent call last):
       ...
       File ".../setuptools-7.0-py2.7.egg/pkg_resources.py", line 2690, 
in scan_list
         raise ValueError(msg, line, "at", line[p:])
     ValueError: ("Expected ',' or end-of-list in", 
'myproject[]==1.1+maurits.3', 'at', '+maurits.3')
   * TIP: you can use a dash instead of a plus in the version pin.


Using latest zc.buildout and setuptools
---------------------------------------

zc.buildout = 2.3.1
setuptools = 8.0.2

- myproject = 1.0
   * all is well

- myproject = 1.1-maurits.1
   * buildout fails:
     Installing myproject 1.1-maurits.1
     Caused installation of a distribution:
     myproject 1.1+maurits.1
     with a different version.
     Got None.
     While:
       Updating mypy.
     Error: There is a version conflict.
     We already have: myproject 1.1+maurits.1
     We require myproject==1.1-maurits.1

   * when I do not specify a version in the buildout config, but the
     latest on server is 1.1-maurits.1, buildout succeeds and importing
     works.  buildout reports this (note the plus-sign):

     Versions had to be automatically picked.
     The following part definition lists the versions picked:
     [versions]
     myproject = 1.1+maurits.1

   * BUG?  pip only warns, buildout may want to do the same.  Maybe
     somehow normalize versions before comparison.

- myproject = 1.1-maurits.2
   * buildout fails:
     Installing myproject 1.1-maurits.2
     Caused installation of a distribution:
     myproject 1.1+maurits.2
     with a different version.
     ...

   * So: same remarks as for 1.1-maurits.1 apply.

- myproject = 1.1-maurits.3
   * buildout fails:
     Error: Couldn't find a distribution for 'myproject==1.1-maurits.3'.

- myproject = 1.1+maurits.3
   * importing works


Conclusions
-----------

- BUG in pip: development version of pip cannot find local version
   identifiers (version numbers with a plus sign).

- TIP: do NOT use setuptools 7.0 or older to create a distribution
   with a local version identifier.  Installation of this may fail,
   because the version according to the filename differs from the
   actual version.

- BUG in buildout?  Where pip only warns about non-matching versions
   from the above tip, buildout actually quits with an error.  Maybe
   somehow normalize versions before comparison.

- TIP: only use local version identifiers when you use setuptools 8 or
   higher both when creating and when using the distribution.  Then
   installing with zc.buildout will work.  Installing with pip can be
   done by pointing to the exact file url.



-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From p.f.moore at gmail.com  Wed Dec 17 01:00:40 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 17 Dec 2014 00:00:40 +0000
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <m6qgl4$bd6$1@ger.gmane.org>
References: <m6qgl4$bd6$1@ger.gmane.org>
Message-ID: <CACac1F8sc_as6X4rToqeV05AcfohoK4BW8Lp_y-HATvrY-3XMA@mail.gmail.com>

On 16 December 2014 at 23:53, Maurits van Rees
<m.van.rees at zestsoftware.nl> wrote:
> BUG in pip: development version of pip cannot find local version
> identifiers (version numbers with a plus sign).

Have you logged that bug with pip? If not, could you? It sounds like
it's something that should probably be a release blocker - it'd be a
shame if pip 6.0 came out and couldn't handle PEP 440 local versions.

Thanks,
Paul

From donald at stufft.io  Wed Dec 17 01:05:59 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 19:05:59 -0500
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <m6qgl4$bd6$1@ger.gmane.org>
References: <m6qgl4$bd6$1@ger.gmane.org>
Message-ID: <FA919B66-3C5A-436E-AD6F-E5B43A0DB4AE@stufft.io>


> On Dec 16, 2014, at 6:53 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
> 
> <words>

Yea, it?s unfortunate that local versions don?t work prior to setuptools 8.0, but the
older versions more or less escape everything that?s not alpha numeric and . and - to
the - character so there?s only so much we can do with that.

It?s correct that existing versions of pip do not support any new syntax with PEP 440,
that will be part of pip 6.0 which I?m hoping to release this week. We were waiting on
setuptools 8 to release with PEP 440 support and now we?re just making sure that setuptools
is calmed down before releasing a pip 6.0.

I agree with Paul, please file a bug for the local versions not working, they _should_
work and it should be a release blocker for 6.0.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From m.van.rees at zestsoftware.nl  Wed Dec 17 01:17:21 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 17 Dec 2014 01:17:21 +0100
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <FA919B66-3C5A-436E-AD6F-E5B43A0DB4AE@stufft.io>
References: <m6qgl4$bd6$1@ger.gmane.org>
 <FA919B66-3C5A-436E-AD6F-E5B43A0DB4AE@stufft.io>
Message-ID: <m6qi2i$u5r$1@ger.gmane.org>

Donald Stufft schreef op 17-12-14 01:05:
>
>> On Dec 16, 2014, at 6:53 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>
>> <words>
>
> Yea, it?s unfortunate that local versions don?t work prior to setuptools 8.0, but the
> older versions more or less escape everything that?s not alpha numeric and . and - to
> the - character so there?s only so much we can do with that.
>
> It?s correct that existing versions of pip do not support any new syntax with PEP 440,
> that will be part of pip 6.0 which I?m hoping to release this week. We were waiting on
> setuptools 8 to release with PEP 440 support and now we?re just making sure that setuptools
> is calmed down before releasing a pip 6.0.
>
> I agree with Paul, please file a bug for the local versions not working, they _should_
> work and it should be a release blocker for 6.0.

Done:

https://github.com/pypa/pip/issues/2203


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From fungi at yuggoth.org  Wed Dec 17 01:36:46 2014
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Wed, 17 Dec 2014 00:36:46 +0000
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <m6qgl4$bd6$1@ger.gmane.org>
References: <m6qgl4$bd6$1@ger.gmane.org>
Message-ID: <20141217003646.GA2497@yuggoth.org>

On 2014-12-17 00:53:08 +0100 (+0100), Maurits van Rees wrote:
[...]
>       File ".../pip/_vendor/pkg_resources.py", line 2583, in scan_list
>         "Expected ',' or end-of-list in",line,"at",line[p:]
>     ValueError: ("Expected ',' or end-of-list in",
> 'myproject==1.1+maurits.3', 'at', '+maurits.3')
[...]

This was basically why OpenStack/PBR opted not to switch to using an
LVI to encapsulate Git SHA details in unreleased dev version
strings (instead we've now moved to putting them into egg-info and
providing a separate tool to query that).
-- 
Jeremy Stanley

From m.van.rees at zestsoftware.nl  Wed Dec 17 01:46:58 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 17 Dec 2014 01:46:58 +0100
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <m6qgl4$bd6$1@ger.gmane.org>
References: <m6qgl4$bd6$1@ger.gmane.org>
Message-ID: <m6qjq2$l3c$1@ger.gmane.org>

Maurits van Rees schreef op 17-12-14 00:53:
> I have created a very basic python project called 'myproject'.  It
> does nothing.  I have released a few versions here:
> http://pypi.zestsoftware.nl/public/packagingtest/

I have now also distributed myproject version 1.1.  (This has a
base.jinja2 file and requires Jinja2[i18n], which I need for checking
a corner case; I may report that later).

Installing 1.1 unexpectedly gives problems, both with pip and
zc.buildout.

- zc.buildout 2.2.5 and setuptools 7.0: all is well.

- zc.buildout 2.3.1 and setuptools 8.0.4:

   * When I specify 'myproject = 1.1' in the buildout config, myproject
     does not get updated.  It sticks at version 1.1+maurits.3.

   * In fact, when I switch back to 1.0, run bin/buildout, again switch
     to 1.1, run bin/buildout, the result is that 1.1+maurits.3 is used.

- pip 1.5.6, setuptools 7.0:

   * pip warns about three different versions:
     $ pip install -U -f 
http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
     Downloading/unpacking myproject==1.1
       Downloading myproject-1.1+maurits.3.zip
       Running setup.py 
(path:/Users/mauritsvanrees/tmp/venv-older/build/myproject/setup.py) 
egg_info for package myproject
       Requested myproject==1.1, but installing version 1.1-maurits.3
   * The warning is correct: not 1.1, but 1.1-maurits.3 is installed.

- pip dev, setuptools 8.0.4:

   * With currently 1.1+maurits.3 installed, pip says:
     $ pip install -U --trusted-host pypi.zestsoftware.nl -f 
http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
     Requirement already up-to-date: myproject==1.1 in 
./lib/python2.7/site-packages

   * With 1.0 currently installed, pip says:
     $ pip install -U --trusted-host pypi.zestsoftware.nl -f 
http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
     Collecting myproject==1.1
       Downloading myproject-1.1+maurits.3.zip

    * Result is indeed that myproject 1.1+maurits.3 is installed
      instead of 1.1.

Given that both zc.buildout and pip have a problem, I am guessing that
there is a bug in setuptools.  I'll report it.


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From donald at stufft.io  Wed Dec 17 01:54:40 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 19:54:40 -0500
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <m6qjq2$l3c$1@ger.gmane.org>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
Message-ID: <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io>


> On Dec 16, 2014, at 7:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
> 
> Maurits van Rees schreef op 17-12-14 00:53:
>> I have created a very basic python project called 'myproject'.  It
>> does nothing.  I have released a few versions here:
>> http://pypi.zestsoftware.nl/public/packagingtest/
> 
> I have now also distributed myproject version 1.1.  (This has a
> base.jinja2 file and requires Jinja2[i18n], which I need for checking
> a corner case; I may report that later).
> 
> Installing 1.1 unexpectedly gives problems, both with pip and
> zc.buildout.
> 
> - zc.buildout 2.2.5 and setuptools 7.0: all is well.
> 
> - zc.buildout 2.3.1 and setuptools 8.0.4:
> 
>  * When I specify 'myproject = 1.1' in the buildout config, myproject
>    does not get updated.  It sticks at version 1.1+maurits.3.

This isn?t a bug, assuming that buildout is translating ``my project = 1.1``
to something like ==1.1. Local versions are used to indicate something that
is compatible with 1.1 and they sort as newer than the same version without
the local version. The reason they match for a ``==1.1`` is because you want
downstream distributors like Debian to be able to set their versions to 1.0+debian.1
without breaking things for people who do ``==1.1``.

> 
>  * In fact, when I switch back to 1.0, run bin/buildout, again switch
>    to 1.1, run bin/buildout, the result is that 1.1+maurits.3 is used.

See above.

> 
> - pip 1.5.6, setuptools 7.0:
> 
>  * pip warns about three different versions:
>    $ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
>    Downloading/unpacking myproject==1.1
>      Downloading myproject-1.1+maurits.3.zip
>      Running setup.py (path:/Users/mauritsvanrees/tmp/venv-older/build/myproject/setup.py) egg_info for package myproject
>      Requested myproject==1.1, but installing version 1.1-maurits.3
>  * The warning is correct: not 1.1, but 1.1-maurits.3 is installed.

That warning might actually be from a stale build directory laying around and not related to the local version at all.

> 
> - pip dev, setuptools 8.0.4:
> 
>  * With currently 1.1+maurits.3 installed, pip says:
>    $ pip install -U --trusted-host pypi.zestsoftware.nl -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
>    Requirement already up-to-date: myproject==1.1 in ./lib/python2.7/site-packages
> 
>  * With 1.0 currently installed, pip says:
>    $ pip install -U --trusted-host pypi.zestsoftware.nl -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
>    Collecting myproject==1.1
>      Downloading myproject-1.1+maurits.3.zip
> 
>   * Result is indeed that myproject 1.1+maurits.3 is installed
>     instead of 1.1.

This behavior is also correct as I mentioned above.

> 
> Given that both zc.buildout and pip have a problem, I am guessing that
> there is a bug in setuptools.  I'll report it.
> 
> 
> -- 
> Maurits van Rees: http://maurits.vanrees.org/
> Zest Software: http://zestsoftware.nl
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From m.van.rees at zestsoftware.nl  Wed Dec 17 02:33:55 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 17 Dec 2014 02:33:55 +0100
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io>
Message-ID: <m6qmi3$tf2$1@ger.gmane.org>

Donald Stufft schreef op 17-12-14 01:54:
>
>> On Dec 16, 2014, at 7:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>
>> Maurits van Rees schreef op 17-12-14 00:53:
>>> I have created a very basic python project called 'myproject'.  It
>>> does nothing.  I have released a few versions here:
>>> http://pypi.zestsoftware.nl/public/packagingtest/
>>
>> I have now also distributed myproject version 1.1.  (This has a
>> base.jinja2 file and requires Jinja2[i18n], which I need for checking
>> a corner case; I may report that later).
>>
>> Installing 1.1 unexpectedly gives problems, both with pip and
>> zc.buildout.
>>
>> - zc.buildout 2.2.5 and setuptools 7.0: all is well.
>>
>> - zc.buildout 2.3.1 and setuptools 8.0.4:
>>
>>   * When I specify 'myproject = 1.1' in the buildout config, myproject
>>     does not get updated.  It sticks at version 1.1+maurits.3.
>
> This isn?t a bug, assuming that buildout is translating ``my project = 1.1``
> to something like ==1.1.

That is indeed what buildout does.  Note that buildout config files are 
basically parsed by ConfigParser, so '==' is not allowed there.  That is 
why buildout uses a different notation here.

> Local versions are used to indicate something that
> is compatible with 1.1 and they sort as newer than the same version without
> the local version. The reason they match for a ``==1.1`` is because you want
> downstream distributors like Debian to be able to set their versions to 1.0+debian.1
> without breaking things for people who do ``==1.1``.

That is most definitely not what I want.  If I tell setuptools or pip or 
buildout that I want version X, then I don't want roughly version X, but 
I want exactly version X.

When I request 1.0rc1 and I get 1.0, that is wrong.
When I request 1.0 and I get 1.0rc1, that is wrong.

To me, the same is true for local version identifiers.

To me, this is the basis for repeatable deployments, knowing that given 
a buildout file or a pip requirements file you get the same versions and 
the same code that you got when you tried it a month earlier.  It avoids 
"it works on my machine" when I get an error that a colleague does not get.


For clarity, how I use internal releases, so how I would want to use 
local version identifiers, is:

1. Use largetrout 1.0 in a project.

2. Notice a bug.

3. See that the bug is already fixed in development, but there is no 
release yet.

4. Checkout largetrout dev, setup.py version is 1.1.dev0.

5. Set version to 1.1+internal1 (an internal release of a snapshot of 
the code that is expected to end up in 1.1).

6. Make an internal release and use it.

7. Five more bugs get fixed in largetrout.

8. largetrout 1.1 is released.

Now when I tell pip or buildout to use largetrout 1.1 I definitely want 
those five bug fixes from step 7 included.

>> - pip 1.5.6, setuptools 7.0:
>>
>>   * pip warns about three different versions:
>>     $ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
>>     Downloading/unpacking myproject==1.1
>>       Downloading myproject-1.1+maurits.3.zip
>>       Running setup.py (path:/Users/mauritsvanrees/tmp/venv-older/build/myproject/setup.py) egg_info for package myproject
>>       Requested myproject==1.1, but installing version 1.1-maurits.3
>>   * The warning is correct: not 1.1, but 1.1-maurits.3 is installed.
>
> That warning might actually be from a stale build directory laying around and not related to the local version at all.

Let me check.  Nope, the same happens with a fresh virtualenv:

mauritsvanrees at procyon:tmp $ virtualenv-2.7 venv-older
Using real prefix '/Users/mauritsvanrees/buildout/python2.7/parts/opt'
New python executable in venv-older/bin/python2.7
Also creating executable in venv-older/bin/python
Installing setuptools, pip...done.
mauritsvanrees at procyon:tmp $ cd venv-older
mauritsvanrees at procyon:venv-older $ . bin/activate
(venv-older)mauritsvanrees at procyon:venv-older $ pip install -U 
setuptools==7.0  # this is optional
Downloading/unpacking setuptools==7.0
   Downloading setuptools-7.0-py2.py3-none-any.whl (534kB): 534kB downloaded
Installing collected packages: setuptools
   Found existing installation: setuptools 3.6
     Uninstalling setuptools:
       Successfully uninstalled setuptools
Successfully installed setuptools
Cleaning up...
(venv-older)mauritsvanrees at procyon:venv-older $ pip list
pip (1.5.6)
setuptools (7.0)
wsgiref (0.1.2)
(venv-older)mauritsvanrees at procyon:venv-older $ pip install -U -f 
http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
Downloading/unpacking myproject==1.1
   http://pypi.zestsoftware.nl/public/packagingtest/ uses an insecure 
transport scheme (http). Consider using https if pypi.zestsoftware.nl 
has it available
   Downloading myproject-1.1+maurits.3.zip
   Running setup.py 
(path:/Users/mauritsvanrees/tmp/venv-older/build/myproject/setup.py) 
egg_info for package myproject

     warning: no files found matching '*rst'
     warning: no previously-included files matching '*.pyc' found 
anywhere in distribution
   Requested myproject==1.1, but installing version 1.1-maurits.3
Installing collected packages: myproject
   Running setup.py install for myproject

     warning: no files found matching '*rst'
     warning: no previously-included files matching '*.pyc' found 
anywhere in distribution
Successfully installed myproject
Cleaning up...
(venv-older)mauritsvanrees at procyon:venv-older $ ls -1 
lib/python2.7/site-packages/
_markerlib
easy_install.py
easy_install.pyc
myproject
myproject-1.1_maurits.3-py2.7.egg-info
pip
pip-1.5.6.dist-info
pkg_resources.py
pkg_resources.pyc
setuptools
setuptools-7.0.dist-info


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From donald at stufft.io  Wed Dec 17 02:49:24 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 20:49:24 -0500
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <m6qmi3$tf2$1@ger.gmane.org>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io> <m6qmi3$tf2$1@ger.gmane.org>
Message-ID: <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>


> On Dec 16, 2014, at 8:33 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
> 
> Donald Stufft schreef op 17-12-14 01:54:
>> 
>>> On Dec 16, 2014, at 7:46 PM, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
>>> 
>>> Maurits van Rees schreef op 17-12-14 00:53:
>>>> I have created a very basic python project called 'myproject'.  It
>>>> does nothing.  I have released a few versions here:
>>>> http://pypi.zestsoftware.nl/public/packagingtest/
>>> 
>>> I have now also distributed myproject version 1.1.  (This has a
>>> base.jinja2 file and requires Jinja2[i18n], which I need for checking
>>> a corner case; I may report that later).
>>> 
>>> Installing 1.1 unexpectedly gives problems, both with pip and
>>> zc.buildout.
>>> 
>>> - zc.buildout 2.2.5 and setuptools 7.0: all is well.
>>> 
>>> - zc.buildout 2.3.1 and setuptools 8.0.4:
>>> 
>>>  * When I specify 'myproject = 1.1' in the buildout config, myproject
>>>    does not get updated.  It sticks at version 1.1+maurits.3.
>> 
>> This isn?t a bug, assuming that buildout is translating ``my project = 1.1``
>> to something like ==1.1.
> 
> That is indeed what buildout does.  Note that buildout config files are basically parsed by ConfigParser, so '==' is not allowed there.  That is why buildout uses a different notation here.
> 
>> Local versions are used to indicate something that
>> is compatible with 1.1 and they sort as newer than the same version without
>> the local version. The reason they match for a ``==1.1`` is because you want
>> downstream distributors like Debian to be able to set their versions to 1.0+debian.1
>> without breaking things for people who do ``==1.1``.
> 
> That is most definitely not what I want.  If I tell setuptools or pip or buildout that I want version X, then I don't want roughly version X, but I want exactly version X.
> 
> When I request 1.0rc1 and I get 1.0, that is wrong.
> When I request 1.0 and I get 1.0rc1, that is wrong.
> 
> To me, the same is true for local version identifiers.
> 
> To me, this is the basis for repeatable deployments, knowing that given a buildout file or a pip requirements file you get the same versions and the same code that you got when you tried it a month earlier.  It avoids "it works on my machine" when I get an error that a colleague does not get.
> 
> 
> For clarity, how I use internal releases, so how I would want to use local version identifiers, is:
> 
> 1. Use largetrout 1.0 in a project.
> 
> 2. Notice a bug.
> 
> 3. See that the bug is already fixed in development, but there is no release yet.
> 
> 4. Checkout largetrout dev, setup.py version is 1.1.dev0.
> 
> 5. Set version to 1.1+internal1 (an internal release of a snapshot of the code that is expected to end up in 1.1).
> 
> 6. Make an internal release and use it.
> 
> 7. Five more bugs get fixed in largetrout.
> 
> 8. largetrout 1.1 is released.
> 
> Now when I tell pip or buildout to use largetrout 1.1 I definitely want those five bug fixes from step 7 included.

So the *primary* use case that motivated local versions is things like when Debian patches a copy of a project they can indicate that they?ve done so by changing the version to 1.0+dfsg1 or so instead of 1.0. A related use case is the one you?re talking about. However the primary use case is what influenced the fact that ==1.1 matches ==1.0+foobar.

Important to note, is that ==1.0+foobar will only install that patched version, not any 1.0 version. You can also approximate the kind of pinning you want with the === (which is really the arbitrary equality indicator, which is generally used for people who want to install a version like ?dog? which we can?t parse). It?s possible that we could add some sort of a ?None? indicator for local versions that says ?1.0 and exactly 1.0? though I?m not sure how off the top of my head (Maybe ==1.0+).

Another thing though, what you probably want to do is something like 1.1.dev0+internal1, which will sort as older than 1.1 whenever it is actually released.

> 
>>> - pip 1.5.6, setuptools 7.0:
>>> 
>>>  * pip warns about three different versions:
>>>    $ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
>>>    Downloading/unpacking myproject==1.1
>>>      Downloading myproject-1.1+maurits.3.zip
>>>      Running setup.py (path:/Users/mauritsvanrees/tmp/venv-older/build/myproject/setup.py) egg_info for package myproject
>>>      Requested myproject==1.1, but installing version 1.1-maurits.3
>>>  * The warning is correct: not 1.1, but 1.1-maurits.3 is installed.
>> 
>> That warning might actually be from a stale build directory laying around and not related to the local version at all.
> 
> Let me check.  Nope, the same happens with a fresh virtualenv:
> 
> mauritsvanrees at procyon:tmp $ virtualenv-2.7 venv-older
> Using real prefix '/Users/mauritsvanrees/buildout/python2.7/parts/opt'
> New python executable in venv-older/bin/python2.7
> Also creating executable in venv-older/bin/python
> Installing setuptools, pip...done.
> mauritsvanrees at procyon:tmp $ cd venv-older
> mauritsvanrees at procyon:venv-older $ . bin/activate
> (venv-older)mauritsvanrees at procyon:venv-older $ pip install -U setuptools==7.0  # this is optional
> Downloading/unpacking setuptools==7.0
>  Downloading setuptools-7.0-py2.py3-none-any.whl (534kB): 534kB downloaded
> Installing collected packages: setuptools
>  Found existing installation: setuptools 3.6
>    Uninstalling setuptools:
>      Successfully uninstalled setuptools
> Successfully installed setuptools
> Cleaning up...
> (venv-older)mauritsvanrees at procyon:venv-older $ pip list
> pip (1.5.6)
> setuptools (7.0)
> wsgiref (0.1.2)
> (venv-older)mauritsvanrees at procyon:venv-older $ pip install -U -f http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.1
> Downloading/unpacking myproject==1.1
>  http://pypi.zestsoftware.nl/public/packagingtest/ uses an insecure transport scheme (http). Consider using https if pypi.zestsoftware.nl has it available
>  Downloading myproject-1.1+maurits.3.zip
>  Running setup.py (path:/Users/mauritsvanrees/tmp/venv-older/build/myproject/setup.py) egg_info for package myproject
> 
>    warning: no files found matching '*rst'
>    warning: no previously-included files matching '*.pyc' found anywhere in distribution
>  Requested myproject==1.1, but installing version 1.1-maurits.3
> Installing collected packages: myproject
>  Running setup.py install for myproject
> 
>    warning: no files found matching '*rst'
>    warning: no previously-included files matching '*.pyc' found anywhere in distribution
> Successfully installed myproject
> Cleaning up...
> (venv-older)mauritsvanrees at procyon:venv-older $ ls -1 lib/python2.7/site-packages/
> _markerlib
> easy_install.py
> easy_install.pyc
> myproject
> myproject-1.1_maurits.3-py2.7.egg-info
> pip
> pip-1.5.6.dist-info
> pkg_resources.py
> pkg_resources.pyc
> setuptools
> setuptools-7.0.dist-info
> 
> 
> -- 
> Maurits van Rees: http://maurits.vanrees.org/
> Zest Software: http://zestsoftware.nl
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From ethan at stoneleaf.us  Wed Dec 17 03:10:43 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 16 Dec 2014 18:10:43 -0800
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io> <m6qmi3$tf2$1@ger.gmane.org>
 <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
Message-ID: <5490E623.9050100@stoneleaf.us>

On 12/16/2014 05:49 PM, Donald Stufft wrote:
> 
> So the *primary* use case that motivated local versions is things like when Debian
> patches a copy of a project they can indicate that they?ve done so by changing the
> version to 1.0+dfsg1 or so instead of 1.0. A related use case is the one you?re
> talking about. However the primary use case is what influenced the fact that ==1.1
> matches ==1.0+foobar.
> 
> Important to note, is that ==1.0+foobar will only install that patched version,
> not any 1.0 version. You can also approximate the kind of pinning you want with
> the === (which is really the arbitrary equality indicator, which is generally used
> for people who want to install a version like ?dog? which we can?t parse). It?s
> possible that we could add some sort of a ?None? indicator for local versions that
> says ?1.0 and exactly 1.0? though I?m not sure how off the top of my head (Maybe
> ==1.0+).

Let me see if I understand correctly:

I release my dbf package as 1.3.

Somebody at Debian fixes a bug, and rather than wait for me to release the next version just slap a local tag on it --
so now there is:

  - 1.3  (mine)
  - 1.3+debian1

At this point, a non-debian user asking for 1.3 will get mine, whereas a debian user would get the 1.3+debian1 version.
 Correct?  Plus, a debian user asking for 1.4 would also get 1.3+debian1?

Now, I fix that bug plus a few more, and make a 1.4 release.  A non-debian user would get my 1.4 release.  Which version
will a debian user get?

  - 1.3+debian1 ?
  - 1.4 ?

If my 1.4 does not sort higher than a 1.3+blahblah anything, that's not good.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141216/631d1660/attachment.sig>

From donald at stufft.io  Wed Dec 17 03:48:18 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 21:48:18 -0500
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <5490E623.9050100@stoneleaf.us>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io> <m6qmi3$tf2$1@ger.gmane.org>
 <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
 <5490E623.9050100@stoneleaf.us>
Message-ID: <7806B51F-9DCA-43FC-A47B-47F4A4E6EDEC@stufft.io>


> On Dec 16, 2014, at 9:10 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> 
> On 12/16/2014 05:49 PM, Donald Stufft wrote:
>> 
>> So the *primary* use case that motivated local versions is things like when Debian
>> patches a copy of a project they can indicate that they?ve done so by changing the
>> version to 1.0+dfsg1 or so instead of 1.0. A related use case is the one you?re
>> talking about. However the primary use case is what influenced the fact that ==1.1
>> matches ==1.0+foobar.
>> 
>> Important to note, is that ==1.0+foobar will only install that patched version,
>> not any 1.0 version. You can also approximate the kind of pinning you want with
>> the === (which is really the arbitrary equality indicator, which is generally used
>> for people who want to install a version like ?dog? which we can?t parse). It?s
>> possible that we could add some sort of a ?None? indicator for local versions that
>> says ?1.0 and exactly 1.0? though I?m not sure how off the top of my head (Maybe
>> ==1.0+).
> 
> Let me see if I understand correctly:
> 
> I release my dbf package as 1.3.
> 
> Somebody at Debian fixes a bug, and rather than wait for me to release the next version just slap a local tag on it --
> so now there is:
> 
>  - 1.3  (mine)
>  - 1.3+debian1
> 
> At this point, a non-debian user asking for 1.3 will get mine, whereas a debian user would get the 1.3+debian1 version.
> Correct?  Plus, a debian user asking for 1.4 would also get 1.3+debian1?
> 
> Now, I fix that bug plus a few more, and make a 1.4 release.  A non-debian user would get my 1.4 release.  Which version
> will a debian user get?
> 
>  - 1.3+debian1 ?
>  - 1.4 ?
> 
> If my 1.4 does not sort higher than a 1.3+blahblah anything, that's not good.
> 

The versions given here would sort like this (oldest to newest): 1.3, 1.3+debian1, 1.4.

Debian users would not get 1.3+debian1 via ``pip install yourthing``, or buildout, or easy_install.

Debian users would (in this hypothetical situation) install ``apt-get python-yourthing`` and get a copy of the library which is your 1.3 with some debian patches applies to it with the version 1.3+debian1.

So you?d only ever get a hypothetical +debian1 from apt-get (although in other situations it may come from pip, but it won?t be on PyPI).

Now if you have 1.3+debian1 installed via apt-get (or any means really), and you?ll get the following behaviors (show with pip):

- pip install yourthing -> 1.3+debian1 satisfies the constraint of anything, so it stays installed.
- pip install yourthing==1.3 -> 1.3+debian1 satisfies the constraint of ==1.3, so it stays installed.
- pip install yourthing>=1.3 -> 1.3+debian1 satisfies the constraint of >= 1.3, so it stays installed.
- pip install yourthing>=1.4 -> 1.3+debian1 does not satisfy the constraint of >=1.4, so pip will upgrade it to 1.4.
- pip install ?upgrade youthing -> You?ve requested an upgrade, so pip will see 1.4 exists and will install it.
- pip install ?upgrade yourthing==1.3 -> You?ve requested an upgrade, but there?s nothing newer than 1.3+debian1 that matches the constraint so it stays installed
- pip install -upgrade youthing>=1.3 -> You?ve requested an upgrade, pip will see 1.4 exists and satisfies the constraint and will install it.
- pip install ?upgrade yourthing>1.4 -> you?ve requested an upgrade and 1.3+debian1 is not >= 1.4, so pip will see 1.4 exists and install it.

Does that make sense? Essentially a +whatever is the _weakest_ version indicator, and 1.3+debian1 can be thought of semantically as ?version 1.3 plus some patches indicates by debian1?. Since there are patches on top of version 1.3, it considers 1.3+debian1 as newer than 1.3, but since 1.4 is newer than 1.3 it considers 1.4 as newer than 1.3+debian1 regardless of the patches.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From ethan at stoneleaf.us  Wed Dec 17 04:25:55 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 16 Dec 2014 19:25:55 -0800
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <7806B51F-9DCA-43FC-A47B-47F4A4E6EDEC@stufft.io>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io> <m6qmi3$tf2$1@ger.gmane.org>
 <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
 <5490E623.9050100@stoneleaf.us>
 <7806B51F-9DCA-43FC-A47B-47F4A4E6EDEC@stufft.io>
Message-ID: <5490F7C3.2010709@stoneleaf.us>

On 12/16/2014 06:48 PM, Donald Stufft wrote:
> 
> Now if you have 1.3+debian1 installed via apt-get (or any means really), and you?ll get the following behaviors (show with pip):
> 
> - pip install yourthing -> 1.3+debian1 satisfies the constraint of anything, so it stays installed.
> - pip install yourthing==1.3 -> 1.3+debian1 satisfies the constraint of ==1.3, so it stays installed.
> - pip install yourthing>=1.3 -> 1.3+debian1 satisfies the constraint of >= 1.3, so it stays installed.
> - pip install yourthing>=1.4 -> 1.3+debian1 does not satisfy the constraint of >=1.4, so pip will upgrade it to 1.4.
> - pip install ?upgrade youthing -> You?ve requested an upgrade, so pip will see 1.4 exists and will install it.
> - pip install ?upgrade yourthing==1.3 -> You?ve requested an upgrade, but there?s nothing newer than 1.3+debian1 that matches the constraint so it stays installed
> - pip install -upgrade youthing>=1.3 -> You?ve requested an upgrade, pip will see 1.4 exists and satisfies the constraint and will install it.
> - pip install ?upgrade yourthing>1.4 -> you?ve requested an upgrade and 1.3+debian1 is not >= 1.4, so pip will see 1.4 exists and install it.
> 
> Does that make sense?

Yes.  Thank you for that _very_ thorough explanation.  :)

To continue with Maurits' use-case, in order to get /exactly/ 1.3, '===' is the operator to use?  Or are we still
discussing that?  Personally, I think

  pip install yourthing is 1.3

;)

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141216/8f111900/attachment-0001.sig>

From ethan at stoneleaf.us  Wed Dec 17 04:31:44 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 16 Dec 2014 19:31:44 -0800
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <5490F7C3.2010709@stoneleaf.us>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io> <m6qmi3$tf2$1@ger.gmane.org>
 <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
 <5490E623.9050100@stoneleaf.us>
 <7806B51F-9DCA-43FC-A47B-47F4A4E6EDEC@stufft.io>
 <5490F7C3.2010709@stoneleaf.us>
Message-ID: <5490F920.4090107@stoneleaf.us>

Oh, to be clear:  There are no guarantees that 1.4 actually includes the bug-fixes in +debian1, correct?  It's just a
big hope?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141216/7cd230ce/attachment.sig>

From donald at stufft.io  Wed Dec 17 04:59:49 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Dec 2014 22:59:49 -0500
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <5490F7C3.2010709@stoneleaf.us>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io> <m6qmi3$tf2$1@ger.gmane.org>
 <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
 <5490E623.9050100@stoneleaf.us>
 <7806B51F-9DCA-43FC-A47B-47F4A4E6EDEC@stufft.io>
 <5490F7C3.2010709@stoneleaf.us>
Message-ID: <B7043599-B8AD-4A0C-BBB2-012D8950565B@stufft.io>


> On Dec 16, 2014, at 10:25 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> 
> On 12/16/2014 06:48 PM, Donald Stufft wrote:
>> 
>> Now if you have 1.3+debian1 installed via apt-get (or any means really), and you?ll get the following behaviors (show with pip):
>> 
>> - pip install yourthing -> 1.3+debian1 satisfies the constraint of anything, so it stays installed.
>> - pip install yourthing==1.3 -> 1.3+debian1 satisfies the constraint of ==1.3, so it stays installed.
>> - pip install yourthing>=1.3 -> 1.3+debian1 satisfies the constraint of >= 1.3, so it stays installed.
>> - pip install yourthing>=1.4 -> 1.3+debian1 does not satisfy the constraint of >=1.4, so pip will upgrade it to 1.4.
>> - pip install ?upgrade youthing -> You?ve requested an upgrade, so pip will see 1.4 exists and will install it.
>> - pip install ?upgrade yourthing==1.3 -> You?ve requested an upgrade, but there?s nothing newer than 1.3+debian1 that matches the constraint so it stays installed
>> - pip install -upgrade youthing>=1.3 -> You?ve requested an upgrade, pip will see 1.4 exists and satisfies the constraint and will install it.
>> - pip install ?upgrade yourthing>1.4 -> you?ve requested an upgrade and 1.3+debian1 is not >= 1.4, so pip will see 1.4 exists and install it.
>> 
>> Does that make sense?
> 
> Yes.  Thank you for that _very_ thorough explanation.  :)
> 
> To continue with Maurits' use-case, in order to get /exactly/ 1.3, '===' is the operator to use?  Or are we still
> discussing that?  Personally, I think
> 
>  pip install yourthing is 1.3
> 

Currently the === operator will give you that, but it?s not really designed for that. It?s designed as an escape hatch for versions we can?t parse. An example of where this matters is that ==1.3 will match version 1.3.0, but ===1.3 will not. So if we want something better than that for ?give me exactly 1.3 without any patches? then that?s under discussion.

And to answer your other question yes, there is no promises that 1.4 contains whatever patches 1.3+debian1 is carrying ontop of 1.3. Those patches might not even be acceptable changes upstream. For instance pip bundles it?s dependencies but debian unbundles them. That kind of patch would not be accepted upstream so in pip?s case Debian would always have a local version (assuming they actually use the local versions and don?t just ignore them).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From m.van.rees at zestsoftware.nl  Wed Dec 17 10:45:46 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 17 Dec 2014 10:45:46 +0100
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <B7043599-B8AD-4A0C-BBB2-012D8950565B@stufft.io>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io> <m6qmi3$tf2$1@ger.gmane.org>
 <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
 <5490E623.9050100@stoneleaf.us>
 <7806B51F-9DCA-43FC-A47B-47F4A4E6EDEC@stufft.io>
 <5490F7C3.2010709@stoneleaf.us>
 <B7043599-B8AD-4A0C-BBB2-012D8950565B@stufft.io>
Message-ID: <m6rjca$n2j$1@ger.gmane.org>

Donald Stufft schreef op 17-12-14 04:59:
>
>> On Dec 16, 2014, at 10:25 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> To continue with Maurits' use-case, in order to get /exactly/ 1.3, '===' is the operator to use?  Or are we still
>> discussing that?  Personally, I think
>>
>>   pip install yourthing is 1.3
>>
>
> Currently the === operator will give you that, but it?s not really designed for that. It?s designed as an escape hatch for versions we can?t parse. An example of where this matters is that ==1.3 will match version 1.3.0, but ===1.3 will not. So if we want something better than that for ?give me exactly 1.3 without any patches? then that?s under discussion.

Current pip 1.5.6 does not recognize the === operator and gives a 
ValueError.  That is expected.

pip dev does recognize it, but I do not see the behaviour you say.  Let 
me try from scratch:

$ virtualenv-2.7 venv-newest
$ cd venv-newest
$ . bin/activate
$ pip install -U setuptools  # gets 8.0.4
$ pip install https://github.com/pypa/pip/tarball/develop#egg=pip-dev
$ pip install -U --trusted-host pypi.zestsoftware.nl -f 
http://pypi.zestsoftware.nl/public/packagingtest/ myproject===1.1
Collecting myproject===1.1
   Downloading myproject-1.1+maurits.3.zip
   Requested myproject===1.1, but installing version 1.1+maurits.3
Installing collected packages: myproject
   Running setup.py install for myproject
Successfully installed myproject
$ ls -1 lib/python2.7/site-packages
_markerlib
easy_install.py
easy_install.pyc
myproject
myproject-1.1+maurits.3-py2.7.egg-info
pip
pip-1.5.6.dist-info
pip-6.0.dev1-py2.7.egg-info
pkg_resources.py
pkg_resources.pyc
setuptools
setuptools-8.0.4.dist-info
$ pip list
myproject (1.1+maurits.3)
pip (6.0.dev1)
setuptools (8.0.4)


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From ncoghlan at gmail.com  Wed Dec 17 11:13:55 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Dec 2014 20:13:55 +1000
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <3A660023-5E2F-48CE-A955-61A592DA9AA2@stufft.io>
References: <CAPDm-FhOGK439C5k+8cnBUSSGG8mp0sYe09nkP3=_YyoTCNiTA@mail.gmail.com>
 <m6noj9$oiu$1@ger.gmane.org> <m6nrs0$bqq$1@ger.gmane.org>
 <61AAC56D-64A5-49F8-B145-B722B46CB491@stufft.io>
 <CAPkN8xK5gcvBQHt6OFasgv=ziHr4pKqJDejUV8bWFdxYto7MmA@mail.gmail.com>
 <B7AAC833-1EB0-4C9A-9831-2910AE3EC4E7@stufft.io>
 <CAPkN8xLfCw-5YShz6XugEBCn_1LvWdimMZoOvqqdRR+_sqZ=Rg@mail.gmail.com>
 <4652E410-7764-4A3C-997F-FA2F6C804567@stufft.io>
 <20141216140817.GS2497@yuggoth.org>
 <CAPkN8xKx7p5c8pLLcxGoCaBvMAoqzR_hnCjXbar9_u8wPtWq8w@mail.gmail.com>
 <20141216154030.GT2497@yuggoth.org>
 <3A660023-5E2F-48CE-A955-61A592DA9AA2@stufft.io>
Message-ID: <CADiSq7fdxmvTwizt9FGOi8m86cOLv8ZvQX0PjWd+=gCQMAxbZg@mail.gmail.com>

On 17 December 2014 at 02:48, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 16, 2014, at 10:40 AM, Jeremy Stanley <fungi at yuggoth.org> wrote:
>>
>> The way I see it, we're all volunteers. We voluntarily work on free
>> software and associated support infrastructure. Some of us are
>> simply lucky enough to find sponsors willing to pay us to do it full
>> time (and while I won't speak for Donald I get the impression he
>> feels the same, whether he's working on PSF or OpenStack projects).
>
> Yea, I?m incredibly lucky to have found someone willing to pay me for my
> hobby. Rackspace pays me to work on Python packaging as well as pays me
> to work on Openstack.

Similarly, these days I get to spend (Red Hat) work time on helping to
get Fedora to play nice with upstream Python packaging (as well as
helping to wrangle other Python integration challenges, including
providing feedback and advice on the Python 3 migration work, and
playing a significant role in maintaining Red Hat's relationship with
the PSF). While most of my work day still goes to Red Hat internal
infrastructure work (including learning to take advantage of new
technologies like Docker and OpenShift), having that ability to divert
time into upstream when necessary makes a huge difference.

Untangling the issues in the Python packaging ecosystem is just an
incredibly complex change management problem, as anything we do to
tighten up the interoperability standards or the underlying security
model is almost certainly going to break things for someone,
somewhere. Those ecosystem specific constraints are thus far more
heavily weighted as a design consideration than interoperability with
third party versioning conventions (although we do aim to accommodate
those where practical).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris at simplistix.co.uk  Wed Dec 17 11:09:56 2014
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 17 Dec 2014 10:09:56 +0000
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <20141216120215.GB22133@fridge.pov.lt>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <20141216120215.GB22133@fridge.pov.lt>
Message-ID: <54915674.6000906@simplistix.co.uk>

On 16/12/2014 12:02, Marius Gedminas wrote:
>> Is there something I?m not aware that is broken currently? I thought the
>> transition was going pretty smoothly overall considering that a core piece
>> of code inside of setuptools was touched.
>
> ~80 zope.* test builds are currently failing, for mysterious reasons due
> to setuptools 8.0.x not interacting well with zc.buildout:
> https://mail.zope.org/pipermail/zope-dev/2014-December/046509.html
>
> Here's a summary of the various errors:
> https://mail.zope.org/pipermail/zope-dev/2014-December/046508.html
>
> Newer builds also added a bunch of warnings of the form
> /var/lib/jenkins/jobs/zopetoolkit_trunk/workspace/lib/python2.7/site-packages/pkg_resources.py:2425:
> RuntimeWarning: 'zc.recipe.testrunner-1.0.5 ()' is being parsed as a
> legacy, non PEP 440, version. You may find odd behavior and sort order.
> In particular it will be sorted as less than 0.0. It is recommend to
> migrate to PEP 440 compatible versions.
>
> Recent threads on distutils-sig@ help explain some of what's happening.
> E.g. the 'zope.app.wsgi<3.11,<4.0dev,>=3.12' probably used to be
> interpreted as an "<3.11 or >=3.12" and just needs to be hunted down and
> replaced with "!=3.11.*", or something like that.
>
> (It's painful when you get requirement conflict errors with no
> indication about the source of those requirements.)

FWIW, I've also been seeing failures in all of my buildout-based library 
testing Jenkins jobs:

This one only on Windows and Python 2.7 only, NOT Python 2.6:

C:\Jenkins\workspace\checker-buildout\aeb5917b>C:\Python27\python.exe 
bootstrap.py
Downloading 
https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.4.zip
Extracting in c:\users\jenkins\appdata\local\temp\tmp1hy01w
Now working in 
c:\users\jenkins\appdata\local\temp\tmp1hy01w\setuptools-8.0.4
Building a Setuptools egg in c:\users\jenkins\appdata\local\temp\tmpzl12vj
warning: no files found matching 'entries*' under directory 
'setuptools\tests'
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
c:\users\jenkins\appdata\local\temp\tmpzl12vj\setuptools-8.0.4-py2.7.egg
Traceback (most recent call last):
   File "bootstrap.py", line 92, in <module>
     ez['use_setuptools'](**setup_args)
   File "<string>", line 140, in use_setuptools
   File "<string>", line 128, in _do_download
   File "build\bdist.win-amd64\egg\setuptools\__init__.py", line 5, in 
<module>
   File "C:\Python27\lib\distutils\core.py", line 20, in <module>
     from distutils.dist import Distribution
   File "C:\Python27\lib\distutils\dist.py", line 10, in <module>
     from email import message_from_file
ImportError: No module named email

...which I see on a couple of jobs, also this, on Python 3, Linux:

Downloading 
https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.1.zip
Extracting in /tmp/tmp71bne9
Now working in /tmp/tmp71bne9/setuptools-8.0.1
Building a Setuptools egg in /tmp/tmpdgs_c0
/tmp/tmpdgs_c0/setuptools-8.0.1-py3.3.egg
Creating directory 
'/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/bin'.
Creating directory 
'/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/parts'.
Creating directory 
'/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/develop-eggs'.
Generated script 
'/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/bin/buildout'.
/tmp/tmpdgs_c0/setuptools-8.0.1-py3.3.egg/pkg_resources.py:130: 
RuntimeWarning: You have iterated over the result of 
pkg_resources.parse_version. This is a legacy behavior which is 
inconsistent with the new version class introduced in setuptools 8.0. 
That class should be used directly instead of attempting to iterate over 
the result.
/var/lib/jenkins/.buildout/eggs/setuptools-8.0.1-py3.3.egg/pkg_resources.py:130: 
RuntimeWarning: You have iterated over the result of 
pkg_resources.parse_version. This is a legacy behavior which is 
inconsistent with the new version class introduced in setuptools 8.0. 
That class should be used directly instead of attempting to iterate over 
the result.
While:
   Installing.
   Checking for upgrades.
   Getting distribution for 'zc.buildout>=2.2.5'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
   File 
"/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py", 
line 1946, in main
     getattr(buildout, command)(args)
   File 
"/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py", 
line 475, in install
     self._maybe_upgrade()
   File 
"/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py", 
line 910, in _maybe_upgrade
     allow_hosts = self._allow_hosts
   File 
"/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", 
line 844, in install
     return installer.install(specs, working_set)
   File 
"/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", 
line 631, in install
     for_buildout_run=for_buildout_run):
   File 
"/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", 
line 473, in _get_dist
     dist, avail = self._satisfied(requirement)
   File 
"/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", 
line 268, in _satisfied
     best_available = self._obtain(req, source)
   File 
"/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py", 
line 427, in _obtain
     if distv > bestv:
TypeError: unorderable types: SetuptoolsVersion() > tuple()

...and this one on Windows Python 3:

C:\Jenkins\workspace\mush-buildout\a1017537>C:\Python33\python.exe 
bootstrap.py
Downloading 
https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.1.zip
Extracting in c:\users\jenkins\appdata\local\temp\tmpkij7_u
Now working in 
c:\users\jenkins\appdata\local\temp\tmpkij7_u\setuptools-8.0.1
Building a Setuptools egg in c:\users\jenkins\appdata\local\temp\tmpgy_a7q
c:\users\jenkins\appdata\local\temp\tmpgy_a7q\setuptools-8.0.1-py3.3.egg
Traceback (most recent call last):
   File "<string>", line 138, in use_setuptools
ImportError: No module named 'pkg_resources'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "bootstrap.py", line 92, in <module>
     ez['use_setuptools'](**setup_args)
   File "<string>", line 140, in use_setuptools
   File "<string>", line 128, in _do_download
   File 
"c:\users\jenkins\appdata\local\temp\tmpgy_a7q\setuptools-8.0.1-py3.3.egg\setuptools\__init__.py", 
line 5, in <module>
   File "C:\Python33\lib\distutils\core.py", line 17, in <module>
     from distutils.dist import Distribution
   File "C:\Python33\lib\distutils\dist.py", line 15, in <module>
     from distutils.fancy_getopt import FancyGetopt, translate_longopt
   File "C:\Python33\lib\distutils\fancy_getopt.py", line 12, in <module>
     import getopt
ImportError: No module named 'getopt'

Links to these builds are here, if you're familiar with Jenkins:

http://jenkins.simplistix.co.uk/job/mush-buildout/
http://jenkins.simplistix.co.uk/job/checker-buildout/

cheers,

Chris

From ncoghlan at gmail.com  Wed Dec 17 11:19:00 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Dec 2014 20:19:00 +1000
Subject: [Distutils] Changing the status of PEP 440 to Provisional
In-Reply-To: <F1ED0F6D-493F-4380-BEE3-F6EA82D5A86E@stufft.io>
References: <CADiSq7dE=E-iN83fMgRj81VRB+ZzOoVF+=fWTsiRULD=fB1PSQ@mail.gmail.com>
 <CADiSq7cggt1BUwVDRCWX3FZy7HCFLd4abvKCBBWqg3VRyzYMOA@mail.gmail.com>
 <F1ED0F6D-493F-4380-BEE3-F6EA82D5A86E@stufft.io>
Message-ID: <CADiSq7cM9FUnRjfjqPhFeTOWpqJ+9yw-779DGHsLvM=Bi05ZDg@mail.gmail.com>

On 17 December 2014 at 09:42, Donald Stufft <donald at stufft.io> wrote:
> Note that as of now Openstack has decided on a different path and currently
> does not need any adjustments to PEP 440 as I had talked about with you, so
> as of right now the only thing I really have to change (unless they change
> their mind or someone else comes forward) is that I?d like to move c -> rc
> in the preferred form. I?m not against making PEP 440 Provisional though and
> I have no idea if more issues will come up in the coming days.

I think it's a good policy change regardless - I doubt this will be
the last time where something that initially seemed like a good idea
doesn't seem so promising once it's exposed to a broader audience.
That's exactly the kind of problem PEP 411 was designed to deal with
for the standard library, so extending it to the PyPA ecosystem makes
a lot of sense.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From m.van.rees at zestsoftware.nl  Wed Dec 17 11:27:25 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 17 Dec 2014 11:27:25 +0100
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io> <m6qmi3$tf2$1@ger.gmane.org>
 <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
Message-ID: <m6rlqd$34v$1@ger.gmane.org>

Donald Stufft schreef op 17-12-14 02:49:
> Another thing though, what you probably want to do is something like 1.1.dev0+internal1, which will sort as older than 1.1 whenever it is actually released.

Sounds like the reasonable way to go.

Let me try, with a pip that has your local version identifier fixes from 
https://github.com/pypa/pip/pull/2205

$ pip install -U --trusted-host pypi.zestsoftware.nl -f 
http://pypi.zestsoftware.nl/public/packagingtest/ 
myproject==1.3.dev0+maurits.1
Collecting myproject==1.3.dev0+maurits.1
   Downloading myproject-1.3.dev0+maurits.1.zip
Requirement already up-to-date: Jinja2[i18n] in 
./lib/python2.7/site-packages (from myproject==1.3.dev0+maurits.1)
   Installing extra requirements: u'i18n'
Requirement already up-to-date: markupsafe in 
./lib/python2.7/site-packages (from 
Jinja2[i18n]->myproject==1.3.dev0+maurits.1)
Requirement already up-to-date: Babel>=0.8 in 
./lib/python2.7/site-packages (from 
Jinja2[i18n]->myproject==1.3.dev0+maurits.1)
Requirement already up-to-date: pytz>=0a in 
./lib/python2.7/site-packages (from 
Babel>=0.8->Jinja2[i18n]->myproject==1.3.dev0+maurits.1)
Installing collected packages: myproject
   Found existing installation: myproject 1.2
     Uninstalling myproject:
       Successfully uninstalled myproject
   Running setup.py install for myproject
Successfully installed myproject

Working fine.

Now I create and upload myproject version 1.3.

$ pip install -U --trusted-host pypi.zestsoftware.nl -f 
http://pypi.zestsoftware.nl/public/packagingtest/ myproject==1.3
Collecting myproject==1.3
   Downloading myproject-1.3.zip
...
Installing collected packages: myproject
   Found existing installation: myproject 1.3.dev0+maurits.1
     Uninstalling myproject:
       Successfully uninstalled myproject
   Running setup.py install for myproject
Successfully installed myproject

Again, working fine.

A '-U myproject' also gives me 1.3, and I can still request 
myproject==1.3.dev0+maurits.1 if I want.

Good!


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From marius at pov.lt  Wed Dec 17 11:36:52 2014
From: marius at pov.lt (Marius Gedminas)
Date: Wed, 17 Dec 2014 12:36:52 +0200
Subject: [Distutils] setuptools 8 changes are great, but ...
In-Reply-To: <54915674.6000906@simplistix.co.uk>
References: <CAPDm-FiMupApdZmWaENO4oMkQsPpU+Cw+5k+JeibXcJ_i40nNQ@mail.gmail.com>
 <188E8319-20E2-495B-ACC9-1030854B53A5@stufft.io>
 <20141216120215.GB22133@fridge.pov.lt>
 <54915674.6000906@simplistix.co.uk>
Message-ID: <20141217103651.GA8810@fridge.pov.lt>

On Wed, Dec 17, 2014 at 10:09:56AM +0000, Chris Withers wrote:
> FWIW, I've also been seeing failures in all of my buildout-based
> library testing Jenkins jobs:
> 
> This one only on Windows and Python 2.7 only, NOT Python 2.6:
> 
> C:\Jenkins\workspace\checker-buildout\aeb5917b>C:\Python27\python.exe
> bootstrap.py
...
> ImportError: No module named email

This sounds like https://github.com/buildout/buildout/issues/217

> ...which I see on a couple of jobs, also this, on Python 3, Linux:
> 
> Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.1.zip
> Extracting in /tmp/tmp71bne9
> Now working in /tmp/tmp71bne9/setuptools-8.0.1
> Building a Setuptools egg in /tmp/tmpdgs_c0
> /tmp/tmpdgs_c0/setuptools-8.0.1-py3.3.egg
> Creating directory
> '/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/bin'.
> Creating directory
> '/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/parts'.
> Creating directory '/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/develop-eggs'.
> Generated script '/var/lib/jenkins/slave/workspace/mush-buildout/0321cca2/bin/buildout'.
> /tmp/tmpdgs_c0/setuptools-8.0.1-py3.3.egg/pkg_resources.py:130:
> RuntimeWarning: You have iterated over the result of
> pkg_resources.parse_version. This is a legacy behavior which is
> inconsistent with the new version class introduced in setuptools
> 8.0. That class should be used directly instead of attempting to
> iterate over the result.
> /var/lib/jenkins/.buildout/eggs/setuptools-8.0.1-py3.3.egg/pkg_resources.py:130:
> RuntimeWarning: You have iterated over the result of
> pkg_resources.parse_version. This is a legacy behavior which is
> inconsistent with the new version class introduced in setuptools
> 8.0. That class should be used directly instead of attempting to
> iterate over the result.
> While:
>   Installing.
>   Checking for upgrades.
>   Getting distribution for 'zc.buildout>=2.2.5'.
> 
> An internal error occurred due to a bug in either zc.buildout or in a
> recipe being used:
> Traceback (most recent call last):
>   File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py",
> line 1946, in main
>     getattr(buildout, command)(args)
>   File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py",
> line 475, in install
>     self._maybe_upgrade()
>   File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/buildout.py",
> line 910, in _maybe_upgrade
>     allow_hosts = self._allow_hosts
>   File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py",
> line 844, in install
>     return installer.install(specs, working_set)
>   File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py",
> line 631, in install
>     for_buildout_run=for_buildout_run):
>   File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py",
> line 473, in _get_dist
>     dist, avail = self._satisfied(requirement)
>   File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py",
> line 268, in _satisfied
>     best_available = self._obtain(req, source)
>   File "/var/lib/jenkins/.buildout/eggs/zc.buildout-2.2.5-py3.3.egg/zc/buildout/easy_install.py",
> line 427, in _obtain
>     if distv > bestv:
> TypeError: unorderable types: SetuptoolsVersion() > tuple()

Haven't seen that one before.

> ...and this one on Windows Python 3:
> 
> C:\Jenkins\workspace\mush-buildout\a1017537>C:\Python33\python.exe
> bootstrap.py
> Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-8.0.1.zip
> Extracting in c:\users\jenkins\appdata\local\temp\tmpkij7_u
> Now working in
> c:\users\jenkins\appdata\local\temp\tmpkij7_u\setuptools-8.0.1
> Building a Setuptools egg in c:\users\jenkins\appdata\local\temp\tmpgy_a7q
> c:\users\jenkins\appdata\local\temp\tmpgy_a7q\setuptools-8.0.1-py3.3.egg
> Traceback (most recent call last):
>   File "<string>", line 138, in use_setuptools
> ImportError: No module named 'pkg_resources'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "bootstrap.py", line 92, in <module>
>     ez['use_setuptools'](**setup_args)
>   File "<string>", line 140, in use_setuptools
>   File "<string>", line 128, in _do_download
>   File "c:\users\jenkins\appdata\local\temp\tmpgy_a7q\setuptools-8.0.1-py3.3.egg\setuptools\__init__.py",
> line 5, in <module>
>   File "C:\Python33\lib\distutils\core.py", line 17, in <module>
>     from distutils.dist import Distribution
>   File "C:\Python33\lib\distutils\dist.py", line 15, in <module>
>     from distutils.fancy_getopt import FancyGetopt, translate_longopt
>   File "C:\Python33\lib\distutils\fancy_getopt.py", line 12, in <module>
>     import getopt
> ImportError: No module named 'getopt'

Might be https://github.com/buildout/buildout/issues/217 again?

> Links to these builds are here, if you're familiar with Jenkins:
> 
> http://jenkins.simplistix.co.uk/job/mush-buildout/
> http://jenkins.simplistix.co.uk/job/checker-buildout/

Marius Gedminas
-- 
  To express oneself
  In seventeen syllables
  Is very diffic
		-- John Cooper Clark.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141217/1957f9b8/attachment.sig>

From ncoghlan at gmail.com  Wed Dec 17 11:37:40 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Dec 2014 20:37:40 +1000
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <5490F920.4090107@stoneleaf.us>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <4F149DEE-7AF5-42AD-883F-BC9A5F3B3065@stufft.io>
 <m6qmi3$tf2$1@ger.gmane.org>
 <814116C2-6E25-47CD-BB93-401F22039BB4@stufft.io>
 <5490E623.9050100@stoneleaf.us>
 <7806B51F-9DCA-43FC-A47B-47F4A4E6EDEC@stufft.io>
 <5490F7C3.2010709@stoneleaf.us> <5490F920.4090107@stoneleaf.us>
Message-ID: <CADiSq7c8nRuL0L-8f6bnfFktc1Gy0bVKATc-bZVp-OtU9mLF_w@mail.gmail.com>

On 17 December 2014 at 13:31, Ethan Furman <ethan at stoneleaf.us> wrote:
> Oh, to be clear:  There are no guarantees that 1.4 actually includes the bug-fixes in +debian1, correct?  It's just a
> big hope?

Correct. The local versions were mostly added such that Linux distros
could indicate to Python specific tools when patches had been applied
using the distro packaging system. This is actually a pretty normal
practice (especially in a long lived distro like RHEL/CentOS - our
version numbers indicate the *baseline* version of a package), but
it's currently completely invisible to the Python level tooling (a
patched distro provided package will currently report the same Python
level version metadata as a pristine upstream package).

The promise we make is that any such patches shouldn't break
compatibility at all (that's the whole point of the baseline+backports
long term support model), which is why the comparison semantics are
defined to ignore them by default. That lets us incorporate security
fixes (for example) without breaking anything, while still providing
an indication that something has changed in the software.

Whether those patches make their way upstream will depend on the patch
and the upstream project. Some distro integration patches (like pip
dependency unbundling) will never go upstream. Others will be
backports of bug fixes from later upstream feature releases to earlier
versions that are integrated into the distro. And some will be patches
that distros apply because we need them quickly for some reason, and
then subsequently work through the process of getting the change (or a
variant thereof) accepted upstream (because carrying downstream
patches long term is expensive, we prefer not to do it if we don't
have to).

So while the usage model Maurits describes here is valid (and should
work with setuptools 8 + pip 6), it isn't really the primary intended
use case - it's aimed at when you're installing Python packages but
using something other than the Python specific tooling to do it (e.g.
apt-get, yum, conda, etc).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From m.van.rees at zestsoftware.nl  Wed Dec 17 13:07:06 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 17 Dec 2014 13:07:06 +0100
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <m6qjq2$l3c$1@ger.gmane.org>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
Message-ID: <m6rrla$3sp$1@ger.gmane.org>

Maurits van Rees schreef op 17-12-14 01:46:
> Maurits van Rees schreef op 17-12-14 00:53:
>> I have created a very basic python project called 'myproject'.  It
>> does nothing.  I have released a few versions here:
>> http://pypi.zestsoftware.nl/public/packagingtest/
>
> I have now also distributed myproject version 1.1.  (This has a
> base.jinja2 file and requires Jinja2[i18n], which I need for checking
> a corner case; I may report that later).

I will report that now.


I ran into some problems earlier with an internal release of Babel.

Problem is that I use Jinja2 and this has a requirement on
'Babel>=0.8'.  This gives extra problems in some cases, maybe more
because entry points are used.  In some cases installing works fine,
but actually using the package gives an error.

For clarity: I don't think the notes below indicate a bug in
setuptools or pip or buildout.  It just shows some possibly unexpected
behaviour.

Latest official Babel release is 1.3.  I have created a few
development releases and added them to
http://pypi.zestsoftware.nl/public/

Some versions there are PEP 440 compliant, others not.  Some may make
sense, other not.  In order of creation:

- setup.py version 2.0a0.zest.20141021: Babel-2.0a0.zest.20141021.zip

- setup.py version 2.0a1.zest.20141021: Babel-2.0a1.zest.20141021.zip

- setup.py version 2.0a1+zest.20141216: Babel-2.0a1-zest.20141216.zip

- setup.py version 2.0a1+zest.201412162: Babel-2.0a1+zest.201412162.zip

Let's try them out.


Babel and buildout
------------------

With the earlier given buildout, I added an extra directory to the 
find-links:

find-links = http://pypi.zestsoftware.nl/public/packagingtest/
     http://pypi.zestsoftware.nl/public/

Then I updated the Babel version in the [versions] section.

- Babel = 2.0a1.zest.20141021
   * Works in setuptools 7.
   * Does not work setuptools 8:
     Error: Bad constraint 2.0a1.zest.20141021 Babel>=0.8

- Babel = 2.0a1-zest.20141216
   * Works in setuptools 7.
   * Does not work setuptools 8:
     Error: Bad constraint 2.0a1.zest.20141021 Babel>=0.8

- Babel = 2.0a1+zest.201412162
   * Does not work in setuptools 8:
     ValueError: ("Expected ',' or end-of-list in", 
'Babel[]==2.0a1+zest.201412162', 'at', '+zest.201412162')
   * Works in setuptools 8


Babel and pip
-------------

- Babel==2.0a1.zest.20141021
   * pip 1.5.6 works fine
   * pip dev fails when using it:
     $ ../venv-newest/bin/python setup.py extract_messages 
--mapping-file=extract.ini --output-file=foo.pot
     running extract_messages
     extracting messages from myproject/__init__.py
     extracting messages from myproject/base.jinja2
     Traceback (most recent call last):
       File "setup.py", line 11, in <module>
         'Jinja2[i18n]',
       File 
"/Users/mauritsvanrees/buildout/python2.7/parts/opt/lib/python2.7/distutils/core.py", 
line 151, in setup
         dist.run_commands()
       File 
"/Users/mauritsvanrees/buildout/python2.7/parts/opt/lib/python2.7/distutils/dist.py", 
line 953, in run_commands
         self.run_command(cmd)
       File 
"/Users/mauritsvanrees/buildout/python2.7/parts/opt/lib/python2.7/distutils/dist.py", 
line 972, in run_command
         cmd_obj.run()
       File 
"/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/babel/messages/frontend.py", 
line 305, in run
         for filename, lineno, message, comments, context in extracted:
       File 
"/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/babel/messages/extract.py", 
line 163, in extract_from_dir
         strip_comment_tags):
       File 
"/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/babel/messages/extract.py", 
line 190, in extract_from_file
         strip_comment_tags))
       File 
"/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/babel/messages/extract.py", 
line 249, in extract
         func = entry_point.load(require=True)
       File 
"/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/pkg_resources.py", 
line 2242, in load
         self.require(env, installer)
       File 
"/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/pkg_resources.py", 
line 2256, in require
         items = working_set.resolve(reqs, env, installer)
       File 
"/Users/mauritsvanrees/tmp/venv-newest/lib/python2.7/site-packages/pkg_resources.py", 
line 751, in resolve
         raise VersionConflict(tmpl % args)
     pkg_resources.VersionConflict: Babel 2.0a1.zest.20141021 is 
installed but Babel>=0.8 is required by []
   * DANGER: a version that previously worked fine and still installs
     fine, is broken when using it.  This may be troublesome.  Mostly
     you should just be aware of it.  Basically, this wrongly named
     internal version number of my Babel distribution does not work
     nicely with setuptools 8 when you also use jinja2.
   * I only wonder if something needs fixing in Babel's usage of the
     entry point.

- Babel==2.0a1-zest.20141216
   * pip 1.5.6 works fine
   * pip dev works fine (with version warning while installing)
   * Note that it works because the setup.py version is a valid 440
     version: 2.0a1+zest.20141216

- Babel==2.0a1+zest.20141216
   * pip 1.5.6 fails to install: plus is not accepted.
   * pip dev fails to install.  Error message is: "No distributions
     matching the version for Babel==2.0a1+zest.20141216".  This is
     logical, because this distribution was created with setuptools 7
     or earlier, which results in a file name that does not exactly
     match the version.

- Babel==2.0a1+zest.201412162
   * pip 1.5.6 fails to install: plus is not accepted.
   * pip dev fails to install.
   * pip dev with pull request #2205 (Expand our regex to support the
     new PEP 440 version characters) works fine.

- pip install 
http://pypi.zestsoftware.nl/public/Babel-2.0a1+zest.201412162.zip
   * pip 1.5.6 works fine
   * pip dev works fine



-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From m.van.rees at zestsoftware.nl  Wed Dec 17 13:40:44 2014
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 17 Dec 2014 13:40:44 +0100
Subject: [Distutils] Local version identifiers from PEP 440 in practice
In-Reply-To: <m6rrla$3sp$1@ger.gmane.org>
References: <m6qgl4$bd6$1@ger.gmane.org> <m6qjq2$l3c$1@ger.gmane.org>
 <m6rrla$3sp$1@ger.gmane.org>
Message-ID: <m6rtkc$4ua$1@ger.gmane.org>

Maurits van Rees schreef op 17-12-14 13:07:
>
> Babel and buildout
> ------------------
>
> With the earlier given buildout, I added an extra directory to the
> find-links:
>
> find-links = http://pypi.zestsoftware.nl/public/packagingtest/
> http://pypi.zestsoftware.nl/public/
>
> Then I updated the Babel version in the [versions] section.
>
> - Babel = 2.0a1.zest.20141021
>    * Works in setuptools 7.
>    * Does not work setuptools 8:
>      Error: Bad constraint 2.0a1.zest.20141021 Babel>=0.8
>
> - Babel = 2.0a1-zest.20141216
>    * Works in setuptools 7.
>    * Does not work setuptools 8:
>      Error: Bad constraint 2.0a1.zest.20141021 Babel>=0.8
>
> - Babel = 2.0a1+zest.201412162
>    * Does not work in setuptools 8:
>      ValueError: ("Expected ',' or end-of-list in",
> 'Babel[]==2.0a1+zest.201412162', 'at', '+zest.201412162')
>    * Works in setuptools 8

Sorry, that last part should have been:
- Babel = 2.0a1+zest.201412162
    * Does not work in setuptools *7*:
      ValueError: ("Expected ',' or end-of-list in",
'Babel[]==2.0a1+zest.201412162', 'at', '+zest.201412162')
    * Works in setuptools 8


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From vinay_sajip at yahoo.co.uk  Wed Dec 17 20:49:19 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 17 Dec 2014 19:49:19 +0000 (UTC)
Subject: [Distutils] [ANN]: distlib 0.2.0 released on PyPI
Message-ID: <304031956.508659.1418845759377.JavaMail.yahoo@jws11114.mail.ir2.yahoo.com>

I've just released version 0.2.0 of distlib on PyPI [1]. For newcomers,
distlib is a library of packaging functionality which is intended to be
usable as the basis for third-party packaging tools.

The main changes in this release are as follows:

    Updated match_hostname to use the latest Python implementation.

    Updates to better support PEP 426 / PEP 440.

    You can now provide interpreter arguments in shebang lines written
    by distlib.

    Removed reference to __PYVENV_LAUNCHER__ (relevant to OS X only).

A more detailed change log is available at [2].

Please try it out, and if you find any problems or have any suggestions for
improvements, please give some feedback using the issue tracker! [3]

Regards,

Vinay Sajip

[1] https://pypi.python.org/pypi/distlib/0.2.0
[2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib
[3] https://bitbucket.org/pypa/distlib/issues/new

From donald at stufft.io  Thu Dec 18 02:55:50 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 17 Dec 2014 20:55:50 -0500
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release
	Candidates
In-Reply-To: <CACac1F9CZYCEpbxXrJuwEOV4EBLxdNqo+0a6yRcQyZ3vTLi1yQ@mail.gmail.com>
References: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
 <CACac1F9CZYCEpbxXrJuwEOV4EBLxdNqo+0a6yRcQyZ3vTLi1yQ@mail.gmail.com>
Message-ID: <0D6AA22C-4021-451F-BF51-A7447DA2B33C@stufft.io>


> On Dec 16, 2014, at 2:51 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 16 December 2014 at 17:56, Donald Stufft <donald at stufft.io> wrote:
>> I?ve had a number of people complain that PEP 440 normalizes a release
>> candidate as 1.0c1 instead of 1.0rc1. Do we want to ammend the PEP to switch
>> the normalization around so that it normalizes 1.0c1 to 1.0rc1? This probably
>> matches what people expect better in general? It wouldn't have any effect on
>> sort order or the like, it would just change which form is preferred.
> 
> I'd typically expect to see 1.0rc1 rather than 1.0c1. I don't think
> it's a crisis either way, but I would mildly prefer it if this change
> was made.
> Paul

Given that it?s a fairly low impact change, We?ve got several pip developers
+1 on it, Jason is OK with it on the setuptools side, and nobody has spoken
out against it I?m going to go ahead and make this change.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From donald at stufft.io  Thu Dec 18 03:17:02 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 17 Dec 2014 21:17:02 -0500
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release
	Candidates
In-Reply-To: <0D6AA22C-4021-451F-BF51-A7447DA2B33C@stufft.io>
References: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
 <CACac1F9CZYCEpbxXrJuwEOV4EBLxdNqo+0a6yRcQyZ3vTLi1yQ@mail.gmail.com>
 <0D6AA22C-4021-451F-BF51-A7447DA2B33C@stufft.io>
Message-ID: <512E3F1A-72A7-40DF-A06A-0AFC6515FEC9@stufft.io>


> On Dec 17, 2014, at 8:55 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Dec 16, 2014, at 2:51 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> 
>> On 16 December 2014 at 17:56, Donald Stufft <donald at stufft.io> wrote:
>>> I?ve had a number of people complain that PEP 440 normalizes a release
>>> candidate as 1.0c1 instead of 1.0rc1. Do we want to ammend the PEP to switch
>>> the normalization around so that it normalizes 1.0c1 to 1.0rc1? This probably
>>> matches what people expect better in general? It wouldn't have any effect on
>>> sort order or the like, it would just change which form is preferred.
>> 
>> I'd typically expect to see 1.0rc1 rather than 1.0c1. I don't think
>> it's a crisis either way, but I would mildly prefer it if this change
>> was made.
>> Paul
> 
> Given that it?s a fairly low impact change, We?ve got several pip developers
> +1 on it, Jason is OK with it on the setuptools side, and nobody has spoken
> out against it I?m going to go ahead and make this change.
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

And done: https://hg.python.org/peps/rev/93db88237edb

Will get the libraries updated next.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From ncoghlan at gmail.com  Thu Dec 18 03:40:25 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Dec 2014 12:40:25 +1000
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release
	Candidates
In-Reply-To: <0D6AA22C-4021-451F-BF51-A7447DA2B33C@stufft.io>
References: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
 <CACac1F9CZYCEpbxXrJuwEOV4EBLxdNqo+0a6yRcQyZ3vTLi1yQ@mail.gmail.com>
 <0D6AA22C-4021-451F-BF51-A7447DA2B33C@stufft.io>
Message-ID: <CADiSq7dcj6iBP1oK_zk_tA81r32aTSJCnbvFcmLE+oxVdXbquQ@mail.gmail.com>

On 18 December 2014 at 11:55, Donald Stufft <donald at stufft.io> wrote:
>
>> On Dec 16, 2014, at 2:51 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> On 16 December 2014 at 17:56, Donald Stufft <donald at stufft.io> wrote:
>>> I?ve had a number of people complain that PEP 440 normalizes a release
>>> candidate as 1.0c1 instead of 1.0rc1. Do we want to ammend the PEP to switch
>>> the normalization around so that it normalizes 1.0c1 to 1.0rc1? This probably
>>> matches what people expect better in general? It wouldn't have any effect on
>>> sort order or the like, it would just change which form is preferred.
>>
>> I'd typically expect to see 1.0rc1 rather than 1.0c1. I don't think
>> it's a crisis either way, but I would mildly prefer it if this change
>> was made.
>> Paul
>
> Given that it?s a fairly low impact change, We?ve got several pip developers
> +1 on it, Jason is OK with it on the setuptools side, and nobody has spoken
> out against it I?m going to go ahead and make this change.

Thanks. I think one key with this particular change is that client
tools were already obliged to accept both formats anyway, as
normalisation for publication is only a SHOULD, rather than a MUST.

Switching the preferred format for publication thus shouldn't break
anything on the consumption side.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From olivier.grisel at ensta.org  Thu Dec 18 11:29:38 2014
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Thu, 18 Dec 2014 11:29:38 +0100
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release
	Candidates
In-Reply-To: <512E3F1A-72A7-40DF-A06A-0AFC6515FEC9@stufft.io>
References: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
 <CACac1F9CZYCEpbxXrJuwEOV4EBLxdNqo+0a6yRcQyZ3vTLi1yQ@mail.gmail.com>
 <0D6AA22C-4021-451F-BF51-A7447DA2B33C@stufft.io>
 <512E3F1A-72A7-40DF-A06A-0AFC6515FEC9@stufft.io>
Message-ID: <CAFvE7K7QYW+KNrtKu4qHaXMX24PgShujESEw6eGE1a5gEjuPhg@mail.gmail.com>

Since PEP 440 was formally accepted in August 2014, would it make
sense to add a change log to document the amendment of the PEP, for
instance in the appendix (maybe with a link to the diff in hg)?

-- 
Olivier

From ncoghlan at gmail.com  Thu Dec 18 13:40:09 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Dec 2014 22:40:09 +1000
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release
	Candidates
In-Reply-To: <CAFvE7K7QYW+KNrtKu4qHaXMX24PgShujESEw6eGE1a5gEjuPhg@mail.gmail.com>
References: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
 <CACac1F9CZYCEpbxXrJuwEOV4EBLxdNqo+0a6yRcQyZ3vTLi1yQ@mail.gmail.com>
 <0D6AA22C-4021-451F-BF51-A7447DA2B33C@stufft.io>
 <512E3F1A-72A7-40DF-A06A-0AFC6515FEC9@stufft.io>
 <CAFvE7K7QYW+KNrtKu4qHaXMX24PgShujESEw6eGE1a5gEjuPhg@mail.gmail.com>
Message-ID: <CADiSq7cABFufNJ-QWtjhysPDePdoFMTfh9yM_2Up4C0aExLnxQ@mail.gmail.com>

On 18 Dec 2014 20:30, "Olivier Grisel" <olivier.grisel at ensta.org> wrote:
>
> Since PEP 440 was formally accepted in August 2014, would it make
> sense to add a change log to document the amendment of the PEP, for
> instance in the appendix (maybe with a link to the diff in hg)?

Yeah, that's a good idea. I'll also include that requirement when I draft
the proposed update for PEP 1.

Cheers,
Nick.

>
> --
> Olivier
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141218/582b3596/attachment.html>

From donald at stufft.io  Thu Dec 18 15:49:58 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 18 Dec 2014 09:49:58 -0500
Subject: [Distutils] Amend PEP 440 with Wider Feedback on Release
	Candidates
In-Reply-To: <512E3F1A-72A7-40DF-A06A-0AFC6515FEC9@stufft.io>
References: <D9361519-2524-4551-8204-EBEAF1C74A53@stufft.io>
 <CACac1F9CZYCEpbxXrJuwEOV4EBLxdNqo+0a6yRcQyZ3vTLi1yQ@mail.gmail.com>
 <0D6AA22C-4021-451F-BF51-A7447DA2B33C@stufft.io>
 <512E3F1A-72A7-40DF-A06A-0AFC6515FEC9@stufft.io>
Message-ID: <1687C12F-F430-45F0-97B3-9810CDAC8775@stufft.io>


> On Dec 17, 2014, at 9:17 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Dec 17, 2014, at 8:55 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>> 
>>> On Dec 16, 2014, at 2:51 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>> 
>>> On 16 December 2014 at 17:56, Donald Stufft <donald at stufft.io> wrote:
>>>> I?ve had a number of people complain that PEP 440 normalizes a release
>>>> candidate as 1.0c1 instead of 1.0rc1. Do we want to ammend the PEP to switch
>>>> the normalization around so that it normalizes 1.0c1 to 1.0rc1? This probably
>>>> matches what people expect better in general? It wouldn't have any effect on
>>>> sort order or the like, it would just change which form is preferred.
>>> 
>>> I'd typically expect to see 1.0rc1 rather than 1.0c1. I don't think
>>> it's a crisis either way, but I would mildly prefer it if this change
>>> was made.
>>> Paul
>> 
>> Given that it?s a fairly low impact change, We?ve got several pip developers
>> +1 on it, Jason is OK with it on the setuptools side, and nobody has spoken
>> out against it I?m going to go ahead and make this change.
>> 
>> ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> 
> And done: https://hg.python.org/peps/rev/93db88237edb
> 
> Will get the libraries updated next.
> 

setuptools 8.1 is out which has this in it and packaging 14.5
is as well.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From mklein0 at gmail.com  Thu Dec 18 18:17:47 2014
From: mklein0 at gmail.com (Marcos Klein)
Date: Thu, 18 Dec 2014 09:17:47 -0800
Subject: [Distutils] Amend PEP 440 regarding timestamp based version
	identifiers and packaging
Message-ID: <CAEgqhcBbcnNHCc9a60b6RCjKUBzyw5Lq8ah_LN7aDQvBG1qtCQ@mail.gmail.com>

I have two update requests for PEP 440.

Could PEP 440 date-based version identifier examples be extend to include
full timestamp version identifiers?

The version identifiers I use in a continuous integration build are derived
from ISO8601 time stamps.

YearMonthDay "." HourMinuteSecond    (UTC)

ex. date -u +'%Y%m%d.%H%M%S'

This version identifier format has the peculiar aspect of having a number
of leading 0s in the version segment when a package is created at the
beginning of a new day.

20141218.000018

I just want to make sure any test cases include this version identifier
format.


This leads me to my second request.

Could the effects of normalized version identifiers be clarified when it
comes to package builds?

The normalization section in PEP 440 only seems to discuss the use of
normalization in parsing and processing of the version identifier.  I was
quite surprised when my package build for the above version identifier
became the following under setuptools 8:

dated_release-20141218.18-p27-none-any.whl

Previous releases of setuptools would build:

dated_release-20141218.000018-p27-any.whl

This is jarring as it is an unexpected interpretation of PEP 440. It is the
classic pointer argument.  I want to call it THIS, but it really is THAT.

--
Marcos Klein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141218/d1290d79/attachment.html>

From ncoghlan at gmail.com  Fri Dec 19 01:03:58 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Dec 2014 10:03:58 +1000
Subject: [Distutils] Amend PEP 440 regarding timestamp based version
 identifiers and packaging
In-Reply-To: <CAEgqhcBbcnNHCc9a60b6RCjKUBzyw5Lq8ah_LN7aDQvBG1qtCQ@mail.gmail.com>
References: <CAEgqhcBbcnNHCc9a60b6RCjKUBzyw5Lq8ah_LN7aDQvBG1qtCQ@mail.gmail.com>
Message-ID: <CADiSq7eFViNK1N9Nus9wG5iLf7HM293qRhj7x9ziwHHj=cL9hg@mail.gmail.com>

On 19 Dec 2014 03:50, "Marcos Klein" <mklein0 at gmail.com> wrote:
>
> I have two update requests for PEP 440.
>
> Could PEP 440 date-based version identifier examples be extend to include
full timestamp version identifiers?

Sure, that's not a change to the semantics, just some additional examples.

> This leads me to my second request.
>
> Could the effects of normalized version identifiers be clarified when it
comes to package builds?
>
> The normalization section in PEP 440 only seems to discuss the use of
normalization in parsing and processing of the version identifier.  I was
quite surprised when my package build for the above version identifier
became the following under setuptools 8:
>
> dated_release-20141218.18-p27-none-any.whl
>
> Previous releases of setuptools would build:
>
> dated_release-20141218.000018-p27-any.whl
>
> This is jarring as it is an unexpected interpretation of PEP 440. It is
the classic pointer argument.  I want to call it THIS, but it really is
THAT.

This is primarily an RFE for setuptools 8+ requesting the ability to skip
the normalisation step. At the PEP level, it's already covered by the fact
that installers are required to be able to do dynamic normalisation.

That said, it's likely worth adding a clarifying paragraph that our
perspective is that while installation tools MUST normalise versions, build
tools SHOULD normalise versions.

Cheers,
Nick.

>
> --
> Marcos Klein
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141219/91008489/attachment.html>

From donald at stufft.io  Fri Dec 19 01:09:42 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 18 Dec 2014 19:09:42 -0500
Subject: [Distutils] Amend PEP 440 regarding timestamp based version
	identifiers and packaging
In-Reply-To: <CADiSq7eFViNK1N9Nus9wG5iLf7HM293qRhj7x9ziwHHj=cL9hg@mail.gmail.com>
References: <CAEgqhcBbcnNHCc9a60b6RCjKUBzyw5Lq8ah_LN7aDQvBG1qtCQ@mail.gmail.com>
 <CADiSq7eFViNK1N9Nus9wG5iLf7HM293qRhj7x9ziwHHj=cL9hg@mail.gmail.com>
Message-ID: <33643629-E985-4798-B9AB-0003D20F66FC@stufft.io>


> On Dec 18, 2014, at 7:03 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> 
> On 19 Dec 2014 03:50, "Marcos Klein" <mklein0 at gmail.com <mailto:mklein0 at gmail.com>> wrote:
> >
> > I have two update requests for PEP 440.
> >
> > Could PEP 440 date-based version identifier examples be extend to include full timestamp version identifiers?
> 
> Sure, that's not a change to the semantics, just some additional examples.
> 
> > This leads me to my second request.
> >
> > Could the effects of normalized version identifiers be clarified when it comes to package builds? 
> >
> > The normalization section in PEP 440 only seems to discuss the use of normalization in parsing and processing of the version identifier.  I was quite surprised when my package build for the above version identifier became the following under setuptools 8:
> >
> > dated_release-20141218.18-p27-none-any.whl
> >
> > Previous releases of setuptools would build:
> >
> > dated_release-20141218.000018-p27-any.whl
> >
> > This is jarring as it is an unexpected interpretation of PEP 440. It is the classic pointer argument.  I want to call it THIS, but it really is THAT.
> 
> This is primarily an RFE for setuptools 8+ requesting the ability to skip the normalisation step. At the PEP level, it's already covered by the fact that installers are required to be able to do dynamic normalisation.
> 
> That said, it's likely worth adding a clarifying paragraph that our perspective is that while installation tools MUST normalise versions, build tools SHOULD normalise versions.
> 
> Cheers,
> Nick.
> 

Originally I directed this here, though I think now it?s because I forgot how numbers work. I was thinking that currently the normalization would result in _wrong_ versions not just ugly versions. I forgot that 0018000 won?t normalize to the same thing as 0000000 so it?s likely safe to do that I think. Originally I was thinking that without doing the int() you?d get different results, something which in hindsight is obviously wrong.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141218/458c26d2/attachment-0001.html>

From vladimir.v.diaz at gmail.com  Mon Dec 22 16:46:01 2014
From: vladimir.v.diaz at gmail.com (Vladimir Diaz)
Date: Mon, 22 Dec 2014 10:46:01 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
Message-ID: <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>

::ping::

Has anyone in the community gotten a chance to review PEP 458 and/or PEP
480?  Community feedback would be appreciated.

Thanks,
Vlad

On Wed, Dec 10, 2014 at 10:16 PM, Vladimir Diaz <vladimir.v.diaz at gmail.com>
wrote:

> Hello everyone,
>
> I am a research programmer at the NYU School of Engineering.  My
> colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting
> community feedback on our proposal, "Surviving a Compromise of PyPI."  The
> two-stage proposal can be reviewed online at:
>
> PEP 458
> http://legacy.python.org/dev/peps/pep-0458/
>
> PEP 480
> http://legacy.python.org/dev/peps/pep-0480/
>
>
> Summary of the Proposal:
>
> "Surviving a Compromise of PyPI" proposes how the Python Package Index
> (PyPI) can be amended to better protect end users from altered or malicious
> packages, and to minimize the extent of PyPI compromises against affected
> users.  The proposed integration allows package managers such as pip to be
> more secure against various types of security attacks on PyPI and defend
> end users from attackers responding to package requests. Specifically,
> these PEPs describe how PyPI processes should be adapted to generate and
> incorporate repository metadata, which are signed text files that describe
> the packages and metadata available on PyPI.  Package managers request
> (along with the packages) the metadata on PyPI to verify the authenticity
> of packages before they are installed.  The changes to PyPI and tools will
> be minimal by leveraging a library, The Update Framework
> <https://github.com/theupdateframework/tuf>, that generates and
> transparently validates the relevant metadata.
>
> The first stage of the proposal (PEP 458
> <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security
> model that supports verification of PyPI packages signed with cryptographic
> keys stored on PyPI, requires no action from developers and end users, and
> protects against malicious CDNs and public mirrors. To support continuous
> delivery of uploaded packages, PyPI administrators sign for uploaded
> packages with an online key stored on PyPI infrastructure. This level of
> security prevents packages from being accidentally or deliberately tampered
> with by a mirror or a CDN because the mirror or CDN will not have any of
> the keys required to sign for projects.
>
> The second stage of the proposal (PEP 480
> <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the
> basic security model (discussed in PEP 458) that supports end-to-end
> verification of signed packages. End-to-end signing allows both PyPI and
> developers to sign for the packages that are downloaded by end users.  If
> the PyPI infrastructure were to be compromised, attackers would be unable
> to serve malicious versions of these packages without access to the
> project's developer key.  As in PEP 458, no additional action is required
> by end users.  However, PyPI administrators will need to periodically
> (perhaps every few months) sign metadata with an offline key.  PEP 480 also
> proposes an easy-to-use key management solution for developers, how to
> interface with a potential build farm on PyPI infrastructure, and discusses
> the security benefits of end-to-end signing.  The second stage of the
> proposal simultaneously supports real-time project registration and
> developer signatures, and when configured to maximize security on PyPI,
> less than 1% of end users will be at risk even if an attacker controls PyPI
> and goes undetected for a month.
>
> We thank Nick Coghlan and Donald Stufft for their valuable contributions,
> and Giovanni Bajo and Anatoly Techtonik for their feedback.
>
> Thanks,
> PEP 458 & 480 authors.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/05cf14e3/attachment.html>

From ncoghlan at gmail.com  Mon Dec 22 17:30:24 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Dec 2014 02:30:24 +1000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
Message-ID: <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>

On 23 December 2014 at 01:46, Vladimir Diaz <vladimir.v.diaz at gmail.com>
wrote:

> ::ping::
>
> Has anyone in the community gotten a chance to review PEP 458 and/or PEP
> 480?  Community feedback would be appreciated.
>

Sorry about that Vladimir - I think the main PyPA devs have been focused on
getting PEP 440 implemented, and the associated setuptools, pip and
virtualenv updates out the door, and now we're into the end of year holiday
period for many of us.

>From my perspective, the split into two PEPs meant most of the areas I have
doubts about have been moved to the end-to-end security model in PEP 480,
leaving PEP 458 to cover the simpler task of securing the link from PyPI to
the end user in such a way that public mirrors of packages can be trusted
to accurately reflect the content published by PyPI.

The new appendix C raises some good questions regarding how we would like
to deal with externally hosted packages. I personally agree with the PEP's
recommendation to require TUF metadata generated with a developer key in
that case, with a slight preference for publishing that metadata (including
the corresponding security delegation) through PyPI. However, I'm open to
practicality arguments suggesting that one of the other options may be more
feasible for maintainers of externally hosted repositories.

The root key management question is the other one that will be interesting,
given the distributed nature of both PSF Infrastructure maintenance and
pip/PyPI development. A partial root key compromise would effectively
become a CVE against pip and CPython (and hence flowing on to Linux
distributions and potentially other redistributors), with the severity
depending on whether or not the signing threshold has been reached.

I suspect before we sign off on that last part, we're going to need to get
quite specific with exactly *who* will hold signing authority over those
root keys (just as the CPython release PEPs specifically name the release
managers for the source tarballs and the platform specific binaries, who
then have signing authority for their respective artifacts)

Regards,
Nick.


> Thanks,
> Vlad
>
> On Wed, Dec 10, 2014 at 10:16 PM, Vladimir Diaz <vladimir.v.diaz at gmail.com
> > wrote:
>
>> Hello everyone,
>>
>> I am a research programmer at the NYU School of Engineering.  My
>> colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting
>> community feedback on our proposal, "Surviving a Compromise of PyPI."  The
>> two-stage proposal can be reviewed online at:
>>
>> PEP 458
>> http://legacy.python.org/dev/peps/pep-0458/
>>
>> PEP 480
>> http://legacy.python.org/dev/peps/pep-0480/
>>
>>
>> Summary of the Proposal:
>>
>> "Surviving a Compromise of PyPI" proposes how the Python Package Index
>> (PyPI) can be amended to better protect end users from altered or malicious
>> packages, and to minimize the extent of PyPI compromises against affected
>> users.  The proposed integration allows package managers such as pip to be
>> more secure against various types of security attacks on PyPI and defend
>> end users from attackers responding to package requests. Specifically,
>> these PEPs describe how PyPI processes should be adapted to generate and
>> incorporate repository metadata, which are signed text files that describe
>> the packages and metadata available on PyPI.  Package managers request
>> (along with the packages) the metadata on PyPI to verify the authenticity
>> of packages before they are installed.  The changes to PyPI and tools will
>> be minimal by leveraging a library, The Update Framework
>> <https://github.com/theupdateframework/tuf>, that generates and
>> transparently validates the relevant metadata.
>>
>> The first stage of the proposal (PEP 458
>> <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security
>> model that supports verification of PyPI packages signed with cryptographic
>> keys stored on PyPI, requires no action from developers and end users, and
>> protects against malicious CDNs and public mirrors. To support continuous
>> delivery of uploaded packages, PyPI administrators sign for uploaded
>> packages with an online key stored on PyPI infrastructure. This level of
>> security prevents packages from being accidentally or deliberately tampered
>> with by a mirror or a CDN because the mirror or CDN will not have any of
>> the keys required to sign for projects.
>>
>> The second stage of the proposal (PEP 480
>> <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the
>> basic security model (discussed in PEP 458) that supports end-to-end
>> verification of signed packages. End-to-end signing allows both PyPI and
>> developers to sign for the packages that are downloaded by end users.  If
>> the PyPI infrastructure were to be compromised, attackers would be unable
>> to serve malicious versions of these packages without access to the
>> project's developer key.  As in PEP 458, no additional action is required
>> by end users.  However, PyPI administrators will need to periodically
>> (perhaps every few months) sign metadata with an offline key.  PEP 480 also
>> proposes an easy-to-use key management solution for developers, how to
>> interface with a potential build farm on PyPI infrastructure, and discusses
>> the security benefits of end-to-end signing.  The second stage of the
>> proposal simultaneously supports real-time project registration and
>> developer signatures, and when configured to maximize security on PyPI,
>> less than 1% of end users will be at risk even if an attacker controls PyPI
>> and goes undetected for a month.
>>
>> We thank Nick Coghlan and Donald Stufft for their valuable contributions,
>> and Giovanni Bajo and Anatoly Techtonik for their feedback.
>>
>> Thanks,
>> PEP 458 & 480 authors.
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/96b47df7/attachment-0001.html>

From vladimir.v.diaz at gmail.com  Mon Dec 22 19:15:36 2014
From: vladimir.v.diaz at gmail.com (Vladimir Diaz)
Date: Mon, 22 Dec 2014 13:15:36 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
 <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
Message-ID: <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>

On Mon, Dec 22, 2014 at 11:30 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 23 December 2014 at 01:46, Vladimir Diaz <vladimir.v.diaz at gmail.com>
> wrote:
>
>> ::ping::
>>
>> Has anyone in the community gotten a chance to review PEP 458 and/or PEP
>> 480?  Community feedback would be appreciated.
>>
>
> Sorry about that Vladimir - I think the main PyPA devs have been focused
> on getting PEP 440 implemented, and the associated setuptools, pip and
> virtualenv updates out the door, and now we're into the end of year holiday
> period for many of us.
>

Considering that the previous drafts of PEP 458 generated quite a bit of
feedback and questions, I was surprised by the lack of responses this time
around (almost relieved :).  Receiving feedback from the main PyP
developers after the holiday period is definitely not a problem.


> From my perspective, the split into two PEPs meant most of the areas I
> have doubts about have been moved to the end-to-end security model in PEP
> 480, leaving PEP 458 to cover the simpler task of securing the link from
> PyPI to the end user in such a way that public mirrors of packages can be
> trusted to accurately reflect the content published by PyPI.
>

I think splitting the proposal into two PEPs was the right decision.  We
hope working with Donald on the end-to-end security model (PEP 480), and
feedback from the community will help to address any remaining questions.
Excluding the end-to-end option from the revised version of PEP 458 also
made room for an overview of the metadata and framework, which was
requested by multiple members of the community.


> The new appendix C raises some good questions regarding how we would like
> to deal with externally hosted packages. I personally agree with the PEP's
> recommendation to require TUF metadata generated with a developer key in
> that case, with a slight preference for publishing that metadata (including
> the corresponding security delegation) through PyPI. However, I'm open to
> practicality arguments suggesting that one of the other options may be more
> feasible for maintainers of externally hosted repositories.
>
> The root key management question is the other one that will be
> interesting, given the distributed nature of both PSF Infrastructure
> maintenance and pip/PyPI development. A partial root key compromise would
> effectively become a CVE against pip and CPython (and hence flowing on to
> Linux distributions and potentially other redistributors), with the
> severity depending on whether or not the signing threshold has been reached.
>
> I suspect before we sign off on that last part, we're going to need to get
> quite specific with exactly *who* will hold signing authority over those
> root keys (just as the CPython release PEPs specifically name the release
> managers for the source tarballs and the platform specific binaries, who
> then have signing authority for their respective artifacts)
>

We can decide in the coming weeks who should be explicitly named, and the
number of individuals that should hold signing authority over the root
keys.  If reaching a reasonable threshold of root keys remains a problem,
we can also assist with meeting this threshold.  I will discuss this matter
with the rest of the authors to make sure we are all on the same page wrt
root key management.

Regards,
Nick.


> Thanks,
> Vlad
>
> On Wed, Dec 10, 2014 at 10:16 PM, Vladimir Diaz <vladimir.v.diaz at gmail.com
> > wrote:
>
>> Hello everyone,
>>
>> I am a research programmer at the NYU School of Engineering.  My
>> colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting
>> community feedback on our proposal, "Surviving a Compromise of PyPI."  The
>> two-stage proposal can be reviewed online at:
>>
>> PEP 458
>> http://legacy.python.org/dev/peps/pep-0458/
>>
>> PEP 480
>> http://legacy.python.org/dev/peps/pep-0480/
>>
>>
>> Summary of the Proposal:
>>
>> "Surviving a Compromise of PyPI" proposes how the Python Package Index
>> (PyPI) can be amended to better protect end users from altered or malicious
>> packages, and to minimize the extent of PyPI compromises against affected
>> users.  The proposed integration allows package managers such as pip to be
>> more secure against various types of security attacks on PyPI and defend
>> end users from attackers responding to package requests. Specifically,
>> these PEPs describe how PyPI processes should be adapted to generate and
>> incorporate repository metadata, which are signed text files that describe
>> the packages and metadata available on PyPI.  Package managers request
>> (along with the packages) the metadata on PyPI to verify the authenticity
>> of packages before they are installed.  The changes to PyPI and tools will
>> be minimal by leveraging a library, The Update Framework
>> <https://github.com/theupdateframework/tuf>, that generates and
>> transparently validates the relevant metadata.
>>
>> The first stage of the proposal (PEP 458
>> <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security
>> model that supports verification of PyPI packages signed with cryptographic
>> keys stored on PyPI, requires no action from developers and end users, and
>> protects against malicious CDNs and public mirrors. To support continuous
>> delivery of uploaded packages, PyPI administrators sign for uploaded
>> packages with an online key stored on PyPI infrastructure. This level of
>> security prevents packages from being accidentally or deliberately tampered
>> with by a mirror or a CDN because the mirror or CDN will not have any of
>> the keys required to sign for projects.
>>
>> The second stage of the proposal (PEP 480
>> <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the
>> basic security model (discussed in PEP 458) that supports end-to-end
>> verification of signed packages. End-to-end signing allows both PyPI and
>> developers to sign for the packages that are downloaded by end users.  If
>> the PyPI infrastructure were to be compromised, attackers would be unable
>> to serve malicious versions of these packages without access to the
>> project's developer key.  As in PEP 458, no additional action is required
>> by end users.  However, PyPI administrators will need to periodically
>> (perhaps every few months) sign metadata with an offline key.  PEP 480 also
>> proposes an easy-to-use key management solution for developers, how to
>> interface with a potential build farm on PyPI infrastructure, and discusses
>> the security benefits of end-to-end signing.  The second stage of the
>> proposal simultaneously supports real-time project registration and
>> developer signatures, and when configured to maximize security on PyPI,
>> less than 1% of end users will be at risk even if an attacker controls PyPI
>> and goes undetected for a month.
>>
>> We thank Nick Coghlan and Donald Stufft for their valuable contributions,
>> and Giovanni Bajo and Anatoly Techtonik for their feedback.
>>
>> Thanks,
>> PEP 458 & 480 authors.
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/b7d348a4/attachment.html>

From donald at stufft.io  Mon Dec 22 19:32:00 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 22 Dec 2014 13:32:00 -0500
Subject: [Distutils] Release: pip 6.0 and virtualenv 12.0
Message-ID: <1924C59B-C450-4F3E-8AA9-419CD4545C1E@stufft.io>

I'm happy to announce the release of pip 6.0 and virtualenv 12.0.

A High level overview of what this brings:

* PEP 440 is fully implemented now and pip will use specifiers and the version
  selection logic as specified there.

* HTTP access will now be cached by default in pip, speeding up repeated
  downloads of the same file automatically.

* Randomized and secure build directories are used by default in most
  situations.

* Accessing an insecure origin (Invalid HTTPS or HTTP) by default is now
  deprecated. For HTTP this will continue to work in pip 6.0 but raise a
  warning and for HTTPS this will not work. You may use
  --trusted-host example.com to re-enable this on a per-host basis.

* Added per-virtualenv and a machine global configuration file, as well as
  support platform standard directories for configuration.

* Support environment markers inside of a requirements file.

* Support environment markers inside of a setuptools extra.

* Automatically retry failed HTTP requests.

* Greatly reduce the verbosity of the pip command by default.

* Updated virtualenv to have setuptools 8.2.1 and pip 6.0 bundled.

* Many many bugfixes and smaller changes.

As always please file any issues with either https://github.com/pypa/pip/issues
or https://github.com/pypa/virtualenv/issues.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From qwcode at gmail.com  Mon Dec 22 21:20:32 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 12:20:32 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
Message-ID: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>

In PEP440,  foo-X.Y.Z does not satisfy the specifier  "foo>X.Y" (although
it satisfies "foo>=X.Y")

for example,  foo-1.7.2 will not satisfy  "foo>1.7", but it will satisfy
"foo>=1.7"

for '>' and '<', PEP440 states that they are "interpreted as implying the
prefix based version exclusion clause != V.*"

the rationale from Donald for this is explained here:
https://bitbucket.org/pypa/setuptools/issue/301/101-in-requirementparse-foo-10-results

in brief, this was done to avoid pre-releases (i.e. something like
"foo-1.7a1") satisfying "foo<1.7".

It seems to me we've just traded one confusion for another, and this seems
worse (to me at least, although I'm trying to let it sink in)

1.7.2 is greater than 1.7, and the specifiers should honor that.  It's hard
to accept otherwise.

Yes, it's true in the past, when people would get a pre-release installed
(for example 1.7a1), when they really wanted something in the 1.6.X series,
that was also confusing.  But now that pip now excludes pre-releases by
default, it seems that confusion is mostly solved practically speaking, so
why solve it (and create a new confusion)?

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/1d35ef09/attachment-0001.html>

From p.f.moore at gmail.com  Mon Dec 22 21:31:44 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 Dec 2014 20:31:44 +0000
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
Message-ID: <CACac1F960FMGuMO8M63AHM1wKTpyvbKej0MnWAPnrT1AHfcwnQ@mail.gmail.com>

On 22 December 2014 at 20:20, Marcus Smith <qwcode at gmail.com> wrote:
> In PEP440,  foo-X.Y.Z does not satisfy the specifier  "foo>X.Y" (although it
> satisfies "foo>=X.Y")
>
> for example,  foo-1.7.2 will not satisfy  "foo>1.7", but it will satisfy
> "foo>=1.7"
>
> for '>' and '<', PEP440 states that they are "interpreted as implying the
> prefix based version exclusion clause != V.*"
>
> the rationale from Donald for this is explained here:
> https://bitbucket.org/pypa/setuptools/issue/301/101-in-requirementparse-foo-10-results
>
> in brief, this was done to avoid pre-releases (i.e. something like
> "foo-1.7a1") satisfying "foo<1.7".
>
> It seems to me we've just traded one confusion for another, and this seems
> worse (to me at least, although I'm trying to let it sink in)
>
> 1.7.2 is greater than 1.7, and the specifiers should honor that.  It's hard
> to accept otherwise.
>
> Yes, it's true in the past, when people would get a pre-release installed
> (for example 1.7a1), when they really wanted something in the 1.6.X series,
> that was also confusing.  But now that pip now excludes pre-releases by
> default, it seems that confusion is mostly solved practically speaking, so
> why solve it (and create a new confusion)?

Hmm, yes, I'd have to say that sounds like a recipe for bug reports.

+1 for updating PEP 440 (and pip) so that 1.7.2 is > 1.7. Maybe add a
note to PEP 440 that installers SHOULD exclude pre-releases by default
(with a note mentioning the 1.7a1 < 1.7 case as the justification).

Paul

From donald at stufft.io  Mon Dec 22 21:33:55 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 22 Dec 2014 15:33:55 -0500
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
Message-ID: <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>


> On Dec 22, 2014, at 3:20 PM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> In PEP440,  foo-X.Y.Z does not satisfy the specifier  "foo>X.Y" (although it satisfies "foo>=X.Y")
> 
> for example,  foo-1.7.2 will not satisfy  "foo>1.7", but it will satisfy "foo>=1.7"
> 
> for '>' and '<', PEP440 states that they are "interpreted as implying the prefix based version exclusion clause != V.*"
> 
> the rationale from Donald for this is explained here:  https://bitbucket.org/pypa/setuptools/issue/301/101-in-requirementparse-foo-10-results <https://bitbucket.org/pypa/setuptools/issue/301/101-in-requirementparse-foo-10-results>
> 
> in brief, this was done to avoid pre-releases (i.e. something like "foo-1.7a1") satisfying "foo<1.7".   
> 
> It seems to me we've just traded one confusion for another, and this seems worse (to me at least, although I'm trying to let it sink in)
> 
> 1.7.2 is greater than 1.7, and the specifiers should honor that.  It's hard to accept otherwise.
> 
> Yes, it's true in the past, when people would get a pre-release installed (for example 1.7a1), when they really wanted something in the 1.6.X series, that was also confusing.  But now that pip now excludes pre-releases by default, it seems that confusion is mostly solved practically speaking, so why solve it (and create a new confusion)?
> 

It?s true that pip doesn?t install pre-releases by default (assuming there is any final releases available), but that doesn?t actually solve the underlying problem - it just masks it. When people put ``foo<8`` in their install_requires they generally do not expect to get ``8.0.dev0``, and in my opinion, they shouldn?t get ``8.0.dev0`` even if someone uses ?pre.

In a way, we have replaced one confusion for another, although I do not believe it to be that bad and I believe that the current situation is better because:

* In my searches of github and my experience, use of the > instead of >= is fairly limited, meanwhile use of < to mean how it?s interpreted now is far more common.
* The new behavior maintains consistency between < and >, so that specifiers that ?look? the same act the same, maintaining consistency between them.
* I think that having the > and < behavior vary is a *worse* confusion, and I believe that the behavior of < is far better than previous.

In particular, <, >, ~=, and, when using a .*, the != and == use the number of dots in the given specifier to indicate the precision of the specifier.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/0b52f5fb/attachment.html>

From chris.jerdonek at gmail.com  Mon Dec 22 21:34:17 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 22 Dec 2014 12:34:17 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
Message-ID: <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>

On Mon, Dec 22, 2014 at 12:20 PM, Marcus Smith <qwcode at gmail.com> wrote:
> In PEP440,  foo-X.Y.Z does not satisfy the specifier  "foo>X.Y" (although it
> satisfies "foo>=X.Y")
>
> for example,  foo-1.7.2 will not satisfy  "foo>1.7", but it will satisfy
> "foo>=1.7"

This makes sense if you view "1.7" as encompassing all of the releases
1.7.0, 1.7.1, 1.7.2, etc. (and their variants).  It's a bit like the
"significant digits" of a number. [1]  You want the release to be
bigger than all releases of that type.

--Chris

[1] http://en.wikipedia.org/wiki/Significant_figures


>
> for '>' and '<', PEP440 states that they are "interpreted as implying the
> prefix based version exclusion clause != V.*"
>
> the rationale from Donald for this is explained here:
> https://bitbucket.org/pypa/setuptools/issue/301/101-in-requirementparse-foo-10-results
>
> in brief, this was done to avoid pre-releases (i.e. something like
> "foo-1.7a1") satisfying "foo<1.7".
>
> It seems to me we've just traded one confusion for another, and this seems
> worse (to me at least, although I'm trying to let it sink in)
>
> 1.7.2 is greater than 1.7, and the specifiers should honor that.  It's hard
> to accept otherwise.
>
> Yes, it's true in the past, when people would get a pre-release installed
> (for example 1.7a1), when they really wanted something in the 1.6.X series,
> that was also confusing.  But now that pip now excludes pre-releases by
> default, it seems that confusion is mostly solved practically speaking, so
> why solve it (and create a new confusion)?
>
> Marcus
>
>
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From p.f.moore at gmail.com  Mon Dec 22 21:43:59 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 Dec 2014 20:43:59 +0000
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
Message-ID: <CACac1F_bur69_BmOMg36Z6bfWB46=nK80wHuJ7pJo+Mx=C9kdg@mail.gmail.com>

On 22 December 2014 at 20:33, Donald Stufft <donald at stufft.io> wrote:
> It?s true that pip doesn?t install pre-releases by default (assuming there
> is any final releases available), but that doesn?t actually solve the
> underlying problem - it just masks it. When people put ``foo<8`` in their
> install_requires they generally do not expect to get ``8.0.dev0``, and in my
> opinion, they shouldn?t get ``8.0.dev0`` even if someone uses ?pre.

I see the point, but when *should* they get 8.0.dev0? It seems to me
that "=8.0" (with --pre) is the correct answer. So <8.0 means 7.9 and
earlier, >8.0 means 8.1 and later, =8.0 means 8.0 and any dev, pre,
post and local releases.

Essentially, any version X.Y.Z lumped together with all of its pre,
post, local and dev versions, (mathematically a version is an
equivalence class of all these) and otherwise ordering is a pretty
straightforward ordering as if the version were a tuple (X, Y, Z).
That's what my intuition says - are there any other ways that PEP 440
doesn't work like this?.

Paul

From qwcode at gmail.com  Mon Dec 22 21:44:55 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 12:44:55 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
Message-ID: <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>

it would fail.  you'd need  ">1.7.0"

On Mon, Dec 22, 2014 at 12:36 PM, James Bennett <ubernostrum at gmail.com>
wrote:

> So, if PyPI has foo-1.7 and foo-1.7.1, does ">1.7" just fail to find
> anything installable?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/267bf6f6/attachment-0001.html>

From donald at stufft.io  Mon Dec 22 21:47:17 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 22 Dec 2014 15:47:17 -0500
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CACac1F_bur69_BmOMg36Z6bfWB46=nK80wHuJ7pJo+Mx=C9kdg@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
 <CACac1F_bur69_BmOMg36Z6bfWB46=nK80wHuJ7pJo+Mx=C9kdg@mail.gmail.com>
Message-ID: <96A944F7-031B-4B61-BFF4-8F227F734615@stufft.io>


> On Dec 22, 2014, at 3:43 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 22 December 2014 at 20:33, Donald Stufft <donald at stufft.io> wrote:
>> It?s true that pip doesn?t install pre-releases by default (assuming there
>> is any final releases available), but that doesn?t actually solve the
>> underlying problem - it just masks it. When people put ``foo<8`` in their
>> install_requires they generally do not expect to get ``8.0.dev0``, and in my
>> opinion, they shouldn?t get ``8.0.dev0`` even if someone uses ?pre.
> 
> I see the point, but when *should* they get 8.0.dev0? It seems to me
> that "=8.0" (with --pre) is the correct answer. So <8.0 means 7.9 and
> earlier, >8.0 means 8.1 and later, =8.0 means 8.0 and any dev, pre,
> post and local releases.
> 
> Essentially, any version X.Y.Z lumped together with all of its pre,
> post, local and dev versions, (mathematically a version is an
> equivalence class of all these) and otherwise ordering is a pretty
> straightforward ordering as if the version were a tuple (X, Y, Z).
> That's what my intuition says - are there any other ways that PEP 440
> doesn't work like this?.
> 

You won?t get 8.0.dev1 with a == operation. You would get it with <=8
and of course you?d get it with a >7 if 8.0.dev1 was the newest available.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From qwcode at gmail.com  Mon Dec 22 22:03:26 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 13:03:26 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
Message-ID: <CAPYWazp19sc0qBg3F=MMb8o1Hi2Sxku9iVyBom=r_QkfU0msEQ@mail.gmail.com>

This makes sense if you view "1.7" as encompassing all of the releases

> 1.7.0, 1.7.1, 1.7.2, etc. (and their variants).  It's a bit like the
> "significant digits" of a number. [1]  You want the release to be
> bigger than all releases of that type.
>

but is this really the underlying concept?
Does the PEP explain this as a justification? Where?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/28cd846d/attachment.html>

From qwcode at gmail.com  Mon Dec 22 22:12:29 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 13:12:29 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
Message-ID: <CAPYWazrUdJf0kvTcWCv9_kioKKm8ZJSwa36=dt3q6Vs6f3C9pg@mail.gmail.com>

> * The new behavior maintains consistency between < and >, so that
> specifiers that ?look? the same act the same, maintaining consistency
> between them.
> * I think that having the > and < behavior vary is a *worse* confusion,
> and I believe that the behavior of < is far better than previous.
>

I'm not following the old inconsistencies you're referring to. Maybe you
can explain those, or reference this in the PEP.
But in any case, the argument isn't over whether PEP440 is better than the
old implementation, in total.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/5a0b3b14/attachment.html>

From graffatcolmingov at gmail.com  Mon Dec 22 22:16:24 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 22 Dec 2014 15:16:24 -0600
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazrUdJf0kvTcWCv9_kioKKm8ZJSwa36=dt3q6Vs6f3C9pg@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
 <CAPYWazrUdJf0kvTcWCv9_kioKKm8ZJSwa36=dt3q6Vs6f3C9pg@mail.gmail.com>
Message-ID: <CAN-Kwu06BJ+3Ogu5jKo6XsZf9sgZ90H0UALJRFtVrq1Pjzx92A@mail.gmail.com>

The other way in which this makes a whole lot more sense is when
considering projects that use dates as versions. Let's take for
example projects that have a released versions like

2013.08.10
2013.10.30
2013.12.03
2014.02.16
2014.09.23

If I say I want foo > 2013, I expect anything in the 2014.*.* series.
I don't expect 2013.*.* The same works for > 1.7

On Mon, Dec 22, 2014 at 3:12 PM, Marcus Smith <qwcode at gmail.com> wrote:
>
>> * The new behavior maintains consistency between < and >, so that
>> specifiers that ?look? the same act the same, maintaining consistency
>> between them.
>> * I think that having the > and < behavior vary is a *worse* confusion,
>> and I believe that the behavior of < is far better than previous.
>
>
> I'm not following the old inconsistencies you're referring to. Maybe you can
> explain those, or reference this in the PEP.
> But in any case, the argument isn't over whether PEP440 is better than the
> old implementation, in total.
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From qwcode at gmail.com  Mon Dec 22 22:22:21 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 13:22:21 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAN-Kwu06BJ+3Ogu5jKo6XsZf9sgZ90H0UALJRFtVrq1Pjzx92A@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
 <CAPYWazrUdJf0kvTcWCv9_kioKKm8ZJSwa36=dt3q6Vs6f3C9pg@mail.gmail.com>
 <CAN-Kwu06BJ+3Ogu5jKo6XsZf9sgZ90H0UALJRFtVrq1Pjzx92A@mail.gmail.com>
Message-ID: <CAPYWazq8LYwokE-8W15QWyxzkhQ92SSsUYE3dp5=iQUVtQhtsQ@mail.gmail.com>

> If I say I want foo > 2013, I expect anything in the 2014.*.* series.
> I don't expect 2013.*.* The same works for > 1.7
>

I wouldn't necessarily expect that, unless the PEP laid down these
conceptual assumptions.
maybe it's in there, I'm just not seeing it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/e490b4ee/attachment.html>

From chris.jerdonek at gmail.com  Mon Dec 22 22:31:39 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 22 Dec 2014 13:31:39 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazp19sc0qBg3F=MMb8o1Hi2Sxku9iVyBom=r_QkfU0msEQ@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAPYWazp19sc0qBg3F=MMb8o1Hi2Sxku9iVyBom=r_QkfU0msEQ@mail.gmail.com>
Message-ID: <CAOTb1wfde8nmLE55vivGOk9+z0Qsq1N+4aOYvG41HyVJfJSu1Q@mail.gmail.com>

On Mon, Dec 22, 2014 at 1:03 PM, Marcus Smith <qwcode at gmail.com> wrote:
>
> This makes sense if you view "1.7" as encompassing all of the releases
>>
>> 1.7.0, 1.7.1, 1.7.2, etc. (and their variants).  It's a bit like the
>> "significant digits" of a number. [1]  You want the release to be
>> bigger than all releases of that type.
>
>
> but is this really the underlying concept?
> Does the PEP explain this as a justification? Where?

I think so.  I just read that part of the PEP.

In the section "Exclusive ordered comparison," it says, "the
comparison operators are < and > and the clause MUST be effectively
interpreted as implying the prefix based version exclusion clause !=
V.* ."

The key here is the reference to "prefix based" version exclusion (as
opposed to "strict" version matching).  If I understand correctly,
this says that when "1.7.2" should be compared with "1.7" for
equality, it should be compared with "1.7.*" as opposed to "1.7.0".
This is described in the "Version matching" section.

The prefix-based ".*" notation encapsulates what I said above about
how  "1.7" can be viewed as encompassing all of the releases 1.7.0,
1.7.1, etc.

--Chris

From chris.jerdonek at gmail.com  Mon Dec 22 23:49:15 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 22 Dec 2014 14:49:15 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazobjNt4x3DGxLgkn2bGR3x8AE5W0HVtz6tP48r8fDKf1A@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAPYWazp19sc0qBg3F=MMb8o1Hi2Sxku9iVyBom=r_QkfU0msEQ@mail.gmail.com>
 <CAOTb1wfde8nmLE55vivGOk9+z0Qsq1N+4aOYvG41HyVJfJSu1Q@mail.gmail.com>
 <CAPYWazobjNt4x3DGxLgkn2bGR3x8AE5W0HVtz6tP48r8fDKf1A@mail.gmail.com>
Message-ID: <CAOTb1weMic3F-3QqZo2POQygKx_kWFkLefJWq==VyWty5AQjuw@mail.gmail.com>

On Mon, Dec 22, 2014 at 2:38 PM, Marcus Smith <qwcode at gmail.com> wrote:
> there's a big difference between stating the concept directly/simply in the
> PEP vs readers/users having to reverse engineer it from "prefix based
> version exclusion clause !=
> V.* ."
>
> If this is what it's about, users need to be clearly informed (in the pip
> and PyPUG docs) that "1.7" effectively means "1.7.*" in specifiers.

FWIW, I agree the wording isn't very easy to follow (it reads more
like a technical standard rather than something to educate).  You have
to unwind the logic from a few sections.

By the way, note that "1.7" doesn't *always* mean "1.7.*" (in fact, it
usually doesn't since the default is the "strict" form).  From the
"Version Matching" section:

"By default, the version matching operator is based on a strict
equality comparison: the specified version must be exactly the same as
the requested version. The only substitution performed is the zero
padding of the release segment to ensure the release segments are
compared with the same length.

"Whether or not strict version matching is appropriate depends on the
specific use case for the version specifier. Automated tools SHOULD at
least issue warnings and MAY reject them entirely when strict version
matches are used inappropriately.

"Prefix matching may be requested instead of strict comparison, by
appending a trailing .* to the version identifier in the version
matching clause."

--Chris




>
> On Mon, Dec 22, 2014 at 1:31 PM, Chris Jerdonek <chris.jerdonek at gmail.com>
> wrote:
>>
>> On Mon, Dec 22, 2014 at 1:03 PM, Marcus Smith <qwcode at gmail.com> wrote:
>> >
>> > This makes sense if you view "1.7" as encompassing all of the releases
>> >>
>> >> 1.7.0, 1.7.1, 1.7.2, etc. (and their variants).  It's a bit like the
>> >> "significant digits" of a number. [1]  You want the release to be
>> >> bigger than all releases of that type.
>> >
>> >
>> > but is this really the underlying concept?
>> > Does the PEP explain this as a justification? Where?
>>
>> I think so.  I just read that part of the PEP.
>>
>> In the section "Exclusive ordered comparison," it says, "the
>> comparison operators are < and > and the clause MUST be effectively
>> interpreted as implying the prefix based version exclusion clause !=
>> V.* ."
>>
>> The key here is the reference to "prefix based" version exclusion (as
>> opposed to "strict" version matching).  If I understand correctly,
>> this says that when "1.7.2" should be compared with "1.7" for
>> equality, it should be compared with "1.7.*" as opposed to "1.7.0".
>> This is described in the "Version matching" section.
>>
>> The prefix-based ".*" notation encapsulates what I said above about
>> how  "1.7" can be viewed as encompassing all of the releases 1.7.0,
>> 1.7.1, etc.
>>
>> --Chris
>
>

From qwcode at gmail.com  Mon Dec 22 23:57:32 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 14:57:32 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAOTb1weMic3F-3QqZo2POQygKx_kWFkLefJWq==VyWty5AQjuw@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAPYWazp19sc0qBg3F=MMb8o1Hi2Sxku9iVyBom=r_QkfU0msEQ@mail.gmail.com>
 <CAOTb1wfde8nmLE55vivGOk9+z0Qsq1N+4aOYvG41HyVJfJSu1Q@mail.gmail.com>
 <CAPYWazobjNt4x3DGxLgkn2bGR3x8AE5W0HVtz6tP48r8fDKf1A@mail.gmail.com>
 <CAOTb1weMic3F-3QqZo2POQygKx_kWFkLefJWq==VyWty5AQjuw@mail.gmail.com>
Message-ID: <CAPYWazpXDZhnsa0r1A+G_zED5KAmXjC78oHmjyv6RR+uuJVT0A@mail.gmail.com>

> By the way, note that "1.7" doesn't *always* mean "1.7.*"


right, got it
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/00f35648/attachment.html>

From qwcode at gmail.com  Mon Dec 22 23:38:29 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 14:38:29 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAOTb1wfde8nmLE55vivGOk9+z0Qsq1N+4aOYvG41HyVJfJSu1Q@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAPYWazp19sc0qBg3F=MMb8o1Hi2Sxku9iVyBom=r_QkfU0msEQ@mail.gmail.com>
 <CAOTb1wfde8nmLE55vivGOk9+z0Qsq1N+4aOYvG41HyVJfJSu1Q@mail.gmail.com>
Message-ID: <CAPYWazobjNt4x3DGxLgkn2bGR3x8AE5W0HVtz6tP48r8fDKf1A@mail.gmail.com>

there's a big difference between stating the concept directly/simply in the
PEP vs readers/users having to reverse engineer it from "prefix based
version exclusion clause !=
V.* ."

If this is what it's about, users need to be clearly informed (in the pip
and PyPUG docs) that "1.7" effectively means "1.7.*" in specifiers.

On Mon, Dec 22, 2014 at 1:31 PM, Chris Jerdonek <chris.jerdonek at gmail.com>
wrote:

> On Mon, Dec 22, 2014 at 1:03 PM, Marcus Smith <qwcode at gmail.com> wrote:
> >
> > This makes sense if you view "1.7" as encompassing all of the releases
> >>
> >> 1.7.0, 1.7.1, 1.7.2, etc. (and their variants).  It's a bit like the
> >> "significant digits" of a number. [1]  You want the release to be
> >> bigger than all releases of that type.
> >
> >
> > but is this really the underlying concept?
> > Does the PEP explain this as a justification? Where?
>
> I think so.  I just read that part of the PEP.
>
> In the section "Exclusive ordered comparison," it says, "the
> comparison operators are < and > and the clause MUST be effectively
> interpreted as implying the prefix based version exclusion clause !=
> V.* ."
>
> The key here is the reference to "prefix based" version exclusion (as
> opposed to "strict" version matching).  If I understand correctly,
> this says that when "1.7.2" should be compared with "1.7" for
> equality, it should be compared with "1.7.*" as opposed to "1.7.0".
> This is described in the "Version matching" section.
>
> The prefix-based ".*" notation encapsulates what I said above about
> how  "1.7" can be viewed as encompassing all of the releases 1.7.0,
> 1.7.1, etc.
>
> --Chris
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/a4b7775f/attachment.html>

From qwcode at gmail.com  Tue Dec 23 06:57:37 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 21:57:37 -0800
Subject: [Distutils] PEP440 examples missing operator?
Message-ID: <CAPYWazrjs=-00fZ5uaDwSzRsezb6UzefCjeA4sUhwNT3YPa6zQ@mail.gmail.com>

the first 3 examples here are confusing.
https://www.python.org/dev/peps/pep-0440/#examples

there's no comparison operator?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/da247b5a/attachment.html>

From qwcode at gmail.com  Tue Dec 23 07:03:53 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 22:03:53 -0800
Subject: [Distutils] PEP440 examples missing operator?
In-Reply-To: <CAPYWazrjs=-00fZ5uaDwSzRsezb6UzefCjeA4sUhwNT3YPa6zQ@mail.gmail.com>
References: <CAPYWazrjs=-00fZ5uaDwSzRsezb6UzefCjeA4sUhwNT3YPa6zQ@mail.gmail.com>
Message-ID: <CAPYWazrr9GMiyfHaM0yC0pfbQq2_N20weCsGmcYsdn8bNZ4ZWg@mail.gmail.com>

oh, "A compatible release clause consists of either a version identifier
without any comparison operator or else the compatible release operator ~="

On Mon, Dec 22, 2014 at 9:57 PM, Marcus Smith <qwcode at gmail.com> wrote:

> the first 3 examples here are confusing.
> https://www.python.org/dev/peps/pep-0440/#examples
>
> there's no comparison operator?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/bd6e0467/attachment.html>

From qwcode at gmail.com  Tue Dec 23 07:23:46 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Dec 2014 22:23:46 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
Message-ID: <CAPYWazoGj9JrL0H+NEzgP4LTB6qv7Mkwh__SYdDz4JmyamHCNA@mail.gmail.com>

>
>
> In particular, <, >, ~=, and, when using a .*, the != and == use the
> number of dots in the given specifier to indicate the precision of the
> specifier.
>

the PEP text is pretty clear on the precision concept for ~= and when using
".*", but not so much for < and <.
how about an example right there in that section that mentions the "!= V.*"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/ed4be73f/attachment-0001.html>

From tshepang at gmail.com  Tue Dec 23 09:44:37 2014
From: tshepang at gmail.com (Tshepang Lekhonkhobe)
Date: Tue, 23 Dec 2014 10:44:37 +0200
Subject: [Distutils] Release: pip 6.0 and virtualenv 12.0
In-Reply-To: <1924C59B-C450-4F3E-8AA9-419CD4545C1E@stufft.io>
References: <1924C59B-C450-4F3E-8AA9-419CD4545C1E@stufft.io>
Message-ID: <CAA77j2AaMyfEFTT7dGy=eM9Jcj6fQAmc6c=0siJgPPteQ_ptBw@mail.gmail.com>

Why the jump from 1.5.6 to 6.0?

From chris.jerdonek at gmail.com  Tue Dec 23 09:55:25 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Tue, 23 Dec 2014 00:55:25 -0800
Subject: [Distutils] Release: pip 6.0 and virtualenv 12.0
In-Reply-To: <CAA77j2AaMyfEFTT7dGy=eM9Jcj6fQAmc6c=0siJgPPteQ_ptBw@mail.gmail.com>
References: <1924C59B-C450-4F3E-8AA9-419CD4545C1E@stufft.io>
 <CAA77j2AaMyfEFTT7dGy=eM9Jcj6fQAmc6c=0siJgPPteQ_ptBw@mail.gmail.com>
Message-ID: <CAOTb1wfJC=Kv2XNRWncpyecSCYbG3se1fJK_H7ZFb0G_ZywnWA@mail.gmail.com>

On Tue, Dec 23, 2014 at 12:44 AM, Tshepang Lekhonkhobe
<tshepang at gmail.com> wrote:
> Why the jump from 1.5.6 to 6.0?

The release notes say (under "6.0"):

"PROCESS Version numbers are now simply X.Y where the leading 1 has
been dropped."

(from https://pip.pypa.io/en/latest/news.html )

I'm guessing this is to start honoring the convention (e.g. from
semver) that backwards incompatible changes should only be made when
the major version number is incremented.

--Chris



> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From donald at stufft.io  Tue Dec 23 10:01:50 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Dec 2014 04:01:50 -0500
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazoGj9JrL0H+NEzgP4LTB6qv7Mkwh__SYdDz4JmyamHCNA@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
 <CAPYWazoGj9JrL0H+NEzgP4LTB6qv7Mkwh__SYdDz4JmyamHCNA@mail.gmail.com>
Message-ID: <368BC192-BA6A-4884-8248-D955C77C8759@stufft.io>


> On Dec 23, 2014, at 1:23 AM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> 
> In particular, <, >, ~=, and, when using a .*, the != and == use the number of dots in the given specifier to indicate the precision of the specifier.
> 
> the PEP text is pretty clear on the precision concept for ~= and when using ".*", but not so much for < and <.
> how about an example right there in that section that mentions the "!= V.*"
> 


We can certainly clarify the PEP test to make it easier to understand.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/eef64699/attachment.html>

From donald at stufft.io  Tue Dec 23 10:02:24 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Dec 2014 04:02:24 -0500
Subject: [Distutils] PEP440 examples missing operator?
In-Reply-To: <CAPYWazrr9GMiyfHaM0yC0pfbQq2_N20weCsGmcYsdn8bNZ4ZWg@mail.gmail.com>
References: <CAPYWazrjs=-00fZ5uaDwSzRsezb6UzefCjeA4sUhwNT3YPa6zQ@mail.gmail.com>
 <CAPYWazrr9GMiyfHaM0yC0pfbQq2_N20weCsGmcYsdn8bNZ4ZWg@mail.gmail.com>
Message-ID: <6275B5AE-D3AB-4714-80E4-4ABC69D63993@stufft.io>


> On Dec 23, 2014, at 1:03 AM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> oh, "A compatible release clause consists of either a version identifier without any comparison operator or else the compatible release operator ~="
> 
> On Mon, Dec 22, 2014 at 9:57 PM, Marcus Smith <qwcode at gmail.com <mailto:qwcode at gmail.com>> wrote:
> the first 3 examples here are confusing.
> https://www.python.org/dev/peps/pep-0440/#examples <https://www.python.org/dev/peps/pep-0440/#examples>
> 
> there's no comparison operator?
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


Heh, that?s actually a hold over from before we made specifiers mandatory. That needs updated.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/54a415cf/attachment.html>

From donald at stufft.io  Tue Dec 23 12:37:09 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Dec 2014 06:37:09 -0500
Subject: [Distutils] Release: pip 6.0 and virtualenv 12.0
In-Reply-To: <CAOTb1wfJC=Kv2XNRWncpyecSCYbG3se1fJK_H7ZFb0G_ZywnWA@mail.gmail.com>
References: <1924C59B-C450-4F3E-8AA9-419CD4545C1E@stufft.io>
 <CAA77j2AaMyfEFTT7dGy=eM9Jcj6fQAmc6c=0siJgPPteQ_ptBw@mail.gmail.com>
 <CAOTb1wfJC=Kv2XNRWncpyecSCYbG3se1fJK_H7ZFb0G_ZywnWA@mail.gmail.com>
Message-ID: <7E3A9792-8EAE-438A-8952-A848C66B5ADD@stufft.io>


> On Dec 23, 2014, at 3:55 AM, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
> 
> On Tue, Dec 23, 2014 at 12:44 AM, Tshepang Lekhonkhobe
> <tshepang at gmail.com> wrote:
>> Why the jump from 1.5.6 to 6.0?
> 
> The release notes say (under "6.0"):
> 
> "PROCESS Version numbers are now simply X.Y where the leading 1 has
> been dropped."
> 
> (from https://pip.pypa.io/en/latest/news.html )
> 
> I'm guessing this is to start honoring the convention (e.g. from
> semver) that backwards incompatible changes should only be made when
> the major version number is incremented.
> 
> --Chris
> 
> 
> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig

It?s a mixture of things.

* We don?t generally have massive breaking releases, we tend to deprecate
  things over time and remove them a few versions later. This means that each
  1.X+1 broke something that worked in 1.X.

* There was an allure to do a big "2.0" style breakage that this completely
  negates.

* Given the first item, what to do for the version after 1.9 was a question,
  we could just do 1.10 but given we didn't want to do a big 2.0 style breakage
  and we didn't have any plans to increment the 1, we'd likely just keep doing
  1.X+1 for a very long time.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From ncoghlan at gmail.com  Tue Dec 23 12:42:44 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Dec 2014 21:42:44 +1000
Subject: [Distutils] PEP440 examples missing operator?
In-Reply-To: <6275B5AE-D3AB-4714-80E4-4ABC69D63993@stufft.io>
References: <CAPYWazrjs=-00fZ5uaDwSzRsezb6UzefCjeA4sUhwNT3YPa6zQ@mail.gmail.com>
 <CAPYWazrr9GMiyfHaM0yC0pfbQq2_N20weCsGmcYsdn8bNZ4ZWg@mail.gmail.com>
 <6275B5AE-D3AB-4714-80E4-4ABC69D63993@stufft.io>
Message-ID: <CADiSq7d5Wv60bXur70hdwu+sScWKOU0D1knRpC-MSGEFWd6ZZg@mail.gmail.com>

On 23 Dec 2014 19:02, "Donald Stufft" <donald at stufft.io> wrote:
>
>
>> On Dec 23, 2014, at 1:03 AM, Marcus Smith <qwcode at gmail.com> wrote:
>>
>> oh, "A compatible release clause consists of either a version identifier
without any comparison operator or else the compatible release operator ~="
>>
>> On Mon, Dec 22, 2014 at 9:57 PM, Marcus Smith <qwcode at gmail.com> wrote:
>>>
>>> the first 3 examples here are confusing.
>>> https://www.python.org/dev/peps/pep-0440/#examples
>>>
>>> there's no comparison operator?
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
> Heh, that?s actually a hold over from before we made specifiers
mandatory. That needs updated.

Should we add a reference to the metadata issue tracker to the PEPs
themselves? That's a better home for bug reports than the mailing list
(Note I'm also happy to move the repo for metadata drafts to GitHub if
folks would prefer that - it's a pretty simple repo, so git-hg bridges
should handle the conversion easily. The issue migration would involve a
bit more work, though).

Cheers,
Nick.

>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/90d0998d/attachment.html>

From donald at stufft.io  Tue Dec 23 12:49:18 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Dec 2014 06:49:18 -0500
Subject: [Distutils] PEP440 examples missing operator?
In-Reply-To: <CADiSq7d5Wv60bXur70hdwu+sScWKOU0D1knRpC-MSGEFWd6ZZg@mail.gmail.com>
References: <CAPYWazrjs=-00fZ5uaDwSzRsezb6UzefCjeA4sUhwNT3YPa6zQ@mail.gmail.com>
 <CAPYWazrr9GMiyfHaM0yC0pfbQq2_N20weCsGmcYsdn8bNZ4ZWg@mail.gmail.com>
 <6275B5AE-D3AB-4714-80E4-4ABC69D63993@stufft.io>
 <CADiSq7d5Wv60bXur70hdwu+sScWKOU0D1knRpC-MSGEFWd6ZZg@mail.gmail.com>
Message-ID: <380F600C-DD5F-4121-8AC8-F8E9EF45C694@stufft.io>

Well you know my preferences is for github ;). I don't mind much either way though since its a simple repo I rarely need to interact with. The main benefit would be consistency with most of the other pypa things. 

I'm +1 on adding a bug tracker link. Maybe peps should support a header for it similar to discussions-to


> On Dec 23, 2014, at 6:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> 
> On 23 Dec 2014 19:02, "Donald Stufft" <donald at stufft.io> wrote:
> >
> >
> >> On Dec 23, 2014, at 1:03 AM, Marcus Smith <qwcode at gmail.com> wrote:
> >>
> >> oh, "A compatible release clause consists of either a version identifier without any comparison operator or else the compatible release operator ~="
> >>
> >> On Mon, Dec 22, 2014 at 9:57 PM, Marcus Smith <qwcode at gmail.com> wrote:
> >>>
> >>> the first 3 examples here are confusing.
> >>> https://www.python.org/dev/peps/pep-0440/#examples
> >>>
> >>> there's no comparison operator?
> >>
> >>
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >> https://mail.python.org/mailman/listinfo/distutils-sig
> >
> > Heh, that?s actually a hold over from before we made specifiers mandatory. That needs updated.
> 
> Should we add a reference to the metadata issue tracker to the PEPs themselves? That's a better home for bug reports than the mailing list (Note I'm also happy to move the repo for metadata drafts to GitHub if folks would prefer that - it's a pretty simple repo, so git-hg bridges should handle the conversion easily. The issue migration would involve a bit more work, though).
> 
> Cheers,
> Nick.
> 
> >
> > ---
> > Donald Stufft
> > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/b80daa8b/attachment.html>

From donald at stufft.io  Tue Dec 23 13:47:59 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Dec 2014 07:47:59 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
 <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
 <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
Message-ID: <410E35B5-B509-48B6-A840-EBB5739948D4@stufft.io>


> On Dec 22, 2014, at 1:15 PM, Vladimir Diaz <vladimir.v.diaz at gmail.com> wrote:
> 
> 
> 
> On Mon, Dec 22, 2014 at 11:30 AM, Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> wrote:
> On 23 December 2014 at 01:46, Vladimir Diaz <vladimir.v.diaz at gmail.com <mailto:vladimir.v.diaz at gmail.com>> wrote:
> ::ping::
> 
> Has anyone in the community gotten a chance to review PEP 458 and/or PEP 480?  Community feedback would be appreciated.
> 
> Sorry about that Vladimir - I think the main PyPA devs have been focused on getting PEP 440 implemented, and the associated setuptools, pip and virtualenv updates out the door, and now we're into the end of year holiday period for many of us.
> 
> Considering that the previous drafts of PEP 458 generated quite a bit of feedback and questions, I was surprised by the lack of responses this time around (almost relieved :).  Receiving feedback from the main PyP developers after the holiday period is definitely not a problem.
> 

This is why I haven?t looked at it yet.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/f9aa800f/attachment-0001.html>

From geoff at electron.me.uk  Mon Dec 22 17:26:16 2014
From: geoff at electron.me.uk (Geoff Clements)
Date: Mon, 22 Dec 2014 16:26:16 +0000
Subject: [Distutils] New User: depends not appearing in control file
Message-ID: <6698128.nGAEjiey7P@scotty>

Hi!

I just started using stdeb but have run into a problem which I can't solve.

Just for background, so you know at what level to aim replies, I'm a long-time 
programmer, over 30 years in fact. I make much of what I do available to 
others but I've only just got around to packaging, so I know absolutely 
nothing about how to package code (but I'm learning!).

I'm using stdeb 0.8.2 installed via pypi because the Ubuntu repos are so 
hopelessly out of date.

I'm building a package for python3 only and uploading to an Ubuntu ppa which I 
have done but with the following problems:

Here's my stdeb.cfg file:

[DEFAULT]
Depends: libav-tools (>= 6:11), lsdvd (>= 0.17), mplayer2 (>= 2.0), python3-
pyqt4 (>= 4.11), python3-pyudev (>= 0.16)
XS-Python-Version: >= 3.4
MIME-Desktop-Files: share/ravenc.desktop
Suite: utopic
Section: universe/video

Now here is the control file in the source package that is produced (minus the 
package description, slightly redacted):

Source: ravenc
Maintainer: Geoff Clements <e-mail address>
Section: universe/video
Priority: optional
Build-Depends: python3-setuptools, python3-all, debhelper (>= 7.4.3)
Standards-Version: 3.9.1
X-Python-Version: >= 3.4

Package: python3-ravenc
Architecture: all
Depends: ${misc:Depends}, ${python3:Depends}

===========

The problem here is that the dependencies have not been carried across to the 
control file.

In addition, when I try to build a binary package I get this warning:

dpkg-gencontrol: warning: Depends field of package python3-ravenc: unknown 
substitution variable ${python3:Depends}

This also appears in the build log if a submit the source packages to Ubuntu 
to include in my ppa.
 
-- 
Geoff

From geoff at electron.me.uk  Tue Dec 23 16:09:55 2014
From: geoff at electron.me.uk (Geoff Clements)
Date: Tue, 23 Dec 2014 15:09:55 +0000
Subject: [Distutils] New User: stdeb, depends not appearing in control file
Message-ID: <1878572.a3UFMIBm3m@scotty>

Hi!

I just started using stdeb but have run into a problem which I can't solve.

Just for background, so you know at what level to aim replies, I'm a long-time 
programmer, over 30 years in fact. I make much of what I do available to 
others but I've only just got around to packaging, so I know absolutely 
nothing about how to package code (but I'm learning!).

I'm using stdeb 0.8.2 installed via pypi because the Ubuntu repos are so 
hopelessly out of date.

I'm building a package for python3 only and uploading to an Ubuntu ppa which I 
have done but with the following problems:

Here's my stdeb.cfg file:

[DEFAULT]
Depends: libav-tools (>= 6:11), lsdvd (>= 0.17), mplayer2 (>= 2.0), python3-
pyqt4 (>= 4.11), python3-pyudev (>= 0.16)
XS-Python-Version: >= 3.4
MIME-Desktop-Files: share/ravenc.desktop
Suite: utopic
Section: universe/video

Now here is the control file in the source package that is produced (minus the 
package description, slightly redacted):

Source: ravenc
Maintainer: Geoff Clements <e-mail address>
Section: universe/video
Priority: optional
Build-Depends: python3-setuptools, python3-all, debhelper (>= 7.4.3)
Standards-Version: 3.9.1
X-Python-Version: >= 3.4

Package: python3-ravenc
Architecture: all
Depends: ${misc:Depends}, ${python3:Depends}

===========

The problem here is that the dependencies have not been carried across to the 
control file.

In addition, when I try to build a binary package I get this warning:

dpkg-gencontrol: warning: Depends field of package python3-ravenc: unknown 
substitution variable ${python3:Depends}

This also appears in the build log if a submit the source packages to Ubuntu 
to include in my ppa.
 
-- 
Geoff

From ubernostrum at gmail.com  Mon Dec 22 21:36:02 2014
From: ubernostrum at gmail.com (James Bennett)
Date: Mon, 22 Dec 2014 12:36:02 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
Message-ID: <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>

So, if PyPI has foo-1.7 and foo-1.7.1, does ">1.7" just fail to find
anything installable?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141222/4600e529/attachment.html>

From graffatcolmingov at gmail.com  Tue Dec 23 18:35:42 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 23 Dec 2014 11:35:42 -0600
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
Message-ID: <CAN-Kwu099soTcbU9kTBNbwXt_6Z76mPsrzqR6mEUtH6TBVVNhQ@mail.gmail.com>

Yes but thanks to Marc it now provides a clear message as to why.

(Sent from my mobile)
On Dec 23, 2014 11:22 AM, "James Bennett" <ubernostrum at gmail.com> wrote:

> So, if PyPI has foo-1.7 and foo-1.7.1, does ">1.7" just fail to find
> anything installable?
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/8dcac71e/attachment.html>

From p.f.moore at gmail.com  Tue Dec 23 18:43:03 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 Dec 2014 17:43:03 +0000
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
 <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
Message-ID: <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>

On 22 December 2014 at 20:44, Marcus Smith <qwcode at gmail.com> wrote:
> it would fail.  you'd need  ">1.7.0"
>
> On Mon, Dec 22, 2014 at 12:36 PM, James Bennett <ubernostrum at gmail.com>
> wrote:
>>
>> So, if PyPI has foo-1.7 and foo-1.7.1, does ">1.7" just fail to find
>> anything installable?

I think the thing I'd missed, which makes this behaviour more
understandable (for me) is that you wouldn't usually get that in
reality. Projects tend to use a fixed number of digits in the version
number, so it'd likely be 1.7.0 and 1.7.1, and you'd be writing
>1.7.0.

Thinking of ">1.7" as "greater than the 1.7 series" sort of helps me as well...

Paul

From chris.barker at noaa.gov  Tue Dec 23 18:36:36 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 23 Dec 2014 09:36:36 -0800
Subject: [Distutils] Role of setuptools and eggs in "modern" distributing...
Message-ID: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>

Hi folks,

I'm trying to package up a complex system and would like to do it the
correct, modern way.

In particular, this involves a bunch of compiled extensions, as well as
dependencies on both the scientific "stack" and common Web app packages.

(can you tell I'm building a web service front-end for computational code?)

This is actually a bit of a trick, because the web development community is
generally doing a good job up supporting PyPi and pip, whereas the
complications of the scientific software tools have folks relying more on
tools like Enthought and Continuum.

So far, we've been doing mostly pip and struggling with build our own for
the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL,
and....). But at the end of all this we'd like to be able to distribute and
make it easy on end users to use our tools.

I figure we we'll do one (or both) of:
- providing a custom "wheel house" with our packages and the dependencies
that are hard to come by
- provide a binstar channel with conda packages of all the same stuff but a
totally different set of "other" packages.

At the moment, I'm working on making conda packages, which brings me to my
questions.

I'm a bit confused about the role of setuptools with pip. On the one hand,
pip depends of setuptools. On the other hand, pip supposed doesn't "do"
eggs, and prefers wheels. But I find that I keep getting eggs whether I
want them or not. IN fact, as far as I can tell, the way to get pip to
instal something from git repo is:

git+https://url_to_the_repo.git#egg=name_of_package

why isn't that "wheel=name_of_package"

and will it work if setuptools was not used in the packages setup.py???

Frankly I've generally found setuptools and eggs to be overly heavy weight
and annoying -- the only reason I use setuptools at all is that I REALLY
like develop mode -- which I can now get with "pip --editable" or  does
that give me setuptools develop mode anyway, i.e. do I need to have used
setuptools.setup for it to work?

So question one: do I need to use setuptools.setup rather than plain old
distutils.setup?

Question 2:

What about setuptools: "install_requires"

I generally like the pip requirements.txt approach. It's up to the
installation tool, not the packaging tool to mange requirements. But then
again, it does make sense to declare the requirements in setup.py. But the
issue at hand is that install_requires is doing some odd things with conda:

conda, of course, is designed to manage dependencies itself, and those are
declared in the conda build meta.yaml file. Note that conda dependencies
can have nothign to do with python -- the whole point of conda -- a conda
pacakge can depend on any other conda package, including C libs, etc.

But the issue at hand is that conda build doesn't re-invent setup.py -- but
rather you generally simple call "setup.py install" from your conda build
script. Hence th issue at hand:

Using setuptools.setup, and specifying "install_requires", then kicks in
setuptools trying to resolve dependencies I don't want it to deal with.

I read Donald's "setup.py vs requirements.txt", and I guess I get it, but
it still seems quite redundant -- I see the point of separating out
 ?abstract dependencies? and  ?concrete dependencies?. However, the nifty
solution of only putting:

--index-url https://pypi.python.org/simple/

in the requirements.txt doesn't work for complex cases. In practice, we
find we need to specify version numbers of some dependencies, and have some
go strait into a git repo, etc. So we need a messy requirements.txt file.

And, in fact, I think that's where is belongs -- the goal of the
requirements.txt file is so pip can "do the right thing" and go grab
everything you need. But, in fact, it also is quite human readable, and
serves quite well as the documentation for the "abstract dependencies" as
well.

So I think the way to go is to keep requirements in requirements.txt, and
leave them out of the setup.py.

Can I dump setuptools.setup, as well??

Sorry, this is a bit rambling and short of clear questions, but I'm trying
to get a handle on what best practices really are these days...

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/ea96bc5e/attachment-0001.html>

From graffatcolmingov at gmail.com  Tue Dec 23 18:45:53 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 23 Dec 2014 11:45:53 -0600
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
 <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
 <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
Message-ID: <CAN-Kwu1fPZND7a-6SY3wi4_DsnzDjZbjKP=+kDrPD=HbCL4o4A@mail.gmail.com>

It also provides consistency with date-based versions. And versions aren't
decimals so thinking of them like that is not exactly useful.
On Dec 23, 2014 11:43 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:

> On 22 December 2014 at 20:44, Marcus Smith <qwcode at gmail.com> wrote:
> > it would fail.  you'd need  ">1.7.0"
> >
> > On Mon, Dec 22, 2014 at 12:36 PM, James Bennett <ubernostrum at gmail.com>
> > wrote:
> >>
> >> So, if PyPI has foo-1.7 and foo-1.7.1, does ">1.7" just fail to find
> >> anything installable?
>
> I think the thing I'd missed, which makes this behaviour more
> understandable (for me) is that you wouldn't usually get that in
> reality. Projects tend to use a fixed number of digits in the version
> number, so it'd likely be 1.7.0 and 1.7.1, and you'd be writing
> >1.7.0.
>
> Thinking of ">1.7" as "greater than the 1.7 series" sort of helps me as
> well...
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/1a7716ff/attachment.html>

From donald at stufft.io  Tue Dec 23 19:37:03 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Dec 2014 13:37:03 -0500
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
Message-ID: <D71801E4-2891-4E40-AF47-B3B132926E3B@stufft.io>


> On Dec 23, 2014, at 12:36 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
> Hi folks,
> 
> I'm trying to package up a complex system and would like to do it the correct, modern way.
> 
> In particular, this involves a bunch of compiled extensions, as well as dependencies on both the scientific "stack" and common Web app packages.
> 
> (can you tell I'm building a web service front-end for computational code?)
> 
> This is actually a bit of a trick, because the web development community is generally doing a good job up supporting PyPi and pip, whereas the complications of the scientific software tools have folks relying more on tools like Enthought and Continuum.
> 
> So far, we've been doing mostly pip and struggling with build our own for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL, and....). But at the end of all this we'd like to be able to distribute and make it easy on end users to use our tools.
> 
> I figure we we'll do one (or both) of:
> - providing a custom "wheel house" with our packages and the dependencies that are hard to come by
> - provide a binstar channel with conda packages of all the same stuff but a totally different set of "other" packages.
> 
> At the moment, I'm working on making conda packages, which brings me to my questions.
> 
> I'm a bit confused about the role of setuptools with pip. On the one hand, pip depends of setuptools. On the other hand, pip supposed doesn't "do" eggs, and prefers wheels. But I find that I keep getting eggs whether I want them or not. IN fact, as far as I can tell, the way to get pip to instal something from git repo is:
> 
> git+https://url_to_the_repo.git#egg=name_of_package <https://url_to_the_repo.git/#egg=name_of_package>
> 
> why isn't that "wheel=name_of_package"
> 
> and will it work if setuptools was not used in the packages setup.py???
> 
> Frankly I've generally found setuptools and eggs to be overly heavy weight and annoying -- the only reason I use setuptools at all is that I REALLY like develop mode -- which I can now get with "pip --editable" or  does that give me setuptools develop mode anyway, i.e. do I need to have used setuptools.setup for it to work?
> 
> So question one: do I need to use setuptools.setup rather than plain old distutils.setup?
> 
> Question 2:
> 
> What about setuptools: "install_requires"
> 
> I generally like the pip requirements.txt approach. It's up to the installation tool, not the packaging tool to mange requirements. But then again, it does make sense to declare the requirements in setup.py. But the issue at hand is that install_requires is doing some odd things with conda:
> 
> conda, of course, is designed to manage dependencies itself, and those are declared in the conda build meta.yaml file. Note that conda dependencies can have nothign to do with python -- the whole point of conda -- a conda pacakge can depend on any other conda package, including C libs, etc.
> 
> But the issue at hand is that conda build doesn't re-invent setup.py -- but rather you generally simple call "setup.py install" from your conda build script. Hence th issue at hand:
> 
> Using setuptools.setup, and specifying "install_requires", then kicks in setuptools trying to resolve dependencies I don't want it to deal with.
> 
> I read Donald's "setup.py vs requirements.txt", and I guess I get it, but it still seems quite redundant -- I see the point of separating out  ?abstract dependencies? and  ?concrete dependencies?. However, the nifty solution of only putting:
> 
> --index-url https://pypi.python.org/simple/ <https://pypi.python.org/simple/>
> 
> in the requirements.txt doesn't work for complex cases. In practice, we find we need to specify version numbers of some dependencies, and have some go strait into a git repo, etc. So we need a messy requirements.txt file.
> 
> And, in fact, I think that's where is belongs -- the goal of the requirements.txt file is so pip can "do the right thing" and go grab everything you need. But, in fact, it also is quite human readable, and serves quite well as the documentation for the "abstract dependencies" as well.
> 
> So I think the way to go is to keep requirements in requirements.txt, and leave them out of the setup.py.
> 
> Can I dump setuptools.setup, as well??
> 
> Sorry, this is a bit rambling and short of clear questions, but I'm trying to get a handle on what best practices really are these days...
> 
> -Chris

I?m going to attempt to read between the lines here a little bit.

The ?egg? name is heavily overloaded in setuptools. It is used all over the place for varying sets of related but distinct concepts. The #egg= thing is one of those setuptools concepts where it used that name for something distinct but similar. Ideally it shouldn?t be #egg= or #wheel= but should be #dist= or something similar since it?s neither an egg or a Wheel and there is an open ticket in pip?s issue tracker to do that.

To make a clarification though, pip itself doesn?t depend on setuptools, it can install from Wheels without setuptools being installed on the system at all. It does however rely on setuptools to be installed if it is installing from a sdist. The reason for this is that pip uses setuptools as a build tool, so when it invokes a setup.py it?s ?building" that distribution (even if it?s just pure python it needs ?built?). However pip does some tricks so that it will always uses setuptools to build the project, regardless of if the project imports setuptools or distutils in their setup.py.

To that aim, install_requires specifies a packages dependencies as well as other metadata for that package, and requirements.txt is just a list of packages to install. The difference is subtle but a requirements.txt isn?t attached to a particular project and the rest of the metadata like name, version, etc.


---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/05562289/attachment.html>

From wichert at wiggy.net  Tue Dec 23 19:40:58 2014
From: wichert at wiggy.net (Wichert Akkerman)
Date: Tue, 23 Dec 2014 19:40:58 +0100
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
 <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
 <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
Message-ID: <6ED0129C-86DF-4F32-A842-E4ACB623F84C@wiggy.net>


> On 23 Dec 2014, at 18:43, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 22 December 2014 at 20:44, Marcus Smith <qwcode at gmail.com> wrote:
>> it would fail.  you'd need  ">1.7.0"
>> 
>> On Mon, Dec 22, 2014 at 12:36 PM, James Bennett <ubernostrum at gmail.com>
>> wrote:
>>> 
>>> So, if PyPI has foo-1.7 and foo-1.7.1, does ">1.7" just fail to find
>>> anything installable?
> 
> I think the thing I'd missed, which makes this behaviour more
> understandable (for me) is that you wouldn't usually get that in
> reality. Projects tend to use a fixed number of digits in the version
> number, so it'd likely be 1.7.0 and 1.7.1, and you'd be writing
>> 1.7.0.

For what it?s worth I usually skip a trailing .0 in my versions since it does not add any extra information. So my packages tend to do 1.6 1.6.1 1.6.2 1.7 1.7.1 etc. etc.

Wichert.


From qwcode at gmail.com  Tue Dec 23 20:17:53 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Tue, 23 Dec 2014 11:17:53 -0800
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
Message-ID: <CAPYWazrOaTZyDuc5Zjy9xfTx1xavZNN7Vex6Ewz4HXYF3-c4nw@mail.gmail.com>

> git+https://url_to_the_repo.git#egg=name_of_package
>
> why isn't that "wheel=name_of_package"
>

the "egg" part here has nothing to do with eggs.  just a vestige of another
time.
see  https://github.com/pypa/pip/issues/1265

and will it work if setuptools was not used in the packages setup.py???
>

yes, it would work.

which I can now get with "pip --editable" or  does that give me setuptools
> develop mode anyway
>

"-e" uses setuptools develop mode.


So question one: do I need to use setuptools.setup rather than plain old
> distutils.setup?
>

the main reason for setuptools is for "install_requires", which is
fundamental to pip dependency resolution.
but in general, it offers more features and it's more maintained than pure
distutils.
The standard advice is to use setuptools over distutils.

I generally like the pip requirements.txt approach. It's up to the
> installation tool, not the packaging tool to mange requirements. But then
> again, it does make sense to declare the requirements in setup.py. But the
> issue at hand is that install_requires is doing some odd things with conda:
>

The Packaging User Guide has a breakdown of install_requires vs
requirements files.
https://packaging.python.org/en/latest/technical.html#install-requires-vs-requirements-files

In brief, requirements files are usually for a whole "environment", whereas
install_requires is for a project.

"install_requires" is critical when publishing projects to PyPI.

Even if you're not publishing, install_requires is helpful to safely
attempt an upgrade of your dependencies.
Instead of installing your app from a frozen requirements file, you would
install normally in a clean environment (based on install_requires) and let
any upgrades occur, and then you can re-freeze your requirements.

But if you are doing mostly vcs installs in your requirements file, then
it's true that maintaining install_requires can be somewhat pointless,
although I would still say it's helpful to keep track of each project's
first level abstract dependencies are, even if the install_requires
declarations aren't being used.

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/0f2169fb/attachment.html>

From qwcode at gmail.com  Tue Dec 23 20:24:34 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Tue, 23 Dec 2014 11:24:34 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <368BC192-BA6A-4884-8248-D955C77C8759@stufft.io>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
 <CAPYWazoGj9JrL0H+NEzgP4LTB6qv7Mkwh__SYdDz4JmyamHCNA@mail.gmail.com>
 <368BC192-BA6A-4884-8248-D955C77C8759@stufft.io>
Message-ID: <CAPYWazrsW5VX8GZ+BoWEkjqtmwkGz4AQsqJ0u9Cy0_uDfRgbdA@mail.gmail.com>

just post edit suggestions here?

https://bitbucket.org/pypa/pypi-metadata-formats  is not up to date anymore

some other location?

On Tue, Dec 23, 2014 at 1:01 AM, Donald Stufft <donald at stufft.io> wrote:

>
> On Dec 23, 2014, at 1:23 AM, Marcus Smith <qwcode at gmail.com> wrote:
>
>
>> In particular, <, >, ~=, and, when using a .*, the != and == use the
>> number of dots in the given specifier to indicate the precision of the
>> specifier.
>>
>
> the PEP text is pretty clear on the precision concept for ~= and when
> using ".*", but not so much for < and <.
> how about an example right there in that section that mentions the "!=
> V.*"
>
>
> We can certainly clarify the PEP test to make it easier to understand.
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/a288ae29/attachment.html>

From qwcode at gmail.com  Tue Dec 23 20:36:15 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Tue, 23 Dec 2014 11:36:15 -0800
Subject: [Distutils] PEP440 examples missing operator?
In-Reply-To: <6275B5AE-D3AB-4714-80E4-4ABC69D63993@stufft.io>
References: <CAPYWazrjs=-00fZ5uaDwSzRsezb6UzefCjeA4sUhwNT3YPa6zQ@mail.gmail.com>
 <CAPYWazrr9GMiyfHaM0yC0pfbQq2_N20weCsGmcYsdn8bNZ4ZWg@mail.gmail.com>
 <6275B5AE-D3AB-4714-80E4-4ABC69D63993@stufft.io>
Message-ID: <CAPYWazob6+11F2RNrjBeydLjNBP83W3-ujs7BYBieXtVi3xkdw@mail.gmail.com>

>
>
> Heh, that?s actually a hold over from before we made specifiers mandatory.
> That needs updated.
>

you mean made operators mandatory?
so the bit about "version identifier without any comparison operator" needs
to be removed?


>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/c1e2ca2d/attachment.html>

From donald at stufft.io  Tue Dec 23 20:38:59 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Dec 2014 14:38:59 -0500
Subject: [Distutils] PEP440 examples missing operator?
In-Reply-To: <CAPYWazob6+11F2RNrjBeydLjNBP83W3-ujs7BYBieXtVi3xkdw@mail.gmail.com>
References: <CAPYWazrjs=-00fZ5uaDwSzRsezb6UzefCjeA4sUhwNT3YPa6zQ@mail.gmail.com>
 <CAPYWazrr9GMiyfHaM0yC0pfbQq2_N20weCsGmcYsdn8bNZ4ZWg@mail.gmail.com>
 <6275B5AE-D3AB-4714-80E4-4ABC69D63993@stufft.io>
 <CAPYWazob6+11F2RNrjBeydLjNBP83W3-ujs7BYBieXtVi3xkdw@mail.gmail.com>
Message-ID: <CF2CDFAD-6FDE-45B4-808D-81171893CD73@stufft.io>


> On Dec 23, 2014, at 2:36 PM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> 
> Heh, that?s actually a hold over from before we made specifiers mandatory. That needs updated.
> 
> you mean made operators mandatory?
> so the bit about "version identifier without any comparison operator" needs to be removed?

Yea.

Originally PEP 426 changed the syntax to be ``foo (==2.0)`` and such, and in that regards it made sense
to indicate a default operation for ``foo (2.0)``. However we removed them because:

* That syntax is backwards incompatible with the syntax that setuptools uses, and it?s sort of a pain to
  enter on the CLI.
* A default isn?t that useful, no matter what we pick it?s going to be wrong, originally PEP 386 had it
  equal to == (but a weird == that doesn?t make sense to me), then PEP 440 originally changed it to
  ~= but that doesn?t make sense for date based releases which would be better implemented as
  >= or >.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/3f7644fb/attachment-0001.html>

From ncoghlan at gmail.com  Wed Dec 24 16:14:53 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Dec 2014 01:14:53 +1000
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazrsW5VX8GZ+BoWEkjqtmwkGz4AQsqJ0u9Cy0_uDfRgbdA@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
 <CAPYWazoGj9JrL0H+NEzgP4LTB6qv7Mkwh__SYdDz4JmyamHCNA@mail.gmail.com>
 <368BC192-BA6A-4884-8248-D955C77C8759@stufft.io>
 <CAPYWazrsW5VX8GZ+BoWEkjqtmwkGz4AQsqJ0u9Cy0_uDfRgbdA@mail.gmail.com>
Message-ID: <CADiSq7eh0=Px8e6ox48nFY0=Eb_NVMxSmjm1rZOnce-MwR3uug@mail.gmail.com>

On 24 Dec 2014 05:27, "Marcus Smith" <qwcode at gmail.com> wrote:
>
> just post edit suggestions here?
>
> https://bitbucket.org/pypa/pypi-metadata-formats  is not up to date
anymore
>
> some other location?

It *should* be being kept in sync with the published versions, but Donald
and I have just been working directly in the main PEP repo recently.

Hence my suggestion of moving it to GitHub - it's more likely to be kept up
to date there, and, unlike the master PEP repo, I'm happy to host the
working drafts and issue tracker for the metadata PEPs wherever folks find
them most convenient to work with.

Cheers,
Nick.

>
> On Tue, Dec 23, 2014 at 1:01 AM, Donald Stufft <donald at stufft.io> wrote:
>>
>>
>>> On Dec 23, 2014, at 1:23 AM, Marcus Smith <qwcode at gmail.com> wrote:
>>>
>>>>
>>>> In particular, <, >, ~=, and, when using a .*, the != and == use the
number of dots in the given specifier to indicate the precision of the
specifier.
>>>
>>>
>>> the PEP text is pretty clear on the precision concept for ~= and when
using ".*", but not so much for < and <.
>>> how about an example right there in that section that mentions the "!=
V.*"
>>>
>>
>> We can certainly clarify the PEP test to make it easier to understand.
>>
>> ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141225/5ffb198e/attachment.html>

From qwcode at gmail.com  Wed Dec 24 18:25:08 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 24 Dec 2014 09:25:08 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CADiSq7eh0=Px8e6ox48nFY0=Eb_NVMxSmjm1rZOnce-MwR3uug@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
 <CAPYWazoGj9JrL0H+NEzgP4LTB6qv7Mkwh__SYdDz4JmyamHCNA@mail.gmail.com>
 <368BC192-BA6A-4884-8248-D955C77C8759@stufft.io>
 <CAPYWazrsW5VX8GZ+BoWEkjqtmwkGz4AQsqJ0u9Cy0_uDfRgbdA@mail.gmail.com>
 <CADiSq7eh0=Px8e6ox48nFY0=Eb_NVMxSmjm1rZOnce-MwR3uug@mail.gmail.com>
Message-ID: <CAPYWazohWysHLMbTB9pQsdPCWa6kZ1Vw_3-xpm83fwwJWXSkUw@mail.gmail.com>

> It *should* be being kept in sync with the published versions, but Donald
> and I have just been working directly in the main PEP repo recently.
>
> Hence my suggestion of moving it to GitHub - it's more likely to be kept
> up to date there, and, unlike the master PEP repo, I'm happy to host the
> working drafts and issue tracker for the metadata PEPs wherever folks find
> them most convenient to work with.
>
I'd suggest pypa/peps on github (i.e. a more general name than
pypi-metadata-formats)
and ideally change the name of the pep src file after it has a pep number.
(i.e. from versioning.rst to pep0440.rst)
also, maybe isolate the named drafts (with no number) in a "drafts" folder.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141224/a4ca85ab/attachment.html>

From chris.barker at noaa.gov  Tue Dec 23 22:04:47 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 23 Dec 2014 13:04:47 -0800
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <D71801E4-2891-4E40-AF47-B3B132926E3B@stufft.io>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
 <D71801E4-2891-4E40-AF47-B3B132926E3B@stufft.io>
Message-ID: <CALGmxEJbw2c2adSua92Y0P_TPYM6uQYEJSap68er4vgEswvfNg@mail.gmail.com>

On Tue, Dec 23, 2014 at 10:37 AM, Donald Stufft <donald at stufft.io> wrote:

> I?m going to attempt to read between the lines here a little bit.
>

Thank you -- you did an excellent job of capturing the gestalt of my
confusion !

The ?egg? name is heavily overloaded in setuptools. It is used all over the
> place for varying sets of related but distinct concepts. The #egg= thing is
> one of those setuptools concepts where it used that name for something
> distinct but similar. Ideally it shouldn?t be #egg= or #wheel= but should
> be #dist= or something similar since it?s neither an egg or a Wheel and
> there is an open ticket in pip?s issue tracker to do that.
>

OK, that clears it up.

Though I still get egg-info files all over the place -- not sure why that
annoys me ;-)


> To make a clarification though, pip itself doesn?t depend on setuptools,
> it can install from Wheels without setuptools being installed on the system
> at all. It does however rely on setuptools to be installed if it is
> installing from a sdist. The reason for this is that pip uses setuptools as
> a build tool, so when it invokes a setup.py it?s ?building" that
> distribution (even if it?s just pure python it needs ?built?). However pip
> does some tricks so that it will always uses setuptools to build the
> project, regardless of if the project imports setuptools or distutils in
> their setup.py.
>

Ah -- so pip needs to use setuptools to build, but a package doesn't have
to explicitly use it in its setup.py.

To that aim, install_requires specifies a packages dependencies as well as
> other metadata for that package, and requirements.txt is just a list of
> packages to install. The difference is subtle but a requirements.txt isn?t
> attached to a particular project and the rest of the metadata like name,
> version, etc.
>

hmm...I agree, but often shipped alongside setup.py -- kind of like the
fact that the name "setup.py" is a conventions rather than a spec, but
expected all over the place.


On Tue, Dec 23, 2014 at 11:17 AM, Marcus Smith <qwcode at gmail.com> wrote:

which I can now get with "pip --editable" or  does that give me setuptools
> develop mode anyway
>

"-e" uses setuptools develop mode.

OK -- though it sounds like pip would do that whether or not I used
setuptools in the setup.py.

the main reason for setuptools is for "install_requires", which is
> fundamental to pip dependency resolution.
> but in general, it offers more features and it's more maintained than pure
> distutils.
> The standard advice is to use setuptools over distutils.


OK -- still not clear how install_requires plays with conda - but it's
common enough that I think conda simply  ignores it (though not silently)

The Packaging User Guide has a breakdown of install_requires vs
> requirements files.
>
> https://packaging.python.org/en/latest/technical.html#install-requires-vs-requirements-files



>
> In brief, requirements files are usually for a whole "environment",
> whereas install_requires is for a project.
>

A note about terminology here (both in this email and The Packaging User
Guide) -- it seems to me that install_requires is about requirements for a
"package" not a "project", and that, in fact, requirements.txt is best used
for "projects".

I guess the distinction may be that a "package" has a setup.py, whereas a
project is somethign you are building that requires perhaps a stack of
unrelated packages.  So you can say : if you want to run my application,
run:

pip install -r requirements.txt

first.

"install_requires" is critical when publishing projects to PyPI.


Good to know -- I may need to go there art some point.

So I'll go with setuptools and install_requires, and see how it all goes.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/e6276a91/attachment-0001.html>

From qwcode at gmail.com  Wed Dec 24 21:50:52 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 24 Dec 2014 12:50:52 -0800
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <CALGmxEJbw2c2adSua92Y0P_TPYM6uQYEJSap68er4vgEswvfNg@mail.gmail.com>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
 <D71801E4-2891-4E40-AF47-B3B132926E3B@stufft.io>
 <CALGmxEJbw2c2adSua92Y0P_TPYM6uQYEJSap68er4vgEswvfNg@mail.gmail.com>
Message-ID: <CAPYWazrD6aPJicEeP3fh7YMa-P89m7FKkR24mV5tGK3NzD+FtQ@mail.gmail.com>

>
> A note about terminology here (both in this email and The Packaging User
> Guide) -- it seems to me that install_requires is about requirements for a
> "package" not a "project",
>

well, read through the PyPUG glossary:
https://packaging.python.org/en/latest/glossary.html

a "project" is anything with an associated setup.py (which will often have
"install_requires" metadata)

a "package" (listed as a "distribution package" in the glossary) is the
distribution of a certain "release" of a "project"



> I guess the distinction may be that a "package" has a setup.py, whereas a
> project is somethign you are building that requires perhaps a stack of
> unrelated packages.
>


Above, I used the word "environment", which was just short hand for the
whole set of installed packages on the Python path for the interpreter used
by your application.   This is often literally a "virtual environment"
created by virtualenv.

To me, the distinction is over which project *owns* the whole environment,
i.e  what is the top-level project that the environment exists for.

Requirements files are typically associated with the project that owns the
environment.


Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141224/e182f291/attachment.html>

From ncoghlan at gmail.com  Thu Dec 25 15:23:48 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Dec 2014 00:23:48 +1000
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazohWysHLMbTB9pQsdPCWa6kZ1Vw_3-xpm83fwwJWXSkUw@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <C7ED9559-CE31-4B04-AFB1-BF6B898207F9@stufft.io>
 <CAPYWazoGj9JrL0H+NEzgP4LTB6qv7Mkwh__SYdDz4JmyamHCNA@mail.gmail.com>
 <368BC192-BA6A-4884-8248-D955C77C8759@stufft.io>
 <CAPYWazrsW5VX8GZ+BoWEkjqtmwkGz4AQsqJ0u9Cy0_uDfRgbdA@mail.gmail.com>
 <CADiSq7eh0=Px8e6ox48nFY0=Eb_NVMxSmjm1rZOnce-MwR3uug@mail.gmail.com>
 <CAPYWazohWysHLMbTB9pQsdPCWa6kZ1Vw_3-xpm83fwwJWXSkUw@mail.gmail.com>
Message-ID: <CADiSq7dYRBqkYVxBvgqkdru+vpFZOg8Wo0jxGr8zgBFouM05Gw@mail.gmail.com>

On 25 Dec 2014 03:25, "Marcus Smith" <qwcode at gmail.com> wrote:
>
>
>> It *should* be being kept in sync with the published versions, but
Donald and I have just been working directly in the main PEP repo recently.
>>
>> Hence my suggestion of moving it to GitHub - it's more likely to be kept
up to date there, and, unlike the master PEP repo, I'm happy to host the
working drafts and issue tracker for the metadata PEPs wherever folks find
them most convenient to work with.
>
> I'd suggest pypa/peps on github (i.e. a more general name than
pypi-metadata-formats)
> and ideally change the name of the pep src file after it has a pep
number. (i.e. from versioning.rst to pep0440.rst)
> also, maybe isolate the named drafts (with no number) in a "drafts"
folder.

Perhaps a "published" subfolder for snapshots of the released versions? The
reason I went with the named drafts (rather than numbered) is because full
spec revisions will get a new PEP number, but I figured it would be
desirable to have a full version history under source control (rather than
having to chase it across multiple files).

Cheers,
Nick.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141226/2c5e2a8d/attachment.html>

From ncoghlan at gmail.com  Thu Dec 25 15:34:01 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Dec 2014 00:34:01 +1000
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <CAPYWazrD6aPJicEeP3fh7YMa-P89m7FKkR24mV5tGK3NzD+FtQ@mail.gmail.com>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
 <D71801E4-2891-4E40-AF47-B3B132926E3B@stufft.io>
 <CALGmxEJbw2c2adSua92Y0P_TPYM6uQYEJSap68er4vgEswvfNg@mail.gmail.com>
 <CAPYWazrD6aPJicEeP3fh7YMa-P89m7FKkR24mV5tGK3NzD+FtQ@mail.gmail.com>
Message-ID: <CADiSq7eFR7jtDydLOxXRhn0MoG-FjU8_xj6H4ssa9uPMWVi6Yg@mail.gmail.com>

On 25 Dec 2014 06:51, "Marcus Smith" <qwcode at gmail.com> wrote:
>
>
>
> Above, I used the word "environment", which was just short hand for the
whole set of installed packages on the Python path for the interpreter used
by your application.   This is often literally a "virtual environment"
created by virtualenv.
>
> To me, the distinction is over which project *owns* the whole
environment, i.e  what is the top-level project that the environment exists
for.
>
> Requirements files are typically associated with the project that owns
the environment.

>From my perspective, it's mainly a question of "Who is responsible for
defining this metadata?".

setup.py -> always the project publisher (and getting too specific annoys
system integrators)
requirements.txt -> always the system integrator (and you can be as
specific as you like)

Web applications just blur the line a lot, as the publisher and integrator
are often the same person or group. For integration into Linux distros and
other larger systems though, we prefer the first kind of metadata, as
that's usually a bit more lenient on the acceptable versions of
dependencies.

Cheers,
Nick.

>
>
> Marcus
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141226/3cd33004/attachment.html>

From jollychang at gmail.com  Fri Dec 26 04:21:17 2014
From: jollychang at gmail.com (William Zhang)
Date: Fri, 26 Dec 2014 11:21:17 +0800
Subject: [Distutils] Can zc.buildout use same comparison operator for
	versions as setup.py
Message-ID: <CAJikR8+76_MK89VUx6eNaBjqndWAqTO=T+KThCY+aJm6DaGpAw@mail.gmail.com>

Hello
Can zc.buildout use same comparison operator for  versions as  setup.py?

*buildout*
It use "show-picked-versions" for  print verison
https://pypi.python.org/pypi/zc.buildout/2.3.1#easier-reporting-and-managing-of-versions-new-in-buildout-2-0

[versions]
buildout-versions = 1.2
setuptools = 0.6c11
zc.buildout = 1.5.0

*setup.py*
It use install_requires for handle package's version
https://docs.python.org/2/distutils/setupscript.html#relationships-between-distributions-and-packages

install_requires=[
  'django == 1.4',
  'South == 0.7.5',
]

Is there any way for "show-picked-versions" print version be used by
setup.py install_requires?

Thanks
William
--------------------
This is the life!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141226/e25f9c48/attachment.html>

From qwcode at gmail.com  Fri Dec 26 22:53:58 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 26 Dec 2014 13:53:58 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
 <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
 <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
Message-ID: <CAPYWazpXEP6NSAFRxt=pU_bB1tCqt=BLM-KUJOogDMrKeP_6ig@mail.gmail.com>

Thinking of ">1.7" as "greater than the 1.7 series" sort of helps me as
> well...
>
>

the problem with thinking of it this way is that you naturally want to
extend the concept to >=, but it doesn't work.
If the concept were consistent,  1.7.dev1 would satisfy  >=1.7, but it
doesn't.
for >=, the concept seems to be zero-padding.
for >, the concept is the series wildcard idea.

this inconsistency concerns me, and makes it hard to edit the PEP to be
more understandable like I would have wanted.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141226/0be5ec47/attachment.html>

From tseaver at palladion.com  Fri Dec 26 23:22:52 2014
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 26 Dec 2014 17:22:52 -0500
Subject: [Distutils] Can zc.buildout use same comparison operator for
 versions as setup.py
In-Reply-To: <CAJikR8+76_MK89VUx6eNaBjqndWAqTO=T+KThCY+aJm6DaGpAw@mail.gmail.com>
References: <CAJikR8+76_MK89VUx6eNaBjqndWAqTO=T+KThCY+aJm6DaGpAw@mail.gmail.com>
Message-ID: <m7kn3t$rg8$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/25/2014 10:21 PM, William Zhang wrote:
> Hello Can zc.buildout use same comparison operator for  versions as
> setup.py?
> 
> *buildout* It use "show-picked-versions" for  print verison 
> https://pypi.python.org/pypi/zc.buildout/2.3.1#easier-reporting-and-managing-of-versions-new-in-buildout-2-0
>
>  [versions] buildout-versions = 1.2 setuptools = 0.6c11 zc.buildout =
> 1.5.0
> 
> *setup.py* It use install_requires for handle package's version 
> https://docs.python.org/2/distutils/setupscript.html#relationships-between-distributions-and-packages
>
>  install_requires=[ 'django == 1.4', 'South == 0.7.5', ]
> 
> Is there any way for "show-picked-versions" print version be used by 
> setup.py install_requires?

zc.buildout uses a configparser/ConfigParser-based config file, which
mandates the single equal sign.  It also doesn't support ranges or other
(new) PEP 440 syntax for versions ('~=', etc.).


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAlSd37wACgkQ+gerLs4ltQ7x7gCePOrTgsoB6fn3kVw7Vlto4t9e
WtMAoKUCK4IVnNjZBaaPhvy7u6C3nIfD
=26gz
-----END PGP SIGNATURE-----


From fred at fdrake.net  Sat Dec 27 02:07:20 2014
From: fred at fdrake.net (Fred Drake)
Date: Fri, 26 Dec 2014 20:07:20 -0500
Subject: [Distutils] Can zc.buildout use same comparison operator for
 versions as setup.py
In-Reply-To: <m7kn3t$rg8$1@ger.gmane.org>
References: <CAJikR8+76_MK89VUx6eNaBjqndWAqTO=T+KThCY+aJm6DaGpAw@mail.gmail.com>
 <m7kn3t$rg8$1@ger.gmane.org>
Message-ID: <CAFT4OTGnSavaFOM9DF_Eyyy1mdRG+muaXZ4sHdzm9yrMZs=SdA@mail.gmail.com>

On Fri, Dec 26, 2014 at 5:22 PM, Tres Seaver <tseaver at palladion.com> wrote:
> zc.buildout uses a configparser/ConfigParser-based config file, which
> mandates the single equal sign.  It also doesn't support ranges or other
> (new) PEP 440 syntax for versions ('~=', etc.).

It may be worth noting the intent of specifying versions in a buildout
config and a version constraint for a software package in setup.py is
different.

Constraints in setup.py are intended to specify what versions a
package is compatible with; they're about the code in the package.
Versions in a buildout config (sometimes called "nails") are about
getting a repeatable build: *this* build should use exactly *this*
version.

I hope that helps.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From chris.jerdonek at gmail.com  Sat Dec 27 06:38:50 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sat, 27 Dec 2014 00:38:50 -0500
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazpXEP6NSAFRxt=pU_bB1tCqt=BLM-KUJOogDMrKeP_6ig@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
 <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
 <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
 <CAPYWazpXEP6NSAFRxt=pU_bB1tCqt=BLM-KUJOogDMrKeP_6ig@mail.gmail.com>
Message-ID: <CAOTb1wd2=DGchPpNPPh0b7J-ZkzcgAqqvSS5rcr1-oUeXTDdDg@mail.gmail.com>

On Fri, Dec 26, 2014 at 4:53 PM, Marcus Smith <qwcode at gmail.com> wrote:
>
>
>> Thinking of ">1.7" as "greater than the 1.7 series" sort of helps me as
>> well...
>>
>
>
> the problem with thinking of it this way is that you naturally want to
> extend the concept to >=, but it doesn't work.
> If the concept were consistent,  1.7.dev1 would satisfy  >=1.7, but it
> doesn't.

I'm pretty sure it's consistent.  For example, "1.7.2" doesn't satisfy
">1.7", but it satisfies ">=1.7" (e.g. because it's "part of the
series").

I believe the inconsistency you're mentioning doesn't have anything to
do with the comparison operator.  It's that pre-releases like "dev1"
are special cases and governed by different rules.

For example, from the "Handling of pre-releases" section,
"Pre-releases of any kind, including developmental releases, are
implicitly excluded from all version specifiers, unless they are
already present on the system, explicitly requested by the user, or if
the only available version that satisfies the version specifier is a
pre-release."  And then it goes on.

--Chris






> for >=, the concept seems to be zero-padding.
> for >, the concept is the series wildcard idea.
>
> this inconsistency concerns me, and makes it hard to edit the PEP to be more
> understandable like I would have wanted.
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From qwcode at gmail.com  Sat Dec 27 08:13:48 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 26 Dec 2014 23:13:48 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAOTb1wd2=DGchPpNPPh0b7J-ZkzcgAqqvSS5rcr1-oUeXTDdDg@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
 <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
 <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
 <CAPYWazpXEP6NSAFRxt=pU_bB1tCqt=BLM-KUJOogDMrKeP_6ig@mail.gmail.com>
 <CAOTb1wd2=DGchPpNPPh0b7J-ZkzcgAqqvSS5rcr1-oUeXTDdDg@mail.gmail.com>
Message-ID: <CAPYWazoZ8u_9ntb24PkPHPWi0CVSLFD=1=iA5_opgxzXkQmB9Q@mail.gmail.com>

> > the problem with thinking of it this way is that you naturally want to
> > extend the concept to >=, but it doesn't work.
> > If the concept were consistent,  1.7.dev1 would satisfy  >=1.7, but it
> > doesn't.
>
> I'm pretty sure it's consistent.  For example, "1.7.2" doesn't satisfy
> ">1.7", but it satisfies ">=1.7" (e.g. because it's "part of the
> series").
>
> I believe the inconsistency you're mentioning doesn't have anything to
> do with the comparison operator.  It's that pre-releases like "dev1"
> are special cases and governed by different rules. [...] are
> implicitly excluded


but look at this (using setuptools 8)

>>> '1.7.dev1' in pkg_resources.Requirement.parse('foo>=1.7')
False
>>> '1.7.dev1' in pkg_resources.Requirement.parse('foo<=1.7')
True

Only the exclusive ordering section speaks of the "!= V.*" bit.
The inclusive section just talks of zero padding and relative ordering.

The packaging implementation that underlies all this seems to bear this out
as well:

inclusive:
https://github.com/pypa/packaging/blob/master/packaging/specifiers.py#L453
exclusive:
https://github.com/pypa/packaging/blob/master/packaging/specifiers.py#L461
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141226/3158eba4/attachment-0001.html>

From jim at zope.com  Sat Dec 27 18:16:40 2014
From: jim at zope.com (Jim Fulton)
Date: Sat, 27 Dec 2014 12:16:40 -0500
Subject: [Distutils] setuptools-8.4.zip?
Message-ID: <CAPDm-FgoBXuRG+h5CpQr-vdk=B-gt119ZQLoDJQ-avmjBLyKdA@mail.gmail.com>

Bootstrapping buildout is failing because buildout uses
https://bootstrap.pypa.io/ez_setup.py
which tried to download:

  https://pypi.python.org/packages/source/s/setuptools/setuptools-8.4.zip

which doesn't exist.

Did someone forget to upload setuptools-8.4.zip?

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From alex.gaynor at gmail.com  Sat Dec 27 19:47:06 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 27 Dec 2014 18:47:06 +0000 (UTC)
Subject: [Distutils] setuptools-8.4.zip?
References: <CAPDm-FgoBXuRG+h5CpQr-vdk=B-gt119ZQLoDJQ-avmjBLyKdA@mail.gmail.com>
Message-ID: <loom.20141227T194553-68@post.gmane.org>

Jim Fulton <jim <at> zope.com> writes:

> 
> Did someone forget to upload setuptools-8.4.zip?
> 
> Jim
> 


I don't really understand how it came about, but the sever was on the wrong
branch or something. Benjamin Peterson whacked it with a stick and now it
should be good.

Alex


From jim at zope.com  Sat Dec 27 23:00:35 2014
From: jim at zope.com (Jim Fulton)
Date: Sat, 27 Dec 2014 17:00:35 -0500
Subject: [Distutils] setuptools-8.4.zip?
In-Reply-To: <loom.20141227T194553-68@post.gmane.org>
References: <CAPDm-FgoBXuRG+h5CpQr-vdk=B-gt119ZQLoDJQ-avmjBLyKdA@mail.gmail.com>
 <loom.20141227T194553-68@post.gmane.org>
Message-ID: <CAPDm-FjYSnnwxPJ0_e-0YU=nEJ4hgGP-wGNMRF=qNFNDL1ZORg@mail.gmail.com>

On Sat, Dec 27, 2014 at 1:47 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> Jim Fulton <jim <at> zope.com> writes:
>
>>
>> Did someone forget to upload setuptools-8.4.zip?
>>
>> Jim
>>
>
>
> I don't really understand how it came about, but the sever was on the wrong
> branch or something. Benjamin Peterson whacked it with a stick and now it
> should be good.

Thanks.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From chris.jerdonek at gmail.com  Sun Dec 28 00:38:50 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sat, 27 Dec 2014 18:38:50 -0500
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAPYWazoZ8u_9ntb24PkPHPWi0CVSLFD=1=iA5_opgxzXkQmB9Q@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
 <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
 <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
 <CAPYWazpXEP6NSAFRxt=pU_bB1tCqt=BLM-KUJOogDMrKeP_6ig@mail.gmail.com>
 <CAOTb1wd2=DGchPpNPPh0b7J-ZkzcgAqqvSS5rcr1-oUeXTDdDg@mail.gmail.com>
 <CAPYWazoZ8u_9ntb24PkPHPWi0CVSLFD=1=iA5_opgxzXkQmB9Q@mail.gmail.com>
Message-ID: <CAOTb1weG0pQ2vqQTfJgS26LuCAgNcfwaP9F9DOPh-ksKnTpspw@mail.gmail.com>

On Sat, Dec 27, 2014 at 2:13 AM, Marcus Smith <qwcode at gmail.com> wrote:
>
>> > the problem with thinking of it this way is that you naturally want to
>> > extend the concept to >=, but it doesn't work.
>> > If the concept were consistent,  1.7.dev1 would satisfy  >=1.7, but it
>> > doesn't.
>>
>> I'm pretty sure it's consistent.  For example, "1.7.2" doesn't satisfy
>> ">1.7", but it satisfies ">=1.7" (e.g. because it's "part of the
>> series").
>>
>> I believe the inconsistency you're mentioning doesn't have anything to
>> do with the comparison operator.  It's that pre-releases like "dev1"
>> are special cases and governed by different rules. [...] are
>> implicitly excluded
>
>
> but look at this (using setuptools 8)
>
>>>> '1.7.dev1' in pkg_resources.Requirement.parse('foo>=1.7')
> False
>>>> '1.7.dev1' in pkg_resources.Requirement.parse('foo<=1.7')
> True

I believe the first example is a consequence of the following two
excerpts from the PEP:

(1) From the "Developmental releases" section: "Developmental releases
are ordered by their numerical component, **immediately before the
corresponding release** [my emphasis]...."

(2) And from the "Inclusive ordered comparison" section: "An inclusive
ordered comparison clause ... will match any version where the
comparison is correct based on the relative position of the candidate
version and the specified version given the consistent ordering
defined by the standard Version scheme. ... As with version matching,
the release segment is zero padded as necessary to ensure the release
segments are compared with the same length."

Since "1.7.dev1" is before its corresponding release "1.7" in the
ordering, "1.7.dev1" does not satisfy "foo>=1.7".  Similarly,
"1.7.dev1" satisfies "foo<=1.7" since "1.7.dev1" precedes "1.7" in the
ordering.

In both cases (by (2) above), you are supposed to compare the
candidate version "1.7.dev1" with the specified version "1.7".

--Chris

>
> Only the exclusive ordering section speaks of the "!= V.*" bit.
> The inclusive section just talks of zero padding and relative ordering.
>
> The packaging implementation that underlies all this seems to bear this out
> as well:
>
> inclusive:
> https://github.com/pypa/packaging/blob/master/packaging/specifiers.py#L453
> exclusive:
> https://github.com/pypa/packaging/blob/master/packaging/specifiers.py#L461
>
>

From qwcode at gmail.com  Sun Dec 28 02:11:12 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sat, 27 Dec 2014 17:11:12 -0800
Subject: [Distutils] PEP440: foo-X.Y.Z does not satisfy "foo>X.Y"?
In-Reply-To: <CAOTb1weG0pQ2vqQTfJgS26LuCAgNcfwaP9F9DOPh-ksKnTpspw@mail.gmail.com>
References: <CAPYWazoWCBgxq-x7f=nTx6ev6FKrE7sUJhx3aX69qfNVm8kLsA@mail.gmail.com>
 <CAOTb1wdBZDbFt2cQ3Ru9iy7DthY02eDew-W3L4yaJvVL5JdLPA@mail.gmail.com>
 <CAL13Cg-o_t5h44PufGoxaFEHHY-XotV5Zwxg=wMBLcFXGtRy6A@mail.gmail.com>
 <CAPYWazo0jvbx00g3hoA9cEZX4zQfY-MrkuSw1R4CXRdD=w60yg@mail.gmail.com>
 <CACac1F-MrZZubxBerVPN5rjY3F2Q-C2U6FEmVBp1vKixw8Tf-g@mail.gmail.com>
 <CAPYWazpXEP6NSAFRxt=pU_bB1tCqt=BLM-KUJOogDMrKeP_6ig@mail.gmail.com>
 <CAOTb1wd2=DGchPpNPPh0b7J-ZkzcgAqqvSS5rcr1-oUeXTDdDg@mail.gmail.com>
 <CAPYWazoZ8u_9ntb24PkPHPWi0CVSLFD=1=iA5_opgxzXkQmB9Q@mail.gmail.com>
 <CAOTb1weG0pQ2vqQTfJgS26LuCAgNcfwaP9F9DOPh-ksKnTpspw@mail.gmail.com>
Message-ID: <CAPYWazoh5_P_J0h=kqzKzpH5fnWieK1=9JkN74kjU0p1pNOkYw@mail.gmail.com>

> > but look at this (using setuptools 8)
> >
> >>>> '1.7.dev1' in pkg_resources.Requirement.parse('foo>=1.7')
> > False
> >>>> '1.7.dev1' in pkg_resources.Requirement.parse('foo<=1.7')
> > True
>
> I believe the first example is a consequence of the following two
> excerpts from the PEP [...] In both cases (by (2) above), you

are supposed to compare the
> candidate version "1.7.dev1" with the specified version "1.7".
>
>
yes, I agree with you.  :  )
my 2 examples weren't meant to exemplify the inconsistency I was referring
to.
they were intended to be response to you quoting the bit about "implicitly
excluded"
Because the 2nd example was True, they're not implicitly excluded.

For fear that my current point is lost now (since it's changed from the
original title), I'm going to start a new thread that sums it up real
brief...

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/bdd01781/attachment.html>

From qwcode at gmail.com  Sun Dec 28 02:35:53 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sat, 27 Dec 2014 17:35:53 -0800
Subject: [Distutils] PEP440: >1.7 vs >=1.7
Message-ID: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>

I'm starting a new thread to state cleanly what my current question/concern
is...

per PEP440 as I understand it:
- for ">1.7", "1.7" means roughly the "1.7 series" or "1.7*"
- for ">=1.7", "1.7" means literally "1.7" (with zero-padding as needed)

While I understand the motivation for the "series" conception of 1.7 to
deal with  prereleases, the resulting inconsistency in meaning for "1.7" is
what concerns me.  It's odd .  Can someone make it not seem odd for "1.7 to
change meanings just because you switch between >= and >?

And for anyone who wants to say that "1.7" also means the "1.7 series" for
>=, note that 1.7 prereleases do not satisfy >=1.7

Would a solution be to make >= also use the series concept?, i.e. make
pre-releases satisfy >=1.7

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/83e37623/attachment-0001.html>

From donald at stufft.io  Sun Dec 28 02:39:26 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 27 Dec 2014 20:39:26 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
Message-ID: <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>


> On Dec 27, 2014, at 8:35 PM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> 
> I'm starting a new thread to state cleanly what my current question/concern is...
> 
> per PEP440 as I understand it:
> - for ">1.7", "1.7" means roughly the "1.7 series" or "1.7*"
> - for ">=1.7", "1.7" means literally "1.7" (with zero-padding as needed)
> 
> While I understand the motivation for the "series" conception of 1.7 to deal with  prereleases, the resulting inconsistency in meaning for "1.7" is what concerns me.  It's odd .  Can someone make it not seem odd for "1.7 to change meanings just because you switch between >= and >?
> 
> And for anyone who wants to say that "1.7" also means the "1.7 series" for >=, note that 1.7 prereleases do not satisfy >=1.7
> 
> Would a solution be to make >= also use the series concept?, i.e. make pre-releases satisfy >=1.7

The equality operator pads zeros, >= has the equality operator as part of it.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/c444d9aa/attachment.html>

From graffatcolmingov at gmail.com  Sun Dec 28 02:51:58 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sat, 27 Dec 2014 19:51:58 -0600
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
Message-ID: <CAN-Kwu2H1zEoK_eG2kHe=hPeM3+cg3rMFY_-d6ztPMW1_Xs-kw@mail.gmail.com>

On Sat, Dec 27, 2014 at 7:35 PM, Marcus Smith <qwcode at gmail.com> wrote:
>
> I'm starting a new thread to state cleanly what my current question/concern
> is...
>
> per PEP440 as I understand it:
> - for ">1.7", "1.7" means roughly the "1.7 series" or "1.7*"

You mean "1.7.*" right? Because 1.70 would satisfy >1.7

> - for ">=1.7", "1.7" means literally "1.7" (with zero-padding as needed)
>
> While I understand the motivation for the "series" conception of 1.7 to deal
> with  prereleases, the resulting inconsistency in meaning for "1.7" is what
> concerns me.  It's odd .  Can someone make it not seem odd for "1.7 to
> change meanings just because you switch between >= and >?
>
> And for anyone who wants to say that "1.7" also means the "1.7 series" for
>>=, note that 1.7 prereleases do not satisfy >=1.7
>
> Would a solution be to make >= also use the series concept?, i.e. make
> pre-releases satisfy >=1.7
>
> Marcus
>
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From qwcode at gmail.com  Sun Dec 28 02:52:35 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sat, 27 Dec 2014 17:52:35 -0800
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAN-Kwu2H1zEoK_eG2kHe=hPeM3+cg3rMFY_-d6ztPMW1_Xs-kw@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <CAN-Kwu2H1zEoK_eG2kHe=hPeM3+cg3rMFY_-d6ztPMW1_Xs-kw@mail.gmail.com>
Message-ID: <CAPYWazrZyuNsiLs_N4uLm3e2HU88h8_SDGrC3hgcLBJuXNw4xA@mail.gmail.com>

You mean "1.7.*" right? Because 1.70 would satisfy >1.7
>

yes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/710d3b92/attachment.html>

From qwcode at gmail.com  Sun Dec 28 02:57:51 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sat, 27 Dec 2014 17:57:51 -0800
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
Message-ID: <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>

>
>
> The equality operator pads zeros, >= has the equality operator as part of
> it.
>


right, but that's not really an answer.
Does the conceptual inconsistency phase you at all?







>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/7d4fdef6/attachment.html>

From donald at stufft.io  Sun Dec 28 03:05:23 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 27 Dec 2014 21:05:23 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
Message-ID: <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>


> On Dec 27, 2014, at 8:57 PM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> 
> The equality operator pads zeros, >= has the equality operator as part of it.
> 
> 
> right, but that's not really an answer.
> Does the conceptual inconsistency phase you at all?
> 


In gives me a minor bit of pause. However any alternative that I can come up
with bothers me more, especially since I don't believe many people actually
even *use* a bare > and any alternative I can come up with has worse behavior
for operators which get much more use.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/32e282af/attachment.html>

From qwcode at gmail.com  Sun Dec 28 03:10:20 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sat, 27 Dec 2014 18:10:20 -0800
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
Message-ID: <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>

>
> In gives me a minor bit of pause. However any alternative that I can come
> up
> with bothers me more, especially since I don't believe many people actually
> even *use* a bare > and any alternative I can come up with has worse
> behavior
> for operators which get much more use.
>

what about making >=,<= also use the series concept? where does that go
wrong?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/1ff9237b/attachment-0001.html>

From donald at stufft.io  Sun Dec 28 03:26:13 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 27 Dec 2014 21:26:13 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
Message-ID: <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>


> On Dec 27, 2014, at 9:10 PM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> In gives me a minor bit of pause. However any alternative that I can come up
> with bothers me more, especially since I don't believe many people actually
> even *use* a bare > and any alternative I can come up with has worse behavior
> for operators which get much more use.
> 
> what about making >=,<= also use the series concept? where does that go wrong?
> 


To be clear, you?re talking about doing something like:

>1.7 is >1.7 AND !=1.7.* which would correlate to >1.7 OR ==1.7.*?

Honestly, the thing that I dislike about that is it takes a behavior which is
less intuitive (I do agree that the behavior of > is less intuitive) and
applies to globally. I don't think people would expect >=1.7 to match 1.7.dev0
and given that I don't think people would expect >=1.7.0 to _not_match
1.7.dev0.

I totally agree that the behavior of > isn't the greatest, I don't think the
solution to that problem is to globally apply that particular wart. The only
*reasonable* solutions I can think of are:

1. Make < and > both act as simple comparison operators, and have >1.7 and
   >1.7.0 both allow 1.7.1. This would include also allowing 1.7.dev0 to be
   <1.7 and <1.7.0.

2. Make < and > both act as "exclusive comparison operators", and which is
   the current behavior.

3. Make < and > both act as simple comparison operators, but include a special
   case that < does not ever match pre-releases of the version mentioned in
   the specifier. So <3 would not match a pre-released like 3.dev0, and <3.1
   would not match a pre-release like 3.1.dev0 but would match a pre-release
   like 3.0.dev0.

It may be that the correct solution is to just treat pre-releases as special
and just switch to 3.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/25c5bb69/attachment.html>

From sontek at gmail.com  Sun Dec 28 05:26:50 2014
From: sontek at gmail.com (John Anderson)
Date: Sat, 27 Dec 2014 20:26:50 -0800
Subject: [Distutils] Detecting rst errors when uploading to pypi
Message-ID: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>

Hey, I'm trying to get my README to render properly on pypi but I haven't
found a way to detect the errors pypi is experiencing.

The package I'm trying to upload is here:
https://pypi.python.org/pypi/pyramid_celery

I've checked the README.rst with restview, rst2html, and
collective.showdocs and they all render the RST just fine.

It seems like I'm not the only one having this issue:
https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way

So what is the recommended route for debugging the problem?  I would really
like to fix it but without know the problem I would just to make random
guesses.

Thanks,
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141227/d29e13f7/attachment.html>

From donald at stufft.io  Sun Dec 28 08:12:24 2014
From: donald at stufft.io (Donald Stufft)
Date: Sun, 28 Dec 2014 02:12:24 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
Message-ID: <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>


> On Dec 27, 2014, at 9:26 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Dec 27, 2014, at 9:10 PM, Marcus Smith <qwcode at gmail.com <mailto:qwcode at gmail.com>> wrote:
>> 
>> In gives me a minor bit of pause. However any alternative that I can come up
>> with bothers me more, especially since I don't believe many people actually
>> even *use* a bare > and any alternative I can come up with has worse behavior
>> for operators which get much more use.
>> 
>> what about making >=,<= also use the series concept? where does that go wrong?
>> 
> 
> 
> To be clear, you?re talking about doing something like:
> 
> >1.7 is >1.7 AND !=1.7.* which would correlate to >1.7 OR ==1.7.*?
> 
> Honestly, the thing that I dislike about that is it takes a behavior which is
> less intuitive (I do agree that the behavior of > is less intuitive) and
> applies to globally. I don't think people would expect >=1.7 to match 1.7.dev0
> and given that I don't think people would expect >=1.7.0 to _not_match
> 1.7.dev0.
> 
> I totally agree that the behavior of > isn't the greatest, I don't think the
> solution to that problem is to globally apply that particular wart. The only
> *reasonable* solutions I can think of are:
> 
> 1. Make < and > both act as simple comparison operators, and have >1.7 and
>    >1.7.0 both allow 1.7.1. This would include also allowing 1.7.dev0 to be
>    <1.7 and <1.7.0.
> 
> 2. Make < and > both act as "exclusive comparison operators", and which is
>    the current behavior.
> 
> 3. Make < and > both act as simple comparison operators, but include a special
>    case that < does not ever match pre-releases of the version mentioned in
>    the specifier. So <3 would not match a pre-released like 3.dev0, and <3.1
>    would not match a pre-release like 3.1.dev0 but would match a pre-release
>    like 3.0.dev0.
> 
> It may be that the correct solution is to just treat pre-releases as special
> and just switch to 3.
> 

After thinking about this some more, I made a PR that adjusts the packaging
library so that is has semantics which might be better overall than what is
currently in PEP 440. This PR is at https://github.com/pypa/packaging/pull/25
if someone wants to play with it at all, but the highlights are:

* 1.7.1 matches >1.7 (previously it did not)
* 1.7.post1 does not match >1.7 (previously it did not)
* 1.7.post1 matches >1.7.post0 (previously it did)
* 3.dev0 does not match <3.0 (previously it did)
* 3.0.dev0 does not match <3.0 (previously it did not)
* 3.0.dev0 matches <3.0rc1 (previously it did)

Instead of having >V and <V impliy !=V.*, this means that:

1. A <V does not match a pre-release of V unless V is itself a pre-release.
2. A >V does not match a post-release of V unless V is itself a post-release.
3. A >V does not match a V+local.version.

Implicitly these three rules are also true, but they are true because of the
ordered nature of the versions and the mathemtical meaning of > and < rather
than any rule mentioned in the PEP:

4. A <V does not match a post-release of V unless V is itself a post-release.
5. A >V does not match a pre-release of V unless V is itself a pre-release.
6. A <V does not match a V+local.version.

If you check out my branch, you can play around with the specifier rules as
I've implemented them now by just doing:

    >>> from packaging.specifiers import SpecifierSet
    >>> SpecifierSet(">1.7").contains("1.7.1", prereleases=True)
    True
    >>> SpecifierSet(">1.7").contains("1.7.post1", prereleases=True)
    False
    >>> SpecifierSet(">1.7.post0").contains("1.7.post1", prereleases=True)
    True
    >>> SpecifierSet(">1.7").contains("1.7+thing.1", prereleases=True)
    False
    >>> SpecifierSet("<3.0.0").contains("3.0.dev0", prereleases=True)
    False
    >>> SpecifierSet("<3").contains("3.0.dev0", prereleases=True)
    False
    >>> SpecifierSet("<3rc1").contains("3.0.dev0", prereleases=True)
    True

Does this better match your expectations? I think I might like these rules
better as they handle zero padding similarly to ==, !=, >=, and <= (sans when
== and != have a .*) which means that a rule like <3.0 won't accept 3.dev0
even though the current PEP means that it does. I think it also follows along
better with what people expect both for > and for <.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/b840cdbf/attachment-0001.html>

From ncoghlan at gmail.com  Sun Dec 28 15:57:08 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Dec 2014 00:57:08 +1000
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
Message-ID: <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>

On 28 Dec 2014 17:12, "Donald Stufft" <donald at stufft.io> wrote:
>
>
>> On Dec 27, 2014, at 9:26 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>>
>>> On Dec 27, 2014, at 9:10 PM, Marcus Smith <qwcode at gmail.com> wrote:
>>>
>>>> In gives me a minor bit of pause. However any alternative that I can
come up
>>>> with bothers me more, especially since I don't believe many people
actually
>>>> even *use* a bare > and any alternative I can come up with has worse
behavior
>>>> for operators which get much more use.
>>>
>>>
>>> what about making >=,<= also use the series concept? where does that go
wrong?
>>>
>>
>> To be clear, you?re talking about doing something like:
>>
>> >1.7 is >1.7 AND !=1.7.* which would correlate to >1.7 OR ==1.7.*?
>>
>> Honestly, the thing that I dislike about that is it takes a behavior
which is
>> less intuitive (I do agree that the behavior of > is less intuitive) and
>> applies to globally. I don't think people would expect >=1.7 to match
1.7.dev0
>> and given that I don't think people would expect >=1.7.0 to _not_match
>> 1.7.dev0.
>>
>> I totally agree that the behavior of > isn't the greatest, I don't think
the
>> solution to that problem is to globally apply that particular wart. The
only
>> *reasonable* solutions I can think of are:
>>
>> 1. Make < and > both act as simple comparison operators, and have >1.7
and
>>    >1.7.0 both allow 1.7.1. This would include also allowing 1.7.dev0 to
be
>>    <1.7 and <1.7.0.
>>
>> 2. Make < and > both act as "exclusive comparison operators", and which
is
>>    the current behavior.
>>
>> 3. Make < and > both act as simple comparison operators, but include a
special
>>    case that < does not ever match pre-releases of the version mentioned
in
>>    the specifier. So <3 would not match a pre-released like 3.dev0, and
<3.1
>>    would not match a pre-release like 3.1.dev0 but would match a
pre-release
>>    like 3.0.dev0.
>>
>> It may be that the correct solution is to just treat pre-releases as
special
>> and just switch to 3.
>>
>
> After thinking about this some more, I made a PR that adjusts the
packaging
> library so that is has semantics which might be better overall than what
is
> currently in PEP 440. This PR is at
https://github.com/pypa/packaging/pull/25
> if someone wants to play with it at all, but the highlights are:
>
> * 1.7.1 matches >1.7 (previously it did not)

This sounds like a straight up bug fix in the packaging module to me - the
PEP 440 zero padding should apply to *all* checks, not just to equality
checks, as you can't sensibly compare release segments with different
numbers of elements.

Hence this sentence in the description of the release segment under the
"Final releases" heading: "When comparing release segments with different
numbers of components, the shorter segment is padded out with additional
zeros as necessary."

The zero padding does get called out again explicitly in the section on
version matching, but that's in the context of explaining why "== 1.7" will
match "1.7.0" and vice-versa, even though version matching is otherwise
strict.

> * 1.7.post1 does not match >1.7 (previously it did not)
> * 1.7.post1 matches >1.7.post0 (previously it did)
> * 3.dev0 does not match <3.0 (previously it did)

This is another bug fix - avoiding that match is one of the key reasons for
the wildcard exclusion in the definition of the exclusive comparison
operators.

Given the next bullet point, it sounds like this may have been another
symptom of incorrectly limiting the release segment zero padding to version
matching operations.

> * 3.0.dev0 does not match <3.0 (previously it did not)
> * 3.0.dev0 matches <3.0rc1 (previously it did)
>
> Instead of having >V and <V impliy !=V.*, this means that:
>
> 1. A <V does not match a pre-release of V unless V is itself a
pre-release.
> 2. A >V does not match a post-release of V unless V is itself a
post-release.
> 3. A >V does not match a V+local.version.

Please don't make the change more complicated than it needs to be - there's
just a bug in the packaging implementation of the PEP semantics. Zero
padding of the release segment should be applied to all operations, and
that currently isn't happening.

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141229/beac1b6b/attachment.html>

From jim at zope.com  Sun Dec 28 16:25:53 2014
From: jim at zope.com (Jim Fulton)
Date: Sun, 28 Dec 2014 10:25:53 -0500
Subject: [Distutils] Can zc.buildout use same comparison operator for
 versions as setup.py
In-Reply-To: <CAJikR8+76_MK89VUx6eNaBjqndWAqTO=T+KThCY+aJm6DaGpAw@mail.gmail.com>
References: <CAJikR8+76_MK89VUx6eNaBjqndWAqTO=T+KThCY+aJm6DaGpAw@mail.gmail.com>
Message-ID: <CAPDm-FiZDL5wYe5Y1jiwbuhe8b7ksVOJ3pJ7RbmxHxiPM7wXyA@mail.gmail.com>

On Thu, Dec 25, 2014 at 10:21 PM, William Zhang <jollychang at gmail.com> wrote:
> Hello
> Can zc.buildout use same comparison operator for  versions as  setup.py?

No. They do technically allow limited ranges, but the primary purpose
is to set/pin/nail specific versions. Fred did a good job of
explaining this.

...

Why do you ask?

> Is there any way for "show-picked-versions" print version be used by
> setup.py install_requires?

If you run buildout in verbose mode (-v), it will will give an
indication of why it's trying to get a particular requirement.  Note
that requirements can come from a variety of places, including
setup.py, the versions section, and requirements stated in egg and
similar parts.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From jim at zope.com  Sun Dec 28 16:33:00 2014
From: jim at zope.com (Jim Fulton)
Date: Sun, 28 Dec 2014 10:33:00 -0500
Subject: [Distutils] Detecting rst errors when uploading to pypi
In-Reply-To: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>
References: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>
Message-ID: <CAPDm-Fjv_mzAt+xVx8O77u4tfDaAF_VKd6b0rzgQBHOwZcQTzw@mail.gmail.com>

On Sat, Dec 27, 2014 at 11:26 PM, John Anderson <sontek at gmail.com> wrote:
> Hey, I'm trying to get my README to render properly on pypi but I haven't
> found a way to detect the errors pypi is experiencing.
>
> The package I'm trying to upload is here:
> https://pypi.python.org/pypi/pyramid_celery
>
> I've checked the README.rst with restview, rst2html, and collective.showdocs
> and they all render the RST just fine.
>
> It seems like I'm not the only one having this issue:
> https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way
>
> So what is the recommended route for debugging the problem?  I would really
> like to fix it but without know the problem I would just to make random
> guesses.

I run:

  setup.py --long-description

and pipe the output into a ReST processor to check for errors.

(I use rst2, https://pypi.python.org/pypi/zc.rst2, which is a thin
but convenient wrapper around docutils for converting ReST
to various formats.)

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From marius at gedmin.as  Sun Dec 28 18:02:07 2014
From: marius at gedmin.as (Marius Gedminas)
Date: Sun, 28 Dec 2014 19:02:07 +0200
Subject: [Distutils] Detecting rst errors when uploading to pypi
In-Reply-To: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>
References: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>
Message-ID: <20141228170207.GC21036@platonas>

On Sat, Dec 27, 2014 at 08:26:50PM -0800, John Anderson wrote:
> Hey, I'm trying to get my README to render properly on pypi but I haven't
> found a way to detect the errors pypi is experiencing.
> 
> The package I'm trying to upload is here:
> https://pypi.python.org/pypi/pyramid_celery

Are you talking about
https://pypi.python.org/pypi/pyramid_celery/2.0.0-rc2 or
https://pypi.python.org/pypi/pyramid_celery/1.4?

The first looks fine now (have you found the error and fixed it
already?) while the second looks like it uses some kind of Markdown
instead of ReStructuredText:

``` python
    pyramid.includes = pyramid_celery
```

> I've checked the README.rst with restview, rst2html, and
> collective.showdocs and they all render the RST just fine.

Just to be sure, did you try restview's --pypi-strict mode?

> It seems like I'm not the only one having this issue:
> https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way
> 
> So what is the recommended route for debugging the problem?  I would really
> like to fix it but without know the problem I would just to make random
> guesses.

restview --long-description is what I use (in the source tree with a
setup.py: this runs setup.py --long-description and parses the output in
--pypi-strict mode).

If that fails to find the error, I try to copy & paste the
long_description text from PyPI into a text file and use restview
--pypi-strict directly.

If that fails, I end up bisecting the long_description text in the web
form, to see where it starts failing.

If it's not my package and I can't edit the long_description on the web,
I tend to give up.

Marius Gedminas
-- 
"Nobody will ever need more than 640k RAM!"
                                -- Bill Gates, 1981
"Windows 95 needs at least 8 MB RAM."
                                -- Bill Gates, 1996
"Nobody will ever need Windows 95."
                                -- logical conclusion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/90019d8f/attachment-0001.sig>

From marius at gedmin.as  Sun Dec 28 17:57:01 2014
From: marius at gedmin.as (Marius Gedminas)
Date: Sun, 28 Dec 2014 18:57:01 +0200
Subject: [Distutils] setuptools-8.4.zip?
In-Reply-To: <CAPDm-FgoBXuRG+h5CpQr-vdk=B-gt119ZQLoDJQ-avmjBLyKdA@mail.gmail.com>
References: <CAPDm-FgoBXuRG+h5CpQr-vdk=B-gt119ZQLoDJQ-avmjBLyKdA@mail.gmail.com>
Message-ID: <20141228165701.GB21036@platonas>

On Sat, Dec 27, 2014 at 12:16:40PM -0500, Jim Fulton wrote:
> Bootstrapping buildout is failing because buildout uses
> https://bootstrap.pypa.io/ez_setup.py
> which tried to download:
> 
>   https://pypi.python.org/packages/source/s/setuptools/setuptools-8.4.zip
> 
> which doesn't exist.
> 
> Did someone forget to upload setuptools-8.4.zip?

It was pulled from PyPI due to a bug:
https://bitbucket.org/pypa/setuptools/issue/314

Marius Gedminas
-- 
* philiKON wonders what niemeyer is committing :)
*** benji_york is now known as benji
<benji> murder?
		-- #zope3-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 173 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/0ef6d04e/attachment.sig>

From qwcode at gmail.com  Sun Dec 28 19:20:34 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 28 Dec 2014 10:20:34 -0800
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
Message-ID: <CAPYWazqaONNByABzOAzu2eu5kabRU6Tp-S5-uB-vjsxJAQ9gtg@mail.gmail.com>

> >
> > * 1.7.1 matches >1.7 (previously it did not)
>
> This sounds like a straight up bug fix in the packaging module to me - the
> PEP 440 zero padding should apply to *all* checks, not just to equality
> checks, as you can't sensibly compare release segments with different
> numbers of elements.
>
OK.  to be clear, I guess you really didn't follow the previous thread?
I specifically raised the concern over 1.7.1 not matching >1.7 (in the
current implementation), but most people were arguing it was a logical
interpretation of PEP440.

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/f0add068/attachment.html>

From chris.jerdonek at gmail.com  Sun Dec 28 20:43:57 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sun, 28 Dec 2014 14:43:57 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAPYWazqaONNByABzOAzu2eu5kabRU6Tp-S5-uB-vjsxJAQ9gtg@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <CAPYWazqaONNByABzOAzu2eu5kabRU6Tp-S5-uB-vjsxJAQ9gtg@mail.gmail.com>
Message-ID: <CAOTb1wdFejWeV=FQsFOrWMJughM34CUJj2+NnGgKB68h8QFkrw@mail.gmail.com>

On Sun, Dec 28, 2014 at 1:20 PM, Marcus Smith <qwcode at gmail.com> wrote:
>
>> >
>> > * 1.7.1 matches >1.7 (previously it did not)
>>
>> This sounds like a straight up bug fix in the packaging module to me - the
>> PEP 440 zero padding should apply to *all* checks, not just to equality
>> checks, as you can't sensibly compare release segments with different
>> numbers of elements.
>
> OK.  to be clear, I guess you really didn't follow the previous thread?
> I specifically raised the concern over 1.7.1 not matching >1.7 (in the
> current implementation), but most people were arguing it was a logical
> interpretation of PEP440.

I think Nick's e-mail clarifies it for me.

In my e-mail, I was reconciling the current behavior with the current
wording of the PEP, which says, "Exclusive ordered comparisons are
similar to inclusive ordered comparisons, except that the comparison
operators are < and > and the clause MUST be effectively interpreted
as implying the prefix based version exclusion clause != V.*."

I now see that the wording is a bit ambiguous (or at least that I was
misinterpreting it).  I interpreted it to mean that prefix-based
version exclusion should be used *instead* of zero-padding, whereas
with Nick's e-mail, I see that the meaning is that prefix-based
exclusion should be used *after* applying zero padding.

The clarified interpretation also addresses an asymmetry of the
previously mentioned (and now apparently incorrect) "series"
interpretation, which I'm not sure was mentioned before.  Namely,
1.7.2 satisfies ">=1.7" but does not satisfy "<=1.7".  With the series
interpretation, the latter wouldn't be consistent (since 1.7.2 is part
of the series under that interpretation).

--Chris






>
> Marcus
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From graffatcolmingov at gmail.com  Sun Dec 28 20:49:15 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sun, 28 Dec 2014 13:49:15 -0600
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAOTb1wdFejWeV=FQsFOrWMJughM34CUJj2+NnGgKB68h8QFkrw@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <CAPYWazqaONNByABzOAzu2eu5kabRU6Tp-S5-uB-vjsxJAQ9gtg@mail.gmail.com>
 <CAOTb1wdFejWeV=FQsFOrWMJughM34CUJj2+NnGgKB68h8QFkrw@mail.gmail.com>
Message-ID: <CAN-Kwu3Jsc8agXU2SF9iEtwsS9qwTd3aBqs9S4Mtt94LLg9GuA@mail.gmail.com>

I personally think that 1.7.1 matching >1.7 muddies some applications
of it being used with date-based versions with this pep also supports.
This (as best I can tell) means that now 2014.09.31 will match >
2014.09 which seems like a rather terrible idea. No one expects a date
to be padded with 0s. I'm also fully against special casing date-based
versions because the whole point of 440 was to make versioning
consistent and reliable and I wholeheartedly want that.

On Sun, Dec 28, 2014 at 1:43 PM, Chris Jerdonek
<chris.jerdonek at gmail.com> wrote:
> On Sun, Dec 28, 2014 at 1:20 PM, Marcus Smith <qwcode at gmail.com> wrote:
>>
>>> >
>>> > * 1.7.1 matches >1.7 (previously it did not)
>>>
>>> This sounds like a straight up bug fix in the packaging module to me - the
>>> PEP 440 zero padding should apply to *all* checks, not just to equality
>>> checks, as you can't sensibly compare release segments with different
>>> numbers of elements.
>>
>> OK.  to be clear, I guess you really didn't follow the previous thread?
>> I specifically raised the concern over 1.7.1 not matching >1.7 (in the
>> current implementation), but most people were arguing it was a logical
>> interpretation of PEP440.
>
> I think Nick's e-mail clarifies it for me.
>
> In my e-mail, I was reconciling the current behavior with the current
> wording of the PEP, which says, "Exclusive ordered comparisons are
> similar to inclusive ordered comparisons, except that the comparison
> operators are < and > and the clause MUST be effectively interpreted
> as implying the prefix based version exclusion clause != V.*."
>
> I now see that the wording is a bit ambiguous (or at least that I was
> misinterpreting it).  I interpreted it to mean that prefix-based
> version exclusion should be used *instead* of zero-padding, whereas
> with Nick's e-mail, I see that the meaning is that prefix-based
> exclusion should be used *after* applying zero padding.
>
> The clarified interpretation also addresses an asymmetry of the
> previously mentioned (and now apparently incorrect) "series"
> interpretation, which I'm not sure was mentioned before.  Namely,
> 1.7.2 satisfies ">=1.7" but does not satisfy "<=1.7".  With the series
> interpretation, the latter wouldn't be consistent (since 1.7.2 is part
> of the series under that interpretation).
>
> --Chris
>
>
>
>
>
>
>>
>> Marcus
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From chris.jerdonek at gmail.com  Sun Dec 28 21:02:07 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sun, 28 Dec 2014 15:02:07 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAN-Kwu3Jsc8agXU2SF9iEtwsS9qwTd3aBqs9S4Mtt94LLg9GuA@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <CAPYWazqaONNByABzOAzu2eu5kabRU6Tp-S5-uB-vjsxJAQ9gtg@mail.gmail.com>
 <CAOTb1wdFejWeV=FQsFOrWMJughM34CUJj2+NnGgKB68h8QFkrw@mail.gmail.com>
 <CAN-Kwu3Jsc8agXU2SF9iEtwsS9qwTd3aBqs9S4Mtt94LLg9GuA@mail.gmail.com>
Message-ID: <CAOTb1weQR-zOEyBW1cJYN0KSQ82hdVmw5_utVJUrjufskxqOew@mail.gmail.com>

On Sunday, December 28, 2014, Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> I personally think that 1.7.1 matching >1.7 muddies some applications
> of it being used with date-based versions with this pep also supports.
> This (as best I can tell) means that now 2014.09.31 will match >
> 2014.09 which seems like a rather terrible idea. No one expects a date
> to be padded with 0s. I'm also fully against special casing date-based
> versions because the whole point of 440 was to make versioning
> consistent and reliable and I wholeheartedly want that.


To add another wrinkle, do you think 2014.09.01 should or shouldn't match
"<=2014.09"?

This is an example of the case I mentioned at the end of my previous email.

--Chris



>
> On Sun, Dec 28, 2014 at 1:43 PM, Chris Jerdonek
> <chris.jerdonek at gmail.com <javascript:;>> wrote:
> > On Sun, Dec 28, 2014 at 1:20 PM, Marcus Smith <qwcode at gmail.com
> <javascript:;>> wrote:
> >>
> >>> >
> >>> > * 1.7.1 matches >1.7 (previously it did not)
> >>>
> >>> This sounds like a straight up bug fix in the packaging module to me -
> the
> >>> PEP 440 zero padding should apply to *all* checks, not just to equality
> >>> checks, as you can't sensibly compare release segments with different
> >>> numbers of elements.
> >>
> >> OK.  to be clear, I guess you really didn't follow the previous thread?
> >> I specifically raised the concern over 1.7.1 not matching >1.7 (in the
> >> current implementation), but most people were arguing it was a logical
> >> interpretation of PEP440.
> >
> > I think Nick's e-mail clarifies it for me.
> >
> > In my e-mail, I was reconciling the current behavior with the current
> > wording of the PEP, which says, "Exclusive ordered comparisons are
> > similar to inclusive ordered comparisons, except that the comparison
> > operators are < and > and the clause MUST be effectively interpreted
> > as implying the prefix based version exclusion clause != V.*."
> >
> > I now see that the wording is a bit ambiguous (or at least that I was
> > misinterpreting it).  I interpreted it to mean that prefix-based
> > version exclusion should be used *instead* of zero-padding, whereas
> > with Nick's e-mail, I see that the meaning is that prefix-based
> > exclusion should be used *after* applying zero padding.
> >
> > The clarified interpretation also addresses an asymmetry of the
> > previously mentioned (and now apparently incorrect) "series"
> > interpretation, which I'm not sure was mentioned before.  Namely,
> > 1.7.2 satisfies ">=1.7" but does not satisfy "<=1.7".  With the series
> > interpretation, the latter wouldn't be consistent (since 1.7.2 is part
> > of the series under that interpretation).
> >
> > --Chris
> >
> >
> >
> >
> >
> >
> >>
> >> Marcus
> >>
> >>
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org <javascript:;>
> >> https://mail.python.org/mailman/listinfo/distutils-sig
> >>
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org <javascript:;>
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/20f827cb/attachment-0001.html>

From donald at stufft.io  Sun Dec 28 21:58:32 2014
From: donald at stufft.io (Donald Stufft)
Date: Sun, 28 Dec 2014 15:58:32 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
Message-ID: <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>


> On Dec 28, 2014, at 9:57 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> 
> On 28 Dec 2014 17:12, "Donald Stufft" <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> >
> >
> >> On Dec 27, 2014, at 9:26 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> >>
> >>
> >>> On Dec 27, 2014, at 9:10 PM, Marcus Smith <qwcode at gmail.com <mailto:qwcode at gmail.com>> wrote:
> >>>
> >>>> In gives me a minor bit of pause. However any alternative that I can come up
> >>>> with bothers me more, especially since I don't believe many people actually
> >>>> even *use* a bare > and any alternative I can come up with has worse behavior
> >>>> for operators which get much more use.
> >>>
> >>>
> >>> what about making >=,<= also use the series concept? where does that go wrong?
> >>>
> >>
> >> To be clear, you?re talking about doing something like:
> >>
> >> >1.7 is >1.7 AND !=1.7.* which would correlate to >1.7 OR ==1.7.*?
> >>
> >> Honestly, the thing that I dislike about that is it takes a behavior which is
> >> less intuitive (I do agree that the behavior of > is less intuitive) and
> >> applies to globally. I don't think people would expect >=1.7 to match 1.7.dev0
> >> and given that I don't think people would expect >=1.7.0 to _not_match
> >> 1.7.dev0.
> >>
> >> I totally agree that the behavior of > isn't the greatest, I don't think the
> >> solution to that problem is to globally apply that particular wart. The only
> >> *reasonable* solutions I can think of are:
> >>
> >> 1. Make < and > both act as simple comparison operators, and have >1.7 and
> >>    >1.7.0 both allow 1.7.1. This would include also allowing 1.7.dev0 to be
> >>    <1.7 and <1.7.0.
> >>
> >> 2. Make < and > both act as "exclusive comparison operators", and which is
> >>    the current behavior.
> >>
> >> 3. Make < and > both act as simple comparison operators, but include a special
> >>    case that < does not ever match pre-releases of the version mentioned in
> >>    the specifier. So <3 would not match a pre-released like 3.dev0, and <3.1
> >>    would not match a pre-release like 3.1.dev0 but would match a pre-release
> >>    like 3.0.dev0.
> >>
> >> It may be that the correct solution is to just treat pre-releases as special
> >> and just switch to 3.
> >>
> >
> > After thinking about this some more, I made a PR that adjusts the packaging
> > library so that is has semantics which might be better overall than what is
> > currently in PEP 440. This PR is at https://github.com/pypa/packaging/pull/25 <https://github.com/pypa/packaging/pull/25>
> > if someone wants to play with it at all, but the highlights are:
> >
> > * 1.7.1 matches >1.7 (previously it did not)
> 
> This sounds like a straight up bug fix in the packaging module to me - the PEP 440 zero padding should apply to *all* checks, not just to equality checks, as you can't sensibly compare release segments with different numbers of elements.
> 
> Hence this sentence in the description of the release segment under the "Final releases" heading: "When comparing release segments with different numbers of components, the shorter segment is padded out with additional zeros as necessary."
> 
> The zero padding does get called out again explicitly in the section on version matching, but that's in the context of explaining why "== 1.7" will match "1.7.0" and vice-versa, even though version matching is otherwise strict.
> 
> > * 1.7.post1 does not match >1.7 (previously it did not)
> > * 1.7.post1 matches >1.7.post0 (previously it did)
> > * 3.dev0 does not match <3.0 (previously it did)
> 
> This is another bug fix - avoiding that match is one of the key reasons for the wildcard exclusion in the definition of the exclusive comparison operators.
> 
> Given the next bullet point, it sounds like this may have been another symptom of incorrectly limiting the release segment zero padding to version matching operations.
> 
> > * 3.0.dev0 does not match <3.0 (previously it did not)
> > * 3.0.dev0 matches <3.0rc1 (previously it did)
> >
> > Instead of having >V and <V impliy !=V.*, this means that:
> >
> > 1. A <V does not match a pre-release of V unless V is itself a pre-release.
> > 2. A >V does not match a post-release of V unless V is itself a post-release.
> > 3. A >V does not match a V+local.version.
> 
> Please don't make the change more complicated than it needs to be - there's just a bug in the packaging implementation of the PEP semantics. Zero padding of the release segment should be applied to all operations, and that currently isn't happening.
> 
> Regards,
> Nick.
> 

The > and < operators are currently implemented exactly how they are defined in
the PEP. If we want to change that, that's fine, but the packaging library is
correct to the PEP currently.

The PEP says:

    Exclusive ordered comparisons are similar to inclusive ordered comparisons,
    except that the comparison operators are < and > and the clause MUST be
    effectively interpreted as implying the prefix based version exclusion
    clause != V.* .

    The exclusive ordered comparison > V MUST NOT match a post-release or
    maintenance release of the given version. Maintenance releases can be
    permitted by using the clause > V.0 , while both post releases and
    maintenance releases can be permitted by using the inclusive ordered comparison >= V.post1 .

    The exclusive ordered comparison < V MUST NOT match a pre-release of the
    given version, even if acceptance of pre-releases is enabled as described
    in the section below.

    Local version identifiers are NOT permitted in this version specifier. 


The packaging lib currently does exactly that, given a specifier like >1.7 it
transforms it to >1.7 and !=1.7.*. The PEP even calls out the fact that a
maintenance release of the same series should not match and if you want it to
match then you should add a zero yourself.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/531932b9/attachment.html>

From sontek at gmail.com  Mon Dec 29 00:54:25 2014
From: sontek at gmail.com (John Anderson)
Date: Sun, 28 Dec 2014 15:54:25 -0800
Subject: [Distutils] Detecting rst errors when uploading to pypi
In-Reply-To: <20141228170207.GC21036@platonas>
References: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>
 <20141228170207.GC21036@platonas>
Message-ID: <CAOdFaDXc_s4osHZ7zidmLA1qBAqYuuJ+xVmA-x1oNrBMg1VjAw@mail.gmail.com>

On Sun, Dec 28, 2014 at 9:02 AM, Marius Gedminas <marius at gedmin.as> wrote:

> On Sat, Dec 27, 2014 at 08:26:50PM -0800, John Anderson wrote:
> > Hey, I'm trying to get my README to render properly on pypi but I haven't
> > found a way to detect the errors pypi is experiencing.
> >
> > The package I'm trying to upload is here:
> > https://pypi.python.org/pypi/pyramid_celery
>
> Are you talking about
> https://pypi.python.org/pypi/pyramid_celery/2.0.0-rc2 or
> https://pypi.python.org/pypi/pyramid_celery/1.4?
>
> The first looks fine now (have you found the error and fixed it
> already?) while the second looks like it uses some kind of Markdown
> instead of ReStructuredText:
>
> ``` python
>     pyramid.includes = pyramid_celery
> ```
>
> > I've checked the README.rst with restview, rst2html, and
> > collective.showdocs and they all render the RST just fine.
>
> Just to be sure, did you try restview's --pypi-strict mode?
>
> > It seems like I'm not the only one having this issue:
> >
> https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way
> >
> > So what is the recommended route for debugging the problem?  I would
> really
> > like to fix it but without know the problem I would just to make random
> > guesses.
>
> restview --long-description is what I use (in the source tree with a
> setup.py: this runs setup.py --long-description and parses the output in
> --pypi-strict mode).
>
> If that fails to find the error, I try to copy & paste the
> long_description text from PyPI into a text file and use restview
> --pypi-strict directly.
>
> If that fails, I end up bisecting the long_description text in the web
> form, to see where it starts failing.
>
> If it's not my package and I can't edit the long_description on the web,
> I tend to give up.
>

I found the error was a .. code-block::  with no language, I fixed the
README with this commit:

https://github.com/sontek/pyramid_celery/commit/fb69dd44143a5b030b48346e6322845194a690fe


restview with --pypi-strict doesn't detect this (but maybe it should?).   I
even tried ripping the source code from pypi directly to catch the error:

https://gist.github.com/sontek/e58c1cf82931e1478cd2

Using it:

python setup.py --long-description | python check_readme.py

but even that didn't cause the error, I assume because I have a more modern
version of pygments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/f025e451/attachment-0001.html>

From sontek at gmail.com  Mon Dec 29 05:52:16 2014
From: sontek at gmail.com (John Anderson)
Date: Sun, 28 Dec 2014 20:52:16 -0800
Subject: [Distutils] Detecting rst errors when uploading to pypi
In-Reply-To: <CAHrZfZB7vax1s_jQLzz1ZZkMOTmnDAnwbkbMX5HKwJDqt-1FFw@mail.gmail.com>
References: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>
 <20141228170207.GC21036@platonas>
 <CAOdFaDXc_s4osHZ7zidmLA1qBAqYuuJ+xVmA-x1oNrBMg1VjAw@mail.gmail.com>
 <CAHrZfZB7vax1s_jQLzz1ZZkMOTmnDAnwbkbMX5HKwJDqt-1FFw@mail.gmail.com>
Message-ID: <CAOdFaDW+0TiJ281xWC3e0QKwzt52P=wHDZ3-Z53cYn_U4CO5Tg@mail.gmail.com>

I've confirmed that the bug I was experiencing is no longer a problem.

Thanks!

On Sun, Dec 28, 2014 at 8:46 PM, Richard Jones <richard at python.org> wrote:

> It has been a while since Pygments was updated in that environment, so
> I've updated it. The PyPI code required a language argument, so I fixed
> that as well.
>
>
>       Richard
>
> On Mon Dec 29 2014 at 10:54:50 AM John Anderson <sontek at gmail.com> wrote:
>
>> On Sun, Dec 28, 2014 at 9:02 AM, Marius Gedminas <marius at gedmin.as>
>> wrote:
>>
>>> On Sat, Dec 27, 2014 at 08:26:50PM -0800, John Anderson wrote:
>>> > Hey, I'm trying to get my README to render properly on pypi but I
>>> haven't
>>> > found a way to detect the errors pypi is experiencing.
>>> >
>>> > The package I'm trying to upload is here:
>>> > https://pypi.python.org/pypi/pyramid_celery
>>>
>>> Are you talking about
>>> https://pypi.python.org/pypi/pyramid_celery/2.0.0-rc2 or
>>> https://pypi.python.org/pypi/pyramid_celery/1.4?
>>>
>>> The first looks fine now (have you found the error and fixed it
>>> already?) while the second looks like it uses some kind of Markdown
>>> instead of ReStructuredText:
>>>
>>> ``` python
>>>     pyramid.includes = pyramid_celery
>>> ```
>>>
>>> > I've checked the README.rst with restview, rst2html, and
>>> > collective.showdocs and they all render the RST just fine.
>>>
>>> Just to be sure, did you try restview's --pypi-strict mode?
>>>
>>> > It seems like I'm not the only one having this issue:
>>> > https://bitbucket.org/pypa/pypi/issue/161/rest-
>>> formatting-fails-and-there-is-no-way
>>> >
>>> > So what is the recommended route for debugging the problem?  I would
>>> really
>>> > like to fix it but without know the problem I would just to make random
>>> > guesses.
>>>
>>> restview --long-description is what I use (in the source tree with a
>>> setup.py: this runs setup.py --long-description and parses the output in
>>> --pypi-strict mode).
>>>
>>> If that fails to find the error, I try to copy & paste the
>>> long_description text from PyPI into a text file and use restview
>>> --pypi-strict directly.
>>>
>>> If that fails, I end up bisecting the long_description text in the web
>>> form, to see where it starts failing.
>>>
>>> If it's not my package and I can't edit the long_description on the web,
>>> I tend to give up.
>>>
>>
>> I found the error was a .. code-block::  with no language, I fixed the
>> README with this commit:
>>
>> https://github.com/sontek/pyramid_celery/commit/
>> fb69dd44143a5b030b48346e6322845194a690fe
>>
>>
>> restview with --pypi-strict doesn't detect this (but maybe it should?).
>> I even tried ripping the source code from pypi directly to catch the error:
>>
>> https://gist.github.com/sontek/e58c1cf82931e1478cd2
>>
>> Using it:
>>
>> python setup.py --long-description | python check_readme.py
>>
>> but even that didn't cause the error, I assume because I have a more
>> modern version of pygments.
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/15de1bed/attachment.html>

From ncoghlan at gmail.com  Mon Dec 29 06:01:56 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Dec 2014 15:01:56 +1000
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
Message-ID: <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>

On 29 December 2014 at 06:58, Donald Stufft <donald at stufft.io> wrote:

> The packaging lib currently does exactly that, given a specifier like >1.7
> it
> transforms it to >1.7 and !=1.7.*. The PEP even calls out the fact that a
> maintenance release of the same series should not match and if you want it
> to
> match then you should add a zero yourself.
>

Oops, I managed to drop the list off my reply to Donald. I won't relay the
entire subsequent conversation, but the short version is:

* the current handling of ">" and "<" for maintenance releases in PEP 440
is incompatible with the previous behaviour of setuptools, which goes
against one of the main guiding principles in the design of PEP 440 (we
aimed to only break compatibility with pkg_resources when we had a really
compelling reason)
* the interaction between the zero padding rules for the release segment
and wildcard prefix matching is sufficiently tricky that both Donald and I
got it wrong at various points in the discussion. When the two lead
designers for an interoperability spec struggle to remember how it works in
particular edge cases, that's a bad sign (that's not always avoidable, as
some real world interoperability problems are just inherently confusing,
but in this case we both now think it represents a genuine mistake in the
PEP)
* as noted in the previous discussion, we have a short window before folks
start shipping pip 6 with more slowly updating projects (like ensurepip and
Linux distros) where we can treat PEP 440 as Provisional (even though PEP 1
doesn't officially include that notion yet). It makes sense to take
advantage of that window, rather than locking in a design we've now
realised we're not happy with.
* I was incorrect in thinking that ordering operations were currently
defined purely in syntactic terms. They're already semantically aware, as
they need to be to handle pre-releases correctly, so making them
appropriately aware of post-release semantics is a smaller change than I
thought
* for those cases (like date-based versions) where excluding releases with
an additional numeric suffix is the right thing to do, an explicit prefix
exclusion will still be possible, and will have the advantage of failing
noisily on older versions of pip and easy_install, rather than silently
installing an unexpected version (which is what will happen for anyone that
starts relying on the current explicit exclusion in PEP 440). It's also
possible to replace the exclusive bound with an inclusive bound that has
the last release segment component incremented.

Next steps:

* I'll update the currently published version of PEP 440 to include a note
regarding its provisional status
* Donald is going to put together a PR to update the interoperability spec
to match the semantics of his proposed change to the packaging library
(that will either be in the existing BitBucket repo, or after moving the
drafts to the new GitHub repo)
* I'll write a new PEP proposing some changes to PEP 1 based on the things
we found challenging in bringing PEP 440 to a close (this is the first PEP
completed under the new delegation of authority model, and we've definitely
found some rough edges that could stand to be smoothed out)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141229/e73a5edd/attachment.html>

From richard at python.org  Mon Dec 29 05:46:36 2014
From: richard at python.org (Richard Jones)
Date: Mon, 29 Dec 2014 04:46:36 +0000
Subject: [Distutils] Detecting rst errors when uploading to pypi
References: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>
 <20141228170207.GC21036@platonas>
 <CAOdFaDXc_s4osHZ7zidmLA1qBAqYuuJ+xVmA-x1oNrBMg1VjAw@mail.gmail.com>
Message-ID: <CAHrZfZB7vax1s_jQLzz1ZZkMOTmnDAnwbkbMX5HKwJDqt-1FFw@mail.gmail.com>

It has been a while since Pygments was updated in that environment, so I've
updated it. The PyPI code required a language argument, so I fixed that as
well.


      Richard

On Mon Dec 29 2014 at 10:54:50 AM John Anderson <sontek at gmail.com> wrote:

> On Sun, Dec 28, 2014 at 9:02 AM, Marius Gedminas <marius at gedmin.as> wrote:
>
>> On Sat, Dec 27, 2014 at 08:26:50PM -0800, John Anderson wrote:
>> > Hey, I'm trying to get my README to render properly on pypi but I
>> haven't
>> > found a way to detect the errors pypi is experiencing.
>> >
>> > The package I'm trying to upload is here:
>> > https://pypi.python.org/pypi/pyramid_celery
>>
>> Are you talking about
>> https://pypi.python.org/pypi/pyramid_celery/2.0.0-rc2 or
>> https://pypi.python.org/pypi/pyramid_celery/1.4?
>>
>> The first looks fine now (have you found the error and fixed it
>> already?) while the second looks like it uses some kind of Markdown
>> instead of ReStructuredText:
>>
>> ``` python
>>     pyramid.includes = pyramid_celery
>> ```
>>
>> > I've checked the README.rst with restview, rst2html, and
>> > collective.showdocs and they all render the RST just fine.
>>
>> Just to be sure, did you try restview's --pypi-strict mode?
>>
>> > It seems like I'm not the only one having this issue:
>> > https://bitbucket.org/pypa/pypi/issue/161/rest-
>> formatting-fails-and-there-is-no-way
>> >
>> > So what is the recommended route for debugging the problem?  I would
>> really
>> > like to fix it but without know the problem I would just to make random
>> > guesses.
>>
>> restview --long-description is what I use (in the source tree with a
>> setup.py: this runs setup.py --long-description and parses the output in
>> --pypi-strict mode).
>>
>> If that fails to find the error, I try to copy & paste the
>> long_description text from PyPI into a text file and use restview
>> --pypi-strict directly.
>>
>> If that fails, I end up bisecting the long_description text in the web
>> form, to see where it starts failing.
>>
>> If it's not my package and I can't edit the long_description on the web,
>> I tend to give up.
>>
>
> I found the error was a .. code-block::  with no language, I fixed the
> README with this commit:
>
> https://github.com/sontek/pyramid_celery/commit/
> fb69dd44143a5b030b48346e6322845194a690fe
>
>
> restview with --pypi-strict doesn't detect this (but maybe it should?).
> I even tried ripping the source code from pypi directly to catch the error:
>
> https://gist.github.com/sontek/e58c1cf82931e1478cd2
>
> Using it:
>
> python setup.py --long-description | python check_readme.py
>
> but even that didn't cause the error, I assume because I have a more
> modern version of pygments.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141229/7f434165/attachment-0001.html>

From donald at stufft.io  Mon Dec 29 07:16:08 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 Dec 2014 01:16:08 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
 <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
Message-ID: <67E164F7-C212-4FA5-A7DD-B12FD81B3471@stufft.io>


> On Dec 29, 2014, at 12:01 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> * Donald is going to put together a PR to update the interoperability spec to match the semantics of his proposed change to the packaging library (that will either be in the existing BitBucket repo, or after moving the drafts to the new GitHub repo)

See: https://github.com/pypa/interoperability-peps/pull/3

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141229/06fac9fb/attachment.html>

From ncoghlan at gmail.com  Mon Dec 29 06:51:42 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Dec 2014 15:51:42 +1000
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
 <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
Message-ID: <CADiSq7cTU+ZbG9TF25ha8SA17Cr2efafyktr1bDJZnpJ1XhqyA@mail.gmail.com>

On 29 December 2014 at 15:01, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Next steps:
>
> * I'll update the currently published version of PEP 440 to include a note
> regarding its provisional status
>

I just pushed this as https://hg.python.org/peps/rev/a532493ba99c

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141229/e3462800/attachment.html>

From qwcode at gmail.com  Mon Dec 29 08:12:23 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 28 Dec 2014 23:12:23 -0800
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <67E164F7-C212-4FA5-A7DD-B12FD81B3471@stufft.io>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
 <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
 <67E164F7-C212-4FA5-A7DD-B12FD81B3471@stufft.io>
Message-ID: <CAPYWazqcBZXLPD4QQkNvcgoTroxLKYnPKxNxaATfXDPEzBp8mA@mail.gmail.com>

how about just pypa/peps for the name?


On Sun, Dec 28, 2014 at 10:16 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Dec 29, 2014, at 12:01 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> * Donald is going to put together a PR to update the interoperability spec
> to match the semantics of his proposed change to the packaging library
> (that will either be in the existing BitBucket repo, or after moving the
> drafts to the new GitHub repo)
>
>
> See: https://github.com/pypa/interoperability-peps/pull/3
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141228/f951d9fe/attachment.html>

From ncoghlan at gmail.com  Mon Dec 29 08:22:46 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Dec 2014 17:22:46 +1000
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAPYWazqcBZXLPD4QQkNvcgoTroxLKYnPKxNxaATfXDPEzBp8mA@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
 <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
 <67E164F7-C212-4FA5-A7DD-B12FD81B3471@stufft.io>
 <CAPYWazqcBZXLPD4QQkNvcgoTroxLKYnPKxNxaATfXDPEzBp8mA@mail.gmail.com>
Message-ID: <CADiSq7dLpYT8Bkxv0AnPEQXeEJfoB6u6ZGpBQpw_NA815ZkdTA@mail.gmail.com>

On 29 December 2014 at 17:12, Marcus Smith <qwcode at gmail.com> wrote:

> how about just pypa/peps for the name?
>

I would find the name clash with the main peps repo very annoying :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141229/8da24f40/attachment.html>

From donald at stufft.io  Mon Dec 29 08:33:00 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 Dec 2014 02:33:00 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CADiSq7dLpYT8Bkxv0AnPEQXeEJfoB6u6ZGpBQpw_NA815ZkdTA@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
 <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
 <67E164F7-C212-4FA5-A7DD-B12FD81B3471@stufft.io>
 <CAPYWazqcBZXLPD4QQkNvcgoTroxLKYnPKxNxaATfXDPEzBp8mA@mail.gmail.com>
 <CADiSq7dLpYT8Bkxv0AnPEQXeEJfoB6u6ZGpBQpw_NA815ZkdTA@mail.gmail.com>
Message-ID: <F39AF1B2-721B-4DF5-BC6B-304FEDD2BA60@stufft.io>


> On Dec 29, 2014, at 2:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 29 December 2014 at 17:12, Marcus Smith <qwcode at gmail.com <mailto:qwcode at gmail.com>> wrote:
> how about just pypa/peps for the name?
> 
> I would find the name clash with the main peps repo very annoying :)
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>   |   Brisbane, Australia


lol, I have dstufft/peps already, and python/peps, what?s a third clash :V

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141229/99324a42/attachment-0001.html>

From chris.jerdonek at gmail.com  Mon Dec 29 20:09:45 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 29 Dec 2014 14:09:45 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
 <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
Message-ID: <CAOTb1wdAa-7fbg9iv-wbUW4FxE9iQxcjg+UOfjsjVK6C+gDnZw@mail.gmail.com>

On Mon, Dec 29, 2014 at 12:01 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> * for those cases (like date-based versions) where excluding releases with
> an additional numeric suffix is the right thing to do, an explicit prefix
> exclusion will still be possible, and will have the advantage of failing
> noisily on older versions of pip and easy_install, rather than silently
> installing an unexpected version (which is what will happen for anyone that
> starts relying on the current explicit exclusion in PEP 440). It's also
> possible to replace the exclusive bound with an inclusive bound that has the
> last release segment component incremented.

Just to confirm, in each of the below is it true that the two
comparisons are exactly equivalent (taking into account pre-release
and post-release rules, etc)?

1) >V.N.* and >=V.N+1
2) <=V.N.* and <V.N+1

And these two should be even easier:

3) <V.* and <V
4) >=V.* and >=V

With the way the PEP is written, it's not completely obvious to me
that these are true.

Thanks,
--Chris

From donald at stufft.io  Mon Dec 29 20:12:57 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 Dec 2014 14:12:57 -0500
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAOTb1wdAa-7fbg9iv-wbUW4FxE9iQxcjg+UOfjsjVK6C+gDnZw@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
 <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
 <CAOTb1wdAa-7fbg9iv-wbUW4FxE9iQxcjg+UOfjsjVK6C+gDnZw@mail.gmail.com>
Message-ID: <DA30A612-14AC-480B-A964-3F0F99908D83@stufft.io>


> On Dec 29, 2014, at 2:09 PM, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
> 
> On Mon, Dec 29, 2014 at 12:01 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> * for those cases (like date-based versions) where excluding releases with
>> an additional numeric suffix is the right thing to do, an explicit prefix
>> exclusion will still be possible, and will have the advantage of failing
>> noisily on older versions of pip and easy_install, rather than silently
>> installing an unexpected version (which is what will happen for anyone that
>> starts relying on the current explicit exclusion in PEP 440). It's also
>> possible to replace the exclusive bound with an inclusive bound that has the
>> last release segment component incremented.
> 
> Just to confirm, in each of the below is it true that the two
> comparisons are exactly equivalent (taking into account pre-release
> and post-release rules, etc)?
> 
> 1) >V.N.* and >=V.N+1
> 2) <=V.N.* and <V.N+1
> 
> And these two should be even easier:
> 
> 3) <V.* and <V
> 4) >=V.* and >=V
> 
> With the way the PEP is written, it's not completely obvious to me
> that these are true.
> 
> Thanks,
> --Chris

The .* operator is not currently allowed other than in == and != in PEP 440,
that was just a suggestion of a possible future enhancement to make it easier
to get the behavior that people likely want for date based releases. Currently
with the suggested updates to PEP 440 they'd need to do something like >V,!=V.*
to get that behavior.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From ncoghlan at gmail.com  Mon Dec 29 23:57:07 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Dec 2014 08:57:07 +1000
Subject: [Distutils] PEP440: >1.7 vs >=1.7
In-Reply-To: <CAOTb1wdAa-7fbg9iv-wbUW4FxE9iQxcjg+UOfjsjVK6C+gDnZw@mail.gmail.com>
References: <CAPYWazogog-zhTCu_D-GPuEuffrfnp+wcghNq21fURFSE7pfBw@mail.gmail.com>
 <EF8CA3DF-F4F0-4FF9-8688-F0D346714E20@stufft.io>
 <CAPYWazqAObF7ODgyyaMbc2VJQ5yixj0K1QwMq_nw=+CNiaAjeQ@mail.gmail.com>
 <BE20796F-2677-4E0E-A928-9807FB005AA4@stufft.io>
 <CAPYWazpga95+vgfJBkU=1g8UNFS5T4F78-1MLD_tUDAsHFxKuw@mail.gmail.com>
 <EB0242CD-4323-4F5D-B9FF-D60D178764A4@stufft.io>
 <C348B936-1C9F-4D0E-995F-3DBE71EF70D4@stufft.io>
 <CADiSq7cO0O6_kZ7f4qyu84yHknN=gM2-01tL7vsXiDY-TaGZeA@mail.gmail.com>
 <AEC6CFF1-7B80-4500-B4CD-67DCED820B02@stufft.io>
 <CADiSq7eykJk029Y+bZj6N_YaAS_k07Sw9E8XQMxqkKdQLV+asA@mail.gmail.com>
 <CAOTb1wdAa-7fbg9iv-wbUW4FxE9iQxcjg+UOfjsjVK6C+gDnZw@mail.gmail.com>
Message-ID: <CADiSq7cvNN_z-tzJafV9SHb+xwQ8UtUFPxGTxFFkg=UEud24cA@mail.gmail.com>

On 30 December 2014 at 05:09, Chris Jerdonek <chris.jerdonek at gmail.com>
wrote:

> On Mon, Dec 29, 2014 at 12:01 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > * for those cases (like date-based versions) where excluding releases
> with
> > an additional numeric suffix is the right thing to do, an explicit prefix
> > exclusion will still be possible, and will have the advantage of failing
> > noisily on older versions of pip and easy_install, rather than silently
> > installing an unexpected version (which is what will happen for anyone
> that
> > starts relying on the current explicit exclusion in PEP 440). It's also
> > possible to replace the exclusive bound with an inclusive bound that has
> the
> > last release segment component incremented.
>
> Just to confirm, in each of the below is it true that the two
> comparisons are exactly equivalent (taking into account pre-release
> and post-release rules, etc)?
>
> 1) >V.N.* and >=V.N+1
> 2) <=V.N.* and <V.N+1
>

If you omit the trailing ".*" (which isn't part of PEP 440 in the context
of ordered comparisons), then the following are equivalent in the currently
published version of the PEP:

1) "> V.N" and ">= V.N+1"
2) "<= V.N" and "< V.N+1"

The current behaviour of those exclusive comparisons is the part that's
both incompatible with the previous behaviour of setuptools/pkg_resources,
and somewhat confusing (largely because the interaction between the zero
padding and the negative prefix matching means that the trailing zero in
the release segment effectively isn't implied when comparing against
maintenance releases).

Donald's proposed adjustment at
https://github.com/pypa/interoperability-peps/pull/3 (which I now agree
with) changes them to be different: unlike ">= V.N+1", "> V.N" will once
again allow "V.N.x" maintenance releases, as it did in
setuptools/pkg_resources releases prior to 8.0.

The exclusive ordered comparison behaviour in the originally accepted
version of the PEP can still be requested explicitly as "< V, != V.*" and
"> V, != V.*" (where the latter has the virtue of failing noisily on older
versions of pkg_resources, rather than silently referring to a different
minimum version).

And these two should be even easier:
>
> 3) <V.* and <V
> 4) >=V.* and >=V
>
> With the way the PEP is written, it's not completely obvious to me
> that these are true.
>

In a hypothetical future version of the PEP that allowed the ".*" wildcard
suffix for ordered comparisons, it would likely have the following
semantics:

"< V.*" ==> "< V, != V.*"
"> V.*" ==> "> V, != V.*"
"<= V.*" ==> "<= V"
">= V.*" ==> ">= V"

However, it's not clear that such an addition would be worth it, given the
updated definition of exclusive comparisons, and the availability of the
more explicit spelling.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141230/3813fdf6/attachment.html>

From reinout at vanrees.org  Tue Dec 30 23:21:00 2014
From: reinout at vanrees.org (Reinout van Rees)
Date: Tue, 30 Dec 2014 23:21:00 +0100
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
Message-ID: <m7v8gc$471$1@ger.gmane.org>

Chris Barker schreef op 23-12-14 om 18:36:
> I'm trying to package up a complex system and would like to do it the
> correct, modern way.
>
> In particular, this involves a bunch of compiled extensions, as well as
> dependencies on both the scientific "stack" and common Web app packages.
>
> (can you tell I'm building a web service front-end for computational code?)
>
> This is actually a bit of a trick, because the web development community
> is generally doing a good job up supporting PyPi and pip, whereas the
> complications of the scientific software tools have folks relying more
> on tools like Enthought and Continuum.
>
> So far, we've been doing mostly pip and struggling with build our own
> for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and
> GDAL, and....). But at the end of all this we'd like to be able to
> distribute and make it easy on end users to use our tools.

Well, we're in a bit of the same boat here. We make django websites, 
which means pretty much well-behaved setup.py-using pure python stuff.

The websites are heavy users of numpy/scipy/pandas/matplotlib and of the 
geo packages like gdal/mapnik/pyproj. Ouch.


The combination we use now is to use buildout (instead of pip) in 
combination with the "syseggrecipe" 
(https://pypi.python.org/pypi/syseggrecipe) buildout add-on. 
Syseggrecipe allows us to depend explicitly on **system** packages for 
gdal/matplotlib/mapnik/scipy/numpy. Which takes care of most of the 
compilation headaches.

The rest of the huge python package stack is just pulled in regularly by 
buildout (using setuptools).


Just throwing this into the mix as a potential solution. Note that 
you'll get to explain buildout to your users, but... :-)



Reinout

-- 
Reinout van Rees                          http://reinout.vanrees.org/
reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"


From reinout at vanrees.org  Tue Dec 30 23:27:02 2014
From: reinout at vanrees.org (Reinout van Rees)
Date: Tue, 30 Dec 2014 23:27:02 +0100
Subject: [Distutils] Detecting rst errors when uploading to pypi
In-Reply-To: <CAPDm-Fjv_mzAt+xVx8O77u4tfDaAF_VKd6b0rzgQBHOwZcQTzw@mail.gmail.com>
References: <CAOdFaDV59Ovim-xxFY2TvzW51mQCv1cxoaV7uYf+o0yuiDrZaQ@mail.gmail.com>
 <CAPDm-Fjv_mzAt+xVx8O77u4tfDaAF_VKd6b0rzgQBHOwZcQTzw@mail.gmail.com>
Message-ID: <m7v8rm$9k0$1@ger.gmane.org>

Jim Fulton schreef op 28-12-14 om 16:33:
>> So what is the recommended route for debugging the problem?  I would really
>> >like to fix it but without know the problem I would just to make random
>> >guesses.
> I run:
>
>    setup.py --long-description
>
> and pipe the output into a ReST processor to check for errors.

Blowing my own horn here... :-)

If you install zest.releaser (which many people use to release their 
packages to save them lots of manual work), as an added bonus you get 
the "longtest" script. This does exactly what Jim suggests: it takes the 
long description, sends it through docutils and even opens it for you in 
your browser. Easiest way to check for errors :-)

See http://zestreleaser.readthedocs.org/


Reinout

-- 
Reinout van Rees                          http://reinout.vanrees.org/
reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"


From solipsis at pitrou.net  Tue Dec 30 23:37:11 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Dec 2014 23:37:11 +0100
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
Message-ID: <20141230233711.185f03c3@fsol>

On Tue, 23 Dec 2014 09:36:36 -0800
Chris Barker <chris.barker at noaa.gov> wrote:
> 
> So far, we've been doing mostly pip and struggling with build our own for
> the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL,
> and....). But at the end of all this we'd like to be able to distribute and
> make it easy on end users to use our tools.
> 
> I figure we we'll do one (or both) of:
> - providing a custom "wheel house" with our packages and the dependencies
> that are hard to come by
> - provide a binstar channel with conda packages of all the same stuff but a
> totally different set of "other" packages.
> 
> At the moment, I'm working on making conda packages, which brings me to my
> questions.

Note you can use pip inside conda environments, which works quite well
at least for pure Python packages (start with "conda install pip").

Regards

Antoine.



From chris.barker at noaa.gov  Wed Dec 31 01:42:42 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 30 Dec 2014 16:42:42 -0800
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <m7v8gc$471$1@ger.gmane.org>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
 <m7v8gc$471$1@ger.gmane.org>
Message-ID: <CALGmxEKbdUTkGi5zUiiF4vhMUjqkK6knEk8UKsZai40p3i8Ezw@mail.gmail.com>

On Tue, Dec 30, 2014 at 2:21 PM, Reinout van Rees <reinout at vanrees.org>
wrote:

> Well, we're in a bit of the same boat here. We make django websites, which
> means pretty much well-behaved setup.py-using pure python stuff.
>
> The websites are heavy users of numpy/scipy/pandas/matplotlib and of the
> geo packages like gdal/mapnik/pyproj. Ouch.
>

yup -- this is pretty much out stack (though pyramid in our case...) Funny,
though, as coming from my background, I see it as a scipy stack app with a
little web stuff, rather than a web app that needs some scipy stuff ;-)

But the core problem here is that the scipy folks have been going to conda
and enthought to solve their pacakgeing problems, and the web folks have
been doing pip, and maybe buildout -- so you get a bit of mess when you mix
them.


> The combination we use now is to use buildout (instead of pip) in
> combination with the "syseggrecipe" (https://pypi.python.org/pypi/
> syseggrecipe) buildout add-on. Syseggrecipe allows us to depend
> explicitly on **system** packages for gdal/matplotlib/mapnik/scipy/numpy.
> Which takes care of most of the compilation headaches.
>

well, it sounds like you are simple punting -- passing off all the complex
stuff to the system, which may work well if the system is up to date linux
with the packages you need available, but pretty worthless on a Mac or
Windows box.

The scipy folks have been doing a pretty good job lately keeping up with
wheels, but there's still a big hole there for the geo stuff.(GDAL,
Shapely, Fiona)

So I've been looking at going the Anaconda route -- it provides the hard
stuff, though it turns out it's a bit ugly when using it as a development
environment for extensions liked against libs that are both in the system
and Anaconda provided.

Antoine Pitrou wrote:

> Note you can use pip inside conda environments, which works quite well
> at least for pure Python packages (start with "conda install pip").


True -- though it gets a bit ugly, as then conda doesn't know about the
packages, so switching environments is a mess, and  conda can't manage the
dependencies. So not ideal. I've actually spend the last two days writing a
script that auto-grabs packages from PyPI, builds conda packages out of
them, and then uploads them to binstar -- so we can have all our
dependencies supported by conda.

I'd love it if Continuum would build a "pip bridge" on binstar that would
do all that automagically if you request a pip-installable package from
binstar.


> Just throwing this into the mix as a potential solution. Note that you'll
> get to explain buildout to your users, but... :-)


yup -- not sure I want to go there yet, either...

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141230/520b2588/attachment.html>

From ncoghlan at gmail.com  Wed Dec 31 02:24:54 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Dec 2014 11:24:54 +1000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
 <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
 <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
Message-ID: <CADiSq7fGvJV_Szf5shyGD0u6m8vtcdTBwrZvyaBTy4Vqgg6+wg@mail.gmail.com>

On 23 December 2014 at 04:15, Vladimir Diaz <vladimir.v.diaz at gmail.com>
wrote:

> On Mon, Dec 22, 2014 at 11:30 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> From my perspective, the split into two PEPs meant most of the areas I
>> have doubts about have been moved to the end-to-end security model in PEP
>> 480, leaving PEP 458 to cover the simpler task of securing the link from
>> PyPI to the end user in such a way that public mirrors of packages can be
>> trusted to accurately reflect the content published by PyPI.
>>
>
> I think splitting the proposal into two PEPs was the right decision.  We
> hope working with Donald on the end-to-end security model (PEP 480), and
> feedback from the community will help to address any remaining questions.
> Excluding the end-to-end option from the revised version of PEP 458 also
> made room for an overview of the metadata and framework, which was
> requested by multiple members of the community.
>

An off-list question from Richard made me realise we should likely retitle
the two PEPs slightly. I'd suggest the following names:

PEP 458: Surviving a compromise of the PyPI CDN
PEP 480: Surviving a compromise of PyPI

That encapsulates the difference between the threat model of the two PEPs
in a way that the current titles don't quite convey (the reduced scope of
PEP 458 in particular means that the current title is actually outright
wrong - protecting against a compromise of PyPI itself is the scope that
was moved to PEP 480).

The reduced scope of PEP 458 also still protects against the compromise of
read-only mirrors, but I don't think we need to try to capture that
directly in the title.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/754074fb/attachment-0001.html>

From ncoghlan at gmail.com  Wed Dec 31 02:18:23 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Dec 2014 11:18:23 +1000
Subject: [Distutils] New User: depends not appearing in control file
In-Reply-To: <6698128.nGAEjiey7P@scotty>
References: <6698128.nGAEjiey7P@scotty>
Message-ID: <CADiSq7cL5mHvSs+v6MLDEPsjXTx22tacQherY7cZ_GgNQN4Kew@mail.gmail.com>

On 23 December 2014 at 02:26, Geoff Clements <geoff at electron.me.uk> wrote:

> Hi!
>
> I just started using stdeb but have run into a problem which I can't solve.
>

Hi Geoff,

I just noticed you didn't get a reply to this or your other question. For
distro specific tools like stdeb, you may need to look for assistance
through Debian and/or Ubuntu specific channels - while we do see folks like
Matthias Klose and Barry Warsaw here on distutils-sig occasionally, I don't
believe there's anyone specifically monitoring this list for questions
about the distro level integration utilities.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/1d1b288c/attachment.html>

From donald at stufft.io  Wed Dec 31 03:21:00 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 Dec 2014 21:21:00 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CADiSq7fGvJV_Szf5shyGD0u6m8vtcdTBwrZvyaBTy4Vqgg6+wg@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
 <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
 <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
 <CADiSq7fGvJV_Szf5shyGD0u6m8vtcdTBwrZvyaBTy4Vqgg6+wg@mail.gmail.com>
Message-ID: <E0C9C4C2-A869-4C89-9047-9C964129B487@stufft.io>


> On Dec 30, 2014, at 8:24 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 23 December 2014 at 04:15, Vladimir Diaz <vladimir.v.diaz at gmail.com <mailto:vladimir.v.diaz at gmail.com>> wrote:
> On Mon, Dec 22, 2014 at 11:30 AM, Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> wrote:
> From my perspective, the split into two PEPs meant most of the areas I have doubts about have been moved to the end-to-end security model in PEP 480, leaving PEP 458 to cover the simpler task of securing the link from PyPI to the end user in such a way that public mirrors of packages can be trusted to accurately reflect the content published by PyPI.
> 
> I think splitting the proposal into two PEPs was the right decision.  We hope working with Donald on the end-to-end security model (PEP 480), and feedback from the community will help to address any remaining questions.  Excluding the end-to-end option from the revised version of PEP 458 also made room for an overview of the metadata and framework, which was requested by multiple members of the community.
> 
> An off-list question from Richard made me realise we should likely retitle the two PEPs slightly. I'd suggest the following names:
> 
> PEP 458: Surviving a compromise of the PyPI CDN

This isn?t exactly right either, because it won?t survive a compromise of the CDN for *uploading*, but it might be close enough not to matter. Perhaps better would be something about not relying on TLS or something.

> PEP 480: Surviving a compromise of PyPI
> 
> That encapsulates the difference between the threat model of the two PEPs in a way that the current titles don't quite convey (the reduced scope of PEP 458 in particular means that the current title is actually outright wrong - protecting against a compromise of PyPI itself is the scope that was moved to PEP 480).
> 
> The reduced scope of PEP 458 also still protects against the compromise of read-only mirrors, but I don't think we need to try to capture that directly in the title.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141230/3ba76792/attachment.html>

From donald at stufft.io  Wed Dec 31 03:32:40 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 Dec 2014 21:32:40 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAHrZfZBFKK=LasZGtTtwvxiecGMQ0R+2REEQZmp+TzRuVDxWxg@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
 <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
 <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
 <CADiSq7fGvJV_Szf5shyGD0u6m8vtcdTBwrZvyaBTy4Vqgg6+wg@mail.gmail.com>
 <E0C9C4C2-A869-4C89-9047-9C964129B487@stufft.io>
 <CAHrZfZBFKK=LasZGtTtwvxiecGMQ0R+2REEQZmp+TzRuVDxWxg@mail.gmail.com>
Message-ID: <6371D126-5B01-484D-A322-34F1EAABF887@stufft.io>


> On Dec 30, 2014, at 9:29 PM, Richard Jones <richard at python.org> wrote:
> 
> Thanks for the clarification, guys.
> 
> Donald, I'm not sure what you mean by "a compromise of the CDN for *uploading*?.

PyPI trusts the CDN to give it the correct bits, without a signature from the author that is being verified uploading just relies on TLS again. The other PEP should close that gap though I believe.

Note: I have yet to read these PEPs so I?m just going by a casual glance of them.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141230/aa28eb72/attachment.html>

From richard at python.org  Wed Dec 31 03:29:01 2014
From: richard at python.org (Richard Jones)
Date: Wed, 31 Dec 2014 02:29:01 +0000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
 <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
 <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
 <CADiSq7fGvJV_Szf5shyGD0u6m8vtcdTBwrZvyaBTy4Vqgg6+wg@mail.gmail.com>
 <E0C9C4C2-A869-4C89-9047-9C964129B487@stufft.io>
Message-ID: <CAHrZfZBFKK=LasZGtTtwvxiecGMQ0R+2REEQZmp+TzRuVDxWxg@mail.gmail.com>

Thanks for the clarification, guys.

Donald, I'm not sure what you mean by "a compromise of the CDN for
*uploading*".


On Wed Dec 31 2014 at 1:21:18 PM Donald Stufft <donald at stufft.io> wrote:

> On Dec 30, 2014, at 8:24 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 23 December 2014 at 04:15, Vladimir Diaz <vladimir.v.diaz at gmail.com>
> wrote:
>
>> On Mon, Dec 22, 2014 at 11:30 AM, Nick Coghlan <ncoghlan at gmail.com>
>> wrote:
>>
>>> From my perspective, the split into two PEPs meant most of the areas I
>>> have doubts about have been moved to the end-to-end security model in PEP
>>> 480, leaving PEP 458 to cover the simpler task of securing the link from
>>> PyPI to the end user in such a way that public mirrors of packages can be
>>> trusted to accurately reflect the content published by PyPI.
>>>
>>
>> I think splitting the proposal into two PEPs was the right decision.  We
>> hope working with Donald on the end-to-end security model (PEP 480), and
>> feedback from the community will help to address any remaining questions.
>> Excluding the end-to-end option from the revised version of PEP 458 also
>> made room for an overview of the metadata and framework, which was
>> requested by multiple members of the community.
>>
>
> An off-list question from Richard made me realise we should likely retitle
> the two PEPs slightly. I'd suggest the following names:
>
> PEP 458: Surviving a compromise of the PyPI CDN
>
>
> This isn?t exactly right either, because it won?t survive a compromise of
> the CDN for *uploading*, but it might be close enough not to matter.
> Perhaps better would be something about not relying on TLS or something.
>
> PEP 480: Surviving a compromise of PyPI
>
> That encapsulates the difference between the threat model of the two PEPs
> in a way that the current titles don't quite convey (the reduced scope of
> PEP 458 in particular means that the current title is actually outright
> wrong - protecting against a compromise of PyPI itself is the scope that
> was moved to PEP 480).
>
> The reduced scope of PEP 458 also still protects against the compromise of
> read-only mirrors, but I don't think we need to try to capture that
> directly in the title.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>  _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/25944784/attachment-0001.html>

From ncoghlan at gmail.com  Wed Dec 31 04:52:34 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Dec 2014 13:52:34 +1000
Subject: [Distutils] Last call for feedback: PEP 440 exclusive ordered
	comparison fix
Message-ID: <CADiSq7eVR1jzJOYLBVDz0Lc822R3m1Wh-RGH6qXC1u2A6i9CNA@mail.gmail.com>

Donald is keen to get the updated versions of packaging/pip/setuptools out
that fix the regression in handling exclusive ordered comparison, so this
is a last call for feedback on those changes before we publish the updated
version of the PEP to python.org (and then update the tools accordingly).

Given the proximity to New Year's celebrations, I currently plan to publish
the updated version some time on Saturday 3rd January (UTC+10).

The actual definition of the change is in
https://github.com/pypa/interoperability-peps/commit/225de0a41270189dbfd4a038e8c6d56129580f8b

The key fix is making "> X.Y" accept "X.Y.Z" maintenance releases again,
which brings PEP 440 into line with the behaviour in pkg_resources prior to
the release of setuptools 8.0.

The change log also notes that this update includes the revision to the
normalised form of release candidates:
https://github.com/pypa/interoperability-peps/commit/68ce85faaca1ee715e2a9f46cdd14ed01a2b1d73

There's at least one more clarification to the PEP text still to be made
based on community feedback (
https://github.com/pypa/interoperability-peps/issues/6), but that doesn't
require any changes to the tools, so I don't mind if it doesn't make it
into this initial update.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/c13a2ca8/attachment.html>

From ncoghlan at gmail.com  Wed Dec 31 05:01:42 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Dec 2014 14:01:42 +1000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <6371D126-5B01-484D-A322-34F1EAABF887@stufft.io>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
 <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
 <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
 <CADiSq7fGvJV_Szf5shyGD0u6m8vtcdTBwrZvyaBTy4Vqgg6+wg@mail.gmail.com>
 <E0C9C4C2-A869-4C89-9047-9C964129B487@stufft.io>
 <CAHrZfZBFKK=LasZGtTtwvxiecGMQ0R+2REEQZmp+TzRuVDxWxg@mail.gmail.com>
 <6371D126-5B01-484D-A322-34F1EAABF887@stufft.io>
Message-ID: <CADiSq7eCF2SS147O7mxmDqMVuRV+WX4gXLhTFTxNzcuocrvdsg@mail.gmail.com>

On 31 December 2014 at 12:32, Donald Stufft <donald at stufft.io> wrote:

> PyPI trusts the CDN to give it the correct bits, without a signature from
> the author that is being verified uploading just relies on TLS again. The
> other PEP should close that gap though I believe.
>

I'm actually not sure what going through the CDN is buying us on the upload
side of things in the first place, given the main pay-off provided by a CDN
is geographically distributed caching of unchanging data for faster
downloads.

So it seems to me that that particular vulnerability could potentially be
fixed more simply by bypassing the CDN entirely for the upload case. That's
simplicity in the *conceptual* sense, though - there may be architectural
issues in the current implementation of PyPI and the related tools that
make it harder in practice than it would be in theory.

Either way, I agree that any kind of upload compromise based attack is also
out of scope for PEP 458 - that's now entirely about ensuring that the bits
delivered to end users are the same bits PyPI published. Making sure that
the bits *PyPI* publishes are the same ones that the *developer* published
is the domain of PEP 480.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/b467417e/attachment.html>

From richard at python.org  Wed Dec 31 05:14:33 2014
From: richard at python.org (Richard Jones)
Date: Wed, 31 Dec 2014 04:14:33 +0000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <CAOyQwLgcuYQxdi6AU3ORBrGQLQk_AeVmyW-SVCjrkdL6Ui8k_Q@mail.gmail.com>
 <CADiSq7eXzMmonVh_zBTAsGzZmN+zJGnL=1b=Pvtduj2nSNui4Q@mail.gmail.com>
 <CAOyQwLjdt2VmeSx44wmTAqw2OM7s7YCeJnU-niYK-H2P+mqn-A@mail.gmail.com>
 <CADiSq7fGvJV_Szf5shyGD0u6m8vtcdTBwrZvyaBTy4Vqgg6+wg@mail.gmail.com>
 <E0C9C4C2-A869-4C89-9047-9C964129B487@stufft.io>
 <CAHrZfZBFKK=LasZGtTtwvxiecGMQ0R+2REEQZmp+TzRuVDxWxg@mail.gmail.com>
 <6371D126-5B01-484D-A322-34F1EAABF887@stufft.io>
 <CADiSq7eCF2SS147O7mxmDqMVuRV+WX4gXLhTFTxNzcuocrvdsg@mail.gmail.com>
Message-ID: <CAHrZfZDFvoirvQcpOKPYGfUnofgymB7Sn-EmgM0t7Y=sxX8p9A@mail.gmail.com>

Now that I think about it, I'm almost certain that Donald and I have had
the "hey, what about an upload.pypi.python.org" conversation in the past,
as a way around issues involving the CDN :)

Still a good idea, in my opinion.


    Richard


On Wed Dec 31 2014 at 3:01:53 PM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 31 December 2014 at 12:32, Donald Stufft <donald at stufft.io> wrote:
>
>> PyPI trusts the CDN to give it the correct bits, without a signature from
>> the author that is being verified uploading just relies on TLS again. The
>> other PEP should close that gap though I believe.
>>
>
> I'm actually not sure what going through the CDN is buying us on the
> upload side of things in the first place, given the main pay-off provided
> by a CDN is geographically distributed caching of unchanging data for
> faster downloads.
>
> So it seems to me that that particular vulnerability could potentially be
> fixed more simply by bypassing the CDN entirely for the upload case. That's
> simplicity in the *conceptual* sense, though - there may be architectural
> issues in the current implementation of PyPI and the related tools that
> make it harder in practice than it would be in theory.
>
> Either way, I agree that any kind of upload compromise based attack is
> also out of scope for PEP 458 - that's now entirely about ensuring that the
> bits delivered to end users are the same bits PyPI published. Making sure
> that the bits *PyPI* publishes are the same ones that the *developer*
> published is the domain of PEP 480.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/8c92e882/attachment.html>

From donald at stufft.io  Wed Dec 31 08:26:16 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 Dec 2014 02:26:16 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
Message-ID: <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>


> On Dec 10, 2014, at 10:16 PM, Vladimir Diaz <vladimir.v.diaz at gmail.com> wrote:
> 
> Hello everyone,
> 
> I am a research programmer at the NYU School of Engineering.  My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI."  The two-stage proposal can be reviewed online at:
> 
> PEP 458
> http://legacy.python.org/dev/peps/pep-0458/ <http://legacy.python.org/dev/peps/pep-0458/>
> 
> PEP 480
> http://legacy.python.org/dev/peps/pep-0480/ <http://legacy.python.org/dev/peps/pep-0480/>
> 
> 
> Summary of the Proposal:
> 
> "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users.  The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI.  Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed.  The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata.
> 
> The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects.  
> 
> The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users.  If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key.  As in PEP 458, no additional action is required by end users.  However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key.  PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing.  The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month.
> 
> We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback.


I?ve just finished (re)reading the white paper, PEP 450, PEP 480, and some of the supporting documentation on the TUF website.

I?m confused about what exactly is contained within the TUF metadata and who signs what in a PEP 480 world.

Currently when you do something like ``pip install FooBar``, pip fetches /simple/FooBar/ to look for potential installation candidates, and when it finds one it downloads it and installs it. This all all ?signed? by online keys via TLS.

1. In a TUF world, would pip still fetch /simple/FooBar/ to discover things to install or would it fetch some TUF metadata to find things to install?
2. If it?s fetching /simple/FooBar/ is that secured by TUF?
3. If it?s secured by TUF who signs the TUF metadata that talks about /simple/FooBar/ in PEP 480 the author or PyPI?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/69e9f58c/attachment-0001.html>

From reinout at vanrees.org  Wed Dec 31 09:39:37 2014
From: reinout at vanrees.org (Reinout van Rees)
Date: Wed, 31 Dec 2014 09:39:37 +0100
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <CALGmxEKbdUTkGi5zUiiF4vhMUjqkK6knEk8UKsZai40p3i8Ezw@mail.gmail.com>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
 <m7v8gc$471$1@ger.gmane.org>
 <CALGmxEKbdUTkGi5zUiiF4vhMUjqkK6knEk8UKsZai40p3i8Ezw@mail.gmail.com>
Message-ID: <m80co7$fjo$1@ger.gmane.org>

Chris Barker schreef op 31-12-14 om 01:42:
>     The combination we use now is to use buildout (instead of pip) in
>     combination with the "syseggrecipe"
>     (https://pypi.python.org/pypi/__syseggrecipe
>     <https://pypi.python.org/pypi/syseggrecipe>) buildout add-on.
>     Syseggrecipe allows us to depend explicitly on **system** packages
>     for gdal/matplotlib/mapnik/scipy/__numpy. Which takes care of most
>     of the compilation headaches.
>
>
> well, it sounds like you are simple punting -- passing off all the
> complex stuff to the system, which may work well if the system is up to
> date linux with the packages you need available, but pretty worthless on
> a Mac or Windows box.

You're completely right! We pass on the should-be-compiled packages to 
the OS, in our case ubuntu.

... which means that I'm using vagrant/vmware on my mac with an ubuntu 
vm on it.

... which means that at a certain point you need a PPA for a newer 
version for GDAL which brings in other new stuff which breaks after half 
a year...

So: not ideal :-)



Reinout

-- 
Reinout van Rees                          http://reinout.vanrees.org/
reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"


From vladimir.v.diaz at gmail.com  Wed Dec 31 17:08:11 2014
From: vladimir.v.diaz at gmail.com (Vladimir Diaz)
Date: Wed, 31 Dec 2014 11:08:11 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
Message-ID: <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>

On Wed, Dec 31, 2014 at 2:26 AM, Donald Stufft <donald at stufft.io> wrote:

>
> On Dec 10, 2014, at 10:16 PM, Vladimir Diaz <vladimir.v.diaz at gmail.com>
> wrote:
>
> Hello everyone,
>
> I am a research programmer at the NYU School of Engineering.  My
> colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting
> community feedback on our proposal, "Surviving a Compromise of PyPI."  The
> two-stage proposal can be reviewed online at:
>
> PEP 458
> http://legacy.python.org/dev/peps/pep-0458/
>
> PEP 480
> http://legacy.python.org/dev/peps/pep-0480/
>
>
> Summary of the Proposal:
>
> "Surviving a Compromise of PyPI" proposes how the Python Package Index
> (PyPI) can be amended to better protect end users from altered or malicious
> packages, and to minimize the extent of PyPI compromises against affected
> users.  The proposed integration allows package managers such as pip to be
> more secure against various types of security attacks on PyPI and defend
> end users from attackers responding to package requests. Specifically,
> these PEPs describe how PyPI processes should be adapted to generate and
> incorporate repository metadata, which are signed text files that describe
> the packages and metadata available on PyPI.  Package managers request
> (along with the packages) the metadata on PyPI to verify the authenticity
> of packages before they are installed.  The changes to PyPI and tools will
> be minimal by leveraging a library, The Update Framework
> <https://github.com/theupdateframework/tuf>, that generates and
> transparently validates the relevant metadata.
>
> The first stage of the proposal (PEP 458
> <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security
> model that supports verification of PyPI packages signed with cryptographic
> keys stored on PyPI, requires no action from developers and end users, and
> protects against malicious CDNs and public mirrors. To support continuous
> delivery of uploaded packages, PyPI administrators sign for uploaded
> packages with an online key stored on PyPI infrastructure. This level of
> security prevents packages from being accidentally or deliberately tampered
> with by a mirror or a CDN because the mirror or CDN will not have any of
> the keys required to sign for projects.
>
> The second stage of the proposal (PEP 480
> <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the
> basic security model (discussed in PEP 458) that supports end-to-end
> verification of signed packages. End-to-end signing allows both PyPI and
> developers to sign for the packages that are downloaded by end users.  If
> the PyPI infrastructure were to be compromised, attackers would be unable
> to serve malicious versions of these packages without access to the
> project's developer key.  As in PEP 458, no additional action is required
> by end users.  However, PyPI administrators will need to periodically
> (perhaps every few months) sign metadata with an offline key.  PEP 480 also
> proposes an easy-to-use key management solution for developers, how to
> interface with a potential build farm on PyPI infrastructure, and discusses
> the security benefits of end-to-end signing.  The second stage of the
> proposal simultaneously supports real-time project registration and
> developer signatures, and when configured to maximize security on PyPI,
> less than 1% of end users will be at risk even if an attacker controls PyPI
> and goes undetected for a month.
>
> We thank Nick Coghlan and Donald Stufft for their valuable contributions,
> and Giovanni Bajo and Anatoly Techtonik for their feedback.
>
>
> I?ve just finished (re)reading the white paper, PEP 450, PEP 480, and some
> of the supporting documentation on the TUF website.
>

Thanks!


>
> I?m confused about what exactly is contained within the TUF metadata and
> who signs what in a PEP 480 world.
>

The following illustration shows what is contained within TUF metadata
(JSON files):
https://github.com/vladimir-v-diaz/pep-on-pypi-with-tuf/raw/master/pep-0458/figure4.pdf
Note: In this illustration, the "snapshot" and "targets" roles are renamed
"release" and "projects", respectively.

If you're interested in what exactly is contained in these JSON files, here
is example metadata:
https://github.com/theupdateframework/tuf/tree/develop/examples/repository/metadata

In a PEP 480 world, project developers sign a single JSON file. For
example, developer(s) for the "Request" project sign their assigned JSON
file named "/targets/claimed/Requests.json".  Specifically, a signature is
generated of the "signed" entry
<https://github.com/theupdateframework/tuf/blob/develop/examples/repository/metadata/targets.json#L9-L49>
of the dictionary.  Once the signature is generated, it is added to the
"signatures" entry
<https://github.com/theupdateframework/tuf/blob/develop/examples/repository/metadata/targets.json#L2-L7>
of the JSON file.

In figure 1 of PEP 480, PyPI signs for every metadata except those listed
under the "roles signed by developer keys" label:
https://github.com/vladimir-v-diaz/pep-maximum-security-model/blob/master/pep-0480/figure1.png


> Currently when you do something like ``pip install FooBar``, pip fetches
> /simple/FooBar/ to look for potential installation candidates, and when it
> finds one it downloads it and installs it. This all all ?signed? by online
> keys via TLS.
>
> 1. In a TUF world, would pip still fetch /simple/FooBar/ to discover
> things to install or would it fetch some TUF metadata to find things to
> install?
>

In the integration/demo we did with pip, we treated each /simple/ html file
as a target (listed the hash and file size of these html index pages in TUF
metadata).  That is, pip still fetched /simple/FooBar/ to discover
distributions to install, but we verified the html files *and*
distributions against TUF metadata.  In PEP 458, we state that "/simple" is
also listed in TUF metadata:
http://legacy.python.org/dev/peps/pep-0458/#pypi-and-tuf-metadata (last
paragraph just before the diagram).

Another option is to avoid crawling/listing the simple index pages and just
search TUF metadata for distributions, but this approach will require
design changes to pip.  We went with the approach (treat the index pages as
targets) that required minimal changes to pip.


> 2. If it?s fetching /simple/FooBar/ is that secured by TUF?
>

Yes, see my response to (1).

3. If it?s secured by TUF who signs the TUF metadata that talks about
> /simple/FooBar/ in PEP 480 the author or PyPI?
>

PEP 480 authors sign for both their project's index page and
distribution(s) (as indicated in the JSON file):

"A claimed or recently-claimed project will need to upload in its
transaction to PyPI not just targets (a simple index as well as
distributions) but also TUF metadata. The project MAY do so by uploading a
ZIP file containing two directories, /metadata/ (containing delegated
targets metadata files) and /targets/ (containing targets such as the
project simple index and distributions that are signed by the delegated
targets metadata)."

See the second paragraph of
http://legacy.python.org/dev/peps/pep-0480/#snapshot-process.

Let me know exactly what needs to change in the PEPs to make everything
explained above clearer.  For example, in PEP 458 we provide a
link/reference
<https://www.python.org/dev/peps/pep-0458/#what-additional-repository-files-are-required-on-pypi>
(last paragraph of this subsection) to the Metadata document
<https://github.com/theupdateframework/tuf/blob/develop/METADATA.md>
indicating the content of the JSON files, but should the illustration
<https://github.com/vladimir-v-diaz/pep-on-pypi-with-tuf/raw/master/pep-0458/figure4.pdf>
I've included in this reply also be added?


>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/afda54c9/attachment.html>

From p.f.moore at gmail.com  Wed Dec 31 17:54:46 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 Dec 2014 16:54:46 +0000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
Message-ID: <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>

On 31 December 2014 at 16:08, Vladimir Diaz <vladimir.v.diaz at gmail.com> wrote:
> Let me know exactly what needs to change in the PEPs to make everything
> explained above clearer.  For example, in PEP 458 we provide a
> link/reference (last paragraph of this subsection) to the Metadata document
> indicating the content of the JSON files, but should the illustration I've
> included in this reply also be added?

I don't know how generally useful this would be, and I can't even
promise I've got any useful comments to make, but I find the proposals
too full of concepts I don't really follow (as someone who isn't a
PyPI admin or a security specialist) to be able to get much from them.
Is there anywhere a document that simply explains, from the point of
view of a package author, what I would need to do that is different
from right now, in order to benefit from the proposal? (I assume the
benefits are "your users can be sure that they get the files you
uploaded, without tampering", and that's sufficient explanation of the
benefit side from my perspective).

For example, you say "PEP 480 authors sign for both their project's
index page and distribution(s)". Does that mean I need to add
something to the command line when I do "setup.py upload"? Can I still
set up an automated build process or will it now need manual entry of
some sort of passphrase in order to work?

I appreciate that the target audience for these PEPs is really PyPI
admins, at the moment, so maybe it's not the right time to look at
them from a project author perspective - if so, then feel free to
ignore these points for now :-)

Thanks,
Paul

From ncoghlan at gmail.com  Wed Dec 31 18:10:20 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 1 Jan 2015 03:10:20 +1000
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <CALGmxEKbdUTkGi5zUiiF4vhMUjqkK6knEk8UKsZai40p3i8Ezw@mail.gmail.com>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
 <m7v8gc$471$1@ger.gmane.org>
 <CALGmxEKbdUTkGi5zUiiF4vhMUjqkK6knEk8UKsZai40p3i8Ezw@mail.gmail.com>
Message-ID: <CADiSq7cACHgyi-O27EFWGjmuT6Wp1VhKTJLWfHPsTJJRg9PdcA@mail.gmail.com>

On 31 Dec 2014 10:43, "Chris Barker" <chris.barker at noaa.gov> wrote:
>
> But the core problem here is that the scipy folks have been going to
conda and enthought to solve their pacakgeing problems, and the web folks
have been doing pip, and maybe buildout -- so you get a bit of mess when
you mix them.

The problem always existed - it's the longstanding conflict between
"platform independent, language specific" tooling and "platform specific,
language independent" tooling.

The former is often preferred on the developer side (since the tools are
oriented towards building individual custom applications rather than
maintaining a suite of applications written by different groups), while the
latter is essential on the system integrator side (since you're dealing
with inevitable technology sprawl in infrastructure that evolves over the
course of years and decades).

One of the best things coming out of the whole "DevOps" notion is the
recognition that language independence and platform independence are aimed
at solving *different problems*, and that what we need is suitable tools
with different roles and responsibilities that interoperate effectively,
rather than attempting to provide a single universal tool and having
different groups of users yelling at each other for wanting to solve the
"wrong" problem.

Tools like conda and zc.buildout fit into that landscape by aiming to
provide a platform & language independent approach to doing *application*
level integration, which tends to have the effect of making them new
platforms in their own right. If you compare them to Linux distributions,
then zc.buildout is a platform that practices the Gentoo style approach of
building everything from source for each application, while conda uses the
more common Debian/Fedora style of defining a binary packaging format that
allows a redistributor to handle the build process on behalf of their users.

Cheers,
Nick.

P.S. conda's actually one of the projects on my preliminary short list for
consideration as a possible userspace package management tool in Fedora:
https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement#Recommending_language_independent_tooling_.28longer_term.29
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150101/341d1823/attachment.html>

From ncoghlan at gmail.com  Wed Dec 31 18:42:30 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 1 Jan 2015 03:42:30 +1000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
 <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
Message-ID: <CADiSq7fdYOuTmq0NJzR2jxYgebYZ+9RMFrLF3E=Oj9WkP-88ug@mail.gmail.com>

On 1 January 2015 at 02:54, Paul Moore <p.f.moore at gmail.com> wrote:

> I appreciate that the target audience for these PEPs is really PyPI
> admins, at the moment, so maybe it's not the right time to look at
> them from a project author perspective - if so, then feel free to
> ignore these points for now :-)
>

I think capturing the UX consequences is important, especially as that's
actually the key driver behind the split into two PEPs. In theory, the UX
consequences for PyPI hosted projects under PEP 458 should be close to nil
- pip should just silently update to using the additional content
validation mechanism behind the scenes, and thus become capable of
detecting additional forms of illicit tampering that can't be detected
through the existing use of transport layer security.

Projects using external hosting *will* see a UX change under PEP 458, and
that's something that will need to be figured out before the PEP can be
accepted.

PEP 480 is far more challenging from a UX perspective, as it brings in the
notion of developer managed keys and the associated signed uploads. Key
management *is* a pain, especially in distributed developer communities.
Trust management in particular is *not* an easy problem to solve - hence
why we still rely on the CA system for TLS (despite its many known attack
vectors), and why we see the complicated situation around SecureBoot, where
community Linux distributions are generally bootstrapped through a shim
signed through Microsoft's developer program, since that allows them to
"just work" on hardware that only has Microsoft's chain of trust checking
keys pre-installed.

PEP 458 is almost certainly a solid enhancement to PyPI's overall security
(assuming we can come up with an acceptable answer for external hosting).
It's significantly less clear that PEP 480 is the right answer for
delegating more signing authority to developer groups - for example, it may
be better to come up with a federated *hosting* model, where external
hosting is the answer if developers choose to use their own signing keys
rather than PyPI's automated online keys. Things like the Rackspace
developer program, or the emerging next generation of Docker-based
Platform-as-a-Service offerings, make it easier to recommend such federated
hosting models.

As a result, my perspective is that it's the UX design concept that will
make or break PEP 480 - the security model of TUF looks great to me, what
gives me pause is concern over the usability and maintainability of signed
uploads for "developers in a hurry".

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150101/4566ca20/attachment.html>

From vladimir.v.diaz at gmail.com  Wed Dec 31 18:43:57 2014
From: vladimir.v.diaz at gmail.com (Vladimir Diaz)
Date: Wed, 31 Dec 2014 12:43:57 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
 <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
Message-ID: <CAOyQwLh39pFre-Gh34BTf9SEbHsnjvQuzAhrTwjVcd_Zfr7XTw@mail.gmail.com>

In the PEP 458 world, package authors are not required to do anything.

On Wed, Dec 31, 2014 at 11:54 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 31 December 2014 at 16:08, Vladimir Diaz <vladimir.v.diaz at gmail.com>
> wrote:
> > Let me know exactly what needs to change in the PEPs to make everything
> > explained above clearer.  For example, in PEP 458 we provide a
> > link/reference (last paragraph of this subsection) to the Metadata
> document
> > indicating the content of the JSON files, but should the illustration
> I've
> > included in this reply also be added?
>
> I don't know how generally useful this would be, and I can't even
> promise I've got any useful comments to make, but I find the proposals
> too full of concepts I don't really follow (as someone who isn't a
> PyPI admin or a security specialist) to be able to get much from them.
> Is there anywhere a document that simply explains, from the point of
> view of a package author, what I would need to do that is different
> from right now, in order to benefit from the proposal? (I assume the
> benefits are "your users can be sure that they get the files you
> uploaded, without tampering", and that's sufficient explanation of the
> benefit side from my perspective).
>
> For example, you say "PEP 480 authors sign for both their project's
> index page and distribution(s)". Does that mean I need to add
> something to the command line when I do "setup.py upload"? Can I still
> set up an automated build process or will it now need manual entry of
> some sort of passphrase in order to work?
>
> I appreciate that the target audience for these PEPs is really PyPI
> admins, at the moment, so maybe it's not the right time to look at
> them from a project author perspective - if so, then feel free to
> ignore these points for now :-)
>

Nick has stated that a PEP specifically intended for the project author
perspective will be drafted after the "PyPI Admin" PEPs.  For now, I think
Nick wants to focus on the bits (metadata) signed and published by PyPI
(PEP 458).  In the PEP 458 model, package authors are not required to sign
and upload metadata; they will upload distributions as it is currently done.

However, community feedback on PEP 480 (which supports signatures provided
by project authors) is still appreciated.  Although we haven't finalized
all of the details on how the third-party signing tools will change, we can
still discuss the overall approach.  For example, should signing a
distribution require only a secondary passphrase, akin to miniLock
<https://minilock.io/>?  Should signing a distribution be optional?  A
manual process?  We hope to reach a design that is suported by the majority
of the Python community.

PEP 480 includes a section that discusses a potential approach to packages
signed by package authors:
https://www.python.org/dev/peps/pep-0480/#automated-signing-solution

Let us know what you think.


> Thanks,
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/7c2585c0/attachment-0001.html>

From chris.barker at noaa.gov  Wed Dec 31 18:41:15 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 31 Dec 2014 09:41:15 -0800
Subject: [Distutils] Role of setuptools and eggs in "modern"
	distributing...
In-Reply-To: <CADiSq7cACHgyi-O27EFWGjmuT6Wp1VhKTJLWfHPsTJJRg9PdcA@mail.gmail.com>
References: <CALGmxEJg+dFH7KOFe_YxnGVut83DATeZ89mTDA=qF=vCrjoFow@mail.gmail.com>
 <m7v8gc$471$1@ger.gmane.org>
 <CALGmxEKbdUTkGi5zUiiF4vhMUjqkK6knEk8UKsZai40p3i8Ezw@mail.gmail.com>
 <CADiSq7cACHgyi-O27EFWGjmuT6Wp1VhKTJLWfHPsTJJRg9PdcA@mail.gmail.com>
Message-ID: <CALGmxE+5oDW1Aw27gb1m_p8E7CZnvGa+n0Y5u=AgjeHC0bZNSg@mail.gmail.com>

On Wed, Dec 31, 2014 at 9:10 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> The problem always existed - it's the longstanding conflict between
> "platform independent, language specific" tooling and "platform specific,
> language independent" tooling.
>
> The former is often preferred on the developer side (since the tools are
> oriented towards building individual custom applications rather than
> maintaining a suite of applications written by different groups), while the
> latter is essential on the system integrator side (since you're dealing
> with inevitable technology sprawl in infrastructure that evolves over the
> course of years and decades).
>
> One of the best things coming out of the whole "DevOps" notion is the
> recognition that language independence and platform independence are aimed
> at solving *different problems*, and that what we need is suitable tools
> with different roles and responsibilities that interoperate effectively,
> rather than attempting to provide a single universal tool and having
> different groups of users yelling at each other for wanting to solve the
> "wrong" problem.
>
> Tools like conda and zc.buildout fit into that landscape by aiming to
> provide a platform & language independent approach to doing *application*
> level integration, which tends to have the effect of making them new
> platforms in their own right.
>
Indeed -- thanks for providing a clear way to talk/think about these
systems.

I guess the trick here is that we want the different level tools to work
well with each-other.

As conda started with python packages in mind, it does a pretty good job
with them. But I still find some conflicts between setuptools and conda --
in particular, if you specify dependencies in setup.py (install_requires),
it can kind of make a mess of things. conda tries to ignore them, but
somehow I've had issues, even though I had specified it all in the conda's
meta.yaml. This is probably a conda bug/issue, but I'm still trying to
figure out how to best set up a python package so that it can be built
installed with the "regular" python tools, and also conda...

Use case -- developing in a conda environment -- so I want to install
dependencies with conda, but the package under development with setuptools
"develop" mode. (conda does not (yet) have a develop mode that works...)

Oh, and there does seem to be an odd (I think non-fatal) issue with
setuptools and conda:

I have package A, with a bunch of stuff listed with "install_requires"

I have all these dependencies installed with conda.

When I run setup.py develop, setuptools goes and dumps all the dependencies
in easy_install.pth.

I have no idea why that is, and it's probably only annoying, rather than a
problem, but I'm not sure what will happen when I upgrade one of those
dependencies with conda.

> If you compare them to Linux distributions, then zc.buildout is a platform
> that practices the Gentoo style approach of building everything from source
> for each application, while conda uses the more common Debian/Fedora style
> of defining a binary packaging format that allows a redistributor to handle
> the build process on behalf of their users.
>
indeed -- and conda actually provides (to my disappointment) very little in
the way of build support -- you need to write platform dependent build
scripts to actually build the packages.

But it does provide a nice way for me to provide a full "just install this"
distribution of my complex, ugly, hard to build packages....

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/dfe6d947/attachment.html>

From p.f.moore at gmail.com  Wed Dec 31 19:04:36 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 Dec 2014 18:04:36 +0000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAOyQwLh39pFre-Gh34BTf9SEbHsnjvQuzAhrTwjVcd_Zfr7XTw@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
 <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
 <CAOyQwLh39pFre-Gh34BTf9SEbHsnjvQuzAhrTwjVcd_Zfr7XTw@mail.gmail.com>
Message-ID: <CACac1F8ra45YH1+Jio=VdNUg7dsADux8oWgYAdN7y0RHS1dHJA@mail.gmail.com>

On 31 December 2014 at 17:43, Vladimir Diaz <vladimir.v.diaz at gmail.com> wrote:
> PEP 480 includes a section that discusses a potential approach to packages
> signed by package authors:
> https://www.python.org/dev/peps/pep-0480/#automated-signing-solution
>
> Let us know what you think.

Thanks for the pointer. I read the section you referred to (admittedly
in isolation). The language is unfamiliar to me, so I'm afraid I
didn't get much from it. For example, I don't know what miniLock is,
so that analogy was no help. Also, the phrase "the sharing of private
keys across multiple machines of each developer" didn't mean much
other than to raise alarms for me that I might not be able to simply
log onto a new machine (a VM, for example, or a work machine) and do a
quick "git clone; hack; python setup.py upload" to release an
emergency fix, as I'd need a private key with me (as opposed to a
password I can remember), and I'd needto do something to "allow key
sharing" . That would be annoying.

The "Enter a secondary password" note struck me as odd. Why would I
need a *second* password? And why wouldn't I just reuse the same
password as I use for PyPI? After all, I'm trusting that password
hasn't been compromised, why make it harder on myself by needing to
remember two passwords?

Terminology-wise, I don't know what "adding a new identity" means. Is
that authorising a second developer? Or could I need to have multiple
"identities" myself? The first is fine, the second isn't (I'm me, why
do I need to have 2 identities just to upload a distribution)?

I'm aware of (and sorry about) the fact that this is very much a
"drive by" scan of one section of the proposal in isolation. I *hope*
it's still useful feedback, even if it's neither thorough nor
particularly thoughtful - I was sort of aiming for "something is
better than nothing", and that's all :-)

Anyway, I'll leave further comment to people with a better
understanding of the issue, although I'm happy to clarify if any of
the above isn't clear.

Paul.

From ncoghlan at gmail.com  Wed Dec 31 19:24:08 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 1 Jan 2015 04:24:08 +1000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CACac1F8ra45YH1+Jio=VdNUg7dsADux8oWgYAdN7y0RHS1dHJA@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
 <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
 <CAOyQwLh39pFre-Gh34BTf9SEbHsnjvQuzAhrTwjVcd_Zfr7XTw@mail.gmail.com>
 <CACac1F8ra45YH1+Jio=VdNUg7dsADux8oWgYAdN7y0RHS1dHJA@mail.gmail.com>
Message-ID: <CADiSq7fDQZM2D7nGrG5HbczR-QWT7cwAii7jzRiWU+JLkDtORA@mail.gmail.com>

On 1 January 2015 at 04:04, Paul Moore <p.f.moore at gmail.com> wrote:

> Anyway, I'll leave further comment to people with a better
> understanding of the issue, although I'm happy to clarify if any of
> the above isn't clear.
>

Expert blindness can be a serious problem when it comes to security design,
so please keep the questions coming. I've come to the realisation that
having done things like blending aspects of the 802.11i WPA2 spec with HF
automatic link establishment to come up with a custom authentication
protocol means I'm no longer qualified to judge what counts as "common
knowledge" in these areas, and the TUF folks leave me in the dust :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150101/fa992080/attachment.html>

From donald at stufft.io  Wed Dec 31 19:42:27 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 Dec 2014 13:42:27 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CACac1F8ra45YH1+Jio=VdNUg7dsADux8oWgYAdN7y0RHS1dHJA@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
 <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
 <CAOyQwLh39pFre-Gh34BTf9SEbHsnjvQuzAhrTwjVcd_Zfr7XTw@mail.gmail.com>
 <CACac1F8ra45YH1+Jio=VdNUg7dsADux8oWgYAdN7y0RHS1dHJA@mail.gmail.com>
Message-ID: <BFC4547E-79E7-4355-8154-1CA4BF1C9676@stufft.io>


> On Dec 31, 2014, at 1:04 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 December 2014 at 17:43, Vladimir Diaz <vladimir.v.diaz at gmail.com> wrote:
>> PEP 480 includes a section that discusses a potential approach to packages
>> signed by package authors:
>> https://www.python.org/dev/peps/pep-0480/#automated-signing-solution
>> 
>> Let us know what you think.
> 
> Thanks for the pointer. I read the section you referred to (admittedly
> in isolation). The language is unfamiliar to me, so I'm afraid I
> didn't get much from it. For example, I don't know what miniLock is,
> so that analogy was no help. Also, the phrase "the sharing of private
> keys across multiple machines of each developer" didn't mean much
> other than to raise alarms for me that I might not be able to simply
> log onto a new machine (a VM, for example, or a work machine) and do a
> quick "git clone; hack; python setup.py upload" to release an
> emergency fix, as I'd need a private key with me (as opposed to a
> password I can remember), and I'd needto do something to "allow key
> sharing" . That would be annoying.
> 
> The "Enter a secondary password" note struck me as odd. Why would I
> need a *second* password? And why wouldn't I just reuse the same
> password as I use for PyPI? After all, I'm trusting that password
> hasn't been compromised, why make it harder on myself by needing to
> remember two passwords?

Just to speak to these two points. The purpose behind having a developer
sign some files is that you can verify that those files were signed by
the person holding the private key belonging to that developer.

In more traditional systems like RSA that key is pretty long, For example
here?s an RSA private key for TLS: https://github.com/python/psf-salt/blob/master/pillar/dev/secrets/tls/certs/loadbalancer.sls#L6-L33.

Obviously managing a file like that is kind of a pain and it prevents things
like being able to spin up a new VM quickly to make a release. Traditionally
only the author themselves have access to the private key so there?s no way
to get a copy from PyPI or something. The reason for this is that possession
of the private key is how you prove that you?re really the author.

One option is something called key escrow which essentially means that you
take your private key, encrypt it with a password and upload the encrypted
blob to PyPI. Then as long as your password is good and the encryption
algorithm is good it doesn?t matter if PyPI gets compromised because the
encrypted data isn?t able to be decrypted without the password known only
to the admin. The particular solution makes a lot of security folks nervous
only because if the password *is* bad or something does happen to break the
encryption algorithm it means that PyPI now has access to keys that it would
be preferential that it didn?t.

There are other signing algorithms, such as Ed25519 where the private keys can
be much shorter than in RSA. Ed25519 in particular requires only 32 bytes of
data for a private key. In addition to that there are particular algorithms that
can take an arbitrary number of characters (aka a user picked password) and derive
from that 32 bytes that can be used as the Ed25519 private key. This reduces
key management down to something as simple as ?remember a second password? without
needing to resort to something like key escrow. Ideally you would not use the
same password as you use for logging into PyPI because you send that password to
PyPI anytime you login which would mean that PyPI would more or less know your
private key.

> 
> Terminology-wise, I don't know what "adding a new identity" means. Is
> that authorising a second developer? Or could I need to have multiple
> "identities" myself? The first is fine, the second isn't (I'm me, why
> do I need to have 2 identities just to upload a distribution)?
> 
> I'm aware of (and sorry about) the fact that this is very much a
> "drive by" scan of one section of the proposal in isolation. I *hope*
> it's still useful feedback, even if it's neither thorough nor
> particularly thoughtful - I was sort of aiming for "something is
> better than nothing", and that's all :-)
> 
> Anyway, I'll leave further comment to people with a better
> understanding of the issue, although I'm happy to clarify if any of
> the above isn't clear.
> 
> Paul.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From p.f.moore at gmail.com  Wed Dec 31 20:05:32 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 Dec 2014 19:05:32 +0000
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <BFC4547E-79E7-4355-8154-1CA4BF1C9676@stufft.io>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
 <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
 <CAOyQwLh39pFre-Gh34BTf9SEbHsnjvQuzAhrTwjVcd_Zfr7XTw@mail.gmail.com>
 <CACac1F8ra45YH1+Jio=VdNUg7dsADux8oWgYAdN7y0RHS1dHJA@mail.gmail.com>
 <BFC4547E-79E7-4355-8154-1CA4BF1C9676@stufft.io>
Message-ID: <CACac1F915tgo-r3AhOa=Gd-vkyDSrHBoFhbJVxiT1gw4a+KH6g@mail.gmail.com>

On 31 December 2014 at 18:42, Donald Stufft <donald at stufft.io> wrote:
> Just to speak to these two points. The purpose behind having a developer
> sign some files is that you can verify that those files were signed by
> the person holding the private key belonging to that developer.
[...]

Thanks for the explanation.

> Ideally you would not use the
> same password as you use for logging into PyPI because you send that password to
> PyPI anytime you login which would mean that PyPI would more or less know your
> private key.

My problem with this logic is that there's another attack vector that
this ignores - what if someone gets access to my PC, which has all of
these passwords in a "saved password" store that I use because it's a
pain to manage so many passwords (I don't, but you get the point ;-))?
I work in a number of secure environments where multiple complex
passwords are mandated - and typically password management becomes
sufficiently hard that people start to use shortcuts, defeating the
object of the whole exercise (heck, end users probably just use
"Password01" everywhere, "because it's too hard to remember all those
passwords"...)

That's not to say that bad security practices justify anything, but on
the other hand human factors do imply that it's not automatically
guaranteed that two passwords are more secure than one. Single sign-on
is a goal for a lot of people for a reason...

Paul

From donald at stufft.io  Wed Dec 31 20:33:23 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 Dec 2014 14:33:23 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CACac1F915tgo-r3AhOa=Gd-vkyDSrHBoFhbJVxiT1gw4a+KH6g@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
 <CACac1F8CLavCT=eu1Rs5VrzBYEu7=_FQGZR20i2WEhBcTfhP7Q@mail.gmail.com>
 <CAOyQwLh39pFre-Gh34BTf9SEbHsnjvQuzAhrTwjVcd_Zfr7XTw@mail.gmail.com>
 <CACac1F8ra45YH1+Jio=VdNUg7dsADux8oWgYAdN7y0RHS1dHJA@mail.gmail.com>
 <BFC4547E-79E7-4355-8154-1CA4BF1C9676@stufft.io>
 <CACac1F915tgo-r3AhOa=Gd-vkyDSrHBoFhbJVxiT1gw4a+KH6g@mail.gmail.com>
Message-ID: <C59CC5F2-FB45-4D99-B98A-6756D9C7FB6B@stufft.io>


> On Dec 31, 2014, at 2:05 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 December 2014 at 18:42, Donald Stufft <donald at stufft.io> wrote:
>> Just to speak to these two points. The purpose behind having a developer
>> sign some files is that you can verify that those files were signed by
>> the person holding the private key belonging to that developer.
> [...]
> 
> Thanks for the explanation.
> 
>> Ideally you would not use the
>> same password as you use for logging into PyPI because you send that password to
>> PyPI anytime you login which would mean that PyPI would more or less know your
>> private key.
> 
> My problem with this logic is that there's another attack vector that
> this ignores - what if someone gets access to my PC, which has all of
> these passwords in a "saved password" store that I use because it's a
> pain to manage so many passwords (I don't, but you get the point ;-))?
> I work in a number of secure environments where multiple complex
> passwords are mandated - and typically password management becomes
> sufficiently hard that people start to use shortcuts, defeating the
> object of the whole exercise (heck, end users probably just use
> "Password01" everywhere, "because it's too hard to remember all those
> passwords"...)
> 
> That's not to say that bad security practices justify anything, but on
> the other hand human factors do imply that it's not automatically
> guaranteed that two passwords are more secure than one. Single sign-on
> is a goal for a lot of people for a reason...

Basically if your private key, whether it?s a file or it?s a password gets
compromised than the person who compromises your private key can masquerade
as you without anyone being able to detect the difference. At that point
the private key will need to be revoked (as in, programmatically announced as
compromised and no longer valid).

There?s basically no way to have a signing solution that doesn?t involve some
secret bit of data that is known only to you and is never sent to PyPI. As with
most things on PyPI we cannot actually mandate that a user practices good
security with their private key. They could, for instance, post it on Twitter
and there isn?t much that we can do about it. The only way to avoid that is to
more or less give up on the idea of surviving a compromise of the PyPI
infrastructure. That isn?t itself a *wrong* answer, since signing does impose
additional cost to the author and it?s not entirely clear that the cost is
worth it.

One of the benefits of having the author sign is that you get a chain of what
is called an offline key. Essentially all trust comes from pip trusting a set
of ?root? keys which are not stored on a server in the PyPI infrastructure and
are instead stored ?offline? in some fashion such as on a USB drive sitting in
a safe deposit box, or in an HSM (for the uninitiated, a HSM is a hardware
device that stores keys but does not allow you to get the key itself back out.
You can use the key by asking the HSM to do operations with it but you can?t
get the key back out. Some HSMs even include things like an acid pack inside of
them where attempting to physically open the HSM will break the acid pack and
destroy the internal memory that holds the key). Then from that root key(s) you
can go from key to key in a chain until you get to the author key, and every
key in that chain is not stored ?online?.

This chain of offline keys means that if someone takes over the machines
running PyPI they can?t trick people into installing something because the keys
that are required to do that don?t exist on PyPI.

Like most things in security however, there is no singular right answer and
everything is a tradeoff. In this particular case the tradeoff we need to make
is between the ability to survive a compromise of the PyPI machines themselves
and the UX for end users. For example, if we make signing mandatory and the UX
is bad then we're going to hamper the ability for authors to publish new
downloads. If that UX is bad enough we can get into a situation where authors
choose not to release things as often as they would otherwise prefer to because
of the pain associated with releasing. There are also concerns that by making
signing mandatory that it's going to have people who have no interest or desire
in properly securing a private key being forced to make one and they might just
use the same password as they use for PyPI or something like "hunter1" or
whatever. In that case for those people and the users of their projects we've
not added much additional security but we've increased complexity and the
chances of something breaking. If we make signing optional however, then the
benefit might not be worth exposing signing to end users. If, for example,
an end users downloads and installs 100 different files from PyPI and 99 of
them the author opted in to signing them and one of them the author didn't then
a compromise of PyPI can compromise that end user through just that one
project.

I don't believe there is any downside to moving away from relying solely on TLS
and using a TUF based scheme where PyPI itself holds the signing keys. That's
a net win since you're changing having the PyPI infrastructure manage TLS keys
for having the PyPI infrastructure manage TUF keys with the added benefit that
TUF "transfers" in ways that TLS simply can't (such as through the CDN or
through mirrors or what have you).

There *may* be enough downside to having authors sign that it doesn't make
sense to expose that. Part of the discussion around PEP 480 should be hammering
out what the UX looks like for the authors, deciding if that UX is good enough
to make it mandatory or to make it strongly encouraged through pip warnings
and PyPI warnings, and ultimately deciding if the tradeoff of the additional
burden on authors is worth it. If it's not worth it to the community as a whole
then we shouldn't accept PEP 480 and should instead focus on ensuring that
we reduce the ability to compromise PyPI itself (which is something we should
do anyways of course). In this regard having the opinion from someone who isn't
an expert is *extremely* helpful because someone like me already knows how to
manage their keys and already does it so for people like me the answer for if
having authors needing to manage some secret bit of data is asking too much is
an easy answer. No it's not too much to ask me to do that because I'm already
doing it (and a lot of developers are via their SSH private keys for instance).

(How's that for a tl;dr?)

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


From donald at stufft.io  Wed Dec 31 20:51:24 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 Dec 2014 14:51:24 -0500
Subject: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
In-Reply-To: <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
References: <CAJYWiei2Zfiw6qvRHnoMYgptGHVVZ3mxCqZy+UB0eCWwiZLVYA@mail.gmail.com>
 <874D1500-0AFE-44FB-83C2-502A79091A7D@stufft.io>
 <CAOyQwLhopOSqszBX1z+Ucia_oAkqbOB=CBVeC=Do-VZOZfS3Qg@mail.gmail.com>
Message-ID: <484EB80E-E812-4B66-812E-285CF6664455@stufft.io>


> On Dec 31, 2014, at 11:08 AM, Vladimir Diaz <vladimir.v.diaz at gmail.com> wrote:
> 
> 
> 
> 
> 
> On Wed, Dec 31, 2014 at 2:26 AM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> 
>> On Dec 10, 2014, at 10:16 PM, Vladimir Diaz <vladimir.v.diaz at gmail.com <mailto:vladimir.v.diaz at gmail.com>> wrote:
>> 
>> Hello everyone,
>> 
>> I am a research programmer at the NYU School of Engineering.  My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI."  The two-stage proposal can be reviewed online at:
>> 
>> PEP 458
>> http://legacy.python.org/dev/peps/pep-0458/ <http://legacy.python.org/dev/peps/pep-0458/>
>> 
>> PEP 480
>> http://legacy.python.org/dev/peps/pep-0480/ <http://legacy.python.org/dev/peps/pep-0480/>
>> 
>> 
>> Summary of the Proposal:
>> 
>> "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users.  The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI.  Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed.  The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata.
>> 
>> The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects.  
>> 
>> The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users.  If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key.  As in PEP 458, no additional action is required by end users.  However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key.  PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing.  The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month.
>> 
>> We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback.
> 
> 
> I?ve just finished (re)reading the white paper, PEP 450, PEP 480, and some of the supporting documentation on the TUF website.
> 
> Thanks!
>  
> 
> I?m confused about what exactly is contained within the TUF metadata and who signs what in a PEP 480 world.
> 
> The following illustration shows what is contained within TUF metadata (JSON files):
> https://github.com/vladimir-v-diaz/pep-on-pypi-with-tuf/raw/master/pep-0458/figure4.pdf <https://github.com/vladimir-v-diaz/pep-on-pypi-with-tuf/raw/master/pep-0458/figure4.pdf>
> Note: In this illustration, the "snapshot" and "targets" roles are renamed "release" and "projects", respectively.
>  
> If you're interested in what exactly is contained in these JSON files, here is example metadata:
> https://github.com/theupdateframework/tuf/tree/develop/examples/repository/metadata <https://github.com/theupdateframework/tuf/tree/develop/examples/repository/metadata>
> 
> In a PEP 480 world, project developers sign a single JSON file. For example, developer(s) for the "Request" project sign their assigned JSON file named "/targets/claimed/Requests.json".  Specifically, a signature is generated of the "signed" entry <https://github.com/theupdateframework/tuf/blob/develop/examples/repository/metadata/targets.json#L9-L49> of the dictionary.  Once the signature is generated, it is added to the "signatures" entry <https://github.com/theupdateframework/tuf/blob/develop/examples/repository/metadata/targets.json#L2-L7> of the JSON file.
> 
> In figure 1 of PEP 480, PyPI signs for every metadata except those listed under the "roles signed by developer keys" label: https://github.com/vladimir-v-diaz/pep-maximum-security-model/blob/master/pep-0480/figure1.png <https://github.com/vladimir-v-diaz/pep-maximum-security-model/blob/master/pep-0480/figure1.png>

Ok, so authors never actually directly sign the package files themselves, they sign for a document that contains a hash of the files they upload?

> 
> 
> Currently when you do something like ``pip install FooBar``, pip fetches /simple/FooBar/ to look for potential installation candidates, and when it finds one it downloads it and installs it. This all all ?signed? by online keys via TLS.
> 
> 1. In a TUF world, would pip still fetch /simple/FooBar/ to discover things to install or would it fetch some TUF metadata to find things to install?
>  
> In the integration/demo we did with pip, we treated each /simple/ html file as a target (listed the hash and file size of these html index pages in TUF metadata).  That is, pip still fetched /simple/FooBar/ to discover distributions to install, but we verified the html files *and* distributions against TUF metadata.  In PEP 458, we state that "/simple" is also listed in TUF metadata: http://legacy.python.org/dev/peps/pep-0458/#pypi-and-tuf-metadata <http://legacy.python.org/dev/peps/pep-0458/#pypi-and-tuf-metadata> (last paragraph just before the diagram).
> 
> Another option is to avoid crawling/listing the simple index pages and just search TUF metadata for distributions, but this approach will require design changes to pip.  We went with the approach (treat the index pages as targets) that required minimal changes to pip.

So I?m personally perfectly happy to make more than minimal changes to pip as I want to get this right rather than just bolt something onto the side. 

>  
> 2. If it?s fetching /simple/FooBar/ is that secured by TUF?
> 
> Yes, see my response to (1).
> 
> 3. If it?s secured by TUF who signs the TUF metadata that talks about /simple/FooBar/ in PEP 480 the author or PyPI?
>  
> PEP 480 authors sign for both their project's index page and distribution(s) (as indicated in the JSON file):
> 
> "A claimed or recently-claimed project will need to upload in its transaction to PyPI not just targets (a simple index as well as distributions) but also TUF metadata. The project MAY do so by uploading a ZIP file containing two directories, /metadata/ (containing delegated targets metadata files) and /targets/ (containing targets such as the project simple index and distributions that are signed by the delegated targets metadata)."
> 
> See the second paragraph of http://legacy.python.org/dev/peps/pep-0480/#snapshot-process <http://legacy.python.org/dev/peps/pep-0480/#snapshot-process>.

So here is my problem. I?m completely on board with the developer signing for the distribution files. I think that makes total sense. However I worry that requiring the developer to sign for what is essentially the ?installer? API (aka how pip discovers things to install) is going to put us in a situation where we cannot evolve the API easily. If we modified this PEP so that an online key signed for /simple/ what security properties would we lose?

It *appears* to me that the problem then would be that a compromise of PyPI can present whatever information they want to pip as to what is available for pip to download and install. This would mean freeze attacks, mix and match attacks. It would also mean that they could, in a future world where pip can use metadata on PyPI to do dependency resolution, tell pip that it needs to download a valid but malicious project as a dependency of a popular project like virtualenv.

However I don?t think they?d be able to actually cause pip to install a malicious copy of a good project and I believe that we can protect against an attacker who poses that key from tricking pip into installing a malicious but valid project as a fake dependency by having pip only use the theoretical future PyPI metadata that lists dependencies as an optimization hint for what it should download and then once it?s actually downloaded a project like virtualenv (which has been validated to be from the real author) peek inside that file and ensure that the metadata inside that matches what PyPI told pip.

Is my assessment correct? Is keeping the ?API? under control of PyPI a reasonable thing to do while keeping the actual distribution files themselves under control of the distribution authors? The reason this worries me is that unlikely a Linux distribution or an application like Firefox or so we don?t have much of a relationship with the people who are uploading things to PyPI. So if we need to evolve the API we are not going to be able to compel our authors to go back and re-generate new signed metadata.

An additional thing I see, it appears that all of the metadata in TUF has an expiration. While I think this makes complete sense for things signed by online keys and things signed by keys that the PyPI administrator and/or PSF board I don?t think this is something we can reasonably do for things signed by authors themselves. An author might publish something and then disappear and never come back and forcing them to resign at some point in the future isn?t something we?re reasonably able to do. Is there a plan for how to handle that?

> 
> Let me know exactly what needs to change in the PEPs to make everything explained above clearer.  For example, in PEP 458 we provide a link/reference <https://www.python.org/dev/peps/pep-0458/#what-additional-repository-files-are-required-on-pypi> (last paragraph of this subsection) to the Metadata document <https://github.com/theupdateframework/tuf/blob/develop/METADATA.md> indicating the content of the JSON files, but should the illustration <https://github.com/vladimir-v-diaz/pep-on-pypi-with-tuf/raw/master/pep-0458/figure4.pdf>I've included in this reply also be added? 

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141231/91fb13f7/attachment-0001.html>