From kamal.mustafa at gmail.com  Mon Apr  1 02:06:45 2013
From: kamal.mustafa at gmail.com (Kamal Bin Mustafa)
Date: Mon, 01 Apr 2013 08:06:45 +0800
Subject: [Distutils]  Help with buildout
In-Reply-To: <CAOdFaDXE_ggJ7_PXPFabb6mJ=uAdKKH9SQXaTg_tD9gU0THGxQ@mail.gmail.com>
References: <CAOdFaDXE_ggJ7_PXPFabb6mJ=uAdKKH9SQXaTg_tD9gU0THGxQ@mail.gmail.com>
Message-ID: <5158CF95.2080109@gmail.com>

Got this when running your buildout so I can't test the generated 
interpreter:-

Error: There is a version conflict.
We already have: colander 0.9.9
but notaliens-web 0.0 requires 'colander>0.9.9'.

notaliens.com/eggs/PasteDeploy-1.5.0-py2.7.egg',

Does it a typo or really correct path ? Assuming you run the buildout at 
the root of your project dir, then it should be relative to
eggs/PasteDeploy-1.5.0-py2.7.egg', no ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130401/d8d6c719/attachment.html>

From mal at egenix.com  Mon Apr  1 22:12:54 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 01 Apr 2013 22:12:54 +0200
Subject: [Distutils] PEP: Improving Python ZIP Application Support
In-Reply-To: <CAG8k2+7reG26DUAoxNFVzAVSYVSmi+o6-56rJqCtt_GH9eWSkw@mail.gmail.com>
References: <CAG8k2+7reG26DUAoxNFVzAVSYVSmi+o6-56rJqCtt_GH9eWSkw@mail.gmail.com>
Message-ID: <5159EA46.9010806@egenix.com>

On 30.03.2013 20:22, Daniel Holth wrote:
> Python ZIP Application Support -
> https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnri78/edit?usp=sharing
> 
> 
> PEP: 4XX
> 
> Title: Improving Python ZIP Application Support
> 
> Author: Daniel Holth <dholth at gmail.com>
> 
> Status: Draft
> 
> Type: Standards Track
> 
> Python-Version: 3.4
> 
> Created: 30 March 2013
> 
> Post-History: 30 March 2013
> 
> 
> Improving Python ZIP Application Support
> 
> 
> Python has had the ability to execute directories or ZIP-format
> archives as scripts since version 2.6. When invoked with a zip file or
> directory as its first argument the interpreter adds that directory to
> sys.path and executes the __main__ module. These archives provide a
> great way to publish software that needs to be distributed as a single
> file script but is complex enough to need to be written as a
> collection of modules.
> 
> 
> This feature is not as popular as it should be for two reasons: a)
> users haven?t heard of it because it wasn?t mentioned in earlier
> versions of Python 2.6 ?What?s New? document, and b) Windows users
> don?t have a file extension (other than .py) to associate with the
> launcher.

c) The feature has never really been documented outside the ticket
where it got added to Python :-)

Ticket for this:
http://bugs.python.org/issue17359

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 01 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2013-03-25: Released mxODBC 3.2.2 ...             http://egenix.com/go40
2013-04-10: Python Meeting Duesseldorf ...                  9 days to go

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ncoghlan at gmail.com  Tue Apr  2 00:40:34 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 2 Apr 2013 08:40:34 +1000
Subject: [Distutils] PEP: Improving Python ZIP Application Support
In-Reply-To: <5159EA46.9010806@egenix.com>
References: <CAG8k2+7reG26DUAoxNFVzAVSYVSmi+o6-56rJqCtt_GH9eWSkw@mail.gmail.com>
	<5159EA46.9010806@egenix.com>
Message-ID: <CADiSq7famsoXCX3WijQUtbCa3JYsNA+ym8myKT+tk2kcksgV5w@mail.gmail.com>

On Tue, Apr 2, 2013 at 6:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> c) The feature has never really been documented outside the ticket
> where it got added to Python :-)
>


Not true, it's covered in the command line docs at
http://docs.python.org/2/using/cmdline.html#interface-options

It's just that people generally assume they know how our CLI works, so they
never read that.

It's also in the runpy.run_path docs, but that's even more obscure.


>
> Ticket for this:
> http://bugs.python.org/issue17359
>

I have updated that to reference the existing documentation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130402/33db267d/attachment.html>

From tk47 at students.poly.edu  Mon Apr  1 23:40:22 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Mon, 1 Apr 2013 17:40:22 -0400
Subject: [Distutils] Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
Message-ID: <5159FEC6.6070204@students.poly.edu>

Hello PyPI,

Hope attendees had a great time at PyCon 2013! We certainly enjoyed 
presenting to you our lightning talk on securing PyPI with TUF 
(https://www.youtube.com/watch?v=2sx1lS6cT3g).

Since then, we have been busy improving TUF and implementing machinery 
to automatically secure PyPI with TUF:

https://github.com/dachshund/pypi.updateframework.com

You may also have noticed that the root metadata for our prototype 
mirror of PyPI+TUF expired yesterday. This aligns nicely with our plan 
for switching our hand-maintained PyPI+TUF mirror with the automatic 
one. We expect to have it ready very soon, and until then, we certainly 
welcome your first impressions on our automation. You could try it on 
your machine right away!

Finally, we are working continuously on improving TUF, especially on 
ensuring that the metadata scales with data. We welcome your feedback on 
these issues and more (https://github.com/akonst/tuf/issues?state=open).

-Trishank


From pombredanne at nexb.com  Tue Apr  2 02:20:26 2013
From: pombredanne at nexb.com (Philippe Ombredanne)
Date: Tue, 2 Apr 2013 02:20:26 +0200
Subject: [Distutils] PEP: Improving Python ZIP Application Support
In-Reply-To: <CADiSq7famsoXCX3WijQUtbCa3JYsNA+ym8myKT+tk2kcksgV5w@mail.gmail.com>
References: <CAG8k2+7reG26DUAoxNFVzAVSYVSmi+o6-56rJqCtt_GH9eWSkw@mail.gmail.com>
	<5159EA46.9010806@egenix.com>
	<CADiSq7famsoXCX3WijQUtbCa3JYsNA+ym8myKT+tk2kcksgV5w@mail.gmail.com>
Message-ID: <CAOFm3uGzbfwF8r2xyowXLbYQrKKt4+o8nPwhaj3FU5vaepK5Qw@mail.gmail.com>

On Tue, Apr 2, 2013 at 12:40 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Apr 2, 2013 at 6:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> c) The feature has never really been documented outside the ticket
>> where it got added to Python :-)
> Not true, it's covered in the command line docs at
> http://docs.python.org/2/using/cmdline.html#interface-options
The doc has a bug BTW:
 "Changed in version 2.5: Directories and zipfiles containing a
__main__.py file at the top level are now considered valid Python
scripts."
I tested it and this does not work in Pythin 2.5, only in 2.6

-- 
Philippe Ombredanne

+1 650 799 0949 | pombredanne at nexB.com
DejaCode Enterprise at http://www.dejacode.com
nexB Inc. at http://www.nexb.com

From ct at gocept.com  Tue Apr  2 22:12:23 2013
From: ct at gocept.com (Christian Theune)
Date: Tue, 2 Apr 2013 22:12:23 +0200
Subject: [Distutils] Re-uploading packages
Message-ID: <kjfe2n$600$1@ger.gmane.org>

Hi,

when developing bandersnatch I saw some checksum errors for the md5sums 
of downloaded package files that I didn't understand.
I just saw another one and just want to check back whether this is 
true: I can go to PyPI, delete a package version, and upload a 
different file later.

True?

This would explain that I can see a file that I downloaded successfully 
changing it's hash over time.

Feels like a bad idea to me, but I guess this is part of the "PyPI 
doesn't have an oppinion" deal ?

Christian



From tk47 at students.poly.edu  Tue Apr  2 22:35:15 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Tue, 2 Apr 2013 16:35:15 -0400
Subject: [Distutils] Re-uploading packages
In-Reply-To: <kjfe2n$600$1@ger.gmane.org>
References: <kjfe2n$600$1@ger.gmane.org>
Message-ID: <515B4103.9070505@students.poly.edu>

On Tue 02 Apr 2013 04:12:23 PM EDT, Christian Theune wrote:
>
> when developing bandersnatch I saw some checksum errors for the
> md5sums of downloaded package files that I didn't understand.

Firstly, thanks for programming bandersnatch --- what an awesome tool!

Secondly, I see these hash mismatches too with bandersnatch. Would be 
good to know what's happening.


From holger at merlinux.eu  Tue Apr  2 22:39:34 2013
From: holger at merlinux.eu (holger krekel)
Date: Tue, 2 Apr 2013 20:39:34 +0000
Subject: [Distutils] Re-uploading packages
In-Reply-To: <kjfe2n$600$1@ger.gmane.org>
References: <kjfe2n$600$1@ger.gmane.org>
Message-ID: <20130402203934.GX9677@merlinux.eu>

On Tue, Apr 02, 2013 at 22:12 +0200, Christian Theune wrote:
> Hi,
> 
> when developing bandersnatch I saw some checksum errors for the
> md5sums of downloaded package files that I didn't understand.
> I just saw another one and just want to check back whether this is
> true: I can go to PyPI, delete a package version, and upload a
> different file later.
> 
> True?

it's certainly possible.  Not sure if i even did something like
this in my early days :)

> This would explain that I can see a file that I downloaded
> successfully changing it's hash over time.

would be cool if bandersnatch can handle this case.
Maybe queue hash mismatches and only error out if the final
file mismatches its hash or so?

best,
holger

> 
> Feels like a bad idea to me, but I guess this is part of the "PyPI
> doesn't have an oppinion" deal ?
> 
> Christian
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

From ct at gocept.com  Tue Apr  2 22:42:13 2013
From: ct at gocept.com (Christian Theune)
Date: Tue, 2 Apr 2013 22:42:13 +0200
Subject: [Distutils] Re-uploading packages
In-Reply-To: <20130402203934.GX9677@merlinux.eu>
References: <kjfe2n$600$1@ger.gmane.org> <20130402203934.GX9677@merlinux.eu>
Message-ID: <0988A45F-8CD1-4160-83A2-F1C9EC2CB0B3@gocept.com>


On Apr 2, 2013, at 10:39 PM, holger krekel <holger at merlinux.eu> wrote:

> On Tue, Apr 02, 2013 at 22:12 +0200, Christian Theune wrote:
>> Hi,
>> 
>> when developing bandersnatch I saw some checksum errors for the
>> md5sums of downloaded package files that I didn't understand.
>> I just saw another one and just want to check back whether this is
>> true: I can go to PyPI, delete a package version, and upload a
>> different file later.
>> 
>> True?
> 
> it's certainly possible.  Not sure if i even did something like
> this in my early days :)
> 
>> This would explain that I can see a file that I downloaded
>> successfully changing it's hash over time.
> 
> would be cool if bandersnatch can handle this case.
> Maybe queue hash mismatches and only error out if the final
> file mismatches its hash or so?

It does that already: it performs a hash-check of existing files to verify whether they are still intact. If they are not, then it logs a warning (disguised as an error) and redownloads.

Whenever it downloads something that doesn't fit the advertised checksum then it actually errors out (and never redistributes the file to downstream clients).

Christian

-- 
Christian Theune ? ct at gocept.com
gocept gmbh & co. kg ? Forsterstra?e 29 ? 06112 Halle (Saale) ? Germany
http://gocept.com ? Tel +49 345 1229889-7
Python, Pyramid, Plone, Zope ? consulting, development, hosting, operations

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4334 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130402/552498e2/attachment.bin>

From richard at python.org  Wed Apr  3 00:55:26 2013
From: richard at python.org (Richard Jones)
Date: Wed, 3 Apr 2013 09:55:26 +1100
Subject: [Distutils] Re-uploading packages
In-Reply-To: <kjfe2n$600$1@ger.gmane.org>
References: <kjfe2n$600$1@ger.gmane.org>
Message-ID: <CAHrZfZDAODUZiDn9PDuxEtjErFoDGpfh2ejhrsAL04Me4A2mXg@mail.gmail.com>

We prevent people from uploading files to replace contents, but not
deleting and re-uploading. That would take additional tracking not built
into the system.


     Richard


On 3 April 2013 07:12, Christian Theune <ct at gocept.com> wrote:

> Hi,
>
> when developing bandersnatch I saw some checksum errors for the md5sums of
> downloaded package files that I didn't understand.
> I just saw another one and just want to check back whether this is true: I
> can go to PyPI, delete a package version, and upload a different file later.
>
> True?
>
> This would explain that I can see a file that I downloaded successfully
> changing it's hash over time.
>
> Feels like a bad idea to me, but I guess this is part of the "PyPI doesn't
> have an oppinion" deal ?
>
> Christian
>
>
> ______________________________**_________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/**mailman/listinfo/distutils-sig<http://mail.python.org/mailman/listinfo/distutils-sig>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130403/7d687c1f/attachment.html>

From p.f.moore at gmail.com  Thu Apr  4 10:05:59 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 4 Apr 2013 09:05:59 +0100
Subject: [Distutils] "packaging-user" mailing list?
In-Reply-To: <CAPYWazoOcRPYyPg_1tHviVg-yikZn9afSeBE4+hw+Z-=zny0nA@mail.gmail.com>
References: <CAPYWazoOcRPYyPg_1tHviVg-yikZn9afSeBE4+hw+Z-=zny0nA@mail.gmail.com>
Message-ID: <CACac1F_6-rCGKngHDK8_73i2tw14mbrKPAOORnJ8p1iHS15thg@mail.gmail.com>

+0

I think a "packaging user" list is more discoverable for people new to
Python who have questions. But the list will only really be of use if the
developers of current tools join the list and provide support there.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130404/ab8888db/attachment.html>

From me at rpatterson.net  Thu Apr  4 17:35:58 2013
From: me at rpatterson.net (Ross Patterson)
Date: Thu, 04 Apr 2013 08:35:58 -0700
Subject: [Distutils] Help with buildout
References: <CAOdFaDXE_ggJ7_PXPFabb6mJ=uAdKKH9SQXaTg_tD9gU0THGxQ@mail.gmail.com>
	<5158CF95.2080109@gmail.com>
Message-ID: <87hajmmhpd.fsf@rpatterson.net>

Kamal Bin Mustafa <kamal.mustafa at gmail.com> writes:

> Got this when running your buildout so I can't test the generated
> interpreter:-
>
> Error: There is a version conflict.
> We already have: colander 0.9.9
> but notaliens-web 0.0 requires 'colander>0.9.9'.
>
> notaliens.com/eggs/PasteDeploy-1.5.0-py2.7.egg',
>
> Does it a typo or really correct path ? Assuming you run the buildout
> at the root of your project dir, then it should be relative to 
> eggs/PasteDeploy-1.5.0-py2.7.egg', no ?

In my experience, this error is usually a conflict with a distribution
installed in the system/base python installation.  Try using a
virtualenv:

    $ virtualenv .
    $ bin/python bootstrap.py
    $ bin/buildout

HTH,
Ross


From andrew.svetlov at gmail.com  Thu Apr  4 17:48:25 2013
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Thu, 4 Apr 2013 18:48:25 +0300
Subject: [Distutils] PyPI is down
Message-ID: <CAL3CFcW_bV1qf9iUGkdAeJvjoA_rW_jb0EhgecdGMETD-N-HhA@mail.gmail.com>

Subj

--
Thanks,
Andrew Svetlov

From brett at python.org  Thu Apr  4 17:54:55 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Apr 2013 11:54:55 -0400
Subject: [Distutils] PyPI is down
In-Reply-To: <CAL3CFcW_bV1qf9iUGkdAeJvjoA_rW_jb0EhgecdGMETD-N-HhA@mail.gmail.com>
References: <CAL3CFcW_bV1qf9iUGkdAeJvjoA_rW_jb0EhgecdGMETD-N-HhA@mail.gmail.com>
Message-ID: <CAP1=2W510XkqC4ig+YjHJhJnFjkkn38+J+AsWv4yuTXdPEZfeg@mail.gmail.com>

Not anymore. Lasted about 5 minutes (at least according the the DOWN/UP
emails sent to the infrastructure list).

Monitoring is set up for PyPI so it is known very quickly when it stops
responding and thus no need to worry about notifying anyone. Noah has done
a good job. =)


On Thu, Apr 4, 2013 at 11:48 AM, Andrew Svetlov <andrew.svetlov at gmail.com>wrote:

> Subj
>
> --
> Thanks,
> Andrew Svetlov
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130404/6fbceca0/attachment.html>

From fred at fdrake.net  Thu Apr  4 18:11:18 2013
From: fred at fdrake.net (Fred Drake)
Date: Thu, 4 Apr 2013 12:11:18 -0400
Subject: [Distutils] PyPI is down
In-Reply-To: <CAP1=2W510XkqC4ig+YjHJhJnFjkkn38+J+AsWv4yuTXdPEZfeg@mail.gmail.com>
References: <CAL3CFcW_bV1qf9iUGkdAeJvjoA_rW_jb0EhgecdGMETD-N-HhA@mail.gmail.com>
	<CAP1=2W510XkqC4ig+YjHJhJnFjkkn38+J+AsWv4yuTXdPEZfeg@mail.gmail.com>
Message-ID: <CAFT4OTFXVaGuDsyoqS+_DDr0iGxNH7Mtbbfet3T0gmaeHLdMQA@mail.gmail.com>

Thanks following up on this!

The server appeared to be flapping from here, and that definitely lasted
longer than 5 minutes (30 or more?).  The flapping could easily have fooled
monitoring, of course.


On Thu, Apr 4, 2013 at 11:54 AM, Brett Cannon <brett at python.org> wrote:

> Not anymore. Lasted about 5 minutes (at least according the the DOWN/UP
> emails sent to the infrastructure list).
>
> Monitoring is set up for PyPI so it is known very quickly when it stops
> responding and thus no need to worry about notifying anyone. Noah has done
> a good job. =)
>
>
> On Thu, Apr 4, 2013 at 11:48 AM, Andrew Svetlov <andrew.svetlov at gmail.com>wrote:
>
>> Subj
>>
>> --
>> Thanks,
>> Andrew Svetlov
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130404/52f5fd60/attachment.html>

From qwcode at gmail.com  Thu Apr  4 18:45:28 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Thu, 4 Apr 2013 09:45:28 -0700
Subject: [Distutils] "packaging-user" mailing list?
In-Reply-To: <CACac1F_6-rCGKngHDK8_73i2tw14mbrKPAOORnJ8p1iHS15thg@mail.gmail.com>
References: <CAPYWazoOcRPYyPg_1tHviVg-yikZn9afSeBE4+hw+Z-=zny0nA@mail.gmail.com>
	<CACac1F_6-rCGKngHDK8_73i2tw14mbrKPAOORnJ8p1iHS15thg@mail.gmail.com>
Message-ID: <CAPYWazr4iknb_0b=qk2DJBoBcUsbmUo5WtTzBti1gxYA0iYWvw@mail.gmail.com>

> I think a "packaging user" list is more discoverable for people new to
> Python who have questions.
>

that sounds like a +1, kinda


> But the list will only really be of use if the developers of current tools
> join the list and provide support there.
>

yes, packaging devs and "experts" would be needed to have good answers.
that's why I was sending this out, to get a sense if people would jump on
the idea.
but that's not really happening so far.  :  )

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130404/29f727dc/attachment.html>

From bwmaister at gmail.com  Thu Apr  4 18:59:02 2013
From: bwmaister at gmail.com (Brandon W Maister)
Date: Thu, 4 Apr 2013 12:59:02 -0400
Subject: [Distutils] "packaging-user" mailing list?
In-Reply-To: <CALeMXf4vYP5fzjs623+uzXdAxRaf_X6NpgKon4=nZhDKbHB=FQ@mail.gmail.com>
References: <CAPYWazoOcRPYyPg_1tHviVg-yikZn9afSeBE4+hw+Z-=zny0nA@mail.gmail.com>
	<CALeMXf4vYP5fzjs623+uzXdAxRaf_X6NpgKon4=nZhDKbHB=FQ@mail.gmail.com>
Message-ID: <CANNZFE=bxa6krneHxvYLRvBE4bA_dC9zAG_FX8htoboYJ_cm_g@mail.gmail.com>

On Sat, Mar 30, 2013 at 12:51 PM, PJ Eby <pje at telecommunity.com> wrote:

> It's actually mostly a support forum that has periodic surges of
> development discussion.  Granted, the current development surge is
> bigger and longer than any that have happened in a while.  ;-)
>


FWIW I've been a lurker on this list for a while, and the first time I saw
someone ask a support question I expected them to get shut down.

brandon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130404/e21293e4/attachment-0001.html>

From brett at python.org  Fri Apr  5 12:27:03 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 5 Apr 2013 06:27:03 -0400
Subject: [Distutils] PyPI is down
In-Reply-To: <CAFT4OTFXVaGuDsyoqS+_DDr0iGxNH7Mtbbfet3T0gmaeHLdMQA@mail.gmail.com>
References: <CAL3CFcW_bV1qf9iUGkdAeJvjoA_rW_jb0EhgecdGMETD-N-HhA@mail.gmail.com>
	<CAP1=2W510XkqC4ig+YjHJhJnFjkkn38+J+AsWv4yuTXdPEZfeg@mail.gmail.com>
	<CAFT4OTFXVaGuDsyoqS+_DDr0iGxNH7Mtbbfet3T0gmaeHLdMQA@mail.gmail.com>
Message-ID: <CAP1=2W7u_ne+Lr78VSn6c3koKwm5M=q9ritRwBuHLYoD92xkag@mail.gmail.com>

On Thu, Apr 4, 2013 at 12:11 PM, Fred Drake <fred at fdrake.net> wrote:

> Thanks following up on this!
>
> The server appeared to be flapping from here, and that definitely lasted
> longer than 5 minutes (30 or more?).  The flapping could easily have fooled
> monitoring, of course.
>

The flapping is from OSUOSL playing with their firewall settings to try and
mitigate risk from the Postgres vulnerability that went public yesterday.
They are obviously aware of it and are trying to get the settings right
until they get an upstream patch which will require some downtime, but then
that should be it.

-Brett


>
>
> On Thu, Apr 4, 2013 at 11:54 AM, Brett Cannon <brett at python.org> wrote:
>
>> Not anymore. Lasted about 5 minutes (at least according the the DOWN/UP
>> emails sent to the infrastructure list).
>>
>> Monitoring is set up for PyPI so it is known very quickly when it stops
>> responding and thus no need to worry about notifying anyone. Noah has done
>> a good job. =)
>>
>>
>> On Thu, Apr 4, 2013 at 11:48 AM, Andrew Svetlov <andrew.svetlov at gmail.com
>> > wrote:
>>
>>> Subj
>>>
>>> --
>>> Thanks,
>>> Andrew Svetlov
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> http://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>
> --
> Fred L. Drake, Jr.    <fred at fdrake.net>
> "A storm broke loose in my mind."  --Albert Einstein
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130405/6b9cd65b/attachment.html>

From noah at coderanger.net  Fri Apr  5 16:45:35 2013
From: noah at coderanger.net (Noah Kantrowitz)
Date: Fri, 5 Apr 2013 07:45:35 -0700
Subject: [Distutils] PyPI is down
In-Reply-To: <CAP1=2W7u_ne+Lr78VSn6c3koKwm5M=q9ritRwBuHLYoD92xkag@mail.gmail.com>
References: <CAL3CFcW_bV1qf9iUGkdAeJvjoA_rW_jb0EhgecdGMETD-N-HhA@mail.gmail.com>
	<CAP1=2W510XkqC4ig+YjHJhJnFjkkn38+J+AsWv4yuTXdPEZfeg@mail.gmail.com>
	<CAFT4OTFXVaGuDsyoqS+_DDr0iGxNH7Mtbbfet3T0gmaeHLdMQA@mail.gmail.com>
	<CAP1=2W7u_ne+Lr78VSn6c3koKwm5M=q9ritRwBuHLYoD92xkag@mail.gmail.com>
Message-ID: <17CB8758-7904-4B7D-97C4-CB6FF29B33D6@coderanger.net>


On Apr 5, 2013, at 3:27 AM, Brett Cannon wrote:

> 
> 
> 
> On Thu, Apr 4, 2013 at 12:11 PM, Fred Drake <fred at fdrake.net> wrote:
> Thanks following up on this!
> 
> The server appeared to be flapping from here, and that definitely lasted longer than 5 minutes (30 or more?).  The flapping could easily have fooled monitoring, of course.
> 
> The flapping is from OSUOSL playing with their firewall settings to try and mitigate risk from the Postgres vulnerability that went public yesterday. They are obviously aware of it and are trying to get the settings right until they get an upstream patch which will require some downtime, but then that should be it.

The upgrade was done yesterday afternoon, it just only resulted in about 30 seconds of downtime so I didn't even both announcing it :)

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130405/a5ad1c31/attachment.pgp>

From chris at simplistix.co.uk  Fri Apr  5 16:52:17 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 05 Apr 2013 11:52:17 -0300
Subject: [Distutils] buildout 2 craziness: "upgrading" to distribute 0.6.26
 and using site-packages from another python install
Message-ID: <515EE521.8090701@simplistix.co.uk>

Hi All,

This is driving me nuts, I hope someone can help.

So, pre-amble, I'm on a Mac, I have a few python installations, the ones 
in play here are:

buzzkill:xlutils chris$ which python
/Library/Frameworks/EPD64.framework/Versions/Current/bin/python
buzzkill:xlutils chris$ which python2.6
/usr/local/bin/python2.6
buzzkill:xlutils chris$ which python2.7
/Library/Frameworks/EPD64.framework/Versions/Current/bin/python2.7

The buildout I'm working with is here:

https://github.com/python-excel/xlutils/blob/master/buildout.cfg

My python2.7 has a site-packages xlrd (it's EPD, which includes a bunch 
of packages), so I'm using python2.6 on this machine, since it doesn't 
(it's a plain source install on python).

Okay, so, as you'll see from github above, I have the latest 
bootstrap.py, and the initial bootstrap looks fine:

buzzkill:xlutils chris$ python2.6 bootstrap.py
Downloading 
http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz
Extracting in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpxnVlCD
Now working in 
/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpxnVlCD/distribute-0.6.35
Building a Distribute egg in 
/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpg4hxVn
/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpg4hxVn/distribute-0.6.35-py2.6.egg
buzzkill:xlutils chris$ cat bin/buildout
#!/usr/local/bin/python2.6

import sys
sys.path[0:0] = [
   '/Users/chris/buildout-eggs/distribute-0.6.35-py2.6.egg',
   '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg',
   ]

import zc.buildout.buildout

if __name__ == '__main__':
     sys.exit(zc.buildout.buildout.main())

Okay, so now try running that buildout:

buzzkill:xlutils chris$ bin/buildout
Upgraded:
   distribute version 0.6.26;
restarting.
Generated script '/Users/chris/LocalGIT/xlutils/bin/buildout'.
Develop: '/Users/chris/LocalGIT/xlutils/.'
Develop: '/Users/chris/LocalGIT/xlutils/../xlrd'
Traceback (most recent call last):
   File "/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpwnpB08", 
line 13, in <module>
 
exec(compile(open('/Users/chris/LocalGIT/xlutils/../xlrd/setup.py').read(), 
'/Users/chris/LocalGIT/xlutils/../xlrd/setup.py', 'exec'))
   File "/Users/chris/LocalGIT/xlutils/../xlrd/setup.py", line 18, in 
<module>
     from xlrd.info import __VERSION__
ImportError: No module named info
While:
   Installing.
   Processing develop directory '/Users/chris/LocalGIT/xlutils/../xlrd'.

An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
<snip>
   File 
"/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg/zc/buildout/easy_install.py", 
line 129, in call_subprocess
     % repr(args)[1:-1])
Exception: Failed to run command:
'/usr/local/bin/python2.6', 
'/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpwnpB08', '-q', 
'develop', '-mxN', '-d', 
'/Users/chris/LocalGIT/xlutils/develop-eggs/tmpUg8rV8build'

Okay, so, a few comments:

- "upgrading" to distribute version 0.6.26 from 0.6.35. Whu?!

- The xlrd in /Users/chris/LocalGIT/xlrd certainly does have an info module.

Now the really crazy bit:

buzzkill:xlutils chris$ cat bin/buildout
#!/usr/local/bin/python2.6

import sys
sys.path[0:0] = [
   '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg',
 
'/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages',
   ]

import zc.buildout.buildout

if __name__ == '__main__':
     sys.exit(zc.buildout.buildout.main())

Come again?! Why is site-packages from a *different python installation* 
ending up in bin/buildout?

Any help here would be very gratefully received...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From chris at python.org  Fri Apr  5 17:11:18 2013
From: chris at python.org (Chris Withers)
Date: Fri, 05 Apr 2013 12:11:18 -0300
Subject: [Distutils] buildout 2 more craziness: still using
 site-packages from another python install
In-Reply-To: <515EE521.8090701@simplistix.co.uk>
References: <515EE521.8090701@simplistix.co.uk>
Message-ID: <515EE996.7090107@python.org>

On 05/04/2013 11:52, Chris Withers wrote:
> Hi All,
>
> This is driving me nuts, I hope someone can help.

Okay, so using my cunning buildout knowledge (huh!) I tried this:

[buildout]
develop = . ../xlrd ../xlwt
parts = test py docs
versions = versions

[versions]
distribute=0.6.35
<snip>

buzzkill:xlutils chris$ bin/buildout
Develop: '/Users/chris/LocalGIT/xlutils/.'
Develop: '/Users/chris/LocalGIT/xlutils/../xlrd'
Develop: '/Users/chris/LocalGIT/xlutils/../xlwt'
Unused options for buildout: 'unzip'.
Updating test.
Updating py.
Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'.
Updating docs.
Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'.

YAY! (or so I thought)

buzzkill:xlutils chris$ bin/test
<snip>
   Ran 327 tests with 2 failures and 27 errors in 0.478 seconds.
Tearing down left over layers:
   Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.

buzzkill:xlutils chris$ cat bin/test
#!/usr/local/bin/python2.6

import sys
sys.path[0:0] = [
   '/Users/chris/LocalGIT/xlutils',
   '/Users/chris/buildout-eggs/zope.testrunner-4.3.3-py2.6.egg',
 
'/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages',
   '/Users/chris/buildout-eggs/zope.exceptions-4.0.6-py2.6.egg',
<snip>

Oh FFS :-( Why is the python2.6 buildout using xlrd from site-packages 
in a python2.7 installation?!

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From marius at pov.lt  Fri Apr  5 17:15:55 2013
From: marius at pov.lt (Marius Gedminas)
Date: Fri, 5 Apr 2013 18:15:55 +0300
Subject: [Distutils] buildout 2 more craziness: still using
 site-packages from another python install
In-Reply-To: <515EE996.7090107@python.org>
References: <515EE521.8090701@simplistix.co.uk> <515EE996.7090107@python.org>
Message-ID: <20130405151555.GB23389@fridge.pov.lt>

On Fri, Apr 05, 2013 at 12:11:18PM -0300, Chris Withers wrote:
> On 05/04/2013 11:52, Chris Withers wrote:
> >Hi All,
> >
> >This is driving me nuts, I hope someone can help.
> 
> Okay, so using my cunning buildout knowledge (huh!) I tried this:
> 
> [buildout]
> develop = . ../xlrd ../xlwt
> parts = test py docs
> versions = versions
> 
> [versions]
> distribute=0.6.35
> <snip>
> 
> buzzkill:xlutils chris$ bin/buildout
> Develop: '/Users/chris/LocalGIT/xlutils/.'
> Develop: '/Users/chris/LocalGIT/xlutils/../xlrd'
> Develop: '/Users/chris/LocalGIT/xlutils/../xlwt'
> Unused options for buildout: 'unzip'.
> Updating test.
> Updating py.
> Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'.
> Updating docs.
> Generated script '/Users/chris/LocalGIT/xlutils/bin/margins'.
> 
> YAY! (or so I thought)
> 
> buzzkill:xlutils chris$ bin/test
> <snip>
>   Ran 327 tests with 2 failures and 27 errors in 0.478 seconds.
> Tearing down left over layers:
>   Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
> 
> buzzkill:xlutils chris$ cat bin/test
> #!/usr/local/bin/python2.6
> 
> import sys
> sys.path[0:0] = [
>   '/Users/chris/LocalGIT/xlutils',
>   '/Users/chris/buildout-eggs/zope.testrunner-4.3.3-py2.6.egg',
> 
> '/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages',
>   '/Users/chris/buildout-eggs/zope.exceptions-4.0.6-py2.6.egg',
> <snip>
> 
> Oh FFS :-( Why is the python2.6 buildout using xlrd from
> site-packages in a python2.7 installation?!

Check develop-eggs/, there may be a distribute.egg-link file that puts
/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages
in your sys.path.

When in doubt, rm -rf develop-eggs before retrying.

Marius Gedminas
-- 
1 + 1 = 3
                -- from a Microsoft advertisement
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130405/d76bfa0a/attachment-0001.pgp>

From marius at pov.lt  Fri Apr  5 17:14:52 2013
From: marius at pov.lt (Marius Gedminas)
Date: Fri, 5 Apr 2013 18:14:52 +0300
Subject: [Distutils] buildout 2 craziness: "upgrading" to distribute
 0.6.26 and using site-packages from another python install
In-Reply-To: <515EE521.8090701@simplistix.co.uk>
References: <515EE521.8090701@simplistix.co.uk>
Message-ID: <20130405151452.GA23389@fridge.pov.lt>

On Fri, Apr 05, 2013 at 11:52:17AM -0300, Chris Withers wrote:
> This is driving me nuts, I hope someone can help.
> 
> So, pre-amble, I'm on a Mac, I have a few python installations, the
> ones in play here are:

<snip>
> buzzkill:xlutils chris$ cat bin/buildout
> #!/usr/local/bin/python2.6
> 
> import sys
> sys.path[0:0] = [
>   '/Users/chris/buildout-eggs/distribute-0.6.35-py2.6.egg',
>   '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg',
>   ]
...
> buzzkill:xlutils chris$ bin/buildout
> Upgraded:
>   distribute version 0.6.26;
> restarting.
> Generated script '/Users/chris/LocalGIT/xlutils/bin/buildout'.
...
> buzzkill:xlutils chris$ cat bin/buildout
> #!/usr/local/bin/python2.6
> 
> import sys
> sys.path[0:0] = [
>   '/Users/chris/buildout-eggs/zc.buildout-2.1.0-py2.6.egg',
>   '/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages',
>   ]

I've seen this on Ubuntu.  IIRC a Python 3-bootstrapped buildout ended
up with /usr/lib/python2.7/dist-packages in sys.path, in my case.

> Come again?! Why is site-packages from a *different python
> installation* ending up in bin/buildout?
> 
> Any help here would be very gratefully received...

Workaround #1: run 'bin/buildout -N' so it won't try to upgrade
distribute.

Workaround #2: always always always wrap your buildouts in a virtualenv.
I started using a 'virtual-bootstrap' script that does

  #!/bin/sh
  rm -rf develop-eggs .installed.cfg python
  virtualenv --distribute python
  python/bin/python bootstrap.py
  bin/buildout "$@"

and haven't encountered any issues yet.

Marius Gedminas
-- 
When in trouble or in doubt,
run in circles, scream and shout.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130405/64b37b15/attachment.pgp>

From chris at python.org  Fri Apr  5 17:31:33 2013
From: chris at python.org (Chris Withers)
Date: Fri, 05 Apr 2013 12:31:33 -0300
Subject: [Distutils] buildout 2 craziness: "upgrading" to distribute
 0.6.26 and using site-packages from another python install
In-Reply-To: <20130405151452.GA23389@fridge.pov.lt>
References: <515EE521.8090701@simplistix.co.uk>
	<20130405151452.GA23389@fridge.pov.lt>
Message-ID: <515EEE55.7050903@python.org>

On 05/04/2013 12:14, Marius Gedminas wrote:
>> Any help here would be very gratefully received...
>
> Workaround #1: run 'bin/buildout -N' so it won't try to upgrade
> distribute.

I just pinned distribute in the buildout.cfg, as I showed, saves 
forgetting to pass -N later.

> Workaround #2: always always always wrap your buildouts in a virtualenv.

How would this have helped here? The python2.6 I ran bootstrap with is 
totally clean, compiled by me...

> I started using a 'virtual-bootstrap' script that does
>
>    #!/bin/sh
>    rm -rf develop-eggs .installed.cfg python
>    virtualenv --distribute python
>    python/bin/python bootstrap.py
>    bin/buildout "$@"
>
> and haven't encountered any issues yet.

My problem is that I *want* to use that python2.7 as a base for a lot of 
my projects. I absolutely don't want to be trying to compile numpy/etc 
on a Mac when I'm already paying Enthought to do that for me ;-)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From chris at python.org  Fri Apr  5 17:36:40 2013
From: chris at python.org (Chris Withers)
Date: Fri, 05 Apr 2013 12:36:40 -0300
Subject: [Distutils] buildout 2 craziness: bootstrap.py doesn't do enough
In-Reply-To: <20130405151555.GB23389@fridge.pov.lt>
References: <515EE521.8090701@simplistix.co.uk> <515EE996.7090107@python.org>
	<20130405151555.GB23389@fridge.pov.lt>
Message-ID: <515EEF88.1070209@python.org>

On 05/04/2013 12:15, Marius Gedminas wrote:
> Check develop-eggs/, there may be a distribute.egg-link file that puts
> /Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages
> in your sys.path.
>
> When in doubt, rm -rf develop-eggs before retrying.

Well, this worked.

My guess as to what happened: I was using python2.7 (which is EPD, 
containing a tonne of packages including an older version of xlrd).

Since that xlrd was getting picked instead of the one in the develop 
line (wasn't that bug supposed to be fixed in buildout 2?!) I switched 
to python2.6, which I knew was clean.

It sounds like I should have had the prescience (!) to do:

rm -rf develop-eggs/ .installed.cfg

...before I did:

python2.6 bootstrap.py

*sigh*

At times like this (when I've just wasted 1.5hrs chasing my tail) I 
really truly hate buildout :-(

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From jim at zope.com  Fri Apr  5 17:48:34 2013
From: jim at zope.com (Jim Fulton)
Date: Fri, 5 Apr 2013 11:48:34 -0400
Subject: [Distutils] buildout 2 craziness: bootstrap.py doesn't do enough
In-Reply-To: <515EEF88.1070209@python.org>
References: <515EE521.8090701@simplistix.co.uk> <515EE996.7090107@python.org>
	<20130405151555.GB23389@fridge.pov.lt>
	<515EEF88.1070209@python.org>
Message-ID: <CAPDm-FgH54-n7wR300WkhrGuOjMCnBUU_CT++JdiygFBdrSehw@mail.gmail.com>

On Fri, Apr 5, 2013 at 11:36 AM, Chris Withers <chris at python.org> wrote:
> On 05/04/2013 12:15, Marius Gedminas wrote:
>>
>> Check develop-eggs/, there may be a distribute.egg-link file that puts
>>
>> /Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages
>> in your sys.path.
>>
>> When in doubt, rm -rf develop-eggs before retrying.
>
>
> Well, this worked.
>
> My guess as to what happened: I was using python2.7 (which is EPD,
> containing a tonne of packages including an older version of xlrd).

BTW, I have no idea what EPD means.

>
> Since that xlrd was getting picked instead of the one in the develop line
> (wasn't that bug supposed to be fixed in buildout 2?!)

There was quite a bit of discussion about this recently and there
was no consensus that develop eggs should be preferred to
over non-develop eggs with higher versions.

> I switched to
> python2.6, which I knew was clean.
>
> It sounds like I should have had the prescience (!) to do:
>
> rm -rf develop-eggs/ .installed.cfg
>
> ...before I did:
>
> python2.6 bootstrap.py
>
> *sigh*
>
> At times like this (when I've just wasted 1.5hrs chasing my tail) I really
> truly hate buildout :-(

It's times like this that I really really hate it when people insist on using
dirty Pythons when they should know better.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From chris at python.org  Fri Apr  5 18:01:15 2013
From: chris at python.org (Chris Withers)
Date: Fri, 05 Apr 2013 13:01:15 -0300
Subject: [Distutils] buildout 2 craziness: bootstrap.py doesn't do enough
In-Reply-To: <CAPDm-FgH54-n7wR300WkhrGuOjMCnBUU_CT++JdiygFBdrSehw@mail.gmail.com>
References: <515EE521.8090701@simplistix.co.uk> <515EE996.7090107@python.org>
	<20130405151555.GB23389@fridge.pov.lt>
	<515EEF88.1070209@python.org>
	<CAPDm-FgH54-n7wR300WkhrGuOjMCnBUU_CT++JdiygFBdrSehw@mail.gmail.com>
Message-ID: <515EF54B.4040109@python.org>

On 05/04/2013 12:48, Jim Fulton wrote:
>> My guess as to what happened: I was using python2.7 (which is EPD,
>> containing a tonne of packages including an older version of xlrd).
>
> BTW, I have no idea what EPD means.

Enthought Python Distribution, I should emphasise *was* using above...

>> Since that xlrd was getting picked instead of the one in the develop line
>> (wasn't that bug supposed to be fixed in buildout 2?!)
>
> There was quite a bit of discussion about this recently and there
> was no consensus that develop eggs should be preferred to
> over non-develop eggs with higher versions.

The non-develop eggs here have significantly *lower* version numbers.
(and really, if I say I'm developing a package, why would a 
non-development version be picked?!)

>> At times like this (when I've just wasted 1.5hrs chasing my tail) I really
>> truly hate buildout :-(
>
> It's times like this that I really really hate it when people insist on using
> dirty Pythons when they should know better.

Two comments, the first an order of magnitude more important than the 
second:

1. I *was using a clean python*. Buildout decided to use some crap *it* 
left lying around from a totally different Python installation when I 
ran python2.6<clean, build form source> bootstrap.py

2. There is some practicality here. EPD provides suitable versions of a 
tonne of packages I can't and often don't even want to try compiling 
from source. Unfortunately, it provides some packages of which I want 
newer versions. I know this is a hard problem, one that Gary Poster 
tried to solve, and I appreciate that implementation didn't turn out well...

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From jim at zope.com  Fri Apr  5 18:16:50 2013
From: jim at zope.com (Jim Fulton)
Date: Fri, 5 Apr 2013 12:16:50 -0400
Subject: [Distutils] buildout 2 craziness: bootstrap.py doesn't do enough
In-Reply-To: <515EF54B.4040109@python.org>
References: <515EE521.8090701@simplistix.co.uk> <515EE996.7090107@python.org>
	<20130405151555.GB23389@fridge.pov.lt>
	<515EEF88.1070209@python.org>
	<CAPDm-FgH54-n7wR300WkhrGuOjMCnBUU_CT++JdiygFBdrSehw@mail.gmail.com>
	<515EF54B.4040109@python.org>
Message-ID: <CAPDm-FiNY5R7rq3VH=7iixV7DKtpYAK0gq9h31KdcGobRyUxow@mail.gmail.com>

On Fri, Apr 5, 2013 at 12:01 PM, Chris Withers <chris at python.org> wrote:
> On 05/04/2013 12:48, Jim Fulton wrote:
>>>
>>> My guess as to what happened: I was using python2.7 (which is EPD,
>>> containing a tonne of packages including an older version of xlrd).
>>
>>
>> BTW, I have no idea what EPD means.
>
>
> Enthought Python Distribution, I should emphasise *was* using above...
>
>
>>> Since that xlrd was getting picked instead of the one in the develop line
>>> (wasn't that bug supposed to be fixed in buildout 2?!)
>>
>>
>> There was quite a bit of discussion about this recently and there
>> was no consensus that develop eggs should be preferred to
>> over non-develop eggs with higher versions.
>
>
> The non-develop eggs here have significantly *lower* version numbers.
> (and really, if I say I'm developing a package, why would a non-development
> version be picked?!)

There are people who agree with you and people who don't. <shrug>

>
>>> At times like this (when I've just wasted 1.5hrs chasing my tail) I
>>> really
>>> truly hate buildout :-(
>>
>>
>> It's times like this that I really really hate it when people insist on
>> using
>> dirty Pythons when they should know better.
>
>
> Two comments, the first an order of magnitude more important than the
> second:
>
> 1. I *was using a clean python*. Buildout decided to use some crap *it* left
> lying around from a totally different Python installation when I ran
> python2.6<clean, build form source> bootstrap.py

But you were using a dirty Python and that left your buildout in a bad
state.  There's not a lot I can (or am willing) to do about that.

>
> 2. There is some practicality here. EPD provides suitable versions of a
> tonne of packages I can't and often don't even want to try compiling from
> source.

I understand.  I sure hope wheels help this situation.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From chris at python.org  Sat Apr  6 02:55:23 2013
From: chris at python.org (Chris Withers)
Date: Fri, 05 Apr 2013 21:55:23 -0300
Subject: [Distutils] buildout 2 craziness: bootstrap.py doesn't do enough
In-Reply-To: <CAPDm-FiNY5R7rq3VH=7iixV7DKtpYAK0gq9h31KdcGobRyUxow@mail.gmail.com>
References: <515EE521.8090701@simplistix.co.uk> <515EE996.7090107@python.org>
	<20130405151555.GB23389@fridge.pov.lt>
	<515EEF88.1070209@python.org>
	<CAPDm-FgH54-n7wR300WkhrGuOjMCnBUU_CT++JdiygFBdrSehw@mail.gmail.com>
	<515EF54B.4040109@python.org>
	<CAPDm-FiNY5R7rq3VH=7iixV7DKtpYAK0gq9h31KdcGobRyUxow@mail.gmail.com>
Message-ID: <515F727B.5060006@python.org>

On 05/04/2013 13:16, Jim Fulton wrote:
>> The non-develop eggs here have significantly *lower* version numbers.
>> (and really, if I say I'm developing a package, why would a non-development
>> version be picked?!)
>
> There are people who agree with you and people who don't.<shrug>

What's your view on it?

My take: how can I ensure that when I put in a develop line, that 
package gets used, regardless of the version number, but *especially* 
when the version number is greater than other packages found, even if 
that version number is a .dev version (which is usually the case..)

>> Two comments, the first an order of magnitude more important than the
>> second:
>>
>> 1. I *was using a clean python*. Buildout decided to use some crap *it* left
>> lying around from a totally different Python installation when I ran
>> python2.6<clean, build form source>  bootstrap.py
>
> But you were using a dirty Python and that left your buildout in a bad
> state.

No, I was using a Python that left my buildout in the state any python, 
clean or dirty, would have done...

> There's not a lot I can (or am willing) to do about that.

Having bootstrap.py really make sure things are started again from 
scratch would likely solve all these problems. (The .installed.cfg stuff 
I remember hitting before, when only using clean pythons)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From m.van.rees at zestsoftware.nl  Fri Apr  5 16:04:45 2013
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Fri, 05 Apr 2013 16:04:45 +0200
Subject: [Distutils] Re-uploading packages
In-Reply-To: <kjfe2n$600$1@ger.gmane.org>
References: <kjfe2n$600$1@ger.gmane.org>
Message-ID: <kjmllp$lpl$1@ger.gmane.org>

Op 02-04-13 22:12, Christian Theune schreef:> Hi,
 >
 > when developing bandersnatch I saw some checksum errors for the md5sums
 > of downloaded package files that I didn't understand.
 > I just saw another one and just want to check back whether this is true:
 > I can go to PyPI, delete a package version, and upload a different file
 > later.
 >
 > True?

I have seen that happen too, a while ago.  I don't think I noticed it 
often.  I did notice it for one or more distribute releases, maybe one 
or two years ago.  I noticed because I am using collective.eggproxy, 
which is basically a pypi mirror that only gets a distribution from pypi 
when it is actually requested by a user:
https://pypi.python.org/pypi/collective.eggproxy
So it is a partial mirror, saving bandwidth and disk space.

What happened was that buildout or easy_install was requesting 
distribute version X.  The mirror had that package locally, but its 
index.html file was updated with a new md5 sum from pypi.  The new md5 
sum did not match the md5 sum of the previously downloaded distribution.

So apparently the distribution got replaced on pypi.  I don't know why. 
  I compared the old and new version of the package.  I think they 
differed slightly in size, but unpacked they were exactly the same, so I 
did not mentioned it at the time.

So: yes, it can happen.

Of course, here I assume that this was not some manual action by one of 
my colleagues on the eggproxy and also not some freak error in 
collective.eggproxy.


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From ct at gocept.com  Sat Apr  6 17:44:39 2013
From: ct at gocept.com (Christian Theune)
Date: Sat, 6 Apr 2013 17:44:39 +0200
Subject: [Distutils] bandersnatch update
Message-ID: <kjpft0$iqu$1@ger.gmane.org>

Hi,

I just did some work moving bandersnatch a bit further towards a 
packaged release.

The recent changesets contain changes that require some manual 
adjustments for existing installations:

* run `bin/python bootstrap.py` and `bin/buildout` again because I 
switched to buildout 2
* you need to update your cronjobs because the commands have been 
renamed to 'bin/bandersnatch mirror' and 'bin/bandersnatch update-stats'
* check the default config file: it now includes the parameters for 
statistic updates

I think I don't need any further breakage for mirror administrators in 
the near future and will be able to make a PIP-installable release in 
the next days.

If you find any further bugs, please tell me on bitbucket.

Also, if enough people start gittipping me, then I'll start adding more 
mirrors on international locations as I can afford them.

Cheers,
Christian



From ralf at systemexit.de  Sat Apr  6 21:56:19 2013
From: ralf at systemexit.de (Ralf Schmitt)
Date: Sat, 06 Apr 2013 21:56:19 +0200
Subject: [Distutils] bandersnatch update
In-Reply-To: <kjpft0$iqu$1@ger.gmane.org> (Christian Theune's message of "Sat, 
	6 Apr 2013 17:44:39 +0200")
References: <kjpft0$iqu$1@ger.gmane.org>
Message-ID: <87fvz31li4.fsf@myhost.lan>

Christian Theune <ct at gocept.com> writes:

> Hi,
>
> I just did some work moving bandersnatch a bit further towards a
> packaged release.
>
> The recent changesets contain changes that require some manual
> adjustments for existing installations:
>
> * run `bin/python bootstrap.py` and `bin/buildout` again because I
> switched to buildout 2
> * you need to update your cronjobs because the commands have been
> renamed to 'bin/bandersnatch mirror' and 'bin/bandersnatch
> update-stats'
> * check the default config file: it now includes the parameters for
> statistic updates
>
> I think I don't need any further breakage for mirror administrators in
> the near future and will be able to make a PIP-installable release in
> the next days.
>
> If you find any further bugs, please tell me on bitbucket.
>

does bandersnatch support some kind of "selective mirroring"? e.g. only
mirror sdists or only a certain list of packages?

From ct at gocept.com  Sun Apr  7 15:36:35 2013
From: ct at gocept.com (Christian Theune)
Date: Sun, 7 Apr 2013 15:36:35 +0200
Subject: [Distutils] bandersnatch update
References: <kjpft0$iqu$1@ger.gmane.org> <87fvz31li4.fsf@myhost.lan>
Message-ID: <kjrsot$qv3$1@ger.gmane.org>

Hi,

On 2013-04-06 19:56:19 +0000, Ralf Schmitt said:
> does bandersnatch support some kind of "selective mirroring"? e.g. only
> mirror sdists or only a certain list of packages?

At the moment it does not. My primary goal is currently to provide a 
stable solution to reliably run mirrors in the sense of PEP 381.

The internal mechanisms of PEP 381 and the way that clients can 
authenticate the mirrored packages is not compatible with a partial 
mirror as the index pages are not generated but taken byte-by-byte so 
they can be verified with PyPIs cryptographic signature on them.

I can see that this would be useful. However, my current knowledge is 
that the way to authenticate the packages will change in the "near" 
future: Richard Jones mentioned that "the update framework" would be 
interesting for this. I do not know yet what the implications for the 
mirrors will be, but I don't want to invest too much time into 
complexity that I need to rework in the near future.

Christian



From ralf at systemexit.de  Sun Apr  7 15:48:55 2013
From: ralf at systemexit.de (Ralf Schmitt)
Date: Sun, 07 Apr 2013 15:48:55 +0200
Subject: [Distutils] bandersnatch update
In-Reply-To: <kjrsot$qv3$1@ger.gmane.org> (Christian Theune's message of "Sun, 
	7 Apr 2013 15:36:35 +0200")
References: <kjpft0$iqu$1@ger.gmane.org> <87fvz31li4.fsf@myhost.lan>
	<kjrsot$qv3$1@ger.gmane.org>
Message-ID: <877gke4fjs.fsf@myhost.lan>

Christian Theune <ct at gocept.com> writes:

> Hi,
>
> On 2013-04-06 19:56:19 +0000, Ralf Schmitt said:
>> does bandersnatch support some kind of "selective mirroring"? e.g. only
>> mirror sdists or only a certain list of packages?
>
> At the moment it does not. My primary goal is currently to provide a
> stable solution to reliably run mirrors in the sense of PEP 381.
>
> The internal mechanisms of PEP 381 and the way that clients can
> authenticate the mirrored packages is not compatible with a partial
> mirror as the index pages are not generated but taken byte-by-byte so
> they can be verified with PyPIs cryptographic signature on them.
>
> I can see that this would be useful. However, my current knowledge is
> that the way to authenticate the packages will change in the "near"
> future: Richard Jones mentioned that "the update framework" would be
> interesting for this. I do not know yet what the implications for the
> mirrors will be, but I don't want to invest too much time into
> complexity that I need to rework in the near future.
>

thanks for the explanation!

-- 
cheers
ralf

From ct at gocept.com  Sun Apr  7 20:43:46 2013
From: ct at gocept.com (Christian Theune)
Date: Sun, 7 Apr 2013 20:43:46 +0200
Subject: [Distutils] Towards a bandersnatch release - updates, PIP,
	contacting mirror owners
Message-ID: <kjseor$2vl$1@ger.gmane.org>

Hi,

I spend some more time on bandersnatch and I'm ready for a stable 
release, I guess.

Updates
------------

* Add a specific user agent to XML-RPC and plain HTTP calls
* Add a specific user agent to the logging to support debugging
* Provide a default (10s) network timeout and a config option to customize
* Clean up README

PIP
---

I'd be happy to cut a release now. Holger Krekel asked me whether I 
could ensure that bandersnatch is pip installable. Looking through the 
PIP documentation I don't see how I can make a simple "pip install 
bandersnatch" repeatable.

Especially if someone would run a public mirror I would prefer if they 
would use an installation method that makes their deployment 
repeatable. From my own experience and background I would prefer 
buildout in the following style:

$ hg clone https://bitbucket.org/ctheune/bandersnatch
$ cd bandersnatch
$ hg co 1.0
$ virtualenv-2.7 .
$ bin/python bootstrap.py
$ bin/buildout
$ bin/bandersnatch

Thoughts?

Contacting mirror owners
----------------------------------

All this work is directed towards getting the existing official mirrors 
back to a "green bar" situation soon. Can someone help me contact the 
existing mirror owners to convince them to switch tools?

Also, I'd be happy to run a couple vhosts around the world to extend 
the network if some people throw in some money (gittip?)

Cheers,
Christian



From robertc at robertcollins.net  Sun Apr  7 21:59:30 2013
From: robertc at robertcollins.net (Robert Collins)
Date: Mon, 8 Apr 2013 07:59:30 +1200
Subject: [Distutils] Towards a bandersnatch release - updates, PIP,
 contacting mirror owners
In-Reply-To: <kjseor$2vl$1@ger.gmane.org>
References: <kjseor$2vl$1@ger.gmane.org>
Message-ID: <CAJ3HoZ00BqLDdE6R9SDkbo-d7jHfVvUK86Us2Mo2qw3W3sfcdQ@mail.gmail.com>

On 8 April 2013 06:43, Christian Theune <ct at gocept.com> wrote:
> Hi,
>
> I spend some more time on bandersnatch and I'm ready for a stable release, I
> guess.
>
> Updates
> ------------
>
> * Add a specific user agent to XML-RPC and plain HTTP calls
> * Add a specific user agent to the logging to support debugging
> * Provide a default (10s) network timeout and a config option to customize
> * Clean up README
>
> PIP
> ---
>
> I'd be happy to cut a release now. Holger Krekel asked me whether I could
> ensure that bandersnatch is pip installable. Looking through the PIP
> documentation I don't see how I can make a simple "pip install bandersnatch"
> repeatable.
>
> Especially if someone would run a public mirror I would prefer if they would
> use an installation method that makes their deployment repeatable. From my
> own experience and background I would prefer buildout in the following
> style:
>
> $ hg clone https://bitbucket.org/ctheune/bandersnatch
> $ cd bandersnatch
> $ hg co 1.0
> $ virtualenv-2.7 .
> $ bin/python bootstrap.py
> $ bin/buildout
> $ bin/bandersnatch
>
> Thoughts?

You can make it pip installable without breaking buildout; then folk
can choose themselves. Environments that are pip based will have a
harder time with more waste if you insist on buildout being the one
true way for your tool.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Cloud Services

From holger at merlinux.eu  Sun Apr  7 22:01:07 2013
From: holger at merlinux.eu (holger krekel)
Date: Sun, 7 Apr 2013 20:01:07 +0000
Subject: [Distutils] Towards a bandersnatch release - updates, PIP,
 contacting mirror owners
In-Reply-To: <kjseor$2vl$1@ger.gmane.org>
References: <kjseor$2vl$1@ger.gmane.org>
Message-ID: <20130407200107.GP19653@merlinux.eu>

Hi Christian,

On Sun, Apr 07, 2013 at 20:43 +0200, Christian Theune wrote:
> Hi,
> 
> I spend some more time on bandersnatch and I'm ready for a stable
> release, I guess.
> 
> Updates
> ------------
> 
> * Add a specific user agent to XML-RPC and plain HTTP calls
> * Add a specific user agent to the logging to support debugging
> * Provide a default (10s) network timeout and a config option to customize
> * Clean up README
> 
> PIP
> ---
> 
> I'd be happy to cut a release now. Holger Krekel asked me whether I
> could ensure that bandersnatch is pip installable. Looking through
> the PIP documentation I don't see how I can make a simple "pip
> install bandersnatch" repeatable.
> 
> Especially if someone would run a public mirror I would prefer if
> they would use an installation method that makes their deployment
> repeatable. From my own experience and background I would prefer
> buildout in the following style:
> 
> $ hg clone https://bitbucket.org/ctheune/bandersnatch
> $ cd bandersnatch
> $ hg co 1.0
> $ virtualenv-2.7 .

As to installability, why not:

  $ bin/python setup.py install
  $ bin/bandersnatch

now? You'd need to import from setuptools instead of
distutils to get the install_requires=[dep1,dep2,...] metadata 
interpreted correctly.  Also, uploading a 1.0 release to pypi
would allow to say "pip install bandersnatch" instead of getting
the hg repo and the setup.py install command.

cheers,
holger

> $ bin/python bootstrap.py
> $ bin/buildout
> $ bin/bandersnatch
> 
> Thoughts?
> 
> Contacting mirror owners
> ----------------------------------
> 
> All this work is directed towards getting the existing official
> mirrors back to a "green bar" situation soon. Can someone help me
> contact the existing mirror owners to convince them to switch tools?
> 
> Also, I'd be happy to run a couple vhosts around the world to extend
> the network if some people throw in some money (gittip?)
> 
> Cheers,
> Christian
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 

From tk47 at students.poly.edu  Mon Apr  8 03:15:35 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Sun, 7 Apr 2013 21:15:35 -0400
Subject: [Distutils] bandersnatch update
In-Reply-To: <kjrsot$qv3$1@ger.gmane.org>
References: <kjpft0$iqu$1@ger.gmane.org> <87fvz31li4.fsf@myhost.lan>
	<kjrsot$qv3$1@ger.gmane.org>
Message-ID: <51621A37.3000405@students.poly.edu>

On 04/07/2013 09:36 AM, Christian Theune wrote:
>
> I can see that this would be useful. However, my current knowledge is
> that the way to authenticate the packages will change in the "near"
> future: Richard Jones mentioned that "the update framework" would be
> interesting for this. I do not know yet what the implications for the
> mirrors will be, but I don't want to invest too much time into
> complexity that I need to rework in the near future.

Indeed, we are working on this as I write this. I am obtaining some 
results on how large TUF metadata would be in describing PyPI. I will 
email some results very soon.


From tk47 at students.poly.edu  Mon Apr  8 03:09:44 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Sun, 7 Apr 2013 21:09:44 -0400
Subject: [Distutils] Towards a bandersnatch release - updates, PIP,
 contacting mirror owners
In-Reply-To: <20130407200107.GP19653@merlinux.eu>
References: <kjseor$2vl$1@ger.gmane.org> <20130407200107.GP19653@merlinux.eu>
Message-ID: <516218D8.7050405@students.poly.edu>

On Sun 07 Apr 2013 04:01:07 PM EDT, holger krekel wrote:
> interpreted correctly.  Also, uploading a 1.0 release to pypi
> would allow to say "pip install bandersnatch" instead of getting
> the hg repo and the setup.py install command.

I agree, a PyPI package would make it easy for pip users.


From davidbennett at ozemail.com.au  Sat Apr  6 16:03:52 2013
From: davidbennett at ozemail.com.au (david bennett)
Date: Sun, 7 Apr 2013 01:03:52 +1100
Subject: [Distutils] Help needed please
Message-ID: <49FBB893A2D3436CA154F155346F507F@marvin>

G'day
AIM: i'm trying to use Drive API with Python 2.7.3 to read and write files.

PROBLEM: is with the install of "easy_install" (ironic, i know)

1. I have enabled the drive api
2. I can see that the google client library must be there because the files are in c:/program files/python/scripts

BUT when i type easy_install into cmd window it is not recognised


-i have tried running the file: easy_install-2.7.exe
-i have updated the directory path in my computer/properties/advanced/ to include C:/Program Files/Python/Scripts
- i have downloaded the PyWin32 installer (i am running XP) for Python 2.7

http://stackoverflow.com/questions/4016151/how-to-use-pythons-easy-install-on-windows-its-not-so-easy?rq=1 mentions that "you should see that your winpexpect package has been installed correctly." winexpect. where is this located? how is this done?

thanks!

Dav-o

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130407/07ffe8dd/attachment.html>

From ct at gocept.com  Mon Apr  8 17:09:45 2013
From: ct at gocept.com (Christian Theune)
Date: Mon, 8 Apr 2013 17:09:45 +0200
Subject: [Distutils] Towards a bandersnatch release - updates, PIP,
	contacting mirror owners
References: <kjseor$2vl$1@ger.gmane.org> <516218D8.7050405@students.poly.edu>
Message-ID: <kjumjl$tsq$1@ger.gmane.org>

Hi,

On 2013-04-08 01:09:44 +0000, Trishank Karthik Kuppusamy said:

> On Sun 07 Apr 2013 04:01:07 PM EDT, holger krekel wrote:
>> interpreted correctly.  Also, uploading a 1.0 release to pypi
>> would allow to say "pip install bandersnatch" instead of getting
>> the hg repo and the setup.py install command.
> 
> I agree, a PyPI package would make it easy for pip users.

I think I made a mistake (by omission) when phrasing my question, 
sorry, let me try to clarify: I'm not asking whether I should upload a 
pip/distribute/easy_install/buildout-compatible package (sdist) to 
PyPI. That's the easy part.

As I have not used pip seriously before, I'm wondering how people 
document service setups for pip users that are repeatable, i.e. where 
all dependency versions will be fixed.

I think the verdict on fixing versions in setup.py was passed a few 
years ago (correctly), so I think pip uses requirements.txt files for 
this.

How do you guide pip users to install a service (not a library) this way?

AFAICT a changed installation instruction would look like this:

$ virtualenv-2.7 bandersnatch
$ cd bandersnatch
$ wget http://someserver/bandersnatch-1.0/requirements.txt
$ bin/pip install -r requirements.txt

This doesn't look too different from my current approach, except that 
you download a (versioned) requirements.txt file instead of getting a 
checkout. Whether you run

$ bin/python bootstrap.py; bin/buildout

or

$ bin/pip install -r requirements.txt

Doesn't make too much of a difference to me. In contrast: running from 
an HG clone gives easier ugprades, actually.

Again, I'm trying to understand why people want to do it this way and 
I'm happy to do some extra work in order to learn something.

What am I missing?

Christian



From ct at gocept.com  Mon Apr  8 10:00:35 2013
From: ct at gocept.com (Christian Theune)
Date: Mon, 8 Apr 2013 10:00:35 +0200
Subject: [Distutils] Towards a bandersnatch release - updates, PIP,
	contacting mirror owners
References: <kjseor$2vl$1@ger.gmane.org> <516218D8.7050405@students.poly.edu>
Message-ID: <kjtter$4km$1@ger.gmane.org>

On 2013-04-08 01:09:44 +0000, Trishank Karthik Kuppusamy said:

> On Sun 07 Apr 2013 04:01:07 PM EDT, holger krekel wrote:
>> interpreted correctly.  Also, uploading a 1.0 release to pypi
>> would allow to say "pip install bandersnatch" instead of getting
>> the hg repo and the setup.py install command.
> 
> I agree, a PyPI package would make it easy for pip users.

Making an sdist and uploading it to PyPI isn't what I have trouble with:

The installation instructions that I provide are intended to help 
people operate (public) mirrors. I want to make it as smooth as 
possible and will likely have to respond to support requests when 
people have problems getting set up etc.

One of the biggest issues doing a service deployment is to get the 
dependency versions right. Fixing all versions in setup.py is too 
inflexible and I personally am happy with the way buildout solves this.

I would be happy to support a pip installation *if* I can make 
instructions that are as straight forward as the ones I'm giving for 
buildout and end up in people having predictable installations.

"pip install bandersnatch" will result in whatever dependency versions 
people happen to get. "pip install -r 
http://bitbucket.org/.../requirements.txt" is just close to the hellish 
"curl | bash".

Again: I'd be happy to support people if they prefer PIP for 
installation. Using a requirements.txt file I don't see much 
improvement from the "virtualenv, get the file, run it". Doing that 
with a tagged Mercurial checkout + buildout at least lets you catch up 
to newer releases more smoothly, no?

Regards,
Christian



From tk47 at students.poly.edu  Mon Apr  8 20:41:17 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Mon, 8 Apr 2013 14:41:17 -0400
Subject: [Distutils] Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <5159FEC6.6070204@students.poly.edu>
References: <5159FEC6.6070204@students.poly.edu>
Message-ID: <51630F4D.2090102@students.poly.edu>

Hello everyone,

I have been testing and refining the pypi.updateframework.com automation 
over the past week, and looking at how much TUF metadata is generated 
for PyPI.

In this email, I am going to focus only on the PyPI data under /simple; 
let us call that "simple data".

Now, if we assume that every developer will have her own key to sign the 
simple data for her package, then this is what the TUF metadata could 
look like:

metadata/targets.txt
====================
Delegation from the targets to the targets/simple role, with the former 
role being responsible for no target data because it has none of its own.

metadata/targets/simple.txt
===========================
Delegation from targets/simple to the targets/simple/packageI role, with 
the former role being responsible for one target datum: simple/index.html.

metadata/targets/simple/packageI.txt
====================================
The targets/simple/packageI role is responsible only for the simple data 
at simple/packageI/index.html.

In this upper bound case, where every developer is responsible for 
signing her own package, one can estimate the metadata size to be like so:

- metadata/targets/targets.txt is, at most, about a few KB, and can be 
safely ignored.
- metadata/targets/simple/packageI.txt is about 1KB.
- metadata/targets/simple.txt is about the sum of all 
metadata/targets/simple/packageI.txt files. (This is a very rough estimate!)

Therefore, if we have 30,000 developer packages on PyPI (roughly the 
current number of packages), then we would have about 29 MB of 
metadata/targets/simple/packageI.txt, and another 29 MB of 
metadata/targets/simple.txt, for a rough total of 58MB. If PyPI has 45GB 
of total data (roughly what I saw from my last mirror), then the simple 
metadata is about 0.13% of total data size.

This may seem like a lot of metadata, but let us remember a few 
important things:

- So far, the metadata is simply uncompressed JSON. We are considering 
metadata compression or difference schemes.
- This assumes the upper bound case, where every package developer is 
responsible for her own package, so that means that we have talk about a 
lot of keys (random data).
- This is a one-time initial download cost. An update to PyPI is 
unlikely to change all the simple data; therefore, updates to the simple 
metadata will be cheap, because a TUF client would only download updated 
metadata. We could amortize the initial simple metadata download cost by 
distributing it with PyPI installers (e.g. pip).

Could we do better? Yes!

As Nick Coghlan has suggested, PyPI could begin adopting TUF by signing 
for all of the developer packages itself. This means that we could reuse 
a key for multiple developer packages instead of dedicating a key per 
package. The tradeoff here is that if one such "shared key" is 
compromised, then multiple packages (but not all of them) could be 
compromised.

In this case, where we use a shared key to sign up to, say, 1,000 
developer packages, then we would have the following simple metadata 
size. First, let us define some terms:

NP = # of developer packages
NPK = # of developer packages signed by a key
NR = # of roles (each responsible for NPK packages) = math.ceil(NP/NPK)
K = average key metadata size
D = average delegated role metadata size given one target path
P = average target path length
T = average simple target (index.html) metadata size

metadata/targets/simple.txt
===========================
Most of the metadata here deals with all of the keys, and the roles, 
used to sign simple data. Therefore, the size of the keys and roles 
metadata will dominate this file.

key metadata size = NR*K
role metadata size = NR*(D+NPK*P)

Takeaway: the lower the NPK (the number of developer packages signed by 
a key), then the higher the NR, and the larger the metadata. We would 
save metadata by setting NPK to, say, 1,000, because then one key could 
describe 1,000 packages.

metadata/targets/simple/roleI.txt
====================================
When NPK=1, then this file would be equivalent to 
metadata/targets/simple/packageI.txt.

It is a small metadata file if we assume that it only talks about the 
simple data (index.html) for one package. Most of the metadata talks 
about key signatures, and target metadata. If we increase NPK, then 
clearly the target metadata would increase in size:

target metadata size = NPK*T < NPK*1KB

Takeaway: the target metadata would increase in size, but it certainly 
will not increase as much as it would have if we had signed each 
developer package with a separate key.

Finally, the question is how the savings in metadata/targets/simple.txt 
would compare to the "growth" of the metadata/targets/simple/roleI.txt 
files. Ultimately, the higher the NPK (and thus the lower the NR), then 
the less would we be talking about keys (random data). Everything else 
would remain the same, because there would still be the same number of 
targets, and thus the same amount of target metadata. So, we would have 
net savings.

I hope this clears some questions about metadata size. If there was 
something confusing because I did not explain it well enough or I got 
something wrong, please be sure to let me know. My machine is nearly 
done generating all the simple metadata, so we can make better estimates 
then.

-Trishank


From carl at oddbird.net  Mon Apr  8 20:07:50 2013
From: carl at oddbird.net (Carl Meyer)
Date: Mon, 08 Apr 2013 12:07:50 -0600
Subject: [Distutils] Towards a bandersnatch release - updates, PIP,
 contacting mirror owners
In-Reply-To: <kjumjl$tsq$1@ger.gmane.org>
References: <kjseor$2vl$1@ger.gmane.org> <516218D8.7050405@students.poly.edu>
	<kjumjl$tsq$1@ger.gmane.org>
Message-ID: <51630776.3040702@oddbird.net>

Hello Christian,

On 04/08/2013 09:09 AM, Christian Theune wrote:
> As I have not used pip seriously before, I'm wondering how people
> document service setups for pip users that are repeatable, i.e. where
> all dependency versions will be fixed.
> 
> I think the verdict on fixing versions in setup.py was passed a few
> years ago (correctly), so I think pip uses requirements.txt files for this.
> 
> How do you guide pip users to install a service (not a library) this way?
> 
> AFAICT a changed installation instruction would look like this:
> 
> $ virtualenv-2.7 bandersnatch
> $ cd bandersnatch
> $ wget http://someserver/bandersnatch-1.0/requirements.txt
> $ bin/pip install -r requirements.txt
>
> This doesn't look too different from my current approach, except that
> you download a (versioned) requirements.txt file instead of getting a
> checkout. Whether you run
> 
> $ bin/python bootstrap.py; bin/buildout
> 
> or
> 
> $ bin/pip install -r requirements.txt
> 
> Doesn't make too much of a difference to me. In contrast: running from
> an HG clone gives easier ugprades, actually.
> 
> Again, I'm trying to understand why people want to do it this way and
> I'm happy to do some extra work in order to learn something.
> 
> What am I missing?

I don't think you're really missing anything, except that for some
people pip is familiar and comfortable and buildout is not. And these
people may want to manage their deployment using existing infrastructure
(e.g. Chef/Puppet recipes) that relies on having a requirements.txt
file, making the two options not so equivalent as they seem to you. For
instance, if someone wanted to deploy bandersnatch to the Heroku
platform-as-a-service without writing a custom build-pack, they would
not even be running either of the commands above themselves, they would
need to provide a requirements.txt file to Heroku.

I don't think this would need to be a particularly complex change on
your part; just upload the sdist, provide a requirements.txt file with
exactly-pinned dependencies for each bandersnatch version, and note its
existence as a second option in your installation instructions.

Carl

From pje at telecommunity.com  Mon Apr  8 21:43:07 2013
From: pje at telecommunity.com (PJ Eby)
Date: Mon, 8 Apr 2013 15:43:07 -0400
Subject: [Distutils] bandersnatch update
In-Reply-To: <kjrsot$qv3$1@ger.gmane.org>
References: <kjpft0$iqu$1@ger.gmane.org> <87fvz31li4.fsf@myhost.lan>
	<kjrsot$qv3$1@ger.gmane.org>
Message-ID: <CALeMXf5SmxErjQkaHBL1sikpUuM2NO57Pn7eyEGa7y+LX6k01A@mail.gmail.com>

On Sun, Apr 7, 2013 at 9:36 AM, Christian Theune <ct at gocept.com> wrote:
> The internal mechanisms of PEP 381 and the way that clients can authenticate
> the mirrored packages is not compatible with a partial mirror as the index
> pages are not generated but taken byte-by-byte so they can be verified with
> PyPIs cryptographic signature on them.

FWIW, those pages use relative URLs for PyPI-hosted packages, which
means you could serve those as a redirect in order to implement
partial mirroring, at least in theory. ;-)

From pje at telecommunity.com  Mon Apr  8 21:30:29 2013
From: pje at telecommunity.com (PJ Eby)
Date: Mon, 8 Apr 2013 15:30:29 -0400
Subject: [Distutils] Help needed please
In-Reply-To: <49FBB893A2D3436CA154F155346F507F@marvin>
References: <49FBB893A2D3436CA154F155346F507F@marvin>
Message-ID: <CALeMXf4Fe9O=UETTPWvgpKK32m42nPYEqQ_sGpxVHvdyUeaqWw@mail.gmail.com>

On Sat, Apr 6, 2013 at 10:03 AM, david bennett
<davidbennett at ozemail.com.au> wrote:
> G'day
> AIM: i'm trying to use Drive API with Python 2.7.3 to read and write files.
>
> PROBLEM: is with the install of "easy_install" (ironic, i know)
>
> 1. I have enabled the drive api
> 2. I can see that the google client library must be there because the files
> are in c:/program files/python/scripts
>
> BUT when i type easy_install into cmd window it is not recognised

Type "path" into the command window and double-check that C:/Program
Files/Python/Scripts is listed.

(Do check, also, that C:/Program Files/Python/Scripts is actually
where easy_install.exe is located -- the standard installation
location for many versions of Python would be C:\Python27, not
C:\Program Files\Python.)


> http://stackoverflow.com/questions/4016151/how-to-use-pythons-easy-install-on-windows-its-not-so-easy?rq=1
> mentions that "you should see that your winpexpect package has been
> installed correctly." winexpect. where is this located? how is this done?

That discussion is actually about installing winpexpect, not about
installing easy_install.  So the problem that person was having is
entirely unrelated to the one you are having.

From jcappos at poly.edu  Tue Apr  9 01:58:40 2013
From: jcappos at poly.edu (Justin Cappos)
Date: Mon, 8 Apr 2013 19:58:40 -0400
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <51630F4D.2090102@students.poly.edu>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
Message-ID: <CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>

FYI: For anyone who wants the executive summary, we think the TUF metadata
will be under 1MB and even with very broad / rapid adoption of TUF in the
next year or two will stay <3MB or so.

Note that this cost is only paid upon the initial run of the client tool.
Everything after that just downloads diffs (or at least will once we fix an
open ticket).

Thanks,
Justin



On Mon, Apr 8, 2013 at 2:41 PM, Trishank Karthik Kuppusamy <
tk47 at students.poly.edu> wrote:

> Hello everyone,
>
> I have been testing and refining the pypi.updateframework.com automation
> over the past week, and looking at how much TUF metadata is generated for
> PyPI.
>
> In this email, I am going to focus only on the PyPI data under /simple;
> let us call that "simple data".
>
> Now, if we assume that every developer will have her own key to sign the
> simple data for her package, then this is what the TUF metadata could look
> like:
>
> metadata/targets.txt
> ====================
> Delegation from the targets to the targets/simple role, with the former
> role being responsible for no target data because it has none of its own.
>
> metadata/targets/simple.txt
> ===========================
> Delegation from targets/simple to the targets/simple/packageI role, with
> the former role being responsible for one target datum: simple/index.html.
>
> metadata/targets/simple/**packageI.txt
> ==============================**======
> The targets/simple/packageI role is responsible only for the simple data
> at simple/packageI/index.html.
>
> In this upper bound case, where every developer is responsible for signing
> her own package, one can estimate the metadata size to be like so:
>
> - metadata/targets/targets.txt is, at most, about a few KB, and can be
> safely ignored.
> - metadata/targets/simple/**packageI.txt is about 1KB.
> - metadata/targets/simple.txt is about the sum of all
> metadata/targets/simple/**packageI.txt files. (This is a very rough
> estimate!)
>
> Therefore, if we have 30,000 developer packages on PyPI (roughly the
> current number of packages), then we would have about 29 MB of
> metadata/targets/simple/**packageI.txt, and another 29 MB of
> metadata/targets/simple.txt, for a rough total of 58MB. If PyPI has 45GB of
> total data (roughly what I saw from my last mirror), then the simple
> metadata is about 0.13% of total data size.
>
> This may seem like a lot of metadata, but let us remember a few important
> things:
>
> - So far, the metadata is simply uncompressed JSON. We are considering
> metadata compression or difference schemes.
> - This assumes the upper bound case, where every package developer is
> responsible for her own package, so that means that we have talk about a
> lot of keys (random data).
> - This is a one-time initial download cost. An update to PyPI is unlikely
> to change all the simple data; therefore, updates to the simple metadata
> will be cheap, because a TUF client would only download updated metadata.
> We could amortize the initial simple metadata download cost by distributing
> it with PyPI installers (e.g. pip).
>
> Could we do better? Yes!
>
> As Nick Coghlan has suggested, PyPI could begin adopting TUF by signing
> for all of the developer packages itself. This means that we could reuse a
> key for multiple developer packages instead of dedicating a key per
> package. The tradeoff here is that if one such "shared key" is compromised,
> then multiple packages (but not all of them) could be compromised.
>
> In this case, where we use a shared key to sign up to, say, 1,000
> developer packages, then we would have the following simple metadata size.
> First, let us define some terms:
>
> NP = # of developer packages
> NPK = # of developer packages signed by a key
> NR = # of roles (each responsible for NPK packages) = math.ceil(NP/NPK)
> K = average key metadata size
> D = average delegated role metadata size given one target path
> P = average target path length
> T = average simple target (index.html) metadata size
>
> metadata/targets/simple.txt
> ===========================
> Most of the metadata here deals with all of the keys, and the roles, used
> to sign simple data. Therefore, the size of the keys and roles metadata
> will dominate this file.
>
> key metadata size = NR*K
> role metadata size = NR*(D+NPK*P)
>
> Takeaway: the lower the NPK (the number of developer packages signed by a
> key), then the higher the NR, and the larger the metadata. We would save
> metadata by setting NPK to, say, 1,000, because then one key could describe
> 1,000 packages.
>
> metadata/targets/simple/roleI.**txt
> ==============================**======
> When NPK=1, then this file would be equivalent to metadata/targets/simple/
> **packageI.txt.
>
> It is a small metadata file if we assume that it only talks about the
> simple data (index.html) for one package. Most of the metadata talks about
> key signatures, and target metadata. If we increase NPK, then clearly the
> target metadata would increase in size:
>
> target metadata size = NPK*T < NPK*1KB
>
> Takeaway: the target metadata would increase in size, but it certainly
> will not increase as much as it would have if we had signed each developer
> package with a separate key.
>
> Finally, the question is how the savings in metadata/targets/simple.txt
> would compare to the "growth" of the metadata/targets/simple/roleI.**txt
> files. Ultimately, the higher the NPK (and thus the lower the NR), then the
> less would we be talking about keys (random data). Everything else would
> remain the same, because there would still be the same number of targets,
> and thus the same amount of target metadata. So, we would have net savings.
>
> I hope this clears some questions about metadata size. If there was
> something confusing because I did not explain it well enough or I got
> something wrong, please be sure to let me know. My machine is nearly done
> generating all the simple metadata, so we can make better estimates then.
>
> -Trishank
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130408/a23d6f60/attachment.html>

From ncoghlan at gmail.com  Tue Apr  9 06:18:10 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 Apr 2013 14:18:10 +1000
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
	<CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
Message-ID: <CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>

On Tue, Apr 9, 2013 at 9:58 AM, Justin Cappos <jcappos at poly.edu> wrote:
> FYI: For anyone who wants the executive summary, we think the TUF metadata
> will be under 1MB and even with very broad / rapid adoption of TUF in the
> next year or two will stay <3MB or so.

Is that after compression? Or did Trishank miscount the number of
digits for the initial email?

Cheers,
Nick.


>
> Note that this cost is only paid upon the initial run of the client tool.
> Everything after that just downloads diffs (or at least will once we fix an
> open ticket).
>
> Thanks,
> Justin
>
>
>
> On Mon, Apr 8, 2013 at 2:41 PM, Trishank Karthik Kuppusamy
> <tk47 at students.poly.edu> wrote:
>>
>> Hello everyone,
>>
>> I have been testing and refining the pypi.updateframework.com automation
>> over the past week, and looking at how much TUF metadata is generated for
>> PyPI.
>>
>> In this email, I am going to focus only on the PyPI data under /simple;
>> let us call that "simple data".
>>
>> Now, if we assume that every developer will have her own key to sign the
>> simple data for her package, then this is what the TUF metadata could look
>> like:
>>
>> metadata/targets.txt
>> ====================
>> Delegation from the targets to the targets/simple role, with the former
>> role being responsible for no target data because it has none of its own.
>>
>> metadata/targets/simple.txt
>> ===========================
>> Delegation from targets/simple to the targets/simple/packageI role, with
>> the former role being responsible for one target datum: simple/index.html.
>>
>> metadata/targets/simple/packageI.txt
>> ====================================
>> The targets/simple/packageI role is responsible only for the simple data
>> at simple/packageI/index.html.
>>
>> In this upper bound case, where every developer is responsible for signing
>> her own package, one can estimate the metadata size to be like so:
>>
>> - metadata/targets/targets.txt is, at most, about a few KB, and can be
>> safely ignored.
>> - metadata/targets/simple/packageI.txt is about 1KB.
>> - metadata/targets/simple.txt is about the sum of all
>> metadata/targets/simple/packageI.txt files. (This is a very rough estimate!)
>>
>> Therefore, if we have 30,000 developer packages on PyPI (roughly the
>> current number of packages), then we would have about 29 MB of
>> metadata/targets/simple/packageI.txt, and another 29 MB of
>> metadata/targets/simple.txt, for a rough total of 58MB. If PyPI has 45GB of
>> total data (roughly what I saw from my last mirror), then the simple
>> metadata is about 0.13% of total data size.
>>
>> This may seem like a lot of metadata, but let us remember a few important
>> things:
>>
>> - So far, the metadata is simply uncompressed JSON. We are considering
>> metadata compression or difference schemes.
>> - This assumes the upper bound case, where every package developer is
>> responsible for her own package, so that means that we have talk about a lot
>> of keys (random data).
>> - This is a one-time initial download cost. An update to PyPI is unlikely
>> to change all the simple data; therefore, updates to the simple metadata
>> will be cheap, because a TUF client would only download updated metadata. We
>> could amortize the initial simple metadata download cost by distributing it
>> with PyPI installers (e.g. pip).
>>
>> Could we do better? Yes!
>>
>> As Nick Coghlan has suggested, PyPI could begin adopting TUF by signing
>> for all of the developer packages itself. This means that we could reuse a
>> key for multiple developer packages instead of dedicating a key per package.
>> The tradeoff here is that if one such "shared key" is compromised, then
>> multiple packages (but not all of them) could be compromised.
>>
>> In this case, where we use a shared key to sign up to, say, 1,000
>> developer packages, then we would have the following simple metadata size.
>> First, let us define some terms:
>>
>> NP = # of developer packages
>> NPK = # of developer packages signed by a key
>> NR = # of roles (each responsible for NPK packages) = math.ceil(NP/NPK)
>> K = average key metadata size
>> D = average delegated role metadata size given one target path
>> P = average target path length
>> T = average simple target (index.html) metadata size
>>
>> metadata/targets/simple.txt
>> ===========================
>> Most of the metadata here deals with all of the keys, and the roles, used
>> to sign simple data. Therefore, the size of the keys and roles metadata will
>> dominate this file.
>>
>> key metadata size = NR*K
>> role metadata size = NR*(D+NPK*P)
>>
>> Takeaway: the lower the NPK (the number of developer packages signed by a
>> key), then the higher the NR, and the larger the metadata. We would save
>> metadata by setting NPK to, say, 1,000, because then one key could describe
>> 1,000 packages.
>>
>> metadata/targets/simple/roleI.txt
>> ====================================
>> When NPK=1, then this file would be equivalent to
>> metadata/targets/simple/packageI.txt.
>>
>> It is a small metadata file if we assume that it only talks about the
>> simple data (index.html) for one package. Most of the metadata talks about
>> key signatures, and target metadata. If we increase NPK, then clearly the
>> target metadata would increase in size:
>>
>> target metadata size = NPK*T < NPK*1KB
>>
>> Takeaway: the target metadata would increase in size, but it certainly
>> will not increase as much as it would have if we had signed each developer
>> package with a separate key.
>>
>> Finally, the question is how the savings in metadata/targets/simple.txt
>> would compare to the "growth" of the metadata/targets/simple/roleI.txt
>> files. Ultimately, the higher the NPK (and thus the lower the NR), then the
>> less would we be talking about keys (random data). Everything else would
>> remain the same, because there would still be the same number of targets,
>> and thus the same amount of target metadata. So, we would have net savings.
>>
>> I hope this clears some questions about metadata size. If there was
>> something confusing because I did not explain it well enough or I got
>> something wrong, please be sure to let me know. My machine is nearly done
>> generating all the simple metadata, so we can make better estimates then.
>>
>> -Trishank
>>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Apr  9 07:19:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 Apr 2013 15:19:53 +1000
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
	<CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
	<CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>
	<CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>
Message-ID: <CADiSq7f0-wjRu+HbLAAx9b5H=sVEvbzzeqpg+5oYi_+cgqNyJg@mail.gmail.com>

On Tue, Apr 9, 2013 at 3:17 PM, Justin Cappos <jcappos at poly.edu> wrote:
> His 29MB and 58MB numbers assume that every developer has their own key
> right now.   We don't think this is likely to happen and propose initially
> signing everything that the developers don't sign with a single PyPI key.
>
> It also assumes there are no abandoned packages / devel account.   I also
> think many devels won't go back and sign all old versions of their software.
> So my number is definitely a back of the envelope calculation using
> Trishank's data.   Trishank's calculations are much more expressive, but are
> the "worst case" size.

OK, that makes sense - thanks for the clarification.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tk47 at students.poly.edu  Tue Apr  9 07:23:24 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Tue, 9 Apr 2013 01:23:24 -0400
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
	<CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
	<CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>
	<CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>
Message-ID: <5163A5CC.8010807@students.poly.edu>

On 4/9/13 1:17 AM, Justin Cappos wrote:
> His 29MB and 58MB numbers assume that every developer has their own key
> right now.   We don't think this is likely to happen and propose
> initially signing everything that the developers don't sign with a
> single PyPI key.
>
> It also assumes there are no abandoned packages / devel account.   I
> also think many devels won't go back and sign all old versions of their
> software.   So my number is definitely a back of the envelope
> calculation using Trishank's data.   Trishank's calculations are much
> more expressive, but are the "worst case" size.

Correct. Justin based his back-of-the-envelope calculation on some very 
rough prior estimates of mine, so they may be a little off. 
Nevertheless, our argument remains: sharing a key across, say, a 
thousand packages will certainly reduce the metadata by quite a bit. 
Combine that with compression or difference schemes, and you get even 
more savings.


From jcappos at poly.edu  Tue Apr  9 07:17:31 2013
From: jcappos at poly.edu (Justin Cappos)
Date: Tue, 9 Apr 2013 01:17:31 -0400
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
	<CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
	<CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>
Message-ID: <CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>

His 29MB and 58MB numbers assume that every developer has their own key
right now.   We don't think this is likely to happen and propose initially
signing everything that the developers don't sign with a single PyPI key.

It also assumes there are no abandoned packages / devel account.   I also
think many devels won't go back and sign all old versions of their
software.   So my number is definitely a back of the envelope calculation
using Trishank's data.   Trishank's calculations are much more expressive,
but are the "worst case" size.

Thanks,
Justin




On Tue, Apr 9, 2013 at 12:18 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Tue, Apr 9, 2013 at 9:58 AM, Justin Cappos <jcappos at poly.edu> wrote:
> > FYI: For anyone who wants the executive summary, we think the TUF
> metadata
> > will be under 1MB and even with very broad / rapid adoption of TUF in the
> > next year or two will stay <3MB or so.
>
> Is that after compression? Or did Trishank miscount the number of
> digits for the initial email?
>
> Cheers,
> Nick.
>
>
> >
> > Note that this cost is only paid upon the initial run of the client tool.
> > Everything after that just downloads diffs (or at least will once we fix
> an
> > open ticket).
> >
> > Thanks,
> > Justin
> >
> >
> >
> > On Mon, Apr 8, 2013 at 2:41 PM, Trishank Karthik Kuppusamy
> > <tk47 at students.poly.edu> wrote:
> >>
> >> Hello everyone,
> >>
> >> I have been testing and refining the pypi.updateframework.comautomation
> >> over the past week, and looking at how much TUF metadata is generated
> for
> >> PyPI.
> >>
> >> In this email, I am going to focus only on the PyPI data under /simple;
> >> let us call that "simple data".
> >>
> >> Now, if we assume that every developer will have her own key to sign the
> >> simple data for her package, then this is what the TUF metadata could
> look
> >> like:
> >>
> >> metadata/targets.txt
> >> ====================
> >> Delegation from the targets to the targets/simple role, with the former
> >> role being responsible for no target data because it has none of its
> own.
> >>
> >> metadata/targets/simple.txt
> >> ===========================
> >> Delegation from targets/simple to the targets/simple/packageI role, with
> >> the former role being responsible for one target datum:
> simple/index.html.
> >>
> >> metadata/targets/simple/packageI.txt
> >> ====================================
> >> The targets/simple/packageI role is responsible only for the simple data
> >> at simple/packageI/index.html.
> >>
> >> In this upper bound case, where every developer is responsible for
> signing
> >> her own package, one can estimate the metadata size to be like so:
> >>
> >> - metadata/targets/targets.txt is, at most, about a few KB, and can be
> >> safely ignored.
> >> - metadata/targets/simple/packageI.txt is about 1KB.
> >> - metadata/targets/simple.txt is about the sum of all
> >> metadata/targets/simple/packageI.txt files. (This is a very rough
> estimate!)
> >>
> >> Therefore, if we have 30,000 developer packages on PyPI (roughly the
> >> current number of packages), then we would have about 29 MB of
> >> metadata/targets/simple/packageI.txt, and another 29 MB of
> >> metadata/targets/simple.txt, for a rough total of 58MB. If PyPI has
> 45GB of
> >> total data (roughly what I saw from my last mirror), then the simple
> >> metadata is about 0.13% of total data size.
> >>
> >> This may seem like a lot of metadata, but let us remember a few
> important
> >> things:
> >>
> >> - So far, the metadata is simply uncompressed JSON. We are considering
> >> metadata compression or difference schemes.
> >> - This assumes the upper bound case, where every package developer is
> >> responsible for her own package, so that means that we have talk about
> a lot
> >> of keys (random data).
> >> - This is a one-time initial download cost. An update to PyPI is
> unlikely
> >> to change all the simple data; therefore, updates to the simple metadata
> >> will be cheap, because a TUF client would only download updated
> metadata. We
> >> could amortize the initial simple metadata download cost by
> distributing it
> >> with PyPI installers (e.g. pip).
> >>
> >> Could we do better? Yes!
> >>
> >> As Nick Coghlan has suggested, PyPI could begin adopting TUF by signing
> >> for all of the developer packages itself. This means that we could
> reuse a
> >> key for multiple developer packages instead of dedicating a key per
> package.
> >> The tradeoff here is that if one such "shared key" is compromised, then
> >> multiple packages (but not all of them) could be compromised.
> >>
> >> In this case, where we use a shared key to sign up to, say, 1,000
> >> developer packages, then we would have the following simple metadata
> size.
> >> First, let us define some terms:
> >>
> >> NP = # of developer packages
> >> NPK = # of developer packages signed by a key
> >> NR = # of roles (each responsible for NPK packages) = math.ceil(NP/NPK)
> >> K = average key metadata size
> >> D = average delegated role metadata size given one target path
> >> P = average target path length
> >> T = average simple target (index.html) metadata size
> >>
> >> metadata/targets/simple.txt
> >> ===========================
> >> Most of the metadata here deals with all of the keys, and the roles,
> used
> >> to sign simple data. Therefore, the size of the keys and roles metadata
> will
> >> dominate this file.
> >>
> >> key metadata size = NR*K
> >> role metadata size = NR*(D+NPK*P)
> >>
> >> Takeaway: the lower the NPK (the number of developer packages signed by
> a
> >> key), then the higher the NR, and the larger the metadata. We would save
> >> metadata by setting NPK to, say, 1,000, because then one key could
> describe
> >> 1,000 packages.
> >>
> >> metadata/targets/simple/roleI.txt
> >> ====================================
> >> When NPK=1, then this file would be equivalent to
> >> metadata/targets/simple/packageI.txt.
> >>
> >> It is a small metadata file if we assume that it only talks about the
> >> simple data (index.html) for one package. Most of the metadata talks
> about
> >> key signatures, and target metadata. If we increase NPK, then clearly
> the
> >> target metadata would increase in size:
> >>
> >> target metadata size = NPK*T < NPK*1KB
> >>
> >> Takeaway: the target metadata would increase in size, but it certainly
> >> will not increase as much as it would have if we had signed each
> developer
> >> package with a separate key.
> >>
> >> Finally, the question is how the savings in metadata/targets/simple.txt
> >> would compare to the "growth" of the metadata/targets/simple/roleI.txt
> >> files. Ultimately, the higher the NPK (and thus the lower the NR), then
> the
> >> less would we be talking about keys (random data). Everything else would
> >> remain the same, because there would still be the same number of
> targets,
> >> and thus the same amount of target metadata. So, we would have net
> savings.
> >>
> >> I hope this clears some questions about metadata size. If there was
> >> something confusing because I did not explain it well enough or I got
> >> something wrong, please be sure to let me know. My machine is nearly
> done
> >> generating all the simple metadata, so we can make better estimates
> then.
> >>
> >> -Trishank
> >>
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > http://mail.python.org/mailman/listinfo/distutils-sig
> >
>
>
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130409/3b3af55e/attachment-0001.html>

From ct at gocept.com  Tue Apr  9 08:41:46 2013
From: ct at gocept.com (Christian Theune)
Date: Tue, 9 Apr 2013 08:41:46 +0200
Subject: [Distutils] bandersnatch update
References: <kjpft0$iqu$1@ger.gmane.org> <87fvz31li4.fsf@myhost.lan>
	<kjrsot$qv3$1@ger.gmane.org>
	<CALeMXf5SmxErjQkaHBL1sikpUuM2NO57Pn7eyEGa7y+LX6k01A@mail.gmail.com>
Message-ID: <kk0d74$pr5$1@ger.gmane.org>

On 2013-04-08 19:43:07 +0000, PJ Eby said:

> On Sun, Apr 7, 2013 at 9:36 AM, Christian Theune <ct at gocept.com> wrote:
>> The internal mechanisms of PEP 381 and the way that clients can authenticate
>> the mirrored packages is not compatible with a partial mirror as the index
>> pages are not generated but taken byte-by-byte so they can be verified with
>> PyPIs cryptographic signature on them.
> 
> FWIW, those pages use relative URLs for PyPI-hosted packages, which
> means you could serve those as a redirect in order to implement
> partial mirroring, at least in theory. ;-)

Heh. I did not see this coming, but I like it. :)

The simplest solution on a static mirror that I can see for this would 
be (with nginx) a "missing files" rule that redirects to PyPI. However, 
that would be bad form as that would cause *any* 404-triggering request 
on a mirror to hit the upstream server - bad for bots.

I'll let this remain science fiction for now. :)

Christian



From reinout at vanrees.org  Tue Apr  9 13:36:59 2013
From: reinout at vanrees.org (Reinout van Rees)
Date: Tue, 09 Apr 2013 13:36:59 +0200
Subject: [Distutils] buildout 2 craziness: bootstrap.py doesn't do enough
In-Reply-To: <515F727B.5060006@python.org>
References: <515EE521.8090701@simplistix.co.uk> <515EE996.7090107@python.org>
	<20130405151555.GB23389@fridge.pov.lt>
	<515EEF88.1070209@python.org>
	<CAPDm-FgH54-n7wR300WkhrGuOjMCnBUU_CT++JdiygFBdrSehw@mail.gmail.com>
	<515EF54B.4040109@python.org>
	<CAPDm-FiNY5R7rq3VH=7iixV7DKtpYAK0gq9h31KdcGobRyUxow@mail.gmail.com>
	<515F727B.5060006@python.org>
Message-ID: <kk0ugp$chp$1@ger.gmane.org>

On 06-04-13 02:55, Chris Withers wrote:
> Having bootstrap.py really make sure things are started again from
> scratch would likely solve all these problems. (The .installed.cfg stuff
> I remember hitting before, when only using clean pythons)

Removing .installed.cfg: not sure. Some recipes fail when installing 
themselves when there's still something left over from the last run. And 
'install' versus 'update' is used when the .installed.cfg is gone.

OTOH, that's somethint that needs fixing in the recipes.

Removing the develop eggs: that actually sounds like a very good idea. 
It would have solved quite some issues with system packages here at the 
office.

(And yes, we use system packages too. Big geographical stack with numpy, 
mapnik, matplotlib, geos, scipy, netcdf... Impractical to compile from 
source. Though I'm sometimes tempted to make a buildout that compiles 
all that stuff for me :-) )


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From dholth at gmail.com  Tue Apr  9 13:47:54 2013
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 9 Apr 2013 07:47:54 -0400
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <5163A5CC.8010807@students.poly.edu>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
	<CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
	<CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>
	<CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>
	<5163A5CC.8010807@students.poly.edu>
Message-ID: <CAG8k2+5gpaWDKhJ0Z3ndfNZ6CpiGjAM=kAqfxkYG41-VOkqwXQ@mail.gmail.com>

What size keys?
On Apr 9, 2013 1:23 AM, "Trishank Karthik Kuppusamy" <tk47 at students.poly.edu>
wrote:

> On 4/9/13 1:17 AM, Justin Cappos wrote:
>
>> His 29MB and 58MB numbers assume that every developer has their own key
>> right now.   We don't think this is likely to happen and propose
>> initially signing everything that the developers don't sign with a
>> single PyPI key.
>>
>> It also assumes there are no abandoned packages / devel account.   I
>> also think many devels won't go back and sign all old versions of their
>> software.   So my number is definitely a back of the envelope
>> calculation using Trishank's data.   Trishank's calculations are much
>> more expressive, but are the "worst case" size.
>>
>
> Correct. Justin based his back-of-the-envelope calculation on some very
> rough prior estimates of mine, so they may be a little off. Nevertheless,
> our argument remains: sharing a key across, say, a thousand packages will
> certainly reduce the metadata by quite a bit. Combine that with compression
> or difference schemes, and you get even more savings.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130409/a98a1d05/attachment.html>

From tk47 at students.poly.edu  Tue Apr  9 15:22:58 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Tue, 9 Apr 2013 09:22:58 -0400
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <CAG8k2+5gpaWDKhJ0Z3ndfNZ6CpiGjAM=kAqfxkYG41-VOkqwXQ@mail.gmail.com>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
	<CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
	<CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>
	<CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>
	<5163A5CC.8010807@students.poly.edu>
	<CAG8k2+5gpaWDKhJ0Z3ndfNZ6CpiGjAM=kAqfxkYG41-VOkqwXQ@mail.gmail.com>
Message-ID: <51641632.7000903@students.poly.edu>

On 4/9/13 7:47 AM, Daniel Holth wrote:
> What size keys?

2048 bits, which is the minimum key size TUF currently allows for 
security purposes. Which range of key sizes do you think PyPI would be 
comfortable with?


From davidbennett at ozemail.com.au  Tue Apr  9 15:21:15 2013
From: davidbennett at ozemail.com.au (david bennett)
Date: Tue, 9 Apr 2013 23:21:15 +1000
Subject: [Distutils] Help needed please
References: <49FBB893A2D3436CA154F155346F507F@marvin>
	<CALeMXf4Fe9O=UETTPWvgpKK32m42nPYEqQ_sGpxVHvdyUeaqWw@mail.gmail.com>
Message-ID: <D3C97BE12A7E4171AA6609AD3F06AB46@marvin>

PJ
awesome, thanks.
the path was set right, but turns out it was set in PythonPath, not Path
do i need to have %PATH% before the various paths?
cheers
dav-o

----- Original Message ----- 
From: "PJ Eby" <pje at telecommunity.com>
To: "david bennett" <davidbennett at ozemail.com.au>
Cc: <distutils-sig at python.org>
Sent: Tuesday, April 09, 2013 5:30 AM
Subject: Re: [Distutils] Help needed please


> On Sat, Apr 6, 2013 at 10:03 AM, david bennett
> <davidbennett at ozemail.com.au> wrote:
>> G'day
>> AIM: i'm trying to use Drive API with Python 2.7.3 to read and write 
>> files.
>>
>> PROBLEM: is with the install of "easy_install" (ironic, i know)
>>
>> 1. I have enabled the drive api
>> 2. I can see that the google client library must be there because the 
>> files
>> are in c:/program files/python/scripts
>>
>> BUT when i type easy_install into cmd window it is not recognised
>
> Type "path" into the command window and double-check that C:/Program
> Files/Python/Scripts is listed.
>
> (Do check, also, that C:/Program Files/Python/Scripts is actually
> where easy_install.exe is located -- the standard installation
> location for many versions of Python would be C:\Python27, not
> C:\Program Files\Python.)
>
>
>> http://stackoverflow.com/questions/4016151/how-to-use-pythons-easy-install-on-windows-its-not-so-easy?rq=1
>> mentions that "you should see that your winpexpect package has been
>> installed correctly." winexpect. where is this located? how is this done?
>
> That discussion is actually about installing winpexpect, not about
> installing easy_install.  So the problem that person was having is
> entirely unrelated to the one you are having.
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.3272 / Virus Database: 3162/6231 - Release Date: 04/07/13
> 


From ct at gocept.com  Tue Apr  9 22:53:47 2013
From: ct at gocept.com (Christian Theune)
Date: Tue, 9 Apr 2013 22:53:47 +0200
Subject: [Distutils] bandersnatch 1.0
Message-ID: <kk1v4m$2e6$1@ger.gmane.org>

Hi,

good news: I have cut a 1.0 release for "bandersnatch", the PEP381 
client that I started as an overhaul of "pep381client" at PyCon 2013.

Here's the news:

* I decided to use a "requirements.txt" to support PIP-based environments.[1]

* Markus Zappke Gr?ndemann noticed that I was missing the actual 
license text and provided me with a pull request.

* Overhauled README, hopefully a bit easier to read.

*

The next step on my agenda is to get all the existing public mirror 
operators on board so that we hopefully will see more "green bar" 
moments when visiting pypi-mirrors.org. Who can help?

Cheers,
Christian

[1] There is some integration magic to automatically generate the 
requirements.txt, have it available for different releases on 
bitbucket, etc. See "buildout.py" and "release.py" in the package code. 
Not for the faint of heart. I also added a jenkins job that verifies 
the requirements.txt, installs the sdist package and runs tests from 
there. 



From ct at gocept.com  Tue Apr  9 22:59:21 2013
From: ct at gocept.com (Christian Theune)
Date: Tue, 9 Apr 2013 22:59:21 +0200
Subject: [Distutils] bandersnatch 1.0
References: <kk1v4m$2e6$1@ger.gmane.org>
Message-ID: <kk1vf4$5kd$1@ger.gmane.org>

Gah - my news reader betrayed me while writing this message ?

On 2013-04-09 20:53:47 +0000, Christian Theune said:

> Hi,
> 
> good news: I have cut a 1.0 release for "bandersnatch", the 
> PEP381client that I started as an overhaul of "pep381client" at PyCon 
> 2013.
> 
> Here's the news:
> 
> * I decided to use a "requirements.txt" to support PIP-based environments.[1]
> 
> * Markus Zappke Gr?ndemann noticed that I was missing the actuallicense 
> text and provided me with a pull request.
> 
> * Overhauled README, hopefully a bit easier to read.
> 
> *

This item does not exist. :)

Except from that typo I meant to say that I guess that there will be 
further bugs and I'll be happy to fix them when they get reported. I 
did see a mirror process getting stuck at some point and I plan to add 
something to aid debugging those cases. If someone runs into it: just 
kill it and start it again is sufficient in all cases I encountered 
while developing it.

Cheers,
Christian



From pje at telecommunity.com  Wed Apr 10 05:14:37 2013
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 9 Apr 2013 23:14:37 -0400
Subject: [Distutils] bandersnatch update
In-Reply-To: <kk0d74$pr5$1@ger.gmane.org>
References: <kjpft0$iqu$1@ger.gmane.org> <87fvz31li4.fsf@myhost.lan>
	<kjrsot$qv3$1@ger.gmane.org>
	<CALeMXf5SmxErjQkaHBL1sikpUuM2NO57Pn7eyEGa7y+LX6k01A@mail.gmail.com>
	<kk0d74$pr5$1@ger.gmane.org>
Message-ID: <CALeMXf6bvA2c7huZj5P6QeUBuOUeb8E9SqRT63KdHCujWLv1vQ@mail.gmail.com>

On Tue, Apr 9, 2013 at 2:41 AM, Christian Theune <ct at gocept.com> wrote:
> The simplest solution on a static mirror that I can see for this would be
> (with nginx) a "missing files" rule that redirects to PyPI. However, that
> would be bad form as that would cause *any* 404-triggering request on a
> mirror to hit the upstream server - bad for bots.

Even if you limited it to just the /packages subtree?  I guess I'm not
sure what scenario results in bot-havoc for a partial mirror.  I mean,
wouldn't partial mirrors be basically private?

From tk47 at students.poly.edu  Wed Apr 10 05:52:04 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Tue, 9 Apr 2013 23:52:04 -0400
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <5163A5CC.8010807@students.poly.edu>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
	<CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
	<CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>
	<CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>
	<5163A5CC.8010807@students.poly.edu>
Message-ID: <5164E1E4.1020000@students.poly.edu>

I have finished generating the /simple metadata and they are about 52MB
--- not too far off from my estimate of 59MB. Remember: this is the
worst-case size for simple metadata.

I have now started generating the /packages metadata. If all goes well,
I should be able to test pip against a realistic TUF-secured PyPI mirror
fairly soon.

All of this is taking longer than I want it to because, well, automation
is generally tricky business! We are getting there :)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130409/fc8a30f6/attachment.pgp>

From ct at gocept.com  Wed Apr 10 06:46:37 2013
From: ct at gocept.com (Christian Theune)
Date: Wed, 10 Apr 2013 06:46:37 +0200
Subject: [Distutils] bandersnatch update
References: <kjpft0$iqu$1@ger.gmane.org> <87fvz31li4.fsf@myhost.lan>
	<kjrsot$qv3$1@ger.gmane.org>
	<CALeMXf5SmxErjQkaHBL1sikpUuM2NO57Pn7eyEGa7y+LX6k01A@mail.gmail.com>
	<kk0d74$pr5$1@ger.gmane.org>
	<CALeMXf6bvA2c7huZj5P6QeUBuOUeb8E9SqRT63KdHCujWLv1vQ@mail.gmail.com>
Message-ID: <kk2qr5$o5i$1@ger.gmane.org>

On 2013-04-10 03:14:37 +0000, PJ Eby said:

> On Tue, Apr 9, 2013 at 2:41 AM, Christian Theune <ct at gocept.com> wrote:
>> The simplest solution on a static mirror that I can see for this would be
>> (with nginx) a "missing files" rule that redirects to PyPI. However, that
>> would be bad form as that would cause *any* 404-triggering request on a
>> mirror to hit the upstream server - bad for bots.
> 
> Even if you limited it to just the /packages subtree?  I guess I'm not
> sure what scenario results in bot-havoc for a partial mirror.  I mean,
> wouldn't partial mirrors be basically private?

Right, that's what I expect in most cases, too. I see projects setting 
up project-specificy PyPI-style index pages so I can see benefit for 
that kind of application of not mirroring PyPI completely but keeping 
it public.

Christian



From lists at zopyx.com  Thu Apr 11 14:44:54 2013
From: lists at zopyx.com (Andreas Jung)
Date: Thu, 11 Apr 2013 14:44:54 +0200
Subject: [Distutils] outdated PyPI mirrors
Message-ID: <5166B046.2010109@zopyx.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to

http://www.pypi-mirrors.org/

3 out of 8 PyPI mirror are out of date or not available.

This is not good for the PyPI reputation.

Of course mirror can have problems for a while but the
maintainers should feel in charge of maintaining
and checking their mirrors regularly.

I propose that an obvious unmaintained mirror should be removed
(temporarily) if it has not been updated after N days. Right now
N = 50 and N = 69 which is hardly acceptable.

Andreas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJRZrBGAAoJEADcfz7u4AZjWVYLwNnpRxXE1QJIgzHxGGu9QpOi
QF2eJBvXHG8SzL8vYle9uD3iSVTUWIL85x1lJNZUmehVguNADHERv70OQMVLiOI7
9jwe1+tvg8HpBxpF1N87eRTdbGrMEHp5sWYioLcRrCX/1JJul3ISuQiB9Gz4BHYY
CnCQO/lSaU26VR32Q3p9CbtWcBhV5jnaZEzr+FuUUwrztHbwz5dtQHobJ9jqec4H
T5Na3tdXFGTbDLMvDF/Kfu8Oylm40r2rKIduufIDBO1juJlJHY/QplFk46onE3rU
swnlr1G978/G/vhq3dzA4rEbYg+315J1wOgu7MskLGLDIdxNSZtcPySpm8Ds1IWr
YywkNhVP+qc70sCAvwWjc/9y1wu0cOLqmDK2/aRck6CUZqpLtiUqWaPQQDtsEYVD
BeUjddiYHVYC3YV5+C62ubFTuaxEZ7nFiEbrnfr1YqcAVSbGABAm63YETUFh5ldV
oG3LRZ8eF3gRHxGq1n0P61Je3iC0iqE=
=cRqU
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 353 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130411/79f05028/attachment.vcf>

From ct at gocept.com  Thu Apr 11 14:54:07 2013
From: ct at gocept.com (Christian Theune)
Date: Thu, 11 Apr 2013 14:54:07 +0200
Subject: [Distutils] outdated PyPI mirrors
References: <5166B046.2010109@zopyx.com>
Message-ID: <kk6bp4$imn$1@ger.gmane.org>

Hi,

On 2013-04-11 12:44:54 +0000, Andreas Jung said:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> According to
> 
> http://www.pypi-mirrors.org/
> 
> 3 out of 8 PyPI mirror are out of date or not available.
> 
> This is not good for the PyPI reputation.
> 
> Of course mirror can have problems for a while but the
> maintainers should feel in charge of maintaining
> and checking their mirrors regularly.
> 
> I propose that an obvious unmaintained mirror should be removed
> (temporarily) if it has not been updated after N days. Right now
> N = 50 and N = 69 which is hardly acceptable.

I started trying to track down the owners this morning. I managed to 
send email to 3 of them based on their I network contacts.  Google 
responded that they don't want mail on that address (wonder whether 
that's ARIN conforming).

The others I'm waiting for a reply and offered to help moving to 
bandersnatch. Lets wait how that turns out, maybe give it a few days 
now that there's a (seemingly) viable alternative to the existing setup.

I'm also pondering getting a view more international VMs under my 
control and set up mirrors in more places. I'll look into that in case 
the other mirrors go away.

Christian



From donald at stufft.io  Thu Apr 11 14:56:00 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 11 Apr 2013 08:56:00 -0400
Subject: [Distutils] outdated PyPI mirrors
In-Reply-To: <kk6bp4$imn$1@ger.gmane.org>
References: <5166B046.2010109@zopyx.com> <kk6bp4$imn$1@ger.gmane.org>
Message-ID: <AB1FC5F7-1BA3-4715-9224-427E88B09641@stufft.io>


On Apr 11, 2013, at 8:54 AM, Christian Theune <ct at gocept.com> wrote:

> Hi,
> 
> On 2013-04-11 12:44:54 +0000, Andreas Jung said:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> According to
>> http://www.pypi-mirrors.org/
>> 3 out of 8 PyPI mirror are out of date or not available.
>> This is not good for the PyPI reputation.
>> Of course mirror can have problems for a while but the
>> maintainers should feel in charge of maintaining
>> and checking their mirrors regularly.
>> I propose that an obvious unmaintained mirror should be removed
>> (temporarily) if it has not been updated after N days. Right now
>> N = 50 and N = 69 which is hardly acceptable.
> 
> I started trying to track down the owners this morning. I managed to send email to 3 of them based on their I network contacts.  Google responded that they don't want mail on that address (wonder whether that's ARIN conforming).
> 
> The others I'm waiting for a reply and offered to help moving to bandersnatch. Lets wait how that turns out, maybe give it a few days now that there's a (seemingly) viable alternative to the existing setup.
> 
> I'm also pondering getting a view more international VMs under my control and set up mirrors in more places. I'll look into that in case the other mirrors go away.
> 
> Christian
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


b.pypi.python.org is run by MvL I think, and it's on GAE so it needed a special script and doesn't use pep381client.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130411/eba52983/attachment.pgp>

From erik.m.bray at gmail.com  Thu Apr 11 19:43:29 2013
From: erik.m.bray at gmail.com (Erik Bray)
Date: Thu, 11 Apr 2013 13:43:29 -0400
Subject: [Distutils] special compiler options for only one file
In-Reply-To: <3B33BD0A-34A7-486C-83E4-71660775AF09@dalkescientific.com>
References: <3B33BD0A-34A7-486C-83E4-71660775AF09@dalkescientific.com>
Message-ID: <CAOTD34YMcuVWoDq=P7ZXgRS-U1MvYMYXfPYPzNRhLp18o6beXw@mail.gmail.com>

On Tue, Mar 19, 2013 at 7:54 PM, Andrew Dalke <dalke at dalkescientific.com> wrote:
> Hi all,
>
>   I have a Python extension which uses CPU-specific features,
> if available. This is done through a run-time check. If the
> hardware supports the POPCNT instruction then it selects one
> implementation of my inner loop, if SSSE3 is available then
> it selects another, otherwise it falls back to generic versions
> of my performance critical kernel. (Some 95%+ of the time is
> spent in this kernel.)
>
> Unfortunately, there's a failure mode I didn't expect. I
> use -mssse3 and -O3 to compile all of the C code, even though
> only one file needs that -mssse3 option.
>
> As a result, the other files are compiled with the expectation
> that SSSE3 will exist. This causes a segfault for the line
>
>      start_target_popcount = (int)(query_popcount * threshold);
>
> because the compiler used fisttpl, which is an SSSE-3 instruction.
> After all, I told it to assume that ssse3 exists.
>
> The Debian packager for my package recently ran into this problem,
> because the test machine has a gcc which understands -mssse3 but
> the machine itself has an older CPU without those instructions.
>
> I'm trying to come up with a solution that can be automated for
> the Debian distribution. I want a solution where the same binary
> can work on older machines and on newer ones
>
> Ideally I would like to say that only one file is compiled
> with the -mssse3 option. Since my selector code isn't part of this
> file, SSSE-3 code will never be executed unless the CPU supports is.
>
> However, I can't figure out any way to tell distutils that
> a set of compiler options are specific to a single file.
>
> Is that even possible?


One possible solution, albeit more complex code-wise than compiling
that single file as a static lib, would be to subclass the compiler
class you want to use and override its _compile() method, which is the
one responsible for compiling a single file.  You can then override
the arguments in there.  Look, for example, at
distutils.unixcompiler.UnixCCompiler._compile.

If you want to support different compiler implementations, you could
even detect at runtime which compiler was selected with the --compiler
option and subclass the appropriate implementation.  Also getting
distutils to accept a compiler subclass requires a little bit of
hackery, but it's not undoable.  If that optioni sounds viable to you
I can delve into more details about how to actually do it, as I've had
to do this sort of thing before myself.  So yes, it can be done.

Erik

From r1chardj0n3s at gmail.com  Fri Apr 12 05:58:16 2013
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Fri, 12 Apr 2013 13:58:16 +1000
Subject: [Distutils] PyPI moved to PyPA on bitbucket
Message-ID: <CAHrZfZAk+-XWG7uccfUgsEt7s=GJuX1xqaHfEOxd5NbkH+k3_A@mail.gmail.com>

Hi all,

To help bring things under a common umbrella project, I've just moved the
PyPI code from Martin's personal repository to the PyPA one. Thanks Donald
for enabling that move, and thanks to Martin for the initial move to hg way
back :-)

I've moved the links for bug tracking to the new repository but the old
bugs in Martin's repository and those in the sf.net tracker are still
there. Not that there's many tuits(round) being spent on working on them...


     Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130412/c1b59d4b/attachment.html>

From ct at gocept.com  Fri Apr 12 09:27:47 2013
From: ct at gocept.com (Christian Theune)
Date: Fri, 12 Apr 2013 09:27:47 +0200
Subject: [Distutils] outdated PyPI mirrors
References: <5166B046.2010109@zopyx.com>
Message-ID: <kk8d1f$sud$1@ger.gmane.org>

On 2013-04-11 12:44:54 +0000, Andreas Jung said:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> According to
> 
> http://www.pypi-mirrors.org/
> 
> 3 out of 8 PyPI mirror are out of date or not available.
> 
> This is not good for the PyPI reputation.
> 
> Of course mirror can have problems for a while but the
> maintainers should feel in charge of maintaining
> and checking their mirrors regularly.
> 
> I propose that an obvious unmaintained mirror should be removed
> (temporarily) if it has not been updated after N days. Right now
> N = 50 and N = 69 which is hardly acceptable.

g seems to have come back to life, yay!

I got a response from someone at the 'd' network (David Crowe) who says 
he isn't in touch with the Python community and doesn't know about the 
installation. Not sure how to follow up on this.

b seems to be a candidate for removal.



From donald at stufft.io  Fri Apr 12 14:47:01 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 12 Apr 2013 08:47:01 -0400
Subject: [Distutils] PyPI moved to PyPA on bitbucket
In-Reply-To: <CAHrZfZAk+-XWG7uccfUgsEt7s=GJuX1xqaHfEOxd5NbkH+k3_A@mail.gmail.com>
References: <CAHrZfZAk+-XWG7uccfUgsEt7s=GJuX1xqaHfEOxd5NbkH+k3_A@mail.gmail.com>
Message-ID: <B95C70D1-3716-463D-8DCE-CAB5892533AE@stufft.io>

Awesome :D

On Apr 11, 2013, at 11:58 PM, Richard Jones <r1chardj0n3s at gmail.com> wrote:

> Hi all,
> 
> To help bring things under a common umbrella project, I've just moved the PyPI code from Martin's personal repository to the PyPA one. Thanks Donald for enabling that move, and thanks to Martin for the initial move to hg way back :-)
> 
> I've moved the links for bug tracking to the new repository but the old bugs in Martin's repository and those in the sf.net tracker are still there. Not that there's many tuits(round) being spent on working on them...
> 
> 
>      Richard
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130412/c40545a1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130412/c40545a1/attachment.pgp>

From vinay_sajip at yahoo.co.uk  Fri Apr 12 19:27:01 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Fri, 12 Apr 2013 17:27:01 +0000 (UTC)
Subject: [Distutils] Packaging and Distribution mini-summit
Message-ID: <loom.20130412T192120-798@post.gmane.org>

I see that the video for the Directions in Packaging panel is up on
pyvideo.org, but I'm curious as to what happened in the mini-summit. Is
someone going to write up what was discussed / if anything was agreed (beyond
just descriptions of the various projects)? If it's already been posted
somewhere, I'd be grateful for the link.

Thanks & regards,

Vinay Sajip


From qwcode at gmail.com  Fri Apr 12 20:03:03 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 12 Apr 2013 11:03:03 -0700
Subject: [Distutils] Packaging and Distribution mini-summit
In-Reply-To: <loom.20130412T192120-798@post.gmane.org>
References: <loom.20130412T192120-798@post.gmane.org>
Message-ID: <CAPYWazoT3s3+hjKuL7qA7++=ahvY8EJ_9wPcATsv-Hj42gwhLQ@mail.gmail.com>

mostly just sharing in the mini-summit.
the most articulate renderings of any plans atm that are in one place, are
what Nick has posted to his personal page
http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/core_packaging_api.html

that is soon to be moved to the developer hub
https://bitbucket.org/pypa/python-packaging-developer-hub

Marcus

On Fri, Apr 12, 2013 at 10:27 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote:

> I see that the video for the Directions in Packaging panel is up on
> pyvideo.org, but I'm curious as to what happened in the mini-summit. Is
> someone going to write up what was discussed / if anything was agreed
> (beyond
> just descriptions of the various projects)? If it's already been posted
> somewhere, I'd be grateful for the link.
>
> Thanks & regards,
>
> Vinay Sajip
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130412/99ff2666/attachment.html>

From tk47 at students.poly.edu  Fri Apr 12 21:32:35 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Fri, 12 Apr 2013 15:32:35 -0400
Subject: [Distutils] [tuf] Re: Automation for creating,
 updating and destroying a TUF-secured PyPI mirror
In-Reply-To: <5164E1E4.1020000@students.poly.edu>
References: <5159FEC6.6070204@students.poly.edu>
	<51630F4D.2090102@students.poly.edu>
	<CAMVss_pNVdSkD8b8pwiyeupOfS4PZGwCvesA-hug7itoJa4DrA@mail.gmail.com>
	<CADiSq7c0WTr6pCQX4PXfHVxOS8TfXZvywL4jvX1paXEzP1Q43A@mail.gmail.com>
	<CAMVss_qN67PbOjFZABRyFVJaXh5g0w-+XJitNTL+0Os8-=BQpg@mail.gmail.com>
	<5163A5CC.8010807@students.poly.edu>
	<5164E1E4.1020000@students.poly.edu>
Message-ID: <51686153.5070304@students.poly.edu>

On 04/09/2013 11:52 PM, Trishank Karthik Kuppusamy wrote:
> I have finished generating the /simple metadata and they are about 52MB
> --- not too far off from my estimate of 59MB. Remember: this is the
> worst-case size for simple metadata.

Okay, so we have finished generating the TUF metadata for a complete (if
not the latest) set of PyPI packages.

Summary of the largest metadata, assuming the worst case of a key per
package on PyPI:

release.txt: 11MB
/simple metadata: 52MB
/packages metadata: 96MB

All in all, the metadata sums to about 159MB. With the data being 45GB,
that works out to the metadata size being 0.35% of the data size.

Remember: this is the worst case for the metadata, where every PyPI
package has its own key, and there is a role for every possible target
subdirectory. The metadata is also uncompressed JSON.

As we have said before, we think we can do better (e.g, by reusing keys
for multiple packages), and we are working on it.

Simultaneously, we are testing a TUF-enabled version of pip against a
TUF-secured PyPI mirror.


From vinay_sajip at yahoo.co.uk  Mon Apr 15 10:40:28 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 15 Apr 2013 08:40:28 +0000 (UTC)
Subject: [Distutils] distlib, pylauncher -> PyPA
Message-ID: <loom.20130415T103751-388@post.gmane.org>

I'm trying to move the distlib and pylauncher repositories to the PyPA
account on BitBucket. I'm getting Internal Server Errors with both repos and
have logged an issue on BitBucket. This is just to let you know that the
transfer will hopefully happen soon, so some BitBucket links might not work
during the transition.

Regards,

Vinay Sajip


From aron at bustleandflurry.com  Sun Apr 14 18:13:50 2013
From: aron at bustleandflurry.com (Aron Bartling)
Date: Sun, 14 Apr 2013 09:13:50 -0700
Subject: [Distutils] PyPi Classifier for Bottle
In-Reply-To: <CAN4i=VwsTQWv+7_nqv4wJQWLn1upti_c8MSxmqXSEMC5VBHyPw@mail.gmail.com>
References: <CAN4i=VwsTQWv+7_nqv4wJQWLn1upti_c8MSxmqXSEMC5VBHyPw@mail.gmail.com>
Message-ID: <CAN4i=VwaDg-3_ymKKNbGjJgsJzOsvnpDYnyk8NSHsUHhop7mvg@mail.gmail.com>

(Second try to distutils-sig instead of catalog-sig)

I'd like to request that PyPi add a classifier for the web framework
Bottle.

"Framework :: Bottle"
I have a plugin ready to upload for the Bottle framework as well as a
couple more in the works.
 https://github.com/bustleandflurry/bottle-api-json-formatting

Thank you,
Aron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130414/86465854/attachment.html>

From richard at python.org  Tue Apr 16 10:01:35 2013
From: richard at python.org (Richard Jones)
Date: Tue, 16 Apr 2013 09:01:35 +0100
Subject: [Distutils] PyPi Classifier for Bottle
In-Reply-To: <CAN4i=VwaDg-3_ymKKNbGjJgsJzOsvnpDYnyk8NSHsUHhop7mvg@mail.gmail.com>
References: <CAN4i=VwsTQWv+7_nqv4wJQWLn1upti_c8MSxmqXSEMC5VBHyPw@mail.gmail.com>
	<CAN4i=VwaDg-3_ymKKNbGjJgsJzOsvnpDYnyk8NSHsUHhop7mvg@mail.gmail.com>
Message-ID: <CAHrZfZC+tpODs43YeF=MiUB76rYKAmGCyZBCsT0scbU98XmEMA@mail.gmail.com>

Added!


     Richard


On 14 April 2013 17:13, Aron Bartling <aron at bustleandflurry.com> wrote:

> (Second try to distutils-sig instead of catalog-sig)
>
> I'd like to request that PyPi add a classifier for the web framework
> Bottle.
>
> "Framework :: Bottle"
> I have a plugin ready to upload for the Bottle framework as well as a
> couple more in the works.
>  https://github.com/bustleandflurry/bottle-api-json-formatting
>
> Thank you,
> Aron
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130416/fa769637/attachment.html>

From p at google-groups-2013.dobrogost.net  Tue Apr 16 00:21:09 2013
From: p at google-groups-2013.dobrogost.net (Piotr Dobrogost)
Date: Mon, 15 Apr 2013 15:21:09 -0700 (PDT)
Subject: [Distutils] Why doesn't distribute_setup.py install the latest
 version of distribute?
Message-ID: <771eab92-857c-46ae-85e8-88a2fc1fbcb7@googlegroups.com>

It seems distribute_setup.py installs specific version of distribute 
(hardcoded inside the file). However get-pip.py installs the latest 
version. Why doesn't distribute_setup.py install the latest version of 
distribute?


Regards
Piotr Dobrogost
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130415/037bd1fb/attachment.html>

From vinay_sajip at yahoo.co.uk  Tue Apr 16 16:52:00 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 16 Apr 2013 14:52:00 +0000 (UTC)
Subject: [Distutils] distlib, pylauncher -> PyPA
References: <loom.20130415T103751-388@post.gmane.org>
Message-ID: <loom.20130416T165038-76@post.gmane.org>

Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:

> I'm trying to move the distlib and pylauncher repositories to the PyPA
> account on BitBucket. I'm getting Internal Server Errors with both repos and
> have logged an issue on BitBucket. This is just to let you know that the
> transfer will hopefully happen soon, so some BitBucket links might not work
> during the transition.

The move is now complete. I've transferred ownership of pylauncher and
distlib to pypa, and forked back to my old repos so that old links should
work as expected.

Regards,

Vinay Sajip


From solipsis at pitrou.net  Wed Apr 17 00:10:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 16 Apr 2013 22:10:25 +0000 (UTC)
Subject: [Distutils] PEP: Improving Python ZIP Application Support
References: <CAG8k2+7reG26DUAoxNFVzAVSYVSmi+o6-56rJqCtt_GH9eWSkw@mail.gmail.com>
	<5159EA46.9010806@egenix.com>
	<CADiSq7famsoXCX3WijQUtbCa3JYsNA+ym8myKT+tk2kcksgV5w@mail.gmail.com>
Message-ID: <loom.20130417T000547-13@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> Not true, it's covered in the command line docs at http://docs.python.
> org/2/using/cmdline.html#interface-options
> 
> It's just that people generally assume they know how our CLI works,
> so they never read that.

Well, people won't open a Web browser to read a CLI's documentation when
they can simply type "python --help".

$ ./python --help
usage: ./python [option] ... [-c cmd | -m mod | file | -] [arg] ...
[...]
file   : program read from script file
-      : program read from stdin (default; interactive mode if a tty)

There's nothing about a zip file here ;-)

And even the much more comprehensive man page doesn't mention it either:

$ grep -i zip Misc/python.man 
$ 

Regards

Antoine.



From ncoghlan at gmail.com  Wed Apr 17 00:13:14 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Apr 2013 08:13:14 +1000
Subject: [Distutils] distlib, pylauncher -> PyPA
In-Reply-To: <loom.20130416T165038-76@post.gmane.org>
References: <loom.20130415T103751-388@post.gmane.org>
	<loom.20130416T165038-76@post.gmane.org>
Message-ID: <CADiSq7dRy43iyJKLm-oNf0zfP0tf0umLLmzdSRZ164bSAwp4SQ@mail.gmail.com>

On 17 Apr 2013 00:52, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:
>
> Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:
>
> > I'm trying to move the distlib and pylauncher repositories to the PyPA
> > account on BitBucket. I'm getting Internal Server Errors with both
repos and
> > have logged an issue on BitBucket. This is just to let you know that the
> > transfer will hopefully happen soon, so some BitBucket links might not
work
> > during the transition.
>
> The move is now complete. I've transferred ownership of pylauncher and
> distlib to pypa, and forked back to my old repos so that old links should
> work as expected.

Thank you!

Things are settling back down for me post PyCon, so I'll hopefully be able
to get the next draft of PEP 426 together some time this week.

Cheers,
Nick.

>
> Regards,
>
> Vinay Sajip
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130417/9ab24330/attachment.html>

From richard at python.org  Wed Apr 17 17:59:57 2013
From: richard at python.org (Richard Jones)
Date: Wed, 17 Apr 2013 16:59:57 +0100
Subject: [Distutils] outdated PyPI mirrors
In-Reply-To: <kk8d1f$sud$1@ger.gmane.org>
References: <5166B046.2010109@zopyx.com>
	<kk8d1f$sud$1@ger.gmane.org>
Message-ID: <CAHrZfZBzc4uU9ygM_vA1Ja3EYO2M+n0h3rCVVSmZDfh5R2BPtg@mail.gmail.com>

I'm afraid I don't have contacts for the mirrors - it's not something I've
been involved with.


     Richard


On 12 April 2013 08:27, Christian Theune <ct at gocept.com> wrote:

> On 2013-04-11 12:44:54 +0000, Andreas Jung said:
>
>  -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> According to
>>
>> http://www.pypi-mirrors.org/
>>
>> 3 out of 8 PyPI mirror are out of date or not available.
>>
>> This is not good for the PyPI reputation.
>>
>> Of course mirror can have problems for a while but the
>> maintainers should feel in charge of maintaining
>> and checking their mirrors regularly.
>>
>> I propose that an obvious unmaintained mirror should be removed
>> (temporarily) if it has not been updated after N days. Right now
>> N = 50 and N = 69 which is hardly acceptable.
>>
>
> g seems to have come back to life, yay!
>
> I got a response from someone at the 'd' network (David Crowe) who says he
> isn't in touch with the Python community and doesn't know about the
> installation. Not sure how to follow up on this.
>
> b seems to be a candidate for removal.
>
>
>
> ______________________________**_________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/**mailman/listinfo/distutils-sig<http://mail.python.org/mailman/listinfo/distutils-sig>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130417/adda48b5/attachment.html>

From pnasrat at gmail.com  Wed Apr 17 18:52:43 2013
From: pnasrat at gmail.com (Paul Nasrat)
Date: Wed, 17 Apr 2013 12:52:43 -0400
Subject: [Distutils] outdated PyPI mirrors
In-Reply-To: <CAHrZfZBzc4uU9ygM_vA1Ja3EYO2M+n0h3rCVVSmZDfh5R2BPtg@mail.gmail.com>
References: <5166B046.2010109@zopyx.com> <kk8d1f$sud$1@ger.gmane.org>
	<CAHrZfZBzc4uU9ygM_vA1Ja3EYO2M+n0h3rCVVSmZDfh5R2BPtg@mail.gmail.com>
Message-ID: <CAGK+S6TatHKBamj7JXXY118JTwKxRODHRFLvKgV3rPWRhSsH+w@mail.gmail.com>

On 17 April 2013 11:59, Richard Jones <richard at python.org> wrote:

> I'm afraid I don't have contacts for the mirrors - it's not something I've
> been involved with.
>

Maybe we need something a bit more formal for the pypi mirrors with
captured contact details, a mailing list etc.

The process for adding https://pypi.python.org/mirrors

For comparison of some other projects mirroring instructions and how to be
added to a mirror list:

http://fedoraproject.org/wiki/Infrastructure/Mirroring
http://www.debian.org/mirror/ftpmirror
http://www.cpan.org/misc/how-to-mirror.html

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130417/099bd72c/attachment.html>

From aclark at aclark.net  Fri Apr 19 13:36:25 2013
From: aclark at aclark.net (Alex Clark)
Date: Fri, 19 Apr 2013 07:36:25 -0400
Subject: [Distutils] Free GSOC idea: Site Setup -> Users & Groups -> [ ]
	Disable logins
Message-ID: <kkra7l$7nc$1@ger.gmane.org>

On IRC yesterday, Spanktar asked a question that comes up every so 
often (paraphrasing):


  "How do I disable logins while I do some maintenance on my Plone site?"


Various suggests were offered, including davisagli's (paraphrasing):


  "I usually disable all the PAS auth plugins in acl_users"


DING DING DING: LIGHT BULB MOMENT. I've seen this question asked enough 
times to think it might be nice if there were a checkbox in Site Setup 
-> Users & Groups to make it easy for end users to perform this action, 
if only someone were willing to do the work?


-- 
Alex Clark ? http://about.me/alex.clark



From techtonik at gmail.com  Fri Apr 19 18:01:51 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 19 Apr 2013 19:01:51 +0300
Subject: [Distutils] [Python Wiki] Update of "CheeseShopDev" by
	RichardJones
In-Reply-To: <20130412034843.6129.7768@virt-ys0nco.psf.osuosl.org>
References: <20130412034843.6129.7768@virt-ys0nco.psf.osuosl.org>
Message-ID: <CAPkN8xLJEgdcY8U2vjxp-nr9CnC4KR5APgCx+g60=0CfnsaOvA@mail.gmail.com>

At last its done. Very nice to see it. Kudos to all people involved in
migrating bug tracker, code repository, lobbying the decision, merging the
points of contacts and making actual change. I expect more people to pop up
to help with PyPI development now. Too bad we don't have statistics to
prove unconvinced that these actions will actually work this way.

-- 
anatoly t.


On Fri, Apr 12, 2013 at 6:48 AM, Python Wiki <wiki at python.org> wrote:

> Dear Wiki user,
>
> You have subscribed to a wiki page or wiki category on "Python Wiki" for
> change notification.
>
> The "CheeseShopDev" page has been changed by RichardJones:
> http://wiki.python.org/moin/CheeseShopDev?action=diff&rev1=71&rev2=72
>
>
>
>    .
> -  The PyPI code hosting is currently undergoing migration. For the time
> being, it is hosted here: https://bitbucket.org/loewis/pypi
> +  The PyPI code is hosted under the Python Packaging Authority project:
> https://bitbucket.org/pypa/pypi
>
>    .
> -  Bug and patch tracker
> http://sourceforge.net/tracker/?group_id=66150&atid=513503
> +  Bug and patch tracker https://bitbucket.org/pypa/pypi/issues
>
>    .
> -  [[http://mail.python.org/mailman/listinfo/catalog-sig|Mailing List]]
> ([[http://dir.gmane.org/gmane.comp.python.catalog|Gmane]] web interface)
> +  [[http://mail.python.org/mailman/listinfo/distutils-sig|Mailing List]]
> ([[http://dir.gmane.org/gmane.comp.python.distutils|Gmane]] web interface)
>
>    .
>    API that is used by easy_install
> http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130419/13284f21/attachment.html>

From aclark at aclark.net  Sat Apr 20 13:58:56 2013
From: aclark at aclark.net (Alex Clark)
Date: Sat, 20 Apr 2013 07:58:56 -0400
Subject: [Distutils] Free GSOC idea: Site Setup -> Users & Groups -> [ ]
	Disable logins
References: <kkra7l$7nc$1@ger.gmane.org>
Message-ID: <kktvtr$9o3$1@ger.gmane.org>

On 2013-04-19 11:36:25 +0000, Alex Clark said:

> On IRC yesterday, Spanktar asked a question that comes up every so
> often (paraphrasing):
> 
> 
>   "How do I disable logins while I do some maintenance on my Plone site?"
> 
> 
> Various suggests were offered, including davisagli's (paraphrasing):
> 
> 
>   "I usually disable all the PAS auth plugins in acl_users"
> 
> 
> DING DING DING: LIGHT BULB MOMENT. I've seen this question asked enough
> times to think it might be nice if there were a checkbox in Site Setup
> -> Users & Groups to make it easy for end users to perform this action,
> if only someone were willing to do the work?


And? wrong list! Sorry for the noise.


-- 
Alex Clark ? http://about.me/alex.clark



From faltet at gmail.com  Sun Apr 21 09:57:11 2013
From: faltet at gmail.com (Francesc Alted)
Date: Sun, 21 Apr 2013 09:57:11 +0200
Subject: [Distutils] Problems with PyPI authentication
Message-ID: <51739BD7.50708@gmail.com>

Hi,

Since the password reset on PyPI site (https://pypi.python.org/pypi), I 
cannot longer log in there. My username used to 'falted' there, and 
although I requested a reset password for this user 
(https://pypi.python.org/pypi?%3Aaction=forgotten_password_form), and 
the form says that a mail has been sent, the thing is that I don't 
receive any e-mail. I am not sure why I'm not receiving this message!

I need to continue providing releases for the numexpr, blosc and other 
packages, but I can't. What can I do for gaining access to my 'falted' 
account again?

Thanks,

-- 
Francesc Alted


From dw at botanicus.net  Mon Apr 22 01:11:55 2013
From: dw at botanicus.net (David Wilson)
Date: Mon, 22 Apr 2013 02:11:55 +0300
Subject: [Distutils] Fixing Cheese Shop search
Message-ID: <CABzuZRMcyk3wT3sDs4oiuHMNWCXhjmRyoi8SmbdaEzKSLMk5eQ@mail.gmail.com>

Hi there,

In a fit of madness caused by another 30 seconds-long PyPI search I
decided to investigate the code, in the hopes of perhaps finding
something simple that would alleviate the extremely long search time.

I discovered what appears to be a function that makes 6 SQL queries
for each term provided by the search user, in turn those queries
expand to what appear to be %SUBSTRING% table scans across the
releases table, which appears to contain upwards of half a gigabyte of
strings.

Now the root cause has been located, what to do about it? I looked at
hacking on the code, but it seems webui.py is already too massive for
its own good, and in any case PostgreSQL's options for efficient
search are quite limited. It might be all-round good if the size of
that module started to drop..

I wrote a crawler to pull a reasonable facsimile of the releases table
on to my machine via the XML-RPC API, then arranged for Xapian to
index only the newest releases for each package. The resulting
full-text index weighs in at a very reasonable 334mb, and searches
complete almost immediately, even on my lowly Intel Atom colocated
server.

I wrote a quick hack Flask app around it, which you can see here:

      http://5.39.91.176:5000/

The indexer takes as input a database produced by the crawler, which
is smart enough to know how to use PyPI's exposed 'changelog' serial
numbers. Basically it is quite trivial and efficient to run this setup
in an incremental indexing mode.

As you can see from the results, even my lowly colo is trouncing what
is currently on PyPI, and so my thoughts tend toward making an
arrangement like this more permanent.

The crawler code weighs in at 150 lines, the indexer a meagre 113
lines, and the Flask example app is 74 lines. Implementing an exact
replica of PyPI's existing scoring function is already partially done
at indexing time, and the rest is quite easy to complete (mostly
cutpasting code).

Updating the Flask example to provide an XML-RPC API (or similar),
then *initially augmenting* the old search facility seems like a good
start, with a view to removing the old feature entirely. Integrating
indexing directly would be pointless, the PyPI code really doesn't
need anything more added to it until it gets at least reorganized a
little.

So for the cost of 334mb of disk, a cron job, and a lowly VPS with
even just 700MB RAM, PyPI's search pains might be solved permanently.
Naturally I'm writing this mail because it bothers me enough to
volunteer help. :)

Prototype code is here: https://bitbucket.org/dmw/pypi-search (note:
relies on a pre-alpha quality DB library I'm hacking on)

Thoughts?

From dw at botanicus.net  Mon Apr 22 15:30:10 2013
From: dw at botanicus.net (David Wilson)
Date: Mon, 22 Apr 2013 16:30:10 +0300
Subject: [Distutils] Fixing Cheese Shop search
In-Reply-To: <CABzuZRMcyk3wT3sDs4oiuHMNWCXhjmRyoi8SmbdaEzKSLMk5eQ@mail.gmail.com>
References: <CABzuZRMcyk3wT3sDs4oiuHMNWCXhjmRyoi8SmbdaEzKSLMk5eQ@mail.gmail.com>
Message-ID: <CABzuZRMBWUKRwGpC+uYG7Vz=YJkTG21D4Ua4PjYG-BbL1wFr_g@mail.gmail.com>

http://pypi.h1.botanicus.net/ is the same demo running behind Apache
with mod_gzip on a Core i7 920.

On 22 April 2013 02:11, David Wilson <dw at botanicus.net> wrote:
> Hi there,
>
> In a fit of madness caused by another 30 seconds-long PyPI search I
> decided to investigate the code, in the hopes of perhaps finding
> something simple that would alleviate the extremely long search time.
>
> I discovered what appears to be a function that makes 6 SQL queries
> for each term provided by the search user, in turn those queries
> expand to what appear to be %SUBSTRING% table scans across the
> releases table, which appears to contain upwards of half a gigabyte of
> strings.
>
> Now the root cause has been located, what to do about it? I looked at
> hacking on the code, but it seems webui.py is already too massive for
> its own good, and in any case PostgreSQL's options for efficient
> search are quite limited. It might be all-round good if the size of
> that module started to drop..
>
> I wrote a crawler to pull a reasonable facsimile of the releases table
> on to my machine via the XML-RPC API, then arranged for Xapian to
> index only the newest releases for each package. The resulting
> full-text index weighs in at a very reasonable 334mb, and searches
> complete almost immediately, even on my lowly Intel Atom colocated
> server.
>
> I wrote a quick hack Flask app around it, which you can see here:
>
>       http://5.39.91.176:5000/
>
> The indexer takes as input a database produced by the crawler, which
> is smart enough to know how to use PyPI's exposed 'changelog' serial
> numbers. Basically it is quite trivial and efficient to run this setup
> in an incremental indexing mode.
>
> As you can see from the results, even my lowly colo is trouncing what
> is currently on PyPI, and so my thoughts tend toward making an
> arrangement like this more permanent.
>
> The crawler code weighs in at 150 lines, the indexer a meagre 113
> lines, and the Flask example app is 74 lines. Implementing an exact
> replica of PyPI's existing scoring function is already partially done
> at indexing time, and the rest is quite easy to complete (mostly
> cutpasting code).
>
> Updating the Flask example to provide an XML-RPC API (or similar),
> then *initially augmenting* the old search facility seems like a good
> start, with a view to removing the old feature entirely. Integrating
> indexing directly would be pointless, the PyPI code really doesn't
> need anything more added to it until it gets at least reorganized a
> little.
>
> So for the cost of 334mb of disk, a cron job, and a lowly VPS with
> even just 700MB RAM, PyPI's search pains might be solved permanently.
> Naturally I'm writing this mail because it bothers me enough to
> volunteer help. :)
>
> Prototype code is here: https://bitbucket.org/dmw/pypi-search (note:
> relies on a pre-alpha quality DB library I'm hacking on)
>
> Thoughts?

From vinay_sajip at yahoo.co.uk  Tue Apr 23 01:38:59 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 22 Apr 2013 23:38:59 +0000 (UTC)
Subject: [Distutils] =?utf-8?q?Replacing_pkg=5Fresources_with_distlib_in_p?=
	=?utf-8?q?ip?=
Message-ID: <loom.20130423T012513-833@post.gmane.org>

I've made progress in my experiment to replace pkg_resources usage in pip
with code in distlib. The basic approach is to replace references to
pkg_resources with references to pip.vendor.distlib.pkg_resources, where the
latter module is a shim to emulate the pkg_resources API using distlib to
actually implement the functionality.

Although I don't have the very latest upstream pip updates merged in yet, I
have test results which mirror the develop branch (one failure, related to my
test environment - test_correct_pip_version). I believe I have now replaced
all uses of pkg_resources (when I mentioned this effort previously on this
list, pip/req.py had not yet been converted).

Note that the pip.vendor.distlib version is slightly different to the
released distlib - the main difference being that the metadata version in the
released distlib is treated as 2.0, whereas for pip I had to change it to
1.3.

The code is at [1] - the use-distlib branch - and a comparison view is at
[2].

I would appreciate comments, especially from the pip maintainers.

Regards,

Vinay Sajip

[1] https://github.com/vsajip/pip
[2] https://github.com/vsajip/pip/compare/use-distlib



From dalke at dalkescientific.com  Tue Apr 23 02:21:30 2013
From: dalke at dalkescientific.com (Andrew Dalke)
Date: Tue, 23 Apr 2013 02:21:30 +0200
Subject: [Distutils] special compiler options for only one file
In-Reply-To: <CAOTD34YMcuVWoDq=P7ZXgRS-U1MvYMYXfPYPzNRhLp18o6beXw@mail.gmail.com>
References: <3B33BD0A-34A7-486C-83E4-71660775AF09@dalkescientific.com>
	<CAOTD34YMcuVWoDq=P7ZXgRS-U1MvYMYXfPYPzNRhLp18o6beXw@mail.gmail.com>
Message-ID: <A463AA9A-71ED-4639-99C2-DC79B86A6B77@dalkescientific.com>

Hi Erik,

On Apr 11, 2013, at 7:43 PM, Erik Bray wrote:
> One possible solution, albeit more complex code-wise than compiling
> that single file as a static lib, would be to subclass the compiler
> class you want to use and override its _compile() method, which is the
> one responsible for compiling a single file.  You can then override
> the arguments in there.  Look, for example, at
> distutils.unixcompiler.UnixCCompiler._compile.

I shudder every time I think about subclassing one of the compiler
classes. I've done it before, based on suggestions from this list,
but I had no sense of how it actually worked.

Instead, I wrote a wrapper script, at the end of this email, which
can be used like this:

   env CC=$PWD/filter_gcc python setup.py build

It sniffs the command-line and keeps "-mssse3" for the right file,
otherwise it removes that option. It then calls gcc with the tweaked
arguments.

Unlike subclassing, I understand how this one works. :)

Cheers,


				Andrew
				dalke at dalkescientific.com


#!/usr/bin/env python

# chemfp by default assumes the --with-ssse3 flag, which enables
# compiler-specific option to enable the SSSE3 intrinsics.
# 
# You can disable this using --without-ssse3.
#
# The code which requires the SSSE3 specific intrinsics will only be
# run when chemfp, at run-time, determines that the CPU supports the
# SSSE3 instruction set. This is as it should be. Unfortunately, the
# -mssse3 compiler flag also tells gcc that it's okay to use
# SSSE3-specific instructions in the rest of the code. This causes a
# bus error if chemfp is then used on a CPU which doesn't support the
# correct instruction set. Unfortunately, I don't have the ability to
# use a non-ssse3 code path for this case.
#
# This is only a problem if:
#   - you use the same binary on multiple platforms, where
#   - some machines do not have the SSSE3 instruction set AND
#   - some machines have the SSSE3 instruction set AND
#   - the machines with the SSSE3 instruction set do not support POPCNT.
#
# (If all of your SSSE3 machines also support POPCNT then it's okay to
# use --without-ssse3, because the POPCNT instruction is always faster
# than the SSSE3-based popcount implementation.)
#
# This script, filter_gcc, is a workaround for the problem. Only one
# file needs the -mssse3 option. The best solution is to tell Python's
# setup.py to compile src/popcount_SSSE3.c with -mssse3 enabled and to
# leave out that flag for the other files. Unfortunately, setup.py
# doesn't make that easy.
#
# filter_gcc acts as a filter layer between setup.py and gcc. It's
# used like this:
#
#   env CC=$PWD/filter_gcc python setup.py build
#
# This tells setup.py to use $PWD/filter_gcc (ie, this script) as an
# alternate C compiler. This script run and checks if setup.py is
# attempting to compile popcount_SSSE3.c. If so, it leaves the -mssse3
# flag in place (if it exists). Otherwise, it removes the flag (if it
# exists).

import sys
import subprocess

CC = "gcc"  # The real C compiler

args = sys.argv[1:]
#print "Called with", args

# Check to see if I should remove the "-mssse3" flag from the args.
remove_mssse3 = True
for arg in args:
    if "popcount_SSSE3.c" in arg:
        remove_mssse3 = False
        break

if remove_mssse3:
    # Go ahead and remove the "-mssse3"
    try:
        args.remove("-mssse3")
    except ValueError:
        # This can happen if someone does:
        #    env CC=$PWD/filter_gcc python setup.py build --without-ssse3
        pass

assert args, "Missing exec args"

# Use the correct C compiler
args = [CC] + args
#print "  -->", " ".join(args)

# Run the new command, and report any errors.
try:
    retcode = subprocess.call(args)
except OSError, err:
    cmd = " ".join(args)
    raise SystemExit("Failed to execute %r: %s" % (cmd, err))

if retcode:
    raise SystemExit(retched)



From qwcode at gmail.com  Tue Apr 23 04:20:47 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Apr 2013 19:20:47 -0700
Subject: [Distutils] pypa-dev mailing list
Message-ID: <CAPYWazqZVg4mALTAba=MR3jKRoZ0ALZD6-zSjcxPMPPmQOD3ig@mail.gmail.com>

fyi, there is dev list now for the "pypa" family of projects (most notably
pip and virtualenv)
"
"pypa-dev"  https://groups.google.com/forum/?fromgroups#!forum/pypa-dev

This is *not* meant to replace distutils-sig.

It's for more "day-to-day" discussions related to maintaining and releasing
the "pypa" projects.

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130422/7bfda875/attachment.html>

From qwcode at gmail.com  Tue Apr 23 04:09:25 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 22 Apr 2013 19:09:25 -0700
Subject: [Distutils] Replacing pkg_resources with distlib in pip
In-Reply-To: <loom.20130423T012513-833@post.gmane.org>
References: <loom.20130423T012513-833@post.gmane.org>
Message-ID: <CAPYWazrkH94FFW2FA6BBuZZxgCo7_eq2yk5jO-UC=fmV0Uid-A@mail.gmail.com>

Hello Vinay:
I would say get it up to date, and then submit a pull.
We can discuss details in the pull and see the travis results.
It doesn't have to be final or an immediate merge candidate to submit a
pull for discussion.
As for considering it for an upcoming pip milestone that's something we/you
can bring up on the pypa-dev list.
Marcus


On Mon, Apr 22, 2013 at 4:38 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote:

> I've made progress in my experiment to replace pkg_resources usage in pip
> with code in distlib. The basic approach is to replace references to
> pkg_resources with references to pip.vendor.distlib.pkg_resources, where
> the
> latter module is a shim to emulate the pkg_resources API using distlib to
> actually implement the functionality.
>
> Although I don't have the very latest upstream pip updates merged in yet, I
> have test results which mirror the develop branch (one failure, related to
> my
> test environment - test_correct_pip_version). I believe I have now replaced
> all uses of pkg_resources (when I mentioned this effort previously on this
> list, pip/req.py had not yet been converted).
>
> Note that the pip.vendor.distlib version is slightly different to the
> released distlib - the main difference being that the metadata version in
> the
> released distlib is treated as 2.0, whereas for pip I had to change it to
> 1.3.
>
> The code is at [1] - the use-distlib branch - and a comparison view is at
> [2].
>
> I would appreciate comments, especially from the pip maintainers.
>
> Regards,
>
> Vinay Sajip
>
> [1] https://github.com/vsajip/pip
> [2] https://github.com/vsajip/pip/compare/use-distlib
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130422/91519cae/attachment-0001.html>

From ncoghlan at gmail.com  Tue Apr 23 14:41:26 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Apr 2013 22:41:26 +1000
Subject: [Distutils] Fixing Cheese Shop search
In-Reply-To: <CABzuZRMBWUKRwGpC+uYG7Vz=YJkTG21D4Ua4PjYG-BbL1wFr_g@mail.gmail.com>
References: <CABzuZRMcyk3wT3sDs4oiuHMNWCXhjmRyoi8SmbdaEzKSLMk5eQ@mail.gmail.com>
	<CABzuZRMBWUKRwGpC+uYG7Vz=YJkTG21D4Ua4PjYG-BbL1wFr_g@mail.gmail.com>
Message-ID: <CADiSq7db4SXtJBn=hhuafec7=RAN0peQtatD9jVs83WT--VFqQ@mail.gmail.com>

On Mon, Apr 22, 2013 at 11:30 PM, David Wilson <dw at botanicus.net> wrote:
>> Prototype code is here: https://bitbucket.org/dmw/pypi-search (note:
>> relies on a pre-alpha quality DB library I'm hacking on)
>>
>> Thoughts?

Hi David,

This certainly sounds intriguing (and even promising), but I believe
Richard (Jones, the main PyPI maintainer) has just returned from a
long business trip, so it may be a while before he catches back up
with distutils-sig.

I'll see if I can nudge some other folks and get you a better answer...

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Apr 23 15:19:28 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Apr 2013 23:19:28 +1000
Subject: [Distutils] Replacing pkg_resources with distlib in pip
In-Reply-To: <loom.20130423T012513-833@post.gmane.org>
References: <loom.20130423T012513-833@post.gmane.org>
Message-ID: <CADiSq7dTKmnsS_Wb8-45OWKAY_Mek48JLUwLBu-WGQvMrUd=8w@mail.gmail.com>

On Tue, Apr 23, 2013 at 9:38 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> I've made progress in my experiment to replace pkg_resources usage in pip
> with code in distlib. The basic approach is to replace references to
> pkg_resources with references to pip.vendor.distlib.pkg_resources, where the
> latter module is a shim to emulate the pkg_resources API using distlib to
> actually implement the functionality.
>
> Although I don't have the very latest upstream pip updates merged in yet, I
> have test results which mirror the develop branch (one failure, related to my
> test environment - test_correct_pip_version). I believe I have now replaced
> all uses of pkg_resources (when I mentioned this effort previously on this
> list, pip/req.py had not yet been converted).

Very promising!

> Note that the pip.vendor.distlib version is slightly different to the
> released distlib - the main difference being that the metadata version in the
> released distlib is treated as 2.0, whereas for pip I had to change it to
> 1.3.

Given the flux in PEP 426, it's probably best to stick with metadata
1.2 for now. There's no such thing as 1.3, and 2.0 is now going to be
a rather different beast (with the on-disk serialisation format
changing to JSON).

Regards,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Tue Apr 23 18:03:25 2013
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Apr 2013 12:03:25 -0400
Subject: [Distutils] Replacing pkg_resources with distlib in pip
In-Reply-To: <loom.20130423T012513-833@post.gmane.org>
References: <loom.20130423T012513-833@post.gmane.org>
Message-ID: <9119FC49-3D49-4BCB-A661-38537DF18F3F@stufft.io>

On Apr 22, 2013, at 7:38 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> Note that the pip.vendor.distlib version is slightly different to the
> released distlib - the main difference being that the metadata version in the
> released distlib is treated as 2.0, whereas for pip I had to change it to
> 1.3.

This part worries me. I don't think should maintain any patches for things inside of pip.vendor.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130423/88df9cc1/attachment.pgp>

From p.f.moore at gmail.com  Tue Apr 23 18:48:07 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 Apr 2013 17:48:07 +0100
Subject: [Distutils] Replacing pkg_resources with distlib in pip
In-Reply-To: <9119FC49-3D49-4BCB-A661-38537DF18F3F@stufft.io>
References: <loom.20130423T012513-833@post.gmane.org>
	<9119FC49-3D49-4BCB-A661-38537DF18F3F@stufft.io>
Message-ID: <CACac1F8ir5Yq76HEN6W2KkU2AOvrrXz++_QnVS+d0goGZCYgmA@mail.gmail.com>

On 23 April 2013 17:03, Donald Stufft <donald at stufft.io> wrote:

> On Apr 22, 2013, at 7:38 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>
> > Note that the pip.vendor.distlib version is slightly different to the
> > released distlib - the main difference being that the metadata version
> in the
> > released distlib is treated as 2.0, whereas for pip I had to change it to
> > 1.3.
>
> This part worries me. I don't think should maintain any patches for things
> inside of pip.vendor.
>

Agreed.
Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130423/01ec2010/attachment.html>

From chris at simplistix.co.uk  Tue Apr 23 22:30:49 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 23 Apr 2013 21:30:49 +0100
Subject: [Distutils] pypa-dev mailing list
In-Reply-To: <CAPYWazqZVg4mALTAba=MR3jKRoZ0ALZD6-zSjcxPMPPmQOD3ig@mail.gmail.com>
References: <CAPYWazqZVg4mALTAba=MR3jKRoZ0ALZD6-zSjcxPMPPmQOD3ig@mail.gmail.com>
Message-ID: <5176EF79.7000206@simplistix.co.uk>



On 23/04/2013 03:20, Marcus Smith wrote:
> fyi, there is dev list now for the "pypa" family of projects (most
> notably pip and virtualenv)
> "
> "pypa-dev" https://groups.google.com/forum/?fromgroups#!forum/pypa-dev
>
> This is *not* meant to replace distutils-sig.
>
> It's for more "day-to-day" discussions related to maintaining and
> releasing the "pypa" projects.

What's happening with python-virtualenv at googlegroups.com?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From qwcode at gmail.com  Tue Apr 23 22:34:05 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Tue, 23 Apr 2013 13:34:05 -0700
Subject: [Distutils] pypa-dev mailing list
In-Reply-To: <5176EF79.7000206@simplistix.co.uk>
References: <CAPYWazqZVg4mALTAba=MR3jKRoZ0ALZD6-zSjcxPMPPmQOD3ig@mail.gmail.com>
	<5176EF79.7000206@simplistix.co.uk>
Message-ID: <CAPYWazoXCLFz+10Dt1NgVSPSV-M6w7S=tu4OA5emZ-FHmuzbHQ@mail.gmail.com>

It's still the user list for now.


> What's happening with python-virtualenv@**googlegroups.com<python-virtualenv at googlegroups.com>
> ?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130423/5a27966c/attachment.html>

From vinay_sajip at yahoo.co.uk  Tue Apr 23 22:39:18 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 23 Apr 2013 20:39:18 +0000 (UTC)
Subject: [Distutils]
	=?utf-8?q?Replacing_pkg=5Fresources_with_distlib_in_p?=
	=?utf-8?q?ip?=
References: <loom.20130423T012513-833@post.gmane.org>
	<9119FC49-3D49-4BCB-A661-38537DF18F3F@stufft.io>
Message-ID: <loom.20130423T223239-822@post.gmane.org>

Donald Stufft <donald <at> stufft.io> writes:


> > released distlib is treated as 2.0, whereas for pip I had to change it to
> > 1.3.
> 
> This part worries me. I don't think should maintain any patches for things
inside of pip.vendor.

Well, the pip test data includes some distributions which include the
post-1.2 metadata fields and have a metadata version of "1.3". The upstream
distlib version is 2.0 (though the format is still key-value, pending
changes to PEP 426). My aim was to limit changes to pip to just
pkg_resources -> pip.vendor.distlib.pkg_resources. Without my renaming "2.0"
to "1.3", some of the pip tests fail because Metadata version 1.3 is
unrecognised.

Note that pip.vendor.distlib.pkg_resources is a custom shim module to cover
pip's use of pkg_resources - it's not a generic replacement for
pkg_resources (because I am only currently interested in supplanting use of
pkg_resources by pip, and not in a wider sphere). Thus it is not part of
upstream distlib (of course, it could be moved to somewhere else in the pip
package hierarchy).

Regards,

Vinay Sajip


From dholth at gmail.com  Tue Apr 23 22:51:10 2013
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 23 Apr 2013 16:51:10 -0400
Subject: [Distutils] Replacing pkg_resources with distlib in pip
In-Reply-To: <loom.20130423T223239-822@post.gmane.org>
References: <loom.20130423T012513-833@post.gmane.org>
	<9119FC49-3D49-4BCB-A661-38537DF18F3F@stufft.io>
	<loom.20130423T223239-822@post.gmane.org>
Message-ID: <CAG8k2+4JYUMe8KnscADoT5VgY2Yb2a2BVBzNANi1Xqk83+u1SA@mail.gmail.com>

On Tue, Apr 23, 2013 at 4:39 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Donald Stufft <donald <at> stufft.io> writes:
>
>
>> > released distlib is treated as 2.0, whereas for pip I had to change it to
>> > 1.3.
>>
>> This part worries me. I don't think should maintain any patches for things
> inside of pip.vendor.
>
> Well, the pip test data includes some distributions which include the
> post-1.2 metadata fields and have a metadata version of "1.3". The upstream
> distlib version is 2.0 (though the format is still key-value, pending
> changes to PEP 426). My aim was to limit changes to pip to just
> pkg_resources -> pip.vendor.distlib.pkg_resources. Without my renaming "2.0"
> to "1.3", some of the pip tests fail because Metadata version 1.3 is
> unrecognised.
>
> Note that pip.vendor.distlib.pkg_resources is a custom shim module to cover
> pip's use of pkg_resources - it's not a generic replacement for
> pkg_resources (because I am only currently interested in supplanting use of
> pkg_resources by pip, and not in a wider sphere). Thus it is not part of
> upstream distlib (of course, it could be moved to somewhere else in the pip
> package hierarchy).
>
> Regards,
>
> Vinay Sajip

Metadata 2.0 was Metadata 1.3 for a time of course.

The document specifies "warn only when (minor) version is not
recognized" though technically that is not a thing for the 1.x series.

The pkg_resources included in Ubuntu 13.04 understands the killed
key/value Metadata-2.0-with-requirements.

From techtonik at gmail.com  Wed Apr 24 11:19:07 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 24 Apr 2013 12:19:07 +0300
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
Message-ID: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>

Having a lot of meaningless options doesn't make meta data any more clear.
Well, they are not meaningless, but their role is fully fulfilled by other
options (Author and Maintainer fields in this particular case).

The use case for the Author field is that if Author wants to be contacted,
(s)he leaves email. That's it. This use case should be described in the
meta-data along with the format that are expected to be recognized by the
software:

Author: anatoly techtonik
Author: anatoly techtonik <techtonik at gmail.com>
Author: anatoly techtonik <techtonik at gmail.com>, Anything Else for Humans,
Or For Future Specs

Here the field content defines its type - it's like duck typing for
specification, which make specifications more pythonic.

Is it good?
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/c6b4a797/attachment.html>

From techtonik at gmail.com  Wed Apr 24 11:30:25 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 24 Apr 2013 12:30:25 +0300
Subject: [Distutils] Enhance 426: Insufficient crediting options
Message-ID: <CAPkN8x+VX_ATzjCKfjD_O8_v9Vk_mntJPxMFzGi1433Nos46Eg@mail.gmail.com>

The PEP 426 meta-data format allows only single Author and single
Maintainer per package, which is very inadequate to the way most popular
packages are developed.

The format should contains recommendation how to handle credits and process
them.

With the proper implementation the packaging format can positively affect
social collaboration and bring more fun into development. I'd be interested
to know the opinion if somebody considers bringing this topic outside of
the distutils sig.
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/ad7e6700/attachment.html>

From vinay_sajip at yahoo.co.uk  Wed Apr 24 11:35:35 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 24 Apr 2013 09:35:35 +0000 (UTC)
Subject: [Distutils]
	=?utf-8?q?Replacing_pkg=5Fresources_with_distlib_in_p?=
	=?utf-8?q?ip?=
References: <loom.20130423T012513-833@post.gmane.org>
	<CAPYWazrkH94FFW2FA6BBuZZxgCo7_eq2yk5jO-UC=fmV0Uid-A@mail.gmail.com>
Message-ID: <loom.20130424T112440-39@post.gmane.org>

Marcus Smith <qwcode <at> gmail.com> writes:

> 
> 
> Hello Vinay:
> I would say get it up to date, and then submit a pull.
> We can discuss details in the pull and see the travis results.?
> It doesn't have to be final or an immediate merge candidate to submit a
pull for discussion.

Done. There are four failures but these appear to be unrelated to my changes;
rather, they point to some problem in my test environment, because the
test virtualenv has both pip-1.3.1 and pip-1.4.dev1 in its site-packages. I
couldn't see why this should be.

Regards,

Vinay Sajip


From g2p.code at gmail.com  Wed Apr 24 15:51:29 2013
From: g2p.code at gmail.com (Gabriel de Perthuis)
Date: Wed, 24 Apr 2013 13:51:29 +0000 (UTC)
Subject: [Distutils] pypa-dev mailing list
References: <CAPYWazqZVg4mALTAba=MR3jKRoZ0ALZD6-zSjcxPMPPmQOD3ig@mail.gmail.com>
Message-ID: <kl8o11$983$1@ger.gmane.org>

> fyi, there is dev list now for the "pypa" family of projects (most
> notably pip and virtualenv)
> "
> "pypa-dev"  https://groups.google.com/forum/?fromgroups#!forum/pypa-dev
> 
> This is *not* meant to replace distutils-sig.
> 
> It's for more "day-to-day" discussions related to maintaining and
> releasing the "pypa" projects.

What names should the two lists have on GMane [1]?

gmane.comp.python.pypa.user and gmane.comp.python.pypa.devel ?

[1] http://gmane.org/subscribe.php


From regebro at gmail.com  Wed Apr 24 16:06:55 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 24 Apr 2013 16:06:55 +0200
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
Message-ID: <CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>

On Wed, Apr 24, 2013 at 11:19 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> Having a lot of meaningless options doesn't make meta data any more clear.
> Well, they are not meaningless, but their role is fully fulfilled by other
> options (Author and Maintainer fields in this particular case).
>
> The use case for the Author field is that if Author wants to be contacted,
> (s)he leaves email. That's it. This use case should be described in the
> meta-data along with the format that are expected to be recognized by the
> software:
>
> Author: anatoly techtonik
> Author: anatoly techtonik <techtonik at gmail.com>
> Author: anatoly techtonik <techtonik at gmail.com>, Anything Else for Humans,
> Or For Future Specs
>
> Here the field content defines its type - it's like duck typing for
> specification, which make specifications more pythonic.
>
> Is it good?

Nope. Having a separate field for email makes it clear that you should
add the email there IMO. However, both author-email and
maintainer-email are redundant, as is author and maintainer. The
relevant info is maintainer, to be honest.

//Lennart

From ubershmekel at gmail.com  Wed Apr 24 16:39:29 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 24 Apr 2013 17:39:29 +0300
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
Message-ID: <CANSw7KyzB3eKyGYshs9hP-U7A2XLTYrww8XzTZyDJb00D2BYAw@mail.gmail.com>

On Wed, Apr 24, 2013 at 5:06 PM, Lennart Regebro <regebro at gmail.com> wrote:

> On Wed, Apr 24, 2013 at 11:19 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
> > Having a lot of meaningless options doesn't make meta data any more
> clear.
> > Well, they are not meaningless, but their role is fully fulfilled by
> other
> > options (Author and Maintainer fields in this particular case).
> >
> > The use case for the Author field is that if Author wants to be
> contacted,
> > (s)he leaves email. That's it. This use case should be described in the
> > meta-data along with the format that are expected to be recognized by the
> > software:
> >
> > Author: anatoly techtonik
> > Author: anatoly techtonik <techtonik at gmail.com>
> > Author: anatoly techtonik <techtonik at gmail.com>, Anything Else for
> Humans,
> > Or For Future Specs
> >
> > Here the field content defines its type - it's like duck typing for
> > specification, which make specifications more pythonic.
> >
> > Is it good?
>
> Nope. Having a separate field for email makes it clear that you should
> add the email there IMO. However, both author-email and
> maintainer-email are redundant, as is author and maintainer. The
> relevant info is maintainer, to be honest.
>
> //Lennart
>

Probably some projects have more than one person handling them. I agree
that the author/maintainer distinction isn't interesting. A list of
maintainer emails may be a good solution.

Not that we can have any semblance of this on pypi, I just wanted to
mention this is very nicely done on github e.g.
https://github.com/mrdoob/three.js/contributors


Yuval Greenfield
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/72256e49/attachment.html>

From dholth at gmail.com  Wed Apr 24 16:45:27 2013
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 24 Apr 2013 10:45:27 -0400
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
Message-ID: <CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>

Keep in mind that this is the structured index metadata, which is only
used to grab requirements (for end users, who usually don't need the
other information at all) and used to generate cheese shop web pages.
AUTHORS.txt works when you don't need an automatic "email the author"
button. Or the project's github information etc.

Someday the JSON version of the metadata will certainly make all these
fields a little nicer to parse though.   Maybe

"authors" : [{'name':'bob', 'email':'bob at example.org'}]

The author/maintainer distinction will stay.


On Wed, Apr 24, 2013 at 10:06 AM, Lennart Regebro <regebro at gmail.com> wrote:
> On Wed, Apr 24, 2013 at 11:19 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>> Having a lot of meaningless options doesn't make meta data any more clear.
>> Well, they are not meaningless, but their role is fully fulfilled by other
>> options (Author and Maintainer fields in this particular case).
>>
>> The use case for the Author field is that if Author wants to be contacted,
>> (s)he leaves email. That's it. This use case should be described in the
>> meta-data along with the format that are expected to be recognized by the
>> software:
>>
>> Author: anatoly techtonik
>> Author: anatoly techtonik <techtonik at gmail.com>
>> Author: anatoly techtonik <techtonik at gmail.com>, Anything Else for Humans,
>> Or For Future Specs
>>
>> Here the field content defines its type - it's like duck typing for
>> specification, which make specifications more pythonic.
>>
>> Is it good?
>
> Nope. Having a separate field for email makes it clear that you should
> add the email there IMO. However, both author-email and
> maintainer-email are redundant, as is author and maintainer. The
> relevant info is maintainer, to be honest.
>
> //Lennart
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

From erik.bernoth at gmail.com  Wed Apr 24 16:55:31 2013
From: erik.bernoth at gmail.com (Erik Bernoth)
Date: Wed, 24 Apr 2013 16:55:31 +0200
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
Message-ID: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>

Hi everybody,

because there is no central place to look at and probably not all decisions
can be found in written form (like the abandoning of the fellow ship list,
when and why Tarek Ziad? decided to work on other things then
distribute/distutils2 and so on) I thought about writing a short summary of
the current state of packaging efforts as I understand it. I hope it has
value for people who need to make decisions in packaging their own projects
as well as help you packaging developers to see it a little bit from your
user's point of view.

All of it might not be correct or it might be incomplete. It simply
reflects my current state of investigation about that topic. I also hope
nobody gets angry, if he sees things differently or considers his effort
not embraced enough.

Short history about my reasons for this thread
---------------------------------------------------------------

End of last year I started a Python project and wanted to make sure users
can install my project without much effort. The main documentation of
Python 2.7 confused me deeply and it only talked about distutils and
setuptools. I simply wanted to add the required other packages, that they
can be automatically installed with my project, without the user manually
installing anything (which already let to problems in my project pre-alpha).

For some reason I ran into the pip installer and added, thanks to the
distutils-sig list [1] , the `install_requires` parameter to the setup()
function. In conclusion my problem was solved, although I still didn't
understand anything!

Concluding that being confused might mean that other people are also
confused, I started an effort to change the distutils part of the core
documentation [2][3]. I hoped for some guidance, but the basic problem for
that initiative was that I simply didn't know enough (and probably still
don't know enough) and nothing could help but investigating. For now I am
finished with that investigation and will think about continuing about a
documentation update in the short future.

I am writing this mail, because I hope I can give others and my future self
an overview of what all the hours of reading half finished documents and
mailing list threads got me until today. If it makes sense and there is a
central place to put such information I'd like to improve this text with
your assistance and put it there.


Documentation Advice For Users
----------------------------------------------

>From all guides to this complicated topic the best one seems to be the
inofficial guide by Scott Torborg: "How To Package Your Python Code" [4].
It refuses to discuss all the different projects and their interaction, but
presents a clear, simple path to follow that gets most things done, at
least much more then I probably want from packaging in the short future.

A great introduction about the complexities of this topic are in the
chapter "Python Packaging" in the open book "The Architecture of Open
Source Applications" [XX]. It's great, because it states different view
points on the topic, the different requirements and contains diagrams. Oh
how we people need diagrams to understand complex relationships!

Other resources aren't that useful in their current states: The core
documentation [5] is confusing, the often cited "Hitchhikers Guide to
Packaging" [6] seems to be a work in progress from 2009(?), and many
documentations from packaging tools themselves are not really complete,
either.

As a side note I like many sections in the distlib documentation [7]. It's
not complete and has some rough edges, but you can understand a lot about
what it should do from a first read, without too much frustration. That's
huge for any tool/framework in it's alpa state! I am a believer that good
human readable text (a.k.a. a good plan) results in good software, so I
have high hopes for this development.



Implementation Advice For Users
----------------------------------------------

In most cases setuptools is the way to go. distutils simply doesn't have
some fundamental features like `install_requires`. Taking other tools
doesn't seem to be necessary anymore, because distribute should be merged
into setuptools [8], so under the hood it's features will be used in the
short future anyway, Distutils2 is not an option due to nobody working on
it anymore (see the mailing list for example [9]), and distlib is currently
still in alpha and will likely be added under the hood into setuptools
anyway, when it's time.



Packaging efforts/project
-----------------------------------

(This only consideres projects close to the mainline development. There are
other tools like bento, or zc.buildout, which might be good or not, but
don't influence the mainline topics very much, so they are omitted in this
overview)

distutils (classic) - the old school, deprecated, pretty much unmaintained
packaging tooling; still the official standard [4] (Note: reasons [13])

setuptools - fork of distutils, which wasn't maintained for a long time but
will now be merged with it's successor [8]. Contains clear improvements
over distutils like requirements handling. Not all design decisions are
still appreciated, while some of them are based on the distutils mistake
and can't be fixed directly. (finding sources for these arguments will be
tough, because nobody cites any previous sources, when making such
arguments, but probably the following 2 threads contain these arguments at
least once: "Status in packaging 3.3" [10], "packaging location?" [11])

distribute - fork of setuptools [12], which is supported to the current
day. I don't know if it has any feature improvements over setuptools but I
guess it should be at least less buggy. Is currently merged back into
setuptools, which is a work in progress. (Note: Merge notification [8])

distutils2/packaging - a complete rewrite(?) [13] considering new
requirements, features and bug fixes. Was too big a task and couldn't be
finished until today. Was planned to be added to Python 3.3, which wasn't
possible, due to lack of work force for the size of the project. Currently
abandoned(?)

distlib - following the fail of distutils2 a fundamental set of futures
shall now be build into a framework that packaging tools like setuptools
can use to build stable, interoperable packaging mechanisms.

distil - a packaging tool like setuptools purely built on top of the
current state of distlib (?)

easy_install - an install script that shipped with setuptools. maintainance
state probably connected to setuptool's (?)

pip - an installer that seems to be developed in parallel with
distribute(?). From my feeling I'd say there is no much talk about this
online, because it simply works as expected.

Sources
--------------------

[1] http://mail.python.org/pipermail/distutils-sig/2013-February/019743.html
[2] http://mail.python.org/pipermail/distutils-sig/2013-February/019799.html
[3] http://mail.python.org/pipermail/docs/2013-March/013550.html
[4] http://www.scotttorborg.com/python-packaging/minimal.html
[5] http://docs.python.org/2.7/distutils/index.html (text is the same in
all following docs' versions as far as I could see)
[6] http://guide.python-distribute.org/index.html
[7] http://distlib.readthedocs.org/en/latest/index.html
[8] http://mail.python.org/pipermail/distutils-sig/2013-March/020126.html
[9]
https://groups.google.com/forum/?fromgroups#!forum/the-fellowship-of-the-packaging
[10]
http://mail.python.org/pipermail/python-dev/2012-June/thread.html#120430
[11]
http://mail.python.org/pipermail/python-dev/2012-September/thread.html#121689
[12] http://ziade.org/category/packaging,%20python.html
[13]
http://ziade.org/2010/03/03/the-fate-of-distutils-pycon-summit-packaging-sprint-detailed-report/
[XX] http://www.aosabook.org/en/packaging.html (added afterwards as
honourable mention)

Feedback welcome!

Cheers,
Erik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/8ec90927/attachment-0001.html>

From ncoghlan at gmail.com  Wed Apr 24 17:14:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Apr 2013 01:14:53 +1000
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
Message-ID: <CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>

On Thu, Apr 25, 2013 at 12:45 AM, Daniel Holth <dholth at gmail.com> wrote:
> Keep in mind that this is the structured index metadata, which is only
> used to grab requirements (for end users, who usually don't need the
> other information at all) and used to generate cheese shop web pages.
> AUTHORS.txt works when you don't need an automatic "email the author"
> button. Or the project's github information etc.
>
> Someday the JSON version of the metadata will certainly make all these
> fields a little nicer to parse though.   Maybe
>
> "authors" : [{'name':'bob', 'email':'bob at example.org'}]
>
> The author/maintainer distinction will stay.

FWIW, the current draft of PEP 426 has this note:

.. note:

    This section currently mimics the flat layout used in past metadata
    versions. Perhaps it would be better to switch to a different format
    closer to that used for the project URL field::

        "contacts": {
                "author": "\"C. Schultz\" <cschultz at example.com>"
                "maintainer": "\"P. Patty\" <ppatty at example.com>"
            }

So yeah, I'm not particularly happy with the current structure for the
contact metadata either, but it's a *long* way down my priority list
of problems to address in the next draft. (My aim to get that posted
last week proved to be overly optimistic. Because this next update is
the key-value -> JSON-compatible structured metadata switch, it's hard
to post a partial update and still have it even vaguely readable)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From qwcode at gmail.com  Wed Apr 24 17:20:50 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 24 Apr 2013 08:20:50 -0700
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
In-Reply-To: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
Message-ID: <CAPYWazq=w7Qyjz5fBBcXu0uzbE=Z7qOv7pUCGWr_1cAceP_8tw@mail.gmail.com>

Erik:
Thanks for posting this.
fyi, the "Hitchhikers Guide to Packaging" was recently forked to this:
https://bitbucket.org/pypa/python-packaging-user-guide
This is a pypa project, and and aims to be somewhat authoritative.
It's currently too far out of date to be a good base for meaningful pull
requests.
I'm working locally on it now, and hope to push updates soonish.
At that point, I'd love for people such as yourself, to help make it as
valuable as possible to users.
I'll post here as things develop.
Marcus

On Wed, Apr 24, 2013 at 7:55 AM, Erik Bernoth <erik.bernoth at gmail.com>wrote:

> Hi everybody,
>
> because there is no central place to look at and probably not all
> decisions can be found in written form (like the abandoning of the fellow
> ship list, when and why Tarek Ziad? decided to work on other things then
> distribute/distutils2 and so on) I thought about writing a short summary of
> the current state of packaging efforts as I understand it. I hope it has
> value for people who need to make decisions in packaging their own projects
> as well as help you packaging developers to see it a little bit from your
> user's point of view.
>
> All of it might not be correct or it might be incomplete. It simply
> reflects my current state of investigation about that topic. I also hope
> nobody gets angry, if he sees things differently or considers his effort
> not embraced enough.
>
> Short history about my reasons for this thread
> ---------------------------------------------------------------
>
> End of last year I started a Python project and wanted to make sure users
> can install my project without much effort. The main documentation of
> Python 2.7 confused me deeply and it only talked about distutils and
> setuptools. I simply wanted to add the required other packages, that they
> can be automatically installed with my project, without the user manually
> installing anything (which already let to problems in my project pre-alpha).
>
> For some reason I ran into the pip installer and added, thanks to the
> distutils-sig list [1] , the `install_requires` parameter to the setup()
> function. In conclusion my problem was solved, although I still didn't
> understand anything!
>
> Concluding that being confused might mean that other people are also
> confused, I started an effort to change the distutils part of the core
> documentation [2][3]. I hoped for some guidance, but the basic problem for
> that initiative was that I simply didn't know enough (and probably still
> don't know enough) and nothing could help but investigating. For now I am
> finished with that investigation and will think about continuing about a
> documentation update in the short future.
>
> I am writing this mail, because I hope I can give others and my future
> self an overview of what all the hours of reading half finished documents
> and mailing list threads got me until today. If it makes sense and there is
> a central place to put such information I'd like to improve this text with
> your assistance and put it there.
>
>
> Documentation Advice For Users
> ----------------------------------------------
>
> From all guides to this complicated topic the best one seems to be the
> inofficial guide by Scott Torborg: "How To Package Your Python Code" [4].
> It refuses to discuss all the different projects and their interaction, but
> presents a clear, simple path to follow that gets most things done, at
> least much more then I probably want from packaging in the short future.
>
> A great introduction about the complexities of this topic are in the
> chapter "Python Packaging" in the open book "The Architecture of Open
> Source Applications" [XX]. It's great, because it states different view
> points on the topic, the different requirements and contains diagrams. Oh
> how we people need diagrams to understand complex relationships!
>
> Other resources aren't that useful in their current states: The core
> documentation [5] is confusing, the often cited "Hitchhikers Guide to
> Packaging" [6] seems to be a work in progress from 2009(?), and many
> documentations from packaging tools themselves are not really complete,
> either.
>
> As a side note I like many sections in the distlib documentation [7]. It's
> not complete and has some rough edges, but you can understand a lot about
> what it should do from a first read, without too much frustration. That's
> huge for any tool/framework in it's alpa state! I am a believer that good
> human readable text (a.k.a. a good plan) results in good software, so I
> have high hopes for this development.
>
>
>
> Implementation Advice For Users
> ----------------------------------------------
>
> In most cases setuptools is the way to go. distutils simply doesn't have
> some fundamental features like `install_requires`. Taking other tools
> doesn't seem to be necessary anymore, because distribute should be merged
> into setuptools [8], so under the hood it's features will be used in the
> short future anyway, Distutils2 is not an option due to nobody working on
> it anymore (see the mailing list for example [9]), and distlib is currently
> still in alpha and will likely be added under the hood into setuptools
> anyway, when it's time.
>
>
>
> Packaging efforts/project
> -----------------------------------
>
> (This only consideres projects close to the mainline development. There
> are other tools like bento, or zc.buildout, which might be good or not, but
> don't influence the mainline topics very much, so they are omitted in this
> overview)
>
> distutils (classic) - the old school, deprecated, pretty much unmaintained
> packaging tooling; still the official standard [4] (Note: reasons [13])
>
> setuptools - fork of distutils, which wasn't maintained for a long time
> but will now be merged with it's successor [8]. Contains clear improvements
> over distutils like requirements handling. Not all design decisions are
> still appreciated, while some of them are based on the distutils mistake
> and can't be fixed directly. (finding sources for these arguments will be
> tough, because nobody cites any previous sources, when making such
> arguments, but probably the following 2 threads contain these arguments at
> least once: "Status in packaging 3.3" [10], "packaging location?" [11])
>
> distribute - fork of setuptools [12], which is supported to the current
> day. I don't know if it has any feature improvements over setuptools but I
> guess it should be at least less buggy. Is currently merged back into
> setuptools, which is a work in progress. (Note: Merge notification [8])
>
> distutils2/packaging - a complete rewrite(?) [13] considering new
> requirements, features and bug fixes. Was too big a task and couldn't be
> finished until today. Was planned to be added to Python 3.3, which wasn't
> possible, due to lack of work force for the size of the project. Currently
> abandoned(?)
>
> distlib - following the fail of distutils2 a fundamental set of futures
> shall now be build into a framework that packaging tools like setuptools
> can use to build stable, interoperable packaging mechanisms.
>
> distil - a packaging tool like setuptools purely built on top of the
> current state of distlib (?)
>
> easy_install - an install script that shipped with setuptools.
> maintainance state probably connected to setuptool's (?)
>
> pip - an installer that seems to be developed in parallel with
> distribute(?). From my feeling I'd say there is no much talk about this
> online, because it simply works as expected.
>
> Sources
> --------------------
>
> [1]
> http://mail.python.org/pipermail/distutils-sig/2013-February/019743.html
> [2]
> http://mail.python.org/pipermail/distutils-sig/2013-February/019799.html
> [3] http://mail.python.org/pipermail/docs/2013-March/013550.html
> [4] http://www.scotttorborg.com/python-packaging/minimal.html
> [5] http://docs.python.org/2.7/distutils/index.html (text is the same in
> all following docs' versions as far as I could see)
> [6] http://guide.python-distribute.org/index.html
> [7] http://distlib.readthedocs.org/en/latest/index.html
> [8] http://mail.python.org/pipermail/distutils-sig/2013-March/020126.html
> [9]
> https://groups.google.com/forum/?fromgroups#!forum/the-fellowship-of-the-packaging
> [10]
> http://mail.python.org/pipermail/python-dev/2012-June/thread.html#120430
> [11]
> http://mail.python.org/pipermail/python-dev/2012-September/thread.html#121689
> [12] http://ziade.org/category/packaging,%20python.html
> [13]
> http://ziade.org/2010/03/03/the-fate-of-distutils-pycon-summit-packaging-sprint-detailed-report/
> [XX] http://www.aosabook.org/en/packaging.html (added afterwards as
> honourable mention)
>
> Feedback welcome!
>
> Cheers,
> Erik
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/cc01061b/attachment-0001.html>

From ncoghlan at gmail.com  Wed Apr 24 17:31:35 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Apr 2013 01:31:35 +1000
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
In-Reply-To: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
Message-ID: <CADiSq7d-7BaEj81qyh-gFF5yEcP8zChfnX7O5FSj8yJz0ZAAng@mail.gmail.com>

Thanks for posting this Erik.

That's not a bad summary, and
http://www.scotttorborg.com/python-packaging/index.html looks like an
excellent resource.

We're currently trying to bring some order to the chaos, as described
at http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/core_packaging_api.html

Our current status is that most of the key projects are being gathered
under the "Python Packaging Authority" banner on Github and BitBucket.

https://github.com/pypa
  * the original PyPA projects (pip and virtualenv)

https://bitbucket.org/pypa/
  * existing projects recently migrated to PyPA (distlib, pypi, pylauncher)
  * new (very incomplete) packaging guide for users
  * new (very incomplete) "working on packaging tools" meta-guide for
contributors
  * the merged setuptools 0.7+ repo is also expected to live here once
Jason declares it ready for public consumption

My essay linked above should eventually migrate to the meta-guide, and
Scott's guide would be a useful link from the user guide (while it's
desirable to have a "default" tutorial, linking to others can also be
helpful for cases where the main guide doesn't make sense to users).

Once the user guide and meta guide are in a better state, we'll update
the stdlib distutils docs with a reference to them as a guide to a
more up to date packaging toolchain.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Apr 24 17:35:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Apr 2013 01:35:24 +1000
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
In-Reply-To: <CADiSq7d-7BaEj81qyh-gFF5yEcP8zChfnX7O5FSj8yJz0ZAAng@mail.gmail.com>
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
	<CADiSq7d-7BaEj81qyh-gFF5yEcP8zChfnX7O5FSj8yJz0ZAAng@mail.gmail.com>
Message-ID: <CADiSq7cH+-xu7FOF3Rdtb8ko5a2oAmNidOtn5R2bsgJxob5Ftw@mail.gmail.com>

On Thu, Apr 25, 2013 at 1:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Once the user guide and meta guide are in a better state, we'll update
> the stdlib distutils docs with a reference to them as a guide to a
> more up to date packaging toolchain.

Oops, I meant to echo Marcus' sentiment that additional hands in
getting those guides to a more reasonable state is definitely welcome!

One of the challenges of the packaging ecosystem has been that those
involved have been busy enough working on the tools, that it's hard to
find the time to hammer the overall documentation (whether user or
developer oriented) into shape.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Wed Apr 24 18:42:37 2013
From: donald at stufft.io (Donald Stufft)
Date: Wed, 24 Apr 2013 12:42:37 -0400
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
Message-ID: <0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>


On Apr 24, 2013, at 11:14 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Thu, Apr 25, 2013 at 12:45 AM, Daniel Holth <dholth at gmail.com> wrote:
>> Keep in mind that this is the structured index metadata, which is only
>> used to grab requirements (for end users, who usually don't need the
>> other information at all) and used to generate cheese shop web pages.
>> AUTHORS.txt works when you don't need an automatic "email the author"
>> button. Or the project's github information etc.
>> 
>> Someday the JSON version of the metadata will certainly make all these
>> fields a little nicer to parse though.   Maybe
>> 
>> "authors" : [{'name':'bob', 'email':'bob at example.org'}]
>> 
>> The author/maintainer distinction will stay.
> 
> FWIW, the current draft of PEP 426 has this note:
> 
> .. note:
> 
>    This section currently mimics the flat layout used in past metadata
>    versions. Perhaps it would be better to switch to a different format
>    closer to that used for the project URL field::
> 
>        "contacts": {
>                "author": "\"C. Schultz\" <cschultz at example.com>"
>                "maintainer": "\"P. Patty\" <ppatty at example.com>"
>            }
> 
> So yeah, I'm not particularly happy with the current structure for the
> contact metadata either, but it's a *long* way down my priority list
> of problems to address in the next draft. (My aim to get that posted
> last week proved to be overly optimistic. Because this next update is
> the key-value -> JSON-compatible structured metadata switch, it's hard
> to post a partial update and still have it even vaguely readable)
> 
> Cheers,
> Nick.
> 
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


Looking at other languages they all seem to accept N number of "authors" or
"contributors". Possibly something like that would be a good to do.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/d0283e88/attachment.pgp>

From donald at stufft.io  Wed Apr 24 18:54:52 2013
From: donald at stufft.io (Donald Stufft)
Date: Wed, 24 Apr 2013 12:54:52 -0400
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
Message-ID: <F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>


On Apr 24, 2013, at 12:42 PM, Donald Stufft <donald at stufft.io> wrote:

> 
> On Apr 24, 2013, at 11:14 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
>> On Thu, Apr 25, 2013 at 12:45 AM, Daniel Holth <dholth at gmail.com> wrote:
>>> Keep in mind that this is the structured index metadata, which is only
>>> used to grab requirements (for end users, who usually don't need the
>>> other information at all) and used to generate cheese shop web pages.
>>> AUTHORS.txt works when you don't need an automatic "email the author"
>>> button. Or the project's github information etc.
>>> 
>>> Someday the JSON version of the metadata will certainly make all these
>>> fields a little nicer to parse though.   Maybe
>>> 
>>> "authors" : [{'name':'bob', 'email':'bob at example.org'}]
>>> 
>>> The author/maintainer distinction will stay.
>> 
>> FWIW, the current draft of PEP 426 has this note:
>> 
>> .. note:
>> 
>>   This section currently mimics the flat layout used in past metadata
>>   versions. Perhaps it would be better to switch to a different format
>>   closer to that used for the project URL field::
>> 
>>       "contacts": {
>>               "author": "\"C. Schultz\" <cschultz at example.com>"
>>               "maintainer": "\"P. Patty\" <ppatty at example.com>"
>>           }
>> 
>> So yeah, I'm not particularly happy with the current structure for the
>> contact metadata either, but it's a *long* way down my priority list
>> of problems to address in the next draft. (My aim to get that posted
>> last week proved to be overly optimistic. Because this next update is
>> the key-value -> JSON-compatible structured metadata switch, it's hard
>> to post a partial update and still have it even vaguely readable)
>> 
>> Cheers,
>> Nick.
>> 
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
> 
> 
> Looking at other languages they all seem to accept N number of "authors" or
> "contributors". Possibly something like that would be a good to do.
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig



To be specific, here's what node.js does

They have an Author field, who represents the primary author of the package,
and then a contributors field that accepts a list of people who have contributed
to the package. NPM will also automatically fill this out if left blank by using an
AUTHORS file assuming the file has 1 per line and each line is in the format
of Name <email> (url).

Ruby has an authors field which is a list of all the authors.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/5607945a/attachment.pgp>

From vinay_sajip at yahoo.co.uk  Wed Apr 24 19:33:54 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 24 Apr 2013 17:33:54 +0000 (UTC)
Subject: [Distutils]
	=?utf-8?q?User_Viewpoint=3A_Packaging_History=2C=09Co?=
	=?utf-8?q?nfusion_and_User_Advice?=
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
Message-ID: <loom.20130424T192556-717@post.gmane.org>

Erik Bernoth <erik.bernoth <at> gmail.com> writes:

> As a side note I like many sections in the distlib documentation [7].

Thanks for this feedback. Since distlib is under active development the
API reference section lags behind, but I've tried to keep the overview and
tutorial sections updated. If you have any specific comments about how
these sections could be improved, please raise an issue on the distlib
tracker with your suggestions.

[11])distribute - fork of setuptools [12], 
> which is supported to the current day. I don't know if it has any 
> feature improvements over setuptools but I guess it should be at least 
> less buggy.

The main feature improvements in distribute are (a) Python 3 support and
(b) better support for recent metadata developments.


> distlib is currently still in alpha and will likely be added under the
> hood into setuptools anyway, when it's time.

Not sure about that - it's meant to supplant setuptools and avoid setup.py
altogether, whereas distutils and setuptools/distribute are pretty much
tied to "python setup.py <command>".

> distil - a packaging tool like setuptools purely built on top of the
> current state of distlib (?)

It's more like pip than setuptools, but it aims to offer all packaging
roles without using "python setup.py <command>". Meant as a testbed for
distlib, but it already performs package building, source packaging,
installation/uninstallation/upgrading pretty well for a tool in its early
stages of development. (Of course more testing is required, and people here
can definitely help with that by trying it out.)

Thanks for your post summarising the state of the packaging union, it will
definitely help to orient new packaging users/packagers in the right direction.

Regards,

Vinay Sajip


From qwcode at gmail.com  Wed Apr 24 21:38:09 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 24 Apr 2013 12:38:09 -0700
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
In-Reply-To: <CADiSq7d-7BaEj81qyh-gFF5yEcP8zChfnX7O5FSj8yJz0ZAAng@mail.gmail.com>
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
	<CADiSq7d-7BaEj81qyh-gFF5yEcP8zChfnX7O5FSj8yJz0ZAAng@mail.gmail.com>
Message-ID: <CAPYWazpi88YVNY+djy1eS7bXq=Kiifma_D24WNDivmsq=xusHQ@mail.gmail.com>

>   * new (very incomplete) packaging guide for users
>   * new (very incomplete) "working on packaging tools" meta-guide for
> contributors
>

Nick, btw, as I've started worked to work on these locally, I'm inclined to
just have one document (the "user guide") that handles both goals.
Why?
1) It's more realistic that we'll be able to maintain *one* document well,
rather than two.
2) They would end up overlapping or cross-linking anyway I think.
3) Most python user's want to know what's in the works anyway. It makes
them feel comfortable to see an active plan.

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/59ed0b0b/attachment.html>

From qwcode at gmail.com  Wed Apr 24 22:08:45 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 24 Apr 2013 13:08:45 -0700
Subject: [Distutils] pypa-dev mailing list
In-Reply-To: <kl8o11$983$1@ger.gmane.org>
References: <CAPYWazqZVg4mALTAba=MR3jKRoZ0ALZD6-zSjcxPMPPmQOD3ig@mail.gmail.com>
	<kl8o11$983$1@ger.gmane.org>
Message-ID: <CAPYWazqn72t9sh7p43yWsDOiiOs3AkcjUPdUZRw_8_OSvvk50Q@mail.gmail.com>

not sure about gmane naming, but my first reaction w/o knowing much would
be like this:

pip/virtualenv user list
http://groups.google.com/group/python-virtualenv  ->
 gmane.comp.python.virtualenv

pypa-dev list
http://groups.google.com/group/pypa-dev  ->  gmane.comp.python.pypa.dev

as for whether "pypa", or python in general, should have a generic
packaging user list, that's come up, but no consensus as of yet, so no
changes so far.
(btw, the ideas that have been tossed around:  "packaging-user",
"pypa-user")

so, I'm not sure about mapping "python-virtualenv" to "pypa.user" at the
moment like you suggested.

Marcus

On Wed, Apr 24, 2013 at 6:51 AM, Gabriel de Perthuis <g2p.code at gmail.com>wrote:

> > fyi, there is dev list now for the "pypa" family of projects (most
> > notably pip and virtualenv)
> > "
> > "pypa-dev"  https://groups.google.com/forum/?fromgroups#!forum/pypa-dev
> >
> > This is *not* meant to replace distutils-sig.
> >
> > It's for more "day-to-day" discussions related to maintaining and
> > releasing the "pypa" projects.
>
> What names should the two lists have on GMane [1]?
>
> gmane.comp.python.pypa.user and gmane.comp.python.pypa.devel ?
>
> [1] http://gmane.org/subscribe.php
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/c0827774/attachment.html>

From g2p.code at gmail.com  Wed Apr 24 23:29:03 2013
From: g2p.code at gmail.com (Gabriel de Perthuis)
Date: Wed, 24 Apr 2013 21:29:03 +0000 (UTC)
Subject: [Distutils] pypa-dev mailing list
References: <CAPYWazqZVg4mALTAba=MR3jKRoZ0ALZD6-zSjcxPMPPmQOD3ig@mail.gmail.com>
	<kl8o11$983$1@ger.gmane.org>
	<CAPYWazqn72t9sh7p43yWsDOiiOs3AkcjUPdUZRw_8_OSvvk50Q@mail.gmail.com>
Message-ID: <kl9iqv$983$2@ger.gmane.org>

> not sure about gmane naming, but my first reaction w/o knowing much
> would be like this:
> 
> pip/virtualenv user list
> http://groups.google.com/group/python-virtualenv  ->
>  gmane.comp.python.virtualenv
> 
> pypa-dev list http://groups.google.com/group/pypa-dev  -> 
> gmane.comp.python.pypa.dev
> 
> as for whether "pypa", or python in general, should have a generic
> packaging user list, that's come up, but no consensus as of yet, so no
> changes so far.
> (btw, the ideas that have been tossed around:  "packaging-user",
> "pypa-user")
> 
> so, I'm not sure about mapping "python-virtualenv" to "pypa.user" at the
> moment like you suggested.

Okay, Gmane subscription requests sent with those names.
If there's consensus to create the new list at some point,
consider adding it to Gmane as well (I'll have forgotten by then,
but it's good to make sure early messages are all archived).

http://gmane.org/subscribe.php


From ncoghlan at gmail.com  Thu Apr 25 02:30:26 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Apr 2013 10:30:26 +1000
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
In-Reply-To: <CAPYWazpi88YVNY+djy1eS7bXq=Kiifma_D24WNDivmsq=xusHQ@mail.gmail.com>
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
	<CADiSq7d-7BaEj81qyh-gFF5yEcP8zChfnX7O5FSj8yJz0ZAAng@mail.gmail.com>
	<CAPYWazpi88YVNY+djy1eS7bXq=Kiifma_D24WNDivmsq=xusHQ@mail.gmail.com>
Message-ID: <CADiSq7cEy6dfNPmfcsUeXtJGehFY_sfORQi192x04Gr+7ixTCw@mail.gmail.com>

On 25 Apr 2013 05:38, "Marcus Smith" <qwcode at gmail.com> wrote:
>
>
>>   * new (very incomplete) packaging guide for users
>>   * new (very incomplete) "working on packaging tools" meta-guide for
>> contributors
>
>
> Nick, btw, as I've started worked to work on these locally, I'm inclined
to just have one document (the "user guide") that handles both goals.
> Why?
> 1) It's more realistic that we'll be able to maintain *one* document
well, rather than two.
> 2) They would end up overlapping or cross-linking anyway I think.
> 3) Most python user's want to know what's in the works anyway. It makes
them feel comfortable to see an active plan.

Sure, having additional sections like "where to find the source" and  "how
to get involved" at the end of the user guide rather than as a completely
separate doc sounds like a good idea. (Mostly for the first and last points
- intersphinx handles cross linking quite well)

Cheers,
Nick.

> Marcus
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130425/e4261b1e/attachment.html>

From techtonik at gmail.com  Thu Apr 25 05:35:20 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 25 Apr 2013 06:35:20 +0300
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
Message-ID: <CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>

On Wed, Apr 24, 2013 at 7:54 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Apr 24, 2013, at 12:42 PM, Donald Stufft <donald at stufft.io> wrote:
>
> >
> > On Apr 24, 2013, at 11:14 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> >> On Thu, Apr 25, 2013 at 12:45 AM, Daniel Holth <dholth at gmail.com>
> wrote:
> >>> Keep in mind that this is the structured index metadata, which is only
> >>> used to grab requirements (for end users, who usually don't need the
> >>> other information at all) and used to generate cheese shop web pages.
> >>> AUTHORS.txt works when you don't need an automatic "email the author"
> >>> button. Or the project's github information etc.
> >>>
> >>> Someday the JSON version of the metadata will certainly make all these
> >>> fields a little nicer to parse though.   Maybe
> >>>
> >>> "authors" : [{'name':'bob', 'email':'bob at example.org'}]
> >>>
> >>> The author/maintainer distinction will stay.
> >>
> >> FWIW, the current draft of PEP 426 has this note:
> >>
> >> .. note:
> >>
> >>   This section currently mimics the flat layout used in past metadata
> >>   versions. Perhaps it would be better to switch to a different format
> >>   closer to that used for the project URL field::
> >>
> >>       "contacts": {
> >>               "author": "\"C. Schultz\" <cschultz at example.com>"
> >>               "maintainer": "\"P. Patty\" <ppatty at example.com>"
> >>           }
> >>
> >> So yeah, I'm not particularly happy with the current structure for the
> >> contact metadata either, but it's a *long* way down my priority list
> >> of problems to address in the next draft. (My aim to get that posted
> >> last week proved to be overly optimistic. Because this next update is
> >> the key-value -> JSON-compatible structured metadata switch, it's hard
> >> to post a partial update and still have it even vaguely readable)
> >>
> >> Cheers,
> >> Nick.
> >>
> >> --
> >> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >> http://mail.python.org/mailman/listinfo/distutils-sig
> >
> >
> > Looking at other languages they all seem to accept N number of "authors"
> or
> > "contributors". Possibly something like that would be a good to do.
> >
> > -----------------
> > Donald Stufft
> > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > http://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> To be specific, here's what node.js does
>
> They have an Author field, who represents the primary author of the
> package,
> and then a contributors field that accepts a list of people who have
> contributed
> to the package. NPM will also automatically fill this out if left blank by
> using an
> AUTHORS file assuming the file has 1 per line and each line is in the
> format
> of Name <email> (url).
>
> Ruby has an authors field which is a list of all the authors.


I see it this way. There are four user stories. In all three author field
is an id of:
1. (copyright)   software authors (collective team or single-person)
2. (credits)      contributors (people who helped)
3. (publishing) package maintainers (people who port stuff to different
systems)
4. (support)     software maintainers (contact points)

I guess the good rule is not overengineer the stuff. We can formalize
everything endlessly and will get XHTML in the end. I prefer practical
HTML5 approach where the data is directed by a user story. Every change
needs an existing user who has requirements and data format should cover
this specific user's story without making any speculations and assumptions
about future use.

To reach the consensus in this discussion, I guess we all agree that roles
in software projects may differ greatly. Even the 4 user stories listed are
not enough to give proper credits to all people involved. I guess we can
reach consensus stating that maintainers of packages in Python world are
not common - authors or contributors package stuff themselves. PyPI is not
Linux distribution where package maintainers play a significant and
different role. So the "author"+ and "contributor"* fields seem to be
reasonable minimum core.

Do we all agree on that minimum as the first step?

Now about the contents. Only Lennart expressed opinion that separate email
may be useful because it forces people to supply this email. If emails were
obligatory, we could consider this point, but my IMO is that this it is not
an reason to increase complexity of the system.
http://www.youtube.com/watch?v=rI8tNMsozo0

I like the ability to supply URL in the Author/Contributor field. It can be
used to link to the project specific way of listing people involved
(including GitHub pages). But for now I'd not formalize it and considered
as an optional feature. Just listed somewhere. I'd like not to formalize
the contents of this field at all leaving the recommendation to supply
content in "name[ email][, optional stuff]" format. Can we reach a
consensus on that?
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130425/9684fc3b/attachment-0001.html>

From techtonik at gmail.com  Thu Apr 25 05:42:47 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 25 Apr 2013 06:42:47 +0300
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
Message-ID: <CAPkN8xJcnY2Li5ZAiOfy4amgmeOLp91npp73iqn8NgCeppHVBA@mail.gmail.com>

On Wed, Apr 24, 2013 at 6:14 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Thu, Apr 25, 2013 at 12:45 AM, Daniel Holth <dholth at gmail.com> wrote:
> > Keep in mind that this is the structured index metadata, which is only
> > used to grab requirements (for end users, who usually don't need the
> > other information at all) and used to generate cheese shop web pages.
> > AUTHORS.txt works when you don't need an automatic "email the author"
> > button. Or the project's github information etc.
> >
> > Someday the JSON version of the metadata will certainly make all these
> > fields a little nicer to parse though.   Maybe
> >
> > "authors" : [{'name':'bob', 'email':'bob at example.org'}]
> >
> > The author/maintainer distinction will stay.
>
> FWIW, the current draft of PEP 426 has this note:
>
> .. note:
>
>     This section currently mimics the flat layout used in past metadata
>     versions. Perhaps it would be better to switch to a different format
>     closer to that used for the project URL field::
>
>         "contacts": {
>                 "author": "\"C. Schultz\" <cschultz at example.com>"
>                 "maintainer": "\"P. Patty\" <ppatty at example.com>"
>             }
>
> So yeah, I'm not particularly happy with the current structure for the
> contact metadata either, but it's a *long* way down my priority list
> of problems to address in the next draft. (My aim to get that posted
> last week proved to be overly optimistic. Because this next update is
> the key-value -> JSON-compatible structured metadata switch, it's hard
> to post a partial update and still have it even vaguely readable)
>

You user story is "contact point". It is quite different from the role of
the author meaning for open source projects. I know at least 5 active open
source projects where people don't like to receive personal email and
prefer to have public tracker or a mailing list as a contact point.

-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130425/725d8918/attachment.html>

From donald at stufft.io  Thu Apr 25 05:54:45 2013
From: donald at stufft.io (Donald Stufft)
Date: Wed, 24 Apr 2013 23:54:45 -0400
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
Message-ID: <837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>


On Apr 24, 2013, at 11:35 PM, anatoly techtonik <techtonik at gmail.com> wrote:

> To reach the consensus in this discussion, I guess we all agree that roles in software projects may differ greatly. Even the 4 user stories listed are not enough to give proper credits to all people involved. I guess we can reach consensus stating that maintainers of packages in Python world are not common - authors or contributors package stuff themselves. PyPI is not Linux distribution where package maintainers play a significant and different role. So the "author"+ and "contributor"* fields seem to be reasonable minimum core.
> 
> Do we all agree on that minimum as the first step?

An author (or authors) and contributors field would be very suitable.

> 
> Now about the contents. Only Lennart expressed opinion that separate email may be useful because it forces people to supply this email. If emails were obligatory, we could consider this point, but my IMO is that this it is not an reason to increase complexity of the system. http://www.youtube.com/watch?v=rI8tNMsozo0 
> 
> I like the ability to supply URL in the Author/Contributor field. It can be used to link to the project specific way of listing people involved (including GitHub pages). But for now I'd not formalize it and considered as an optional feature. Just listed somewhere. I'd like not to formalize the contents of this field at all leaving the recommendation to supply content in "name[ email][, optional stuff]" format. Can we reach a consensus on that?

I like how Node.js packages handled this. The canonical form is {"name": "Donald Stufft", "email": "donald at stufft.io", "url": "http://github.com/dstufft/"} (and what I think the internal representation should be), but the tooling understands strings like "Donald Stufft <donald at stufft.io> (http://github.com/dstufft/)". And both the email and url data is optional.

> -- 
> anatoly t. 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/0702d713/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/0702d713/attachment.pgp>

From ncoghlan at gmail.com  Thu Apr 25 09:13:35 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Apr 2013 17:13:35 +1000
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
Message-ID: <CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>

On Thu, Apr 25, 2013 at 1:54 PM, Donald Stufft <donald at stufft.io> wrote:
> I like how Node.js packages handled this. The canonical form is {"name":
> "Donald Stufft", "email": "donald at stufft.io", "url":
> "http://github.com/dstufft/"} (and what I think the internal representation
> should be), but the tooling understands strings like "Donald Stufft
> <donald at stufft.io> (http://github.com/dstufft/)". And both the email and url
> data is optional.

How about the following as the canonical PEP 426 representation:

    "contact": {
        "name": "Donald Stufft",
        "email": "donald at stufft.io",
        "url": "http://github.com/dstufft/"
    }
    "contributors": [<list of individual contact entries>]

The point of the neutral "contact" entry is that it may be a larger entity like:

    "contact": {
        "name": "Python Packaging Authority",
        "email": "pypa-dev at googlegroups.com",
        "url": "http://github.com/pypa"
    }

or the distutils-sig+BitBucket variant of that:

   "contact": {
        "name": "Python Packaging Authority/Distutils SIG",
        "email": "distutils-sig at python.org",
        "url": "http://bitbucket.org/pypa"
    }

As I work on the PEP 426 conversion to JSON as the on-disk format, I'm
also becoming convinced that we'll want at least a draft of the
replacement for PEP 390 (static input metadata) before I accept the
metadata 2.0 spec.

The key difference relative to PEP 390 is that the new PEP (for a
"pymeta.cfg" file) will be solely intended as an *input* format to a
build tool that populates pymeta.json when creating an sdist or wheel
file, whereas PEP 390 wanted to use the same format both for the
human-friendly input *and* as the machine readable interchange format.
Working on the PEP 426 conversion has confirmed my opinion that
splitting them is the right way to go, and also that a multi-file
input format is likely a better option than trying to cram everything
into one file.

Those two PEPs will then define "this is where we want to get to"
based on everything we've learned from the status quo and from looking
at other languages. Then the real work begins of figuring out how the
hell we can actually get there in a reasonable amount of time :)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Thu Apr 25 13:06:14 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 25 Apr 2013 07:06:14 -0400
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
	<CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
Message-ID: <435664F0-4F19-4C38-BEAC-40A80AFA49EE@stufft.io>


On Apr 25, 2013, at 3:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Thu, Apr 25, 2013 at 1:54 PM, Donald Stufft <donald at stufft.io> wrote:
>> I like how Node.js packages handled this. The canonical form is {"name":
>> "Donald Stufft", "email": "donald at stufft.io", "url":
>> "http://github.com/dstufft/"} (and what I think the internal representation
>> should be), but the tooling understands strings like "Donald Stufft
>> <donald at stufft.io> (http://github.com/dstufft/)". And both the email and url
>> data is optional.
> 
> How about the following as the canonical PEP 426 representation:
> 
>    "contact": {
>        "name": "Donald Stufft",
>        "email": "donald at stufft.io",
>        "url": "http://github.com/dstufft/"
>    }
>    "contributors": [<list of individual contact entries>]
> 
> The point of the neutral "contact" entry is that it may be a larger entity like:
> 
>    "contact": {
>        "name": "Python Packaging Authority",
>        "email": "pypa-dev at googlegroups.com",
>        "url": "http://github.com/pypa"
>    }
> 
> or the distutils-sig+BitBucket variant of that:
> 
>   "contact": {
>        "name": "Python Packaging Authority/Distutils SIG",
>        "email": "distutils-sig at python.org",
>        "url": "http://bitbucket.org/pypa"
>    }


contact instead of author works for me. That's practically how it's implemented in distutils ATM anyways (Maintainer overrides author and gets sent _as_ the author).

> 
> As I work on the PEP 426 conversion to JSON as the on-disk format, I'm
> also becoming convinced that we'll want at least a draft of the
> replacement for PEP 390 (static input metadata) before I accept the
> metadata 2.0 spec.
> 
> The key difference relative to PEP 390 is that the new PEP (for a
> "pymeta.cfg" file) will be solely intended as an *input* format to a
> build tool that populates pymeta.json when creating an sdist or wheel
> file, whereas PEP 390 wanted to use the same format both for the
> human-friendly input *and* as the machine readable interchange format.
> Working on the PEP 426 conversion has confirmed my opinion that
> splitting them is the right way to go, and also that a multi-file
> input format is likely a better option than trying to cram everything
> into one file.

I'm not sold on the pymeta.cfg file. Part of the purpose of defining the sdist format instead of the tool that creates the sdist format is that we allow any type of "input file" that someone wants to make a tool that does it. Further more package creation should ideally be moved out of the stdlib and I don't see much of a point to creating a PEP for a file that should never leave the developers computer.

> 
> Those two PEPs will then define "this is where we want to get to"
> based on everything we've learned from the status quo and from looking
> at other languages. Then the real work begins of figuring out how the
> hell we can actually get there in a reasonable amount of time :)
> 
> Cheers,
> Nick.
> 
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130425/3fef77e8/attachment.pgp>

From erik.bernoth at gmail.com  Thu Apr 25 13:35:33 2013
From: erik.bernoth at gmail.com (Erik Bernoth)
Date: Thu, 25 Apr 2013 13:35:33 +0200
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
In-Reply-To: <CADiSq7cEy6dfNPmfcsUeXtJGehFY_sfORQi192x04Gr+7ixTCw@mail.gmail.com>
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
	<CADiSq7d-7BaEj81qyh-gFF5yEcP8zChfnX7O5FSj8yJz0ZAAng@mail.gmail.com>
	<CAPYWazpi88YVNY+djy1eS7bXq=Kiifma_D24WNDivmsq=xusHQ@mail.gmail.com>
	<CADiSq7cEy6dfNPmfcsUeXtJGehFY_sfORQi192x04Gr+7ixTCw@mail.gmail.com>
Message-ID: <CAB46HO=Fyf3xwh3mkE=A1OGe0RxViesCTc-CzPOCMApbkWAMcA@mail.gmail.com>

Thanks everybody for your feedback!

On Wed, Apr 24, 2013 at 5:31 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> That's not a bad summary, and
> http://www.scotttorborg.com/python-packaging/index.html looks like an
> excellent resource.
>

I'm going to write him an email pointing him to this thread. Maybe he's
interested in giving the text to the Python community. As far as I could
see it wasn't CC-BY-SA'd or something.

>
> We're currently trying to bring some order to the chaos, as described
> at
http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/core_packaging_api.html
>

I love it! A good overview. It also clarifies general problems and
requirements for a packaging ecosystem. I also like the approach of taking
a step back and solving a problem that people faced, who tried to solve the
obvious problem. This approach is often quite useful in problem solving. A
lot of people can tackle the obvious parts of a problem set, but few people
are able to find the core problems and present them in a human readable
form. While reading I had the urge to convince you to go one more step back
in hope of solving problems a lot of people will face in the next steps,
which might not be obvious to you.

As I understand correctly you already have a lot of experience in and
around an RPM based packaging ecosystem. So you already know the
architecture of a solution to a problem set that is quite close to what the
python community has at the moment. This is not obvious to other people,
though. I bet an analysis of existing and working packaging ecosystems
would be really helpful in making plans. In the best case scenario it might
be possible to find a Lego-like system of best practices which can be
combined this way or another to weigh the pros and cons of each best
practice. I think this might be a much better approach then making a plan
out of what people guess might be a solution to existing problems. It's
hard work, compared to the documents and overviews you are preparing now it
might be even more inconvenient than writing those docs compared to
programming. But the pay off might also be enormous. I think it's like
writing PEPs something that doesn't result in working software directly but
enables a lot of other people to write better code, which indirectly solves
the coding challenge automatically.

What do you guys think about that?

>
> Our current status is that most of the key projects are being gathered
> under the "Python Packaging Authority" banner on Github and BitBucket.
>

I was already wondering what PyPA means.

>
> My essay linked above should eventually migrate to the meta-guide, and
> Scott's guide would be a useful link from the user guide (while it's
> desirable to have a "default" tutorial, linking to others can also be
> helpful for cases where the main guide doesn't make sense to users).
>
> Once the user guide and meta guide are in a better state, we'll update
> the stdlib distutils docs with a reference to them as a guide to a
> more up to date packaging toolchain.
>

How can I get involved in that work?

Both User Guide and Developer Hub didn't receive any commit this month, or
I am not able to review the repositories well enough? I'm new to bitbucket
and hg, so I have to ask: Is the current state of development checked in
there somewhere?
Also I don't see any Issues. Where are the current milestones (or
footsteps) written down and the progress tracked?

Regards,
Erik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130425/09def63e/attachment.html>

From g2p.code at gmail.com  Wed Apr 24 23:25:23 2013
From: g2p.code at gmail.com (Gabriel)
Date: Wed, 24 Apr 2013 23:25:23 +0200
Subject: [Distutils] pypa-dev mailing list
In-Reply-To: <CAPYWazqn72t9sh7p43yWsDOiiOs3AkcjUPdUZRw_8_OSvvk50Q@mail.gmail.com>
References: <CAPYWazqZVg4mALTAba=MR3jKRoZ0ALZD6-zSjcxPMPPmQOD3ig@mail.gmail.com>
	<kl8o11$983$1@ger.gmane.org>
	<CAPYWazqn72t9sh7p43yWsDOiiOs3AkcjUPdUZRw_8_OSvvk50Q@mail.gmail.com>
Message-ID: <51784DC3.5090304@gmail.com>

Okay, Gmane subscription requests sent with those names.
If there's consensus to create the new list at some point,
consider adding it to Gmane as well (I'll have forgotten by then,
but it's good to make sure early messages are all archived).

> not sure about gmane naming, but my first reaction w/o knowing much
> would be like this:
>
> pip/virtualenv user list
> http://groups.google.com/group/python-virtualenv  ->
>  gmane.comp.python.virtualenv
>
> pypa-dev list
> http://groups.google.com/group/pypa-dev  ->  gmane.comp.python.pypa.dev
>
> as for whether "pypa", or python in general, should have a generic
> packaging user list, that's come up, but no consensus as of yet, so no
> changes so far.
> (btw, the ideas that have been tossed around:  "packaging-user",
> "pypa-user")
>
> so, I'm not sure about mapping "python-virtualenv" to "pypa.user" at
> the moment like you suggested.
>
> Marcus

http://gmane.org/subscribe.php

From vinay_sajip at yahoo.co.uk  Thu Apr 25 14:30:50 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 25 Apr 2013 12:30:50 +0000 (UTC)
Subject: [Distutils]
	=?utf-8?q?Simplify_426=3A_Deprecate_Author-email_and?=
	=?utf-8?q?=09Maintainer-email?=
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
	<CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
Message-ID: <loom.20130425T141958-977@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:

> splitting them is the right way to go, and also that a multi-file
> input format is likely a better option than trying to cram everything
> into one file.

There are areas of overlap, if you consider "archiver", "builder" and
"installer" roles. For example, the source files and in-package data are
needed by both builder and archiver. The beauty of JSON is that it allows you
to have your cake and eat it, to an extent. For example, if you consider that
inputs to the different roles are dicts, it's not a big deal if you have a
higher-level dict which contains the others as sub-dicts. So I don't see a
big issue with having one file as long as the schema is clearly defined so
that a specific role could pull out the relevant stuff.

Also note that in the PEP 397 implementation (dictConfig) there is already
the ability to have cross-references to shared sections in a dict serialised
as JSON, so there's no need to compromise on DRY even when there are data
overlaps.

I've added some support for dict-based configuration in distlib (a backport
of the generic configuration stuff from 2.7/3.2 dictConfig, to support 2.6)
but the use of that's not yet documented.

Regards,

Vinay Sajip


From dholth at gmail.com  Thu Apr 25 15:58:13 2013
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 25 Apr 2013 09:58:13 -0400
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <loom.20130425T141958-977@post.gmane.org>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
	<CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
	<loom.20130425T141958-977@post.gmane.org>
Message-ID: <CAG8k2+5OJVvcWQDYR4aK9yr_d-28FS9zJ1-Br-jhTL7d3PzNJA@mail.gmail.com>

I would prefer to see PEP 390 withdrawn and I think this has been
suggested before. The metadata is already sourced from different files
depending on your build system.

The .ini format and our parsers for it are really awful. I always
resented having to learn it in order to use distutils/setuptools when
every other language (RFC822, Python, JSON) is both better and already
familiar.

FWIW bdist_wheel does something half-PEP-390 inspired with setup.cfg:

[metadata]
provides-extra =
    tool
    signatures
    faster-signatures
requires-dist =
    distribute (>= 0.6.34)
    argparse; python_version == '2.6'
    keyring; extra == 'signatures'
    dirspec; sys.platform != 'win32' and extra == 'signatures'
    ed25519ll; extra == 'faster-signatures'
license-file = LICENSE.txt

(https://bitbucket.org/dholth/wheel/src/tip/setup.cfg?at=default)


On Thu, Apr 25, 2013 at 8:30 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>
>> splitting them is the right way to go, and also that a multi-file
>> input format is likely a better option than trying to cram everything
>> into one file.
>
> There are areas of overlap, if you consider "archiver", "builder" and
> "installer" roles. For example, the source files and in-package data are
> needed by both builder and archiver. The beauty of JSON is that it allows you
> to have your cake and eat it, to an extent. For example, if you consider that
> inputs to the different roles are dicts, it's not a big deal if you have a
> higher-level dict which contains the others as sub-dicts. So I don't see a
> big issue with having one file as long as the schema is clearly defined so
> that a specific role could pull out the relevant stuff.
>
> Also note that in the PEP 397 implementation (dictConfig) there is already
> the ability to have cross-references to shared sections in a dict serialised
> as JSON, so there's no need to compromise on DRY even when there are data
> overlaps.
>
> I've added some support for dict-based configuration in distlib (a backport
> of the generic configuration stuff from 2.7/3.2 dictConfig, to support 2.6)
> but the use of that's not yet documented.
>
> Regards,
>
> Vinay Sajip
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

From ronaldoussoren at mac.com  Thu Apr 25 16:22:09 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 25 Apr 2013 16:22:09 +0200
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CAG8k2+5OJVvcWQDYR4aK9yr_d-28FS9zJ1-Br-jhTL7d3PzNJA@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
	<CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
	<loom.20130425T141958-977@post.gmane.org>
	<CAG8k2+5OJVvcWQDYR4aK9yr_d-28FS9zJ1-Br-jhTL7d3PzNJA@mail.gmail.com>
Message-ID: <EB45AB51-B80C-4065-82E6-CE6A30C8BD93@mac.com>


On 25 Apr, 2013, at 15:58, Daniel Holth <dholth at gmail.com> wrote:

> I would prefer to see PEP 390 withdrawn and I think this has been
> suggested before. The metadata is already sourced from different files
> depending on your build system.


I wouldn't mind a PEP 390 update when the 2.0 metadata format is done
that limits it explicitly to distutils (and possibly setuptools), with the 
express purpose of making it possible to use a setup.cfg with a minimal 
setup.py file when using a new enough distutils version.

In particular, a python distribution without extensions shouldn't require
anything beyond this:

   from distutils.core import setup

   setup()

This would make it easier to teach (GUI) tools, like IDLE, to maintain
the input data used by distutils because they won't have to try to
parse and update Python code.

Using PEP 390 beyond distutils/setuptools isn't useful, there is no need
to restrict other tools in this way as long as they can generate output
in the right format (wheel and sdist archives).


> 
> The .ini format and our parsers for it are really awful. I always
> resented having to learn it in order to use distutils/setuptools when
> every other language (RFC822, Python, JSON) is both better and already
> familiar.

Funnily, I tend to use ini files a lot at $work because the are good
enough for simple configuration and power-users/admins with a windows
background are more likely to understand them then json or (shudder) xml
files.

> 
> FWIW bdist_wheel does something half-PEP-390 inspired with setup.cfg:
> 
> [metadata]
> provides-extra =
>    tool
>    signatures
>    faster-signatures
> requires-dist =
>    distribute (>= 0.6.34)
>    argparse; python_version == '2.6'
>    keyring; extra == 'signatures'
>    dirspec; sys.platform != 'win32' and extra == 'signatures'
>    ed25519ll; extra == 'faster-signatures'
> license-file = LICENSE.txt

I use something simular in a number of my packages, with a single setup.py 
file shared between them and all metadata in setup.cfg.

Ronald

> 
> (https://bitbucket.org/dholth/wheel/src/tip/setup.cfg?at=default)
> 
> 
> On Thu, Apr 25, 2013 at 8:30 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>> 
>>> splitting them is the right way to go, and also that a multi-file
>>> input format is likely a better option than trying to cram everything
>>> into one file.
>> 
>> There are areas of overlap, if you consider "archiver", "builder" and
>> "installer" roles. For example, the source files and in-package data are
>> needed by both builder and archiver. The beauty of JSON is that it allows you
>> to have your cake and eat it, to an extent. For example, if you consider that
>> inputs to the different roles are dicts, it's not a big deal if you have a
>> higher-level dict which contains the others as sub-dicts. So I don't see a
>> big issue with having one file as long as the schema is clearly defined so
>> that a specific role could pull out the relevant stuff.
>> 
>> Also note that in the PEP 397 implementation (dictConfig) there is already
>> the ability to have cross-references to shared sections in a dict serialised
>> as JSON, so there's no need to compromise on DRY even when there are data
>> overlaps.
>> 
>> I've added some support for dict-based configuration in distlib (a backport
>> of the generic configuration stuff from 2.7/3.2 dictConfig, to support 2.6)
>> but the use of that's not yet documented.
>> 
>> Regards,
>> 
>> Vinay Sajip
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


From mal at egenix.com  Thu Apr 25 16:42:50 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 25 Apr 2013 16:42:50 +0200
Subject: [Distutils] HTTPS and certificate check update for distribute ?
Message-ID: <517940EA.1020908@egenix.com>

The latest pip supports HTTPS URLs and certificate checks
(according to the change log).

Will there be a release of distribute that implements the
same changes ?

The current 0.6.36 still defaults to the HTTP PyPI address
and doesn't do certificate checks.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 25 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2013-04-17: Released eGenix mx Base 3.2.6 ...     http://egenix.com/go43

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ncoghlan at gmail.com  Thu Apr 25 16:47:25 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Apr 2013 00:47:25 +1000
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <EB45AB51-B80C-4065-82E6-CE6A30C8BD93@mac.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
	<CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
	<loom.20130425T141958-977@post.gmane.org>
	<CAG8k2+5OJVvcWQDYR4aK9yr_d-28FS9zJ1-Br-jhTL7d3PzNJA@mail.gmail.com>
	<EB45AB51-B80C-4065-82E6-CE6A30C8BD93@mac.com>
Message-ID: <CADiSq7f1K7Kr-_CPct8g-50MsEAguMEKfGY+MSXPKn1xYF_-sw@mail.gmail.com>

I plan to reject PEP 390, since that feature won't be added to the stdlib.

The reason we will need at least a minimal replacement for it is so that
there's a standard place to define and retrieve the archiver hook that
creates the sdist (and hence pymeta.json) from a source tarball or VCS
checkout (as well as determine the necessary dependencies for that step).

My reason for wanting to flesh out a more comprehensive pymeta.cfg spec,
though, is as a sanity check on the proposed PEP 426 metadata. If I can't
come up with a clean multi-file input format, then I will be highly
suspicious of the underlying data model.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130426/45b7ba99/attachment.html>

From donald at stufft.io  Thu Apr 25 17:01:41 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 25 Apr 2013 11:01:41 -0400
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CADiSq7f1K7Kr-_CPct8g-50MsEAguMEKfGY+MSXPKn1xYF_-sw@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
	<CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
	<loom.20130425T141958-977@post.gmane.org>
	<CAG8k2+5OJVvcWQDYR4aK9yr_d-28FS9zJ1-Br-jhTL7d3PzNJA@mail.gmail.com>
	<EB45AB51-B80C-4065-82E6-CE6A30C8BD93@mac.com>
	<CADiSq7f1K7Kr-_CPct8g-50MsEAguMEKfGY+MSXPKn1xYF_-sw@mail.gmail.com>
Message-ID: <6EDD281D-CE11-4827-B84F-8FE35D577E6F@stufft.io>


On Apr 25, 2013, at 10:47 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I plan to reject PEP 390, since that feature won't be added to the stdlib.
> 
> The reason we will need at least a minimal replacement for it is so that there's a standard place to define and retrieve the archiver hook that creates the sdist (and hence pymeta.json) from a source tarball or VCS checkout (as well as determine the necessary dependencies for that step).
> 
A minimal file that only requires what's needed for bootstrapping the archiver is fine. 
> My reason for wanting to flesh out a more comprehensive pymeta.cfg spec, though, is as a sanity check on the proposed PEP 426 metadata. If I can't come up with a clean multi-file input format, then I will be highly suspicious of the underlying data model.
> 
I don't see anything wrong with making this spec but I don't think it should be conflated with the PEP for a standard way to bootstrap the archiver. I also don't think it belongs as a PEP as its not a proposal to enhance python just a test of the new metadata and an example of what an archiver _could_ use as its format. 
> Cheers,
> Nick.
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130425/4e44e1a8/attachment-0001.html>

From pombredanne at nexb.com  Thu Apr 25 17:23:03 2013
From: pombredanne at nexb.com (Philippe Ombredanne)
Date: Thu, 25 Apr 2013 17:23:03 +0200
Subject: [Distutils] HTTPS and certificate check update for distribute ?
In-Reply-To: <517940EA.1020908@egenix.com>
References: <517940EA.1020908@egenix.com>
Message-ID: <CAOFm3uE5Km5KBvKPRzii2jjU7SMw4-jg4wysKNzNkYOo0gFVLg@mail.gmail.com>

On Thu, Apr 25, 2013 at 4:42 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> The latest pip supports HTTPS URLs and certificate checks
> (according to the change log).
> Will there be a release of distribute that implements the
> same changes ?
And FWIW, the same question would be relevant for buildout that
depends on distribute now.
--
Philippe Ombredanne

+1 650 799 0949 | pombredanne at nexB.com
DejaCode Enterprise at http://www.dejacode.com
nexB Inc. at http://www.nexb.com

From pje at telecommunity.com  Fri Apr 26 02:12:17 2013
From: pje at telecommunity.com (PJ Eby)
Date: Thu, 25 Apr 2013 20:12:17 -0400
Subject: [Distutils] Environment marker expression types
Message-ID: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>

I was just fiddling with an experimental environment marker
implementation for setuptools, and ran across a bit of a quirk in the
spec.  It says that the value of 'extra' is 'None', but that
comparisons can only be done on strings.

This leads to two problems: first, the syntax doesn't have a way to
spell 'None', so you can't actually compare it to something.  Second,
if you compare it to a string using the equality operators, IIUC
you'll get a TypeError under Python 3.

Should it actually be defaulting 'extra' to an empty string?  That
would actually make a lot more sense, and avoid the issue of
implementing non-string comparisons, None literals, etc.

(Doing extras in this way actually has another problem, btw, which is
that it's insane to do a != comparison on an 'extra'.  And there are
probably other insane operations on extras, because unlike everything
else, the extra would need to be dynamically evaluated.  I think it
would probably be an improvement to remove extras from the environment
marker system altogether and just have a mapping of extras instead,
ala setuptools.  But that can be treated as a separate issue from the
'None' problem.)

From ncoghlan at gmail.com  Fri Apr 26 03:50:47 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Apr 2013 11:50:47 +1000
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
Message-ID: <CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>

On Fri, Apr 26, 2013 at 10:12 AM, PJ Eby <pje at telecommunity.com> wrote:
> I was just fiddling with an experimental environment marker
> implementation for setuptools, and ran across a bit of a quirk in the
> spec.  It says that the value of 'extra' is 'None', but that
> comparisons can only be done on strings.
>
> This leads to two problems: first, the syntax doesn't have a way to
> spell 'None', so you can't actually compare it to something.

It's not explained well in the current PEP, but there's an implied
"extra is None" if extra isn't mentioned in the environment marker (or
there's no environment marker at all)

> Second,
> if you compare it to a string using the equality operators, IIUC
> you'll get a TypeError under Python 3.

Note that == and != don't emit TypeError with non-comparable types -
if both sides return NotImplemented from the corresponding magic
methods, then the comparison is just False for == and True for !=.
It's only the ordering comparisons that now emit TypeError rather than
attempting to guess an appropriate answer.

> Should it actually be defaulting 'extra' to an empty string?  That
> would actually make a lot more sense, and avoid the issue of
> implementing non-string comparisons, None literals, etc.

Once you account for the "extras not mentioned implies extras is
None", the current system is at least self-consistent.

The main advantage of combining the systems is that it allows the
extras to *also* participate in the environment marker system (for
example, only adding certain dependencies if you're on Windows *and*

> (Doing extras in this way actually has another problem, btw, which is
> that it's insane to do a != comparison on an 'extra'.  And there are
> probably other insane operations on extras, because unlike everything
> else, the extra would need to be dynamically evaluated.  I think it
> would probably be an improvement to remove extras from the environment
> marker system altogether and just have a mapping of extras instead,
> ala setuptools.  But that can be treated as a separate issue from the
> 'None' problem.)

While working on the latest PEP 426 update, I've actually been
pondering the problem of how to handle the environment marker system
in general. Specifically, I *don't* really want anyone to need to
check any environment markers in the metadata passed to the
post-install hook, or even in the metadata recorded in the
installation database. The installer should be resolving all of those
at install time. However, then you get the interesting question of
what happens if you share an installation database across
heterogeneous machines via a network file system, implying that either
these things *have* to be evaluated at run time, solely in order to
accommodate that niche use case, or else we need to constrain that use
case by saying that it is up to the people setting it up to ensure
that the environments are *sufficiently* similar for things to keep
working.

Anyway, I have a draft design that I plan to use for the initial
rewrite, but it's one of the areas where I'll be most interested in
robust feedback:

1. All unconditional metadata goes in the top level fields
2. A new "conditional" top level field is added
3. The new field points to a mapping that uses environment markers as keys
4. Assuming that the only fields that participate in the environment
marker system are represented as lists (as is currently the case),
then the values in the conditional mapping will themselves be mappings
from field names to lists that describe the additional entries to be
appended to create the full list.
5. The installer evaluates the conditional metadata as appropriate
before writing the metadata out to the installation database and
invoking the post-install hook. Post-installation, the top level
metadata includes both the unconditional metadata *and* any
conditional metadata that applies to the current system.

For example, if the project metadata looked like this:

    "requires": ["projectA", "projectB"]
    "conditional": {
        "sys.platform == 'win32'": {
            "requires": ["pywin32 (>1.0)"]
        }
    }

Then the installed metadata would be unchanged on Linux or Mac OS X,
but would look like this on Windows:

    "requires": ["projectA", "projectB", "pywin32 (>1.0)"]
    "conditional": {
        "sys.platform == 'win32'": {
            "requires": ["pywin32 (>1.0)"]
        }
    }

This *does* mean I am inclined to split out the "extras" system as its
own top-level field that recapitulates this structure in miniature:
the extras fields would always remain separate from the top-level
ones, but could also include a set of conditional entries that were
evaluated and potentially added at install time:

    "extras" : {
        "test" : {
            "requires": ["projectA", "projectB"]
            "conditional": {
                "sys.platform == 'win32'": {
                    "requires": ["pywin32 (>1.0)"]
                }
            }
        }
    }

Anyway, this kind of problem is part of the reason the rewrite is
taking a while. Structured metadata provides the opportunity to make
things cleaner and more consistent compared to the relatively ad hoc
approaches in metadata 1.2, but it's still an ongoing challenge to
avoid runaway complexity.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Fri Apr 26 03:56:58 2013
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 25 Apr 2013 21:56:58 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
Message-ID: <CAG8k2+6V2+J1L2EGrXLJouFce3Bq=8JFoRyksRQUJBmsi_yiug@mail.gmail.com>

On Thu, Apr 25, 2013 at 8:12 PM, PJ Eby <pje at telecommunity.com> wrote:
> I was just fiddling with an experimental environment marker
> implementation for setuptools, and ran across a bit of a quirk in the
> spec.  It says that the value of 'extra' is 'None', but that
> comparisons can only be done on strings.
>
> This leads to two problems: first, the syntax doesn't have a way to
> spell 'None', so you can't actually compare it to something.  Second,
> if you compare it to a string using the equality operators, IIUC
> you'll get a TypeError under Python 3.
>
> Should it actually be defaulting 'extra' to an empty string?  That
> would actually make a lot more sense, and avoid the issue of
> implementing non-string comparisons, None literals, etc.
>
> (Doing extras in this way actually has another problem, btw, which is
> that it's insane to do a != comparison on an 'extra'.  And there are
> probably other insane operations on extras, because unlike everything
> else, the extra would need to be dynamically evaluated.  I think it
> would probably be an improvement to remove extras from the environment
> marker system altogether and just have a mapping of extras instead,
> ala setuptools.  But that can be treated as a separate issue from the
> 'None' problem.)

I wondered when someone was going to bring that up.

Markerlib, currently a part of distribute as _markerlib, works by
compiling each unique conditional (environment marker) to a function
using the AST module. One of the arguments to the function is "extra"
and the rest are the ordinary environment marker names like os and
python version.

(Markerlib also takes advantage of the fact that in AST/eval land
variable names can have dots in them without triggering attribute
access which I think is kindof cool.)

Once the marker is compiled it is really fast to evaluate a few times
which we do for extras (number of extras declared + 1 times). The way
they are evaluated  / defined is designed to limit mischief like
installing different requirements if two extras are requested at once
instead of installing one extra, and then later installing the other.

It wouldn't make sense to write "None" in the environment marker
itself. We ask you not to mention "extra" at all in that case. Once
it's a function I can pass None as a parameter to guarantee it won't
match anything you wrote in your marker.

Long story short IMO the concrete requirements fall out more or less
effortlessly. I liked flattening the per-environment and per-extras
conditional requirements into a single mechanism, and I had no choice
pre-JSON-metadata.


Hello Nick!

It could also be more like

requires :  [ { condition : "foo == 'gurgle'", requirements : [ "bar",
"baz" ] } , { requirements : [ "quux", "splort" ] } ]

for a conditional and unconditional requirement.

This syntax would also avoid putting data in keys which seems to be fashionable.

From ncoghlan at gmail.com  Fri Apr 26 04:22:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Apr 2013 12:22:18 +1000
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CAG8k2+6V2+J1L2EGrXLJouFce3Bq=8JFoRyksRQUJBmsi_yiug@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CAG8k2+6V2+J1L2EGrXLJouFce3Bq=8JFoRyksRQUJBmsi_yiug@mail.gmail.com>
Message-ID: <CADiSq7e=wHOvku0h3xxtf2pgGfLWtYjFiiL9a9hn_=Lmze6SkA@mail.gmail.com>

On Fri, Apr 26, 2013 at 11:56 AM, Daniel Holth <dholth at gmail.com> wrote:
> requires :  [ { condition : "foo == 'gurgle'", requirements : [ "bar",
> "baz" ] } , { requirements : [ "quux", "splort" ] } ]
>
> for a conditional and unconditional requirement.
>
> This syntax would also avoid putting data in keys which seems to be fashionable.

Good point, I realised "conditional" would be better as a list of
mappings with conditions as a field rather than a mapping with
conditions as keys:


    "requires": ["projectA", "projectB"]
    "conditional": [
        {
            "condition": "sys.platform == 'win32'",
            "requires": ["pywin32 (>1.0)"]
        }
   ]

I definitely prefer separating out the conditional data from the
unconditional data, though - keep in mind that this spec also defines
the runtime API for accessing this info as structured metadata.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From pje at telecommunity.com  Fri Apr 26 08:05:18 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 26 Apr 2013 02:05:18 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
Message-ID: <CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>

On Thu, Apr 25, 2013 at 9:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Note that == and != don't emit TypeError with non-comparable types -
> if both sides return NotImplemented from the corresponding magic
> methods, then the comparison is just False for == and True for !=.
> It's only the ordering comparisons that now emit TypeError rather than
> attempting to guess an appropriate answer.

Ah.  Ok, good.

> The main advantage of combining the systems is that it allows the
> extras to *also* participate in the environment marker system (for
> example, only adding certain dependencies if you're on Windows *and*

I think you accidentally a word, there.  ;-)

ISTM that it isn't necessary for 'extra' to be part of environment
markers in order to allow extras to have environment markers; as long
as environment markers are part of requirements, and extras define
requirements, that's more than sufficient.  ISTM that extras in
environment markers is a kludge to avoid special syntax for extras,
and if we're switching to a structured format, there's no real problem
with having a dedicated structure for extras.

(Incidentally, this would also improve performance, since it would
then not be necessary to parse all conditionals in order to find out
what extras are available!)


> While working on the latest PEP 426 update, I've actually been
> pondering the problem of how to handle the environment marker system
> in general. Specifically, I *don't* really want anyone to need to
> check any environment markers in the metadata passed to the
> post-install hook, or even in the metadata recorded in the
> installation database. The installer should be resolving all of those
> at install time. However, then you get the interesting question of
> what happens if you share an installation database across
> heterogeneous machines via a network file system, implying that either
> these things *have* to be evaluated at run time, solely in order to
> accommodate that niche use case, or else we need to constrain that use
> case by saying that it is up to the people setting it up to ensure
> that the environments are *sufficiently* similar for things to keep
> working.

I'd just as soon stick with dynamic; in fact I've just implemented
that in a setuptools dev branch today.  Basically, I extended the
syntax of requires.txt to allow sections to be named e.g.
"[ssl:python_version in '2.3, 2.4, 2.5']".  That is, "[extra:marker]",
where 'extra' can be blank to designate non-extra conditional
requirements, and an extra can have multiple sections, with or without
conditionals.

The reason I'm implementing this isn't as a speculative feature, it's
because the setuptools SSL support I'm working on has
platform-specific requirements (wincertstore, ctypes) despite
setuptools *itself* being platform-independent.  This is kind of an
ugly hole in the way dependencies work today: you can't sanely package
a platform-independent package with platform-specific dependencies.
So I need some way of doing this in order to ship SSL support in
setuptools 0.6c12 (and of course 0.7+).

As it happens, the code to implement marker handling was short; <120
lines, for code that works across 2.3-2.7 and will probably work
unchanged for 3.x.  And the evaluation only needs to take place when
dependencies are actually processed, which is pretty darn infrequent
in most programs.


> 5. The installer evaluates the conditional metadata as appropriate
> before writing the metadata out to the installation database and
> invoking the post-install hook. Post-installation, the top level
> metadata includes both the unconditional metadata *and* any
> conditional metadata that applies to the current system.

ISTM that this does away with some of the benefits of e.g. fat wheels
and the ability to share source across multiple Python versions (as
distros sometimes like to do).  It also seems to have very little
gain, since if you are reading the metadata, you've probably already
imported distlib anyway.  ;-)

In general, I favor the minimum possible transformations between
wheels and on-disk format, simply because it means fewer conditional
paths in APIs that need to support both.  If you're going to resolve
the dependencies in a wheel, you have to parse the conditionals, so
again it doesn't save much to not parse them.

If you want to increase parsing efficiency in the presence of
conditionals, I would suggest nesting the conditionals in individual
fields, or at least using a structure like:

    { 'conditionals': {'requires': [....], 'other-field': [...]}}

As this means that you only have to parse the conditions for fields
you're actually accessing.  That keeps parsing to a minimum if a given
query is only looking at unconditional fields.

In general, I don't really see performance as a big deal, though: on
an old 1.4Ghz Athlon running 2.x, I can parse and evaluate the marker
"sys.platform=='win32' and python_version in '2.3, 2.4'" over 10000
times per second, without having made any serious attempts at speeding
that up, yet.

Conditionals are not all that common (and don't exist *as*
conditionals in the wild as yet), and actual requirement resolution is
not a high-frequency event, especially in a non-egg or buildout-based
environment.  (Buildout hardcodes dependencies in scripts, and non-egg
environments don't usually evaluate *any* dependencies at runtime, so
only package management tools will pay the price of evaluation.)

But of course, this is all based on setuptools' requires.txt approach,
which doesn't need markers in order to know what extras exist.  OTOH,
if you code extras *only* as markers, then it does indeed create a big
(and wholly unnecesary) can of worms with respect to parsing.

So, my suggestion would be to drop the overloading of extras as
markers, and structure all conditional data such that only the
conditionals needed to satisfy a particular inquiry ever need to be
parsed.  This should optimize the common case (infrequent dependency
parsing and infrequent conditionals in a simple unpacked structure)
while still enabling advanced cases such as cross-version or
cross-platform sharing to work correctly.

From ncoghlan at gmail.com  Fri Apr 26 08:52:50 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Apr 2013 16:52:50 +1000
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
Message-ID: <CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>

On Fri, Apr 26, 2013 at 4:05 PM, PJ Eby <pje at telecommunity.com> wrote:
> On Thu, Apr 25, 2013 at 9:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> 5. The installer evaluates the conditional metadata as appropriate
>> before writing the metadata out to the installation database and
>> invoking the post-install hook. Post-installation, the top level
>> metadata includes both the unconditional metadata *and* any
>> conditional metadata that applies to the current system.
>
> ISTM that this does away with some of the benefits of e.g. fat wheels
> and the ability to share source across multiple Python versions (as
> distros sometimes like to do).  It also seems to have very little
> gain, since if you are reading the metadata, you've probably already
> imported distlib anyway.  ;-)

OK, I'm convinced - no install time transforms, if you need the full
set of dependencies at run time, you have to poke around in the
conditionals field to see if there is anything relevant.

> In general, I favor the minimum possible transformations between
> wheels and on-disk format, simply because it means fewer conditional
> paths in APIs that need to support both.  If you're going to resolve
> the dependencies in a wheel, you have to parse the conditionals, so
> again it doesn't save much to not parse them.
>
> If you want to increase parsing efficiency in the presence of
> conditionals, I would suggest nesting the conditionals in individual
> fields, or at least using a structure like:
>
>     { 'conditionals': {'requires': [....], 'other-field': [...]}}
>
> As this means that you only have to parse the conditions for fields
> you're actually accessing.  That keeps parsing to a minimum if a given
> query is only looking at unconditional fields.

Yes, I think I prefer this layout for runtime consumption. Getting the
full value for a field would then just be something like:

    # Core dependencies
    deps = meta["requires"]
    # Check dependencies for any specified optional components
    for extra_name in extras_of_interest:
        extra = meta["extras"].get(extra_name)
        if not extra:
            continue
        deps.extend(meta["extras"].get("requires", ()))
    # Check conditional dependencies
    maybe_deps = meta["conditional"].get("requires")
    if maybe_deps:
        for dep in maybe_deps:
            if some_marker_evaluation_api(dep["condition"]):
                deps.extend(dep["entry"])

Good enough for a first draft, anyway :)

distlib would probably evolve some metadata abstraction that hides all
that fiddling behind a property.

> So, my suggestion would be to drop the overloading of extras as
> markers, and structure all conditional data such that only the
> conditionals needed to satisfy a particular inquiry ever need to be
> parsed.  This should optimize the common case (infrequent dependency
> parsing and infrequent conditionals in a simple unpacked structure)
> while still enabling advanced cases such as cross-version or
> cross-platform sharing to work correctly.

Yes, I'm definitely sold on that part.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From erik.m.bray at gmail.com  Fri Apr 26 19:57:27 2013
From: erik.m.bray at gmail.com (Erik Bray)
Date: Fri, 26 Apr 2013 13:57:27 -0400
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
In-Reply-To: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
Message-ID: <CAOTD34bmZv6dtKsJd_0PpAvtVoWdTOQxzgnKpPFEA-1vsFzEww@mail.gmail.com>

Hi, I just wanted to point out a couple small corrections to an
otherwise good summary of the situation.

On Wed, Apr 24, 2013 at 10:55 AM, Erik Bernoth <erik.bernoth at gmail.com> wrote:
> distutils (classic) - the old school, deprecated, pretty much unmaintained
> packaging tooling; still the official standard [4] (Note: reasons [13])

distutils isn't deprecated--almost all of the other mentioned projects
rely on distutils on some level, though there seems to be some effort
to undo that reliance and relegate distutils to just a build tool that
can be swapped out for others.  The problem with distutils is not so
much that it's deprecated, as that there are so many other projects
built on top of it, many of them making use of poorly documented
internal interfaces, that it's very very difficult to change or
improve distutils without breaking lots of other projects.

> setuptools - fork of distutils, which wasn't maintained for a long time but
> will now be merged with it's successor [8].

Setuptools is not a fork of distutils. It's one of the aforementioned
products built on top of distutils that are very volatile to changes
in distutils.  Much of the functionality of setuptools is based on
subclasses of some of the class interfaces in distutils.  But it would
not be accurate to describe it as a fork.  (distribute however *is* a
fork of setuptools).

Thanks,
Erik

From pje at telecommunity.com  Fri Apr 26 22:24:38 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 26 Apr 2013 16:24:38 -0400
Subject: [Distutils] User Viewpoint: Packaging History,
	Confusion and User Advice
In-Reply-To: <loom.20130424T192556-717@post.gmane.org>
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
	<loom.20130424T192556-717@post.gmane.org>
Message-ID: <CALeMXf42av5CM_iYr3brWMXwJrcLj_OCkn-GsmihugAg8Tt6gg@mail.gmail.com>

On Wed, Apr 24, 2013 at 1:33 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Erik Bernoth <erik.bernoth <at> gmail.com> writes:
>> distlib is currently still in alpha and will likely be added under the
>> hood into setuptools anyway, when it's time.
>
> Not sure about that - it's meant to supplant setuptools and avoid setup.py
> altogether, whereas distutils and setuptools/distribute are pretty much
> tied to "python setup.py <command>".

Well, I wouldn't necessarily rule out using it to implement (some
portion of) the pkg_resources API, at least on Python 3, as part of
the phasing-it-out process, after packaging feature development has
ended for setuptools on Python 2, or at least Python <2.6.  (The
distinction being that distlib seems to be heavily reliant on features
that only exist in 2.6+.)

Or, to put it somewhat differently, if pkg_resources ever ends up in
the stdlib, it should probably happen in the form of a distlib
wrapper.  ;-)

From pje at telecommunity.com  Fri Apr 26 22:43:27 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 26 Apr 2013 16:43:27 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
Message-ID: <CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>

On Fri, Apr 26, 2013 at 2:52 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Yes, I think I prefer this layout for runtime consumption. Getting the
> full value for a field would then just be something like:
>
>     # Core dependencies
>     deps = meta["requires"]
>     # Check dependencies for any specified optional components
>     for extra_name in extras_of_interest:
>         extra = meta["extras"].get(extra_name)
>         if not extra:
>             continue
>         deps.extend(meta["extras"].get("requires", ()))
>     # Check conditional dependencies
>     maybe_deps = meta["conditional"].get("requires")
>     if maybe_deps:
>         for dep in maybe_deps:
>             if some_marker_evaluation_api(dep["condition"]):
>                 deps.extend(dep["entry"])
>
> Good enough for a first draft, anyway :)

I think it's missing the part where it needs to get the conditionals
for the extras and extend those as well, but yeah.


> distlib would probably evolve some metadata abstraction that hides all
> that fiddling behind a property.

Indeed.  In pkg_resources' case, there already was such a property,
which I've modified to evaluate the conditionals during the metadata
load, where I expect the I/O overhead to outweigh the 90 or so
microseconds of calculation time to process a platform+python-version
conditional.

Actually, while I'm on that subject, I wonder what the parsing
overhead is going to be for the JSON metadata?  I guess as long as
it's done in C, it'll probably be ok.  I expect the main runtime
performance issue for metadata will just be avoiding reading the
metadata in the first place, by using filename-embedded info so a
directory read can load the critical info for all available packages,
not just the one of current interest.  Beyond that, keeping the
description in a separate file will keep any loads that do happen fast
(how often is the long description currently needed by anything other
than PyPI?), and entry keeping entry points in separate files should
minimize the number of loads when searching for entry points.  (Since
in general the entry points you're looking for are needles in a
haystack, it pays to not have to inspect every stalk of hay, but
instead proceed directly to the bits that are metal.  ;-) )

From vinay_sajip at yahoo.co.uk  Sat Apr 27 00:48:11 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Fri, 26 Apr 2013 22:48:11 +0000 (UTC)
Subject: [Distutils]
	=?utf-8?q?User_Viewpoint=3A_Packaging_History=2C=09Co?=
	=?utf-8?q?nfusion_and_User_Advice?=
References: <CAB46HOkbzMWEFYmtge08xansANWSx1c_4qOfJoOdK6PZ_CpKtw@mail.gmail.com>
	<loom.20130424T192556-717@post.gmane.org>
	<CALeMXf42av5CM_iYr3brWMXwJrcLj_OCkn-GsmihugAg8Tt6gg@mail.gmail.com>
Message-ID: <loom.20130427T004014-676@post.gmane.org>

PJ Eby <pje <at> telecommunity.com> writes:

> ended for setuptools on Python 2, or at least Python <2.6.  (The
> distinction being that distlib seems to be heavily reliant on features
> that only exist in 2.6+.)

The reason for setting 2.6 as the minimum version supported is that one can
easily have a single codebase for 2.6+ and 3.x. Once that minimum is set,
one might as well use the features available, where appropriate.

> Or, to put it somewhat differently, if pkg_resources ever ends up in
> the stdlib, it should probably happen in the form of a distlib
> wrapper.  

The subset of pkg_resources used by pip has already been wrapped in a
compatibility shim called "pip.compat.pkg_resources" [1] which uses distlib
to implement the functions used. With this shim in place all pip tests pass,
but it's early days :-)

However, I view that as a transitional stage which allows pip to gradually
move over to calling distlib directly, while not rocking the boat too much.

Regards,

Vinay Sajip

[1] https://github.com/vsajip/pip/blob/use-distlib/pip/compat/pkg_resources.py


From pje at telecommunity.com  Sat Apr 27 03:15:10 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 26 Apr 2013 21:15:10 -0400
Subject: [Distutils] Are chained comparisons allowed in environment markers?
Message-ID: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>

I notice that distlib appears to support chaining of comparisons, but
the PEP does not.  ISTM that there is no sane use case for chaining,
given the nature of comparisons that are allowed.  (That is, if you
perform one equality or contains comparison between two values, then
you know how the next comparison would turn out.)

If chained comparisons are disallowed in the PEP, then distlib
probably shouldn't be allowing them either, to prevent interop issues
w/tools that only support what's in the PEP.

From techtonik at gmail.com  Sat Apr 27 08:19:23 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 27 Apr 2013 09:19:23 +0300
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <6EDD281D-CE11-4827-B84F-8FE35D577E6F@stufft.io>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
	<CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
	<loom.20130425T141958-977@post.gmane.org>
	<CAG8k2+5OJVvcWQDYR4aK9yr_d-28FS9zJ1-Br-jhTL7d3PzNJA@mail.gmail.com>
	<EB45AB51-B80C-4065-82E6-CE6A30C8BD93@mac.com>
	<CADiSq7f1K7Kr-_CPct8g-50MsEAguMEKfGY+MSXPKn1xYF_-sw@mail.gmail.com>
	<6EDD281D-CE11-4827-B84F-8FE35D577E6F@stufft.io>
Message-ID: <CAPkN8x+Q5CueZLp_NxhMo0v4MXN5rncLqanUDr667FfpJr+yMw@mail.gmail.com>

Thread hijacked.

-- 
anatoly t.


On Thu, Apr 25, 2013 at 6:01 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Apr 25, 2013, at 10:47 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> I plan to reject PEP 390, since that feature won't be added to the stdlib.
>
> The reason we will need at least a minimal replacement for it is so that
> there's a standard place to define and retrieve the archiver hook that
> creates the sdist (and hence pymeta.json) from a source tarball or VCS
> checkout (as well as determine the necessary dependencies for that step).
>
> A minimal file that only requires what's needed for bootstrapping the
> archiver is fine.
>
> My reason for wanting to flesh out a more comprehensive pymeta.cfg spec,
> though, is as a sanity check on the proposed PEP 426 metadata. If I can't
> come up with a clean multi-file input format, then I will be highly
> suspicious of the underlying data model.
>
> I don't see anything wrong with making this spec but I don't think it
> should be conflated with the PEP for a standard way to bootstrap the
> archiver. I also don't think it belongs as a PEP as its not a proposal to
> enhance python just a test of the new metadata and an example of what an
> archiver _could_ use as its format.
>
> Cheers,
> Nick.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/6b115436/attachment-0001.html>

From techtonik at gmail.com  Sat Apr 27 08:39:44 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 27 Apr 2013 09:39:44 +0300
Subject: [Distutils] Simplify 426: Deprecate Author-email and
	Maintainer-email
In-Reply-To: <CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
References: <CAPkN8x+vzbGDyEtSGPV_ibFOKeYJTqLEVOniMN=zBe_uYXASiQ@mail.gmail.com>
	<CAL0kPAWf6vZwrdbUquirYvkPxX9MwezrvNSMytPpDbgetHeqTg@mail.gmail.com>
	<CAG8k2+5Q4QVyArrd2Ac21Qba8_ee3xis2ZgMRAPZ_MLEx4+rDQ@mail.gmail.com>
	<CADiSq7eOKcr7QEm8wEaRtuDMuQmQcK1uZNcuk4h7x-70EEDqNg@mail.gmail.com>
	<0383F2C9-3EA2-4496-BB99-4781803B02F1@stufft.io>
	<F3D6D0D0-5BE6-4430-99C0-D9137CBD0685@stufft.io>
	<CAPkN8x+0aGQao9Tt2oO7coHaJKH7Yw4ze04tTKxMhwWz3KuUKA@mail.gmail.com>
	<837685F4-FE83-42AB-AEDC-A9C5C85E89F8@stufft.io>
	<CADiSq7cjSqX=sDQPZmVbe1VX3hDARSHepBMUXQG=Eznf6LtV1g@mail.gmail.com>
Message-ID: <CAPkN8xKwfzg68s4EeyzDgR7=FuuMDTgrP_ot+s6LHXniFjNfLw@mail.gmail.com>

On Thu, Apr 25, 2013 at 10:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Thu, Apr 25, 2013 at 1:54 PM, Donald Stufft <donald at stufft.io> wrote:
> > I like how Node.js packages handled this. The canonical form is {"name":
> > "Donald Stufft", "email": "donald at stufft.io", "url":
> > "http://github.com/dstufft/"} (and what I think the internal
> representation
> > should be), but the tooling understands strings like "Donald Stufft
> > <donald at stufft.io> (http://github.com/dstufft/)". And both the email
> and url
> > data is optional.
>
> How about the following as the canonical PEP 426 representation:
>
>     "contact": {
>         "name": "Donald Stufft",
>         "email": "donald at stufft.io",
>         "url": "http://github.com/dstufft/"
>     }
>     "contributors": [<list of individual contact entries>]
>
> The point of the neutral "contact" entry is that it may be a larger entity
> like:
>
>     "contact": {
>         "name": "Python Packaging Authority",
>         "email": "pypa-dev at googlegroups.com",
>         "url": "http://github.com/pypa"
>     }
>
> or the distutils-sig+BitBucket variant of that:
>
>    "contact": {
>         "name": "Python Packaging Authority/Distutils SIG",
>         "email": "distutils-sig at python.org",
>         "url": "http://bitbucket.org/pypa"
>     }
>

I guess we're still talking about Author field? I don't like the
substitution of author with contact point. For the software catalog people
need to know other packages made by the author, not the packages made by
contact point.

I also don't like the approach to "normalize metadata format" to the 5th
form. What is the goal of the format? Why it is not recorded in PEP 426. If
the goal is to make DB queries - then normalization is ok. If the goal is
to make it readable then we need to make format easier to type. For DB
catalog you will need to parse metadata and build indexes anyway. There is
no need to embed a bomb to kill all rabbits to the meta format.

For further work on this there needs to be a goal. If the goal is to help
people to see who made what and encourage open source collaboration -
that's one thing. If the goal is to make catalog a corporative asset out of
packages and contact points, then it is another. I'd stick with Author and
Contributor semantic for now and add all other stuff later. If further
formalization is needed - there is a common notion of project People and
Roles. IMO it is a better way to go.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/58e94f9b/attachment.html>

From techtonik at gmail.com  Sat Apr 27 08:49:13 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 27 Apr 2013 09:49:13 +0300
Subject: [Distutils] .ini vs JSON vs YAML meta data format (Was: Simplify
 426: Deprecate Author-email and Maintainer-email)
Message-ID: <CAPkN8xK4V7UyGjKarLfdBXMnns6hOYkkigKfc39x8uGJo5c4Og@mail.gmail.com>

On Thu, Apr 25, 2013 at 4:58 PM, Daniel Holth <dholth at gmail.com> wrote:

> I would prefer to see PEP 390 withdrawn and I think this has been
> suggested before. The metadata is already sourced from different files
> depending on your build system.
>
> The .ini format and our parsers for it are really awful. I always
> resented having to learn it in order to use distutils/setuptools when
> every other language (RFC822, Python, JSON) is both better and already
> familiar.
>

Everyone here has an excellent user stories. I am stunned that there is not
process to help PEP authors collect them? Original user stories are much
better to learn and draft new standards than when reworked into PEP
specification text.


> FWIW bdist_wheel does something half-PEP-390 inspired with setup.cfg:
>
> [metadata]
> provides-extra =
>     tool
>     signatures
>     faster-signatures
> requires-dist =
>     distribute (>= 0.6.34)
>     argparse; python_version == '2.6'
>     keyring; extra == 'signatures'
>     dirspec; sys.platform != 'win32' and extra == 'signatures'
>     ed25519ll; extra == 'faster-signatures'
> license-file = LICENSE.txt
>
> (https://bitbucket.org/dholth/wheel/src/tip/setup.cfg?at=default)


Why reinvent own format when there is already YAML with indented sections?
Yes, I have to see examples to write in this format, but it is readable and
familiar across a broad range of products. It is better than JSON at least,
because it doesn't require to wrap anything in quotes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/e744a7db/attachment.html>

From vinay_sajip at yahoo.co.uk  Sat Apr 27 11:24:35 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 09:24:35 +0000 (UTC)
Subject: [Distutils] .ini vs JSON vs YAML meta data format (Was:
	Simplify 426: Deprecate Author-email and Maintainer-email)
References: <CAPkN8xK4V7UyGjKarLfdBXMnns6hOYkkigKfc39x8uGJo5c4Og@mail.gmail.com>
Message-ID: <loom.20130427T111629-983@post.gmane.org>

anatoly techtonik <techtonik <at> gmail.com> writes:
> Why reinvent own format when there is already YAML with indented sections?
> Yes, I have to see examples to write in this format, but it is readable and
> familiar across a broad range of products. It is better than JSON at least,
> because it doesn't require to wrap anything in quotes.

I too originally thought YAML might be better, but unfortunately the most
mature implementation there is (PyYAML) is still not quite ready, as there
are many open issues around dump() and load(). See for example

http://pyyaml.org/ticket/264

"yaml.load() fails to load a dict just saved by yaml.dump()"

Together with the security issues around YAML (which bit the Rails community
not that long ago) means that JSON is probably a better bet for the moment.

Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Sat Apr 27 11:46:59 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 09:46:59 +0000 (UTC)
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
Message-ID: <loom.20130427T112859-313@post.gmane.org>

PJ Eby <pje <at> telecommunity.com> writes:

> I notice that distlib appears to support chaining of comparisons, but
> the PEP does not.  ISTM that there is no sane use case for chaining,
> given the nature of comparisons that are allowed.  (That is, if you
> perform one equality or contains comparison between two values, then
> you know how the next comparison would turn out.)
> 
> If chained comparisons are disallowed in the PEP, then distlib
> probably shouldn't be allowing them either, to prevent interop issues
> w/tools that only support what's in the PEP.

Sorry if I'm being dense, but can you confirm that what you mean by
"chaining" is something like "a < b <= c"?
 
Things can be tightened up once PEP 426 is finalised. At the moment, distlib
allows expressions like "python_version >= '2.6'" or "python_version < '3.5'"
which are also not mentioned in the PEP. I don't recall seeing any discussion
around why allowing inequalities might be a bad idea, but perhaps someone
can point me to it? Perhaps it should be mentioned in the PEP.

Regards,

Vinay Sajip




From donald at stufft.io  Sat Apr 27 16:07:02 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 27 Apr 2013 10:07:02 -0400
Subject: [Distutils] .ini vs JSON vs YAML meta data format (Was:
	Simplify 426: Deprecate Author-email and Maintainer-email)
In-Reply-To: <loom.20130427T111629-983@post.gmane.org>
References: <CAPkN8xK4V7UyGjKarLfdBXMnns6hOYkkigKfc39x8uGJo5c4Og@mail.gmail.com>
	<loom.20130427T111629-983@post.gmane.org>
Message-ID: <60E93012-15A7-41AC-B755-6BEFFFFC7D82@stufft.io>


On Apr 27, 2013, at 5:24 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> anatoly techtonik <techtonik <at> gmail.com> writes:
>> Why reinvent own format when there is already YAML with indented sections?
>> Yes, I have to see examples to write in this format, but it is readable and
>> familiar across a broad range of products. It is better than JSON at least,
>> because it doesn't require to wrap anything in quotes.
> 
> I too originally thought YAML might be better, but unfortunately the most
> mature implementation there is (PyYAML) is still not quite ready, as there
> are many open issues around dump() and load(). See for example
> 
> http://pyyaml.org/ticket/264
> 
> "yaml.load() fails to load a dict just saved by yaml.dump()"
> 
> Together with the security issues around YAML (which bit the Rails community
> not that long ago) means that JSON is probably a better bet for the moment.
> 
> Regards,
> 
> Vinay Sajip
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


Luckily the JSON is only the format that is used inside of the sdist. It does not need to be the user facing format. PArt of the work is making it so that one tool does not own the entire process so that people are free to make their own tools that generate sdists and wheels and what not. These tools could easily consume YAML and then spit out a sdist/wheel with JSON inside.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/a3081aed/attachment.pgp>

From pje at telecommunity.com  Sat Apr 27 16:47:59 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 27 Apr 2013 10:47:59 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <loom.20130427T112859-313@post.gmane.org>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
Message-ID: <CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>

On Sat, Apr 27, 2013 at 5:46 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Sorry if I'm being dense, but can you confirm that what you mean by
> "chaining" is something like "a < b <= c"?

Yes.


> Things can be tightened up once PEP 426 is finalised.

Not if anybody's going to be using distlib between now and then,
unless you want to break backwards compatibility.  Perhaps distlib is
sufficiently experimental at the moment for this not to be a problem,
but since I'm implementing this in setuptools, backwards compatibility
will be an issue from the moment I release it.  ;-)

> At the moment, distlib
> allows expressions like "python_version >= '2.6'" or "python_version < '3.5'"
> which are also not mentioned in the PEP. I don't recall seeing any discussion
> around why allowing inequalities might be a bad idea, but perhaps someone
> can point me to it? Perhaps it should be mentioned in the PEP.

I don't know of any reasons myself.  Perhaps it was thought that
restricting the syntax would make it easier to implement?  For Python,
that's not especially true.

The marker syntax has some other annoying quirks that make it more
difficult to implement and weird to spell, anyway, like mixing actual
things ('sys.platform') with non-real things (e.g. 'platform.machine'
that's really 'platform.machine()').  It would make a lot more sense
(and simplify implementations) to just say 'platform_machine' or even
'machine' in the first place.

OTOH, I hope that we can finalize at least the environment marker
syntax pronto, so I don't end up with a dead-end version of the
syntax.

From ncoghlan at gmail.com  Sat Apr 27 16:57:28 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 Apr 2013 00:57:28 +1000
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
Message-ID: <CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>

On Sat, Apr 27, 2013 at 6:43 AM, PJ Eby <pje at telecommunity.com> wrote:
> Actually, while I'm on that subject, I wonder what the parsing
> overhead is going to be for the JSON metadata?  I guess as long as
> it's done in C, it'll probably be ok.  I expect the main runtime
> performance issue for metadata will just be avoiding reading the
> metadata in the first place, by using filename-embedded info so a
> directory read can load the critical info for all available packages,
> not just the one of current interest.  Beyond that, keeping the
> description in a separate file will keep any loads that do happen fast
> (how often is the long description currently needed by anything other
> than PyPI?)

The current draft still has the long description in the main metadata
file, but you're right, replacing that with a filename relative to the
metadata file may be a better idea. I'll mark it as an open question.

> and entry keeping entry points in separate files should
> minimize the number of loads when searching for entry points.  (Since
> in general the entry points you're looking for are needles in a
> haystack, it pays to not have to inspect every stalk of hay, but
> instead proceed directly to the bits that are metal.  ;-) )

Keeping extensions (which will include entry points) in a separate
file is another potentially useful idea.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Sat Apr 27 17:00:06 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 27 Apr 2013 11:00:06 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
Message-ID: <3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>


On Apr 27, 2013, at 10:57 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Sat, Apr 27, 2013 at 6:43 AM, PJ Eby <pje at telecommunity.com> wrote:
>> Actually, while I'm on that subject, I wonder what the parsing
>> overhead is going to be for the JSON metadata?  I guess as long as
>> it's done in C, it'll probably be ok.  I expect the main runtime
>> performance issue for metadata will just be avoiding reading the
>> metadata in the first place, by using filename-embedded info so a
>> directory read can load the critical info for all available packages,
>> not just the one of current interest.  Beyond that, keeping the
>> description in a separate file will keep any loads that do happen fast
>> (how often is the long description currently needed by anything other
>> than PyPI?)
> 
> The current draft still has the long description in the main metadata
> file, but you're right, replacing that with a filename relative to the
> metadata file may be a better idea. I'll mark it as an open question.
> 
>> and entry keeping entry points in separate files should
>> minimize the number of loads when searching for entry points.  (Since
>> in general the entry points you're looking for are needles in a
>> haystack, it pays to not have to inspect every stalk of hay, but
>> instead proceed directly to the bits that are metal.  ;-) )
> 
> Keeping extensions (which will include entry points) in a separate
> file is another potentially useful idea.
> 
> Cheers,
> Nick.

And here is where I wish JSON had a link type so that this data could either be inlined, or a link to the externalized data could be included.

> 
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/cd54f47d/attachment.pgp>

From ncoghlan at gmail.com  Sat Apr 27 17:12:45 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 Apr 2013 01:12:45 +1000
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
Message-ID: <CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>

On Sun, Apr 28, 2013 at 12:47 AM, PJ Eby <pje at telecommunity.com> wrote:
> On Sat, Apr 27, 2013 at 5:46 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> At the moment, distlib
>> allows expressions like "python_version >= '2.6'" or "python_version < '3.5'"
>> which are also not mentioned in the PEP. I don't recall seeing any discussion
>> around why allowing inequalities might be a bad idea, but perhaps someone
>> can point me to it? Perhaps it should be mentioned in the PEP.
>
> I don't know of any reasons myself.  Perhaps it was thought that
> restricting the syntax would make it easier to implement?  For Python,
> that's not especially true.

I don't know for sure either - I pretty much inherited it as is from
PEP 345, and otherwise haven't messed with it. I already have a note
in the next draft saying that we might want access to the version
specifier comparisons to make it easier to place constraints on the
Python version.

> The marker syntax has some other annoying quirks that make it more
> difficult to implement and weird to spell, anyway, like mixing actual
> things ('sys.platform') with non-real things (e.g. 'platform.machine'
> that's really 'platform.machine()').  It would make a lot more sense
> (and simplify implementations) to just say 'platform_machine' or even
> 'machine' in the first place.
>
> OTOH, I hope that we can finalize at least the environment marker
> syntax pronto, so I don't end up with a dead-end version of the
> syntax.

I have my hands full with other aspects of the PEP 426 update, but if
you and Vinay want to thrash out a modified version of the environment
marker syntax (such as always using underscores for names which don't
correspond directly to usable Python expressions), I'd certainly be
willing to consider it. It's pretty decoupled from the rest of the
PEP, which just says in a couple of places "environment marker strings
can go here", and then defines the syntax for them once.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Apr 27 17:17:43 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 Apr 2013 01:17:43 +1000
Subject: [Distutils] Environment marker expression types
In-Reply-To: <3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
	<3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
Message-ID: <CADiSq7d2nJB6q-f0osWb+B=nbjSkBCQNM+53G245ZfsCVsWmyQ@mail.gmail.com>

On Sun, Apr 28, 2013 at 1:00 AM, Donald Stufft <donald at stufft.io> wrote:
>> Keeping extensions (which will include entry points) in a separate
>> file is another potentially useful idea.
>>
>> Cheers,
>> Nick.
>
> And here is where I wish JSON had a link type so that this data could either be inlined, or a link to the externalized data could be included.

If we move any content out to a separate file, the field itself will
stay, it will just contain a path relative to the metadata file. (It
does raise an interesting question in terms of filesystem abstractions
and the import system, but a "resolve metadata link" API could
probably handle that in the long run).

It's not especially pretty, but it's better than having to use
os.listdir and rely on predefined file names.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Sat Apr 27 17:34:28 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sat, 27 Apr 2013 11:34:28 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
Message-ID: <CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>

Surely getting farther away from python by trying to prohibit useless
makers just makes the implementation needlessly complex.
On Apr 27, 2013 11:13 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

> On Sun, Apr 28, 2013 at 12:47 AM, PJ Eby <pje at telecommunity.com> wrote:
> > On Sat, Apr 27, 2013 at 5:46 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>
> wrote:
> >> At the moment, distlib
> >> allows expressions like "python_version >= '2.6'" or "python_version <
> '3.5'"
> >> which are also not mentioned in the PEP. I don't recall seeing any
> discussion
> >> around why allowing inequalities might be a bad idea, but perhaps
> someone
> >> can point me to it? Perhaps it should be mentioned in the PEP.
> >
> > I don't know of any reasons myself.  Perhaps it was thought that
> > restricting the syntax would make it easier to implement?  For Python,
> > that's not especially true.
>
> I don't know for sure either - I pretty much inherited it as is from
> PEP 345, and otherwise haven't messed with it. I already have a note
> in the next draft saying that we might want access to the version
> specifier comparisons to make it easier to place constraints on the
> Python version.
>
> > The marker syntax has some other annoying quirks that make it more
> > difficult to implement and weird to spell, anyway, like mixing actual
> > things ('sys.platform') with non-real things (e.g. 'platform.machine'
> > that's really 'platform.machine()').  It would make a lot more sense
> > (and simplify implementations) to just say 'platform_machine' or even
> > 'machine' in the first place.
> >
> > OTOH, I hope that we can finalize at least the environment marker
> > syntax pronto, so I don't end up with a dead-end version of the
> > syntax.
>
> I have my hands full with other aspects of the PEP 426 update, but if
> you and Vinay want to thrash out a modified version of the environment
> marker syntax (such as always using underscores for names which don't
> correspond directly to usable Python expressions), I'd certainly be
> willing to consider it. It's pretty decoupled from the rest of the
> PEP, which just says in a couple of places "environment marker strings
> can go here", and then defines the syntax for them once.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/64d7f0cf/attachment.html>

From vinay_sajip at yahoo.co.uk  Sat Apr 27 17:57:30 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 15:57:30 +0000 (UTC)
Subject: [Distutils]
	=?utf-8?q?Are_chained_comparisons_allowed_in_environm?=
	=?utf-8?q?ent=09markers=3F?=
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
Message-ID: <loom.20130427T175412-232@post.gmane.org>

Daniel Holth <dholth <at> gmail.com> writes:

> Surely getting farther away from python by trying to prohibit useless
> makers just makes the implementation needlessly complex.

I'm not quite sure what you mean by "useless" markers. For example, distlib
checks for e.g. "'2' == '2'" and e.g. "'os.name' == 'os.name'" as these are
pointless to include in an environment marker, but that's only on the basis
that (probable) errors shouldn't pass silently.

Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Sat Apr 27 18:03:19 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 16:03:19 +0000 (UTC)
Subject: [Distutils] Environment marker expression types
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
Message-ID: <loom.20130427T175855-800@post.gmane.org>

PJ Eby <pje <at> telecommunity.com> writes:

> not just the one of current interest.  Beyond that, keeping the
> description in a separate file will keep any loads that do happen fast
> (how often is the long description currently needed by anything other
> than PyPI?), and entry keeping entry points in separate files should
> minimize the number of loads when searching for entry points.  (Since
> in general the entry points you're looking for are needles in a
> haystack, it pays to not have to inspect every stalk of hay, but
> instead proceed directly to the bits that are metal.   )

Presumably the overhead for entry points is only relevant for scanning
installed distributions looking for entry points? In my JSON metadata format,
I currently have the entry points data as an "exports" sub-dictionary in the
main meta-dictionary, but this is written to a separate file during
installation so that only that small part of the metadata needs to be checked
when e.g. iterating over entry points in a group.

Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Sat Apr 27 18:15:15 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 16:15:15 +0000 (UTC)
Subject: [Distutils] Environment marker expression types
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
	<3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
Message-ID: <loom.20130427T180348-232@post.gmane.org>

Donald Stufft <donald <at> stufft.io> writes:

> And here is where I wish JSON had a link type so that this data could
> either be inlined, or a link to the externalized data could be included.

While raw JSON does not include a link type, this needn't be a problem in
practice. For example, distlib already contains mechanisms which support
cross-references and references to external data, as per the example code in
[1]. I will shortly be adding to this a mechanism to integrate data in
external files in a standardised way.

The distlib code is an extension of the mechanisms already in the stdlib in
logging.config and used by dictConfig, which are documented at [2].

Regards,

Vinay Sajip

[1] https://gist.github.com/vsajip/5473612
[2]
http://docs.python.org/2/library/logging.config.html#access-to-internal-objects



From dholth at gmail.com  Sat Apr 27 18:16:44 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sat, 27 Apr 2013 12:16:44 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <loom.20130427T175412-232@post.gmane.org>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
Message-ID: <CAG8k2+6RSb3N9iz23AZGWdjz=gywaXEfW7JWqM43_2biENk0jg@mail.gmail.com>

It seems slow to check. Do we also check for python_version == 'squid' and
throw an error because the version can't parse? I think adults should be
able to write true and false in these silly ways.

+1 on always using _ instead of.l . In makers though.
On Apr 27, 2013 11:57 AM, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:

> Daniel Holth <dholth <at> gmail.com> writes:
>
> > Surely getting farther away from python by trying to prohibit useless
> > makers just makes the implementation needlessly complex.
>
> I'm not quite sure what you mean by "useless" markers. For example, distlib
> checks for e.g. "'2' == '2'" and e.g. "'os.name' == 'os.name'" as these
> are
> pointless to include in an environment marker, but that's only on the basis
> that (probable) errors shouldn't pass silently.
>
> Regards,
>
> Vinay Sajip
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/d63a0cf7/attachment-0001.html>

From dholth at gmail.com  Sat Apr 27 18:54:18 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sat, 27 Apr 2013 12:54:18 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <loom.20130427T180348-232@post.gmane.org>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
	<3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
	<loom.20130427T180348-232@post.gmane.org>
Message-ID: <CAG8k2+42QAvjTGi397S3h7inoX-WZZvOSYD_zVT0yXE-UKCLRQ@mail.gmail.com>

Do we really have to optimize the metadata for runtime parsing, or can
we have a distlib-specific indexing operation triggered by the
installer?

On Sat, Apr 27, 2013 at 12:15 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Donald Stufft <donald <at> stufft.io> writes:
>
>> And here is where I wish JSON had a link type so that this data could
>> either be inlined, or a link to the externalized data could be included.
>
> While raw JSON does not include a link type, this needn't be a problem in
> practice. For example, distlib already contains mechanisms which support
> cross-references and references to external data, as per the example code in
> [1]. I will shortly be adding to this a mechanism to integrate data in
> external files in a standardised way.
>
> The distlib code is an extension of the mechanisms already in the stdlib in
> logging.config and used by dictConfig, which are documented at [2].
>
> Regards,
>
> Vinay Sajip
>
> [1] https://gist.github.com/vsajip/5473612
> [2]
> http://docs.python.org/2/library/logging.config.html#access-to-internal-objects
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

From pje at telecommunity.com  Sat Apr 27 19:11:35 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 27 Apr 2013 13:11:35 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CADiSq7d2nJB6q-f0osWb+B=nbjSkBCQNM+53G245ZfsCVsWmyQ@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
	<3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
	<CADiSq7d2nJB6q-f0osWb+B=nbjSkBCQNM+53G245ZfsCVsWmyQ@mail.gmail.com>
Message-ID: <CALeMXf5T2RgjcwrqWnicFV9N-h0h0+Z2W6gxWWOp+EL-8gQeGw@mail.gmail.com>

On Sat, Apr 27, 2013 at 11:17 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> It's not especially pretty, but it's better than having to use
> os.listdir and rely on predefined file names.

For entry points at least, having a predefined file name is the
*entire point*: i.e. to allow the presence of entry points to be
detected without having to read the metadata.  If you have to read the
metadata to find the filename, then you have to read the whole
haystack (metadata for every package) to find the needles (the ones
that have entry points).

From pje at telecommunity.com  Sat Apr 27 19:41:34 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 27 Apr 2013 13:41:34 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <loom.20130427T175412-232@post.gmane.org>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
Message-ID: <CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>

On Sat, Apr 27, 2013 at 11:57 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Daniel Holth <dholth <at> gmail.com> writes:
>
>> Surely getting farther away from python by trying to prohibit useless
>> makers just makes the implementation needlessly complex.
>
> I'm not quite sure what you mean by "useless" markers. For example, distlib
> checks for e.g. "'2' == '2'" and e.g. "'os.name' == 'os.name'" as these are
> pointless to include in an environment marker, but that's only on the basis
> that (probable) errors shouldn't pass silently.

My test suite actually contains checks like that in order to verify
and/or logic, i.e., a bunch of "'x'=='x' and 'x'=='y'" type stuff, to
ensure that operator precedence is handled correctly, so I wouldn't
want to prohibit those in the spec.

I'm definitely in favor of switching to '_'; it'd let me delete some
lines from my implementation as well as making things clearer, and on
top of that it restores compatibility with the PEP 390 marker syntax.
(For whatever good that is, given that it's apparently being rejected
soon.  ;-) )

Anyway, I propose keeping the current spec but explicitly
*prohibiting* chained comparisons and concatenated string constants
(e.g. 'os.name=="win" "32"', which is valid Python 2 code.)  I would
also happily prohibit the use of '\' in strings, as that would let me
get rid of an eval in my implementation.  ;-)

Hm.  Actually i can get rid of that eval already if I use an
appropriate codec to decode the string...  but that brings up another
important question: are environment markers Unicode on Python 2?  Do
we need to support any non-ASCII characters in any of the provided
variables?  What happens if someone uses an r'', u'', or b'' string
constant?

I think we need to be a bit pickier about what subset of Python we're
supporting, so that we don't end up with implementation-defined
behavior.  Ideally, this would include dumbing strings down to ASCII
without backslashes or prefix characters.  If in the future a
non-ASCII usecase occurs for these strings, we could loosen the spec
then. (Hopefully after setuptools is out of the picture, since it's
still stuck using 8-bit files with unspecified encodings.  ;-) )

From dholth at gmail.com  Sat Apr 27 19:45:57 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sat, 27 Apr 2013 13:45:57 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
Message-ID: <CAG8k2+6eAjEQ=kNmHZPrds+KOUARiy3wPeRQSUb0TvS0E-dz2g@mail.gmail.com>

How do you feel about parenthesis?

It's probably not that hard to prohibit chained comparisons. Do we need an
abnf?

What's wrong with eval?
On Apr 27, 2013 1:42 PM, "PJ Eby" <pje at telecommunity.com> wrote:

> On Sat, Apr 27, 2013 at 11:57 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>
> wrote:
> > Daniel Holth <dholth <at> gmail.com> writes:
> >
> >> Surely getting farther away from python by trying to prohibit useless
> >> makers just makes the implementation needlessly complex.
> >
> > I'm not quite sure what you mean by "useless" markers. For example,
> distlib
> > checks for e.g. "'2' == '2'" and e.g. "'os.name' == 'os.name'" as these
> are
> > pointless to include in an environment marker, but that's only on the
> basis
> > that (probable) errors shouldn't pass silently.
>
> My test suite actually contains checks like that in order to verify
> and/or logic, i.e., a bunch of "'x'=='x' and 'x'=='y'" type stuff, to
> ensure that operator precedence is handled correctly, so I wouldn't
> want to prohibit those in the spec.
>
> I'm definitely in favor of switching to '_'; it'd let me delete some
> lines from my implementation as well as making things clearer, and on
> top of that it restores compatibility with the PEP 390 marker syntax.
> (For whatever good that is, given that it's apparently being rejected
> soon.  ;-) )
>
> Anyway, I propose keeping the current spec but explicitly
> *prohibiting* chained comparisons and concatenated string constants
> (e.g. 'os.name=="win" "32"', which is valid Python 2 code.)  I would
> also happily prohibit the use of '\' in strings, as that would let me
> get rid of an eval in my implementation.  ;-)
>
> Hm.  Actually i can get rid of that eval already if I use an
> appropriate codec to decode the string...  but that brings up another
> important question: are environment markers Unicode on Python 2?  Do
> we need to support any non-ASCII characters in any of the provided
> variables?  What happens if someone uses an r'', u'', or b'' string
> constant?
>
> I think we need to be a bit pickier about what subset of Python we're
> supporting, so that we don't end up with implementation-defined
> behavior.  Ideally, this would include dumbing strings down to ASCII
> without backslashes or prefix characters.  If in the future a
> non-ASCII usecase occurs for these strings, we could loosen the spec
> then. (Hopefully after setuptools is out of the picture, since it's
> still stuck using 8-bit files with unspecified encodings.  ;-) )
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/19c6ea2a/attachment.html>

From vinay_sajip at yahoo.co.uk  Sat Apr 27 20:35:56 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 19:35:56 +0100 (BST)
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
Message-ID: <1367087756.88201.YahooMailNeo@web171406.mail.ir2.yahoo.com>


> My test suite actually contains checks like that in order to verify

> and/or logic, i.e., a bunch of "'x'=='x' and 
> 'x'=='y'" type stuff, to
> ensure that operator precedence is handled correctly, so I wouldn't
> want to prohibit those in the spec.

That's OK by me, I'm not too hung up on having those checks.

> I'm definitely in favor of switching to '_'; it'd let me delete 
> some
> lines from my implementation as well as making things clearer, and on
> top of that it restores compatibility with the PEP 390 marker syntax.
> (For whatever good that is, given that it's apparently being rejected
> soon.? ;-) )

So IIUC, that just means that the variables we support in expressions are:

* python_version, python_full_version, extra as in the PEP already.
* os_name instead of os.name.
* sys_platform instead of sys.platform
* platform_version instead of platform.version
* platform_machine instead of platform.machine
* platform_python_implementation instead of platform.python_implementation

How about just python_implementation for the last one?

> Anyway, I propose keeping the current spec but explicitly
> *prohibiting* chained comparisons and concatenated string constants
> (e.g. 'os.name=="win" "32"', which is valid Python 2 
> code.)? I would
> also happily prohibit the use of '\' in strings, as that would let 
> me
> get rid of an eval in my implementation.? ;-)

Why should chained comparisons be disallowed? If we assume that inequalities are allowed, then surely "'v1' <= some_var < 'v2'" is a reasonable short-hand for "'v1' <= some_var and some_var < 'v2'"?

> Hm.? Actually i can get rid of that eval already if I use an
> appropriate codec to decode the string...? but that brings up another
> important question: are environment markers Unicode on Python 2?? Do
> we need to support any non-ASCII characters in any of the provided
> variables?? What happens if someone uses an r'', u'', or 
> b'' string
> constant?

I would say that we shouldn't allow r'', u'' or b'' prefixes in the spec. Logically, the values are text, i.e. Unicode. Both stdlib json and simplejson support Unicode escapes (\uxxxx) in string literals and handle them as expected. I don't see why we need the prefixes, since we're reading this data from text files rather than source code.

> I think we need to be a bit pickier about what subset of Python we're
> supporting, so that we don't end up with implementation-defined
> behavior.? Ideally, this would include dumbing strings down to ASCII
> without backslashes or prefix characters.? If in the future a
> non-ASCII usecase occurs for these strings, we could loosen the spec
> then. (Hopefully after setuptools is out of the picture, since it's
> still stuck using 8-bit files with unspecified encodings.? ;-) )

I'd like to know what practical problems there would be with supporting Unicode escapes. As I understand it, you can convert to Unicode on 2.x by just using s.decode('unicode_escape').

Regards,

Vinay


From vinay_sajip at yahoo.co.uk  Sat Apr 27 20:44:09 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 19:44:09 +0100 (BST)
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CAG8k2+42QAvjTGi397S3h7inoX-WZZvOSYD_zVT0yXE-UKCLRQ@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
	<3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
	<loom.20130427T180348-232@post.gmane.org>
	<CAG8k2+42QAvjTGi397S3h7inoX-WZZvOSYD_zVT0yXE-UKCLRQ@mail.gmail.com>
Message-ID: <1367088249.43465.YahooMailNeo@web171405.mail.ir2.yahoo.com>

> From: Daniel Holth <dholth at gmail.com>

> Do we really have to optimize the metadata for runtime parsing, or can
> we have a distlib-specific indexing operation triggered by the
> installer?

It seems a little early to say how that operation would work, since the metadata format is not finalised yet. You could regard the distlib installer code writing the EXPORTS file as a sort of indexing operation, although specific to entry points (exports) only. Obviously this could be generalised to cover other areas.

Regards,

Vinay Sajip

From pje at telecommunity.com  Sat Apr 27 21:19:24 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 27 Apr 2013 15:19:24 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <1367087756.88201.YahooMailNeo@web171406.mail.ir2.yahoo.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
	<1367087756.88201.YahooMailNeo@web171406.mail.ir2.yahoo.com>
Message-ID: <CALeMXf5W2DO4_LpdFc79Kddcvrfb4xSeqo7RbbhrNNom2gATtg@mail.gmail.com>

On Sat, Apr 27, 2013 at 2:35 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> So IIUC, that just means that the variables we support in expressions are:
>
> * python_version, python_full_version, extra as in the PEP already.
> * os_name instead of os.name.
> * sys_platform instead of sys.platform
> * platform_version instead of platform.version
> * platform_machine instead of platform.machine
> * platform_python_implementation instead of platform.python_implementation
>
> How about just python_implementation for the last one?

+1


> Why should chained comparisons be disallowed? If we assume that inequalities are allowed, then surely "'v1' <= some_var < 'v2'" is a reasonable short-hand for "'v1' <= some_var and some_var < 'v2'"?

I was not proposing that we support inequalities.  They aren't so
problematic in themselves, as they are inviting various questions and
issues such as why they don't do any sort of version parsing.


> I'd like to know what practical problems there would be with supporting Unicode escapes. As I understand it, you can convert to Unicode on 2.x by just using s.decode('unicode_escape').

It's not just the escapes, it's supporting Unicode at all.  Does that
mean all of the variables have to be Unicode too, to prevent errors
when non-ASCII characters are present?  Under Python 3 these issues
are all moot because str==unicode.  Under 2.x it's just a giant can of
worms to introduce unicode instances in a library that's all strs at
the moment.

My preferred way to deal with the whole thing is to allow single or
triple-quoted strings, disallow backslashes, and assume ASCII-only.
That lets me avoid eval *and* string decoding on 2.x.  (My
implementation is actually single-source for 2.3 up through 3.2 at
least, but does so by assuming all the strings it handles are the same
type.)

From techtonik at gmail.com  Sat Apr 27 22:09:21 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 27 Apr 2013 23:09:21 +0300
Subject: [Distutils] .ini vs JSON vs YAML meta data format (Was:
 Simplify 426: Deprecate Author-email and Maintainer-email)
In-Reply-To: <loom.20130427T111629-983@post.gmane.org>
References: <CAPkN8xK4V7UyGjKarLfdBXMnns6hOYkkigKfc39x8uGJo5c4Og@mail.gmail.com>
	<loom.20130427T111629-983@post.gmane.org>
Message-ID: <CAPkN8xLaFVjzjQ5zZRf3A7GqESkFa3+ZkDOjzjo1hESDatd5tA@mail.gmail.com>

On Sat, Apr 27, 2013 at 12:24 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote:

> anatoly techtonik <techtonik <at> gmail.com> writes:
> > Why reinvent own format when there is already YAML with indented
> sections?
> > Yes, I have to see examples to write in this format, but it is readable
> and
> > familiar across a broad range of products. It is better than JSON at
> least,
> > because it doesn't require to wrap anything in quotes.
>
> I too originally thought YAML might be better, but unfortunately the most
> mature implementation there is (PyYAML) is still not quite ready, as there
> are many open issues around dump() and load(). See for example
>
> http://pyyaml.org/ticket/264
>
> "yaml.load() fails to load a dict just saved by yaml.dump()"
>

This one is huge and requires example to be run to see the actual error. Is
that a minimal example? I don't even have an idea fails there.


> Together with the security issues around YAML (which bit the Rails
> community
> not that long ago) means that JSON is probably a better bet for the moment.
>

Google AppEngine uses it. Ansible uses it. I'd like to know more about that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/aa30f0e2/attachment.html>

From pje at telecommunity.com  Sun Apr 28 00:11:24 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 27 Apr 2013 18:11:24 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CAG8k2+42QAvjTGi397S3h7inoX-WZZvOSYD_zVT0yXE-UKCLRQ@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
	<3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
	<loom.20130427T180348-232@post.gmane.org>
	<CAG8k2+42QAvjTGi397S3h7inoX-WZZvOSYD_zVT0yXE-UKCLRQ@mail.gmail.com>
Message-ID: <CALeMXf6g_C3BfUwOOBZcwS5PUCcN5OkpVnmQ7fRYVYM68YNcgA@mail.gmail.com>

On Sat, Apr 27, 2013 at 12:54 PM, Daniel Holth <dholth at gmail.com> wrote:
> Do we really have to optimize the metadata for runtime parsing, or can
> we have a distlib-specific indexing operation triggered by the
> installer?

As long as the same thing is done inside a zipped wheel, duplication
of the data is ok by me.  (IOW, if it's a *build-time* step, not an
install step.)  Technically, though, it probably makes the most sense
to do it when the machine-readable metadata is generated, in order to
support in-place builds ala "setup.py develop" and "setup.py test".

From dholth at gmail.com  Sun Apr 28 00:13:36 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sat, 27 Apr 2013 18:13:36 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CALeMXf6g_C3BfUwOOBZcwS5PUCcN5OkpVnmQ7fRYVYM68YNcgA@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
	<3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
	<loom.20130427T180348-232@post.gmane.org>
	<CAG8k2+42QAvjTGi397S3h7inoX-WZZvOSYD_zVT0yXE-UKCLRQ@mail.gmail.com>
	<CALeMXf6g_C3BfUwOOBZcwS5PUCcN5OkpVnmQ7fRYVYM68YNcgA@mail.gmail.com>
Message-ID: <CAG8k2+7TtJZcOnqn3EdqqV89bKb-XfFwTMFKwt8zdmNZHn9K5Q@mail.gmail.com>

So who's going to check how much long strings in json really cost?
On Apr 27, 2013 6:11 PM, "PJ Eby" <pje at telecommunity.com> wrote:

> On Sat, Apr 27, 2013 at 12:54 PM, Daniel Holth <dholth at gmail.com> wrote:
> > Do we really have to optimize the metadata for runtime parsing, or can
> > we have a distlib-specific indexing operation triggered by the
> > installer?
>
> As long as the same thing is done inside a zipped wheel, duplication
> of the data is ok by me.  (IOW, if it's a *build-time* step, not an
> install step.)  Technically, though, it probably makes the most sense
> to do it when the machine-readable metadata is generated, in order to
> support in-place builds ala "setup.py develop" and "setup.py test".
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/25e4de42/attachment-0001.html>

From vinay_sajip at yahoo.co.uk  Sun Apr 28 00:19:48 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 23:19:48 +0100 (BST)
Subject: [Distutils] .ini vs JSON vs YAML meta data format (Was:
	Simplify 426: Deprecate Author-email and Maintainer-email)
In-Reply-To: <CAPkN8xLaFVjzjQ5zZRf3A7GqESkFa3+ZkDOjzjo1hESDatd5tA@mail.gmail.com>
References: <CAPkN8xK4V7UyGjKarLfdBXMnns6hOYkkigKfc39x8uGJo5c4Og@mail.gmail.com>
	<loom.20130427T111629-983@post.gmane.org>
	<CAPkN8xLaFVjzjQ5zZRf3A7GqESkFa3+ZkDOjzjo1hESDatd5tA@mail.gmail.com>
Message-ID: <1367101188.12225.YahooMailNeo@web171402.mail.ir2.yahoo.com>






>This one is huge and requires example to be run to see the actual error. Is that a minimal example? I don't even have an idea fails there.


Obviously small files might work, but that's no help. I didn't have the time to spend to make the example smaller. It is not *that* large, but it is the size of metadata that we have to handle. There are plenty of other issues on the tracker relating to dumping()/loading().

>Google AppEngine uses it. Ansible uses it. I'd like to know more about that.


That doesn't matter. For dealing with packaging metadata (the application I was using it for) it failed (on more than a trivial number of metadata samples) and that means it's not currently fit for the purpose under discussion.
Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Sun Apr 28 00:38:58 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 27 Apr 2013 23:38:58 +0100 (BST)
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <CALeMXf5W2DO4_LpdFc79Kddcvrfb4xSeqo7RbbhrNNom2gATtg@mail.gmail.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
	<1367087756.88201.YahooMailNeo@web171406.mail.ir2.yahoo.com>
	<CALeMXf5W2DO4_LpdFc79Kddcvrfb4xSeqo7RbbhrNNom2gATtg@mail.gmail.com>
Message-ID: <1367102338.71133.YahooMailNeo@web171406.mail.ir2.yahoo.com>

> I was not proposing that we support inequalities.? They aren't so

> problematic in themselves, as they are inviting various questions and
> issues such as why they don't do any sort of version parsing.

I'm not sure that platform_version inequalities make sense to support, because the semantics across different platform versions aren't clear cut, even on a given platform. But as far as python_version is concerned, there is an understanding about compatibility and Python X.Y versions. It doesn't seem like we need to do version parsing for python_version in the same way as we do for packages on PyPI.

> My preferred way to deal with the whole thing is to allow single or
> triple-quoted strings, disallow backslashes, and assume ASCII-only.

Why do we need triple-quoted strings in environment markers?

> That lets me avoid eval *and* string decoding on 2.x.? (My
> implementation is actually single-source for 2.3 up through 3.2 at
> least, but does so by assuming all the strings it handles are the same
> type.)

I thought you wouldn't need eval if you could use e.g. s.decode('unicode_escape') - are there problems with doing this in older Python versions?

Also, given that setuptools and distribute are merging,? I presume that means that there will need to be some more flexibility in dealing with Unicode in 2.x code. I'm also hoping that some use can be made of my distribute3 fork, which runs on 2.x and 3.x without alteration or need for 2to3, and so has to deal with both bytes and Unicode.

Regards,

Vinay Sajip


From pje at telecommunity.com  Sun Apr 28 02:02:31 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 27 Apr 2013 20:02:31 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <1367102338.71133.YahooMailNeo@web171406.mail.ir2.yahoo.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
	<1367087756.88201.YahooMailNeo@web171406.mail.ir2.yahoo.com>
	<CALeMXf5W2DO4_LpdFc79Kddcvrfb4xSeqo7RbbhrNNom2gATtg@mail.gmail.com>
	<1367102338.71133.YahooMailNeo@web171406.mail.ir2.yahoo.com>
Message-ID: <CALeMXf5BYOvgnpz0BoeOs6kD-fe_NOogVuw_YBMjMd56gCkzwg@mail.gmail.com>

On Sat, Apr 27, 2013 at 6:38 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> I was not proposing that we support inequalities.  They aren't so
>
>> problematic in themselves, as they are inviting various questions and
>> issues such as why they don't do any sort of version parsing.
>
> I'm not sure that platform_version inequalities make sense to support, because the semantics across different platform versions aren't clear cut, even on a given platform. But as far as python_version is concerned, there is an understanding about compatibility and Python X.Y versions. It doesn't seem like we need to do version parsing for python_version in the same way as we do for packages on PyPI.

So, can we keep inequalities (and therefore chained comparisons) out
of the spec then?  ;-)


>> My preferred way to deal with the whole thing is to allow single or
>> triple-quoted strings, disallow backslashes, and assume ASCII-only.
>
> Why do we need triple-quoted strings in environment markers?

We don't necessarily; I suggested allowing them only because if you
don't have backslashes and you need two kinds of quotes you'd
otherwise be out of luck.  But we could totally drop those, too.



>> That lets me avoid eval *and* string decoding on 2.x.  (My
>> implementation is actually single-source for 2.3 up through 3.2 at
>> least, but does so by assuming all the strings it handles are the same
>> type.)
>
> I thought you wouldn't need eval if you could use e.g. s.decode('unicode_escape') - are there problems with doing this in older Python versions?

It makes a single-source version of the code tougher, because strings
don't have a decode() method in Python 3.


> Also, given that setuptools and distribute are merging,  I presume that means that there will need to be some more flexibility in dealing with Unicode in 2.x code. I'm also hoping that some use can be made of my distribute3 fork, which runs on 2.x and 3.x without alteration or need for 2to3, and so has to deal with both bytes and Unicode.

I definitely want to move to a single source strategy ASAP, but that's
precisely why I'd prefer not to decode something that's already a
string of version-appropriate type.  ;-)

Mainly, I just want to keep the code size small, without opening too
many interop problems or backward compatibility issues.  If we outlaw
absolutely everything in the first version of the spec (and enforce
it), we don't end up with implementation-defined behavior, but can
still loosen restrictions later if an actual use case appears.

From lele at metapensiero.it  Sun Apr 28 02:05:18 2013
From: lele at metapensiero.it (Lele Gaifax)
Date: Sun, 28 Apr 2013 02:05:18 +0200
Subject: [Distutils] .ini vs JSON vs YAML meta data format
References: <CAPkN8xK4V7UyGjKarLfdBXMnns6hOYkkigKfc39x8uGJo5c4Og@mail.gmail.com>
	<loom.20130427T111629-983@post.gmane.org>
	<CAPkN8xLaFVjzjQ5zZRf3A7GqESkFa3+ZkDOjzjo1hESDatd5tA@mail.gmail.com>
	<1367101188.12225.YahooMailNeo@web171402.mail.ir2.yahoo.com>
Message-ID: <87sj2ba5a9.fsf@nautilus.nautilus>

Vinay Sajip <vinay_sajip at yahoo.co.uk> writes:

>>This one is huge and requires example to be run to see the actual
>>error. Is that a minimal example? I don't even have an idea fails
>>there.
>
>
> Obviously small files might work, but that's no help. I didn't have
> the time to spend to make the example smaller. It is not *that* large,
> but it is the size of metadata that we have to handle. There are
> plenty of other issues on the tracker relating to dumping()/loading().
>

Looking at the script, it exhibits the problem only when you tweak the
style of the emitter, ie for aesthetic reasons I presume, so *that*
cannot be the main reason IMO in this thread's context.

The security issues are a much bigger problem (but there is a
safe_load() that maybe can alleviate those); maybe speed issues are also
worth a consideration.

ciao, lele.
-- 
nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri
real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia.
lele at metapensiero.it  |                 -- Fortunato Depero, 1929.


From pje at telecommunity.com  Sun Apr 28 02:11:48 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 27 Apr 2013 20:11:48 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <CAG8k2+6eAjEQ=kNmHZPrds+KOUARiy3wPeRQSUb0TvS0E-dz2g@mail.gmail.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
	<CAG8k2+6eAjEQ=kNmHZPrds+KOUARiy3wPeRQSUb0TvS0E-dz2g@mail.gmail.com>
Message-ID: <CALeMXf5F3cfuRxHZFqkyNx9mvi3+px7=pUyRPVniyjZCXh=Kbw@mail.gmail.com>

On Sat, Apr 27, 2013 at 1:45 PM, Daniel Holth <dholth at gmail.com> wrote:
> How do you feel about parenthesis?

Without them, you have to express everything in DNF (disjunctive
normal form), which usually means a lot of subexpression duplication
when you want to express something like  "(a or b) and (c or d)"; it
would otherwise expand to: "a and c or a and d or b and c or b and
d"...  assuming I didn't make an error in the expansion.  ;-)  (Also,
it's much harder to read and interpret correctly, even with such a
relatively simple expression, and is absolutely not DRY.)


> It's probably not that hard to prohibit chained comparisons. Do we need an
> abnf?

Maybe.  If so, I would use a subset of the actual Python grammar,
since it would help in translation for implementations that abuse
Python's parser as part of the evaluation process.  (Which is all of
them, at the moment: distlib uses Python 2.6+'s "ast" module, while my
work on setuptools is using the "parser" module, which is available in
at least 2.3 through 3.2.)


> What's wrong with eval?

It's slow, and prompts some people to be paranoid about whether you're
thereby introducing some sort of to-be-discovered-in-future security
hole.  In truth, I've realized that I can do away with it entirely,
anyway; the last usage I had was for string constants, but you can
actually just strip off the quotes.  Using eval was just a way to do
that without needing to examine what kinds of quotes, decode escapes,
etc.

From pje at telecommunity.com  Sun Apr 28 02:27:40 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 27 Apr 2013 20:27:40 -0400
Subject: [Distutils] Environment marker expression types
In-Reply-To: <CAG8k2+7TtJZcOnqn3EdqqV89bKb-XfFwTMFKwt8zdmNZHn9K5Q@mail.gmail.com>
References: <CALeMXf71w9V78Va3h0+Z6uROBp3AqRK9Wk_8Y5z+qqWPOkgG1A@mail.gmail.com>
	<CADiSq7d2Ag+7oh1U-foBFX9bobojXj6H6PWie7xHrV4SojSwdg@mail.gmail.com>
	<CALeMXf4rOJJUR-MVJiEdhF+_mq77hXipnWujhHYMt40Y5idj8g@mail.gmail.com>
	<CADiSq7dWCzc9XnA53v9-eQVfXSGodoaeE15++A4Wry3aMuZ=GQ@mail.gmail.com>
	<CALeMXf6f8i3kYDTfVNb6u3gtTQPOYLiqV2++3EL4PV8qbbny0g@mail.gmail.com>
	<CADiSq7dT8NWhQnGDkSTXzOu-=_a0Gt7Ly8PJMQXoSQoHZtsWGQ@mail.gmail.com>
	<3B032F06-B3DC-47CC-8489-3DD4BF583140@stufft.io>
	<loom.20130427T180348-232@post.gmane.org>
	<CAG8k2+42QAvjTGi397S3h7inoX-WZZvOSYD_zVT0yXE-UKCLRQ@mail.gmail.com>
	<CALeMXf6g_C3BfUwOOBZcwS5PUCcN5OkpVnmQ7fRYVYM68YNcgA@mail.gmail.com>
	<CAG8k2+7TtJZcOnqn3EdqqV89bKb-XfFwTMFKwt8zdmNZHn9K5Q@mail.gmail.com>
Message-ID: <CALeMXf40-fK-PmGPjgZ_k1_3zQuf83ieb5G_MbqeD6w8SAznwA@mail.gmail.com>

On Sat, Apr 27, 2013 at 6:13 PM, Daniel Holth <dholth at gmail.com> wrote:
> So who's going to check how much long strings in json really cost?

I'm not sure how that's relevant, except for READMEs.  Right now the
long description is how you get a decent homepage for your project on
PyPI; but there are no other use cases for that information being
carried in the metadata file.  A good-sized README can add 32K to what
would otherwise be a 1K file, at least in the PKG-INFO format.  Also,
a README usually exists as a separate file to start with; do we really
need to read that file and then embed it in another file, so that it
doesn't have to be uploaded as a second attachment when posting to
PyPI?

Really, a README should be installed as documentation, not as part of
the package metadata.  I don't believe most packaging systems put that
kind of info in metadata; it's typically a line or a paragraph or two
at most, i.e. a description, not a web page.

From donald at stufft.io  Sun Apr 28 02:50:45 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 27 Apr 2013 20:50:45 -0400
Subject: [Distutils] .ini vs JSON vs YAML meta data format
In-Reply-To: <87sj2ba5a9.fsf@nautilus.nautilus>
References: <CAPkN8xK4V7UyGjKarLfdBXMnns6hOYkkigKfc39x8uGJo5c4Og@mail.gmail.com>
	<loom.20130427T111629-983@post.gmane.org>
	<CAPkN8xLaFVjzjQ5zZRf3A7GqESkFa3+ZkDOjzjo1hESDatd5tA@mail.gmail.com>
	<1367101188.12225.YahooMailNeo@web171402.mail.ir2.yahoo.com>
	<87sj2ba5a9.fsf@nautilus.nautilus>
Message-ID: <E92EDFA4-1697-484C-A5A4-6FA7C8F2B0CD@stufft.io>


Reminder that the JSON file format is a data exchange format and does not need to be the user facing format.

We should use JSON because it's already in the Python stdlib (and has been since 2.6) and it does not have any of a numerous amount of issues.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130427/97c9c039/attachment.pgp>

From vinay_sajip at yahoo.co.uk  Sun Apr 28 11:12:16 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sun, 28 Apr 2013 10:12:16 +0100 (BST)
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <CALeMXf5BYOvgnpz0BoeOs6kD-fe_NOogVuw_YBMjMd56gCkzwg@mail.gmail.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
	<1367087756.88201.YahooMailNeo@web171406.mail.ir2.yahoo.com>
	<CALeMXf5W2DO4_LpdFc79Kddcvrfb4xSeqo7RbbhrNNom2gATtg@mail.gmail.com>
	<1367102338.71133.YahooMailNeo@web171406.mail.ir2.yahoo.com>
	<CALeMXf5BYOvgnpz0BoeOs6kD-fe_NOogVuw_YBMjMd56gCkzwg@mail.gmail.com>
Message-ID: <1367140336.28912.YahooMailNeo@web171405.mail.ir2.yahoo.com>



> From: PJ Eby <pje at telecommunity.com>

> So, can we keep inequalities (and therefore chained comparisons) out
> of the spec then?? ;-)

But it seems a reasonable expectation to have "python_version > '2.5'" in the spec, which needs no version parsing to work, and likewise it seems not unreasonable to also allow "'2.4' <= python_version < '2.6'". What's the actual difficulty in allowing this? It seems a reasonable use case.

> It makes a single-source version of the code tougher, because strings
> don't have a decode() method in Python 3.

I must be missing something - why not something like

def ensure_unicode(s):
??? if not isinstance(s, text_type): s = s.decode('unicode_escape')
??? return s

assuming you only ever pass bytestrings or Unicode in s, and text_type is unicode on 2.x and str on 3.x?

> I definitely want to move to a single source strategy ASAP, but that's
> precisely why I'd prefer not to decode something that's already a
> string of version-appropriate type.? ;-)

Sadly, some type checking is unavoidable if you can't control what people pass in to your code, but it seems easy enough to deal with from a pragmatic point of view.

> Mainly, I just want to keep the code size small, without opening too
> many interop problems or backward compatibility issues.? If we outlaw
> absolutely everything in the first version of the spec (and enforce
> it), we don't end up with implementation-defined behavior, but can
> still loosen restrictions later if an actual use case appears.

I'm OK with restricting to ASCII (though, from having done numerous single-source ports, the cross-platform handling of Unicode/bytes seems to be a solved problem), but I don't think inequalities need to be removed. (It's easy to disable in distlib's implementation, so that's not the issue - it's whether it's a useful thing to have in the spec.) Let's see what others say.

Regards,

Vinay


From ncoghlan at gmail.com  Sun Apr 28 15:59:17 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 Apr 2013 23:59:17 +1000
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <1367140336.28912.YahooMailNeo@web171405.mail.ir2.yahoo.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
	<1367087756.88201.YahooMailNeo@web171406.mail.ir2.yahoo.com>
	<CALeMXf5W2DO4_LpdFc79Kddcvrfb4xSeqo7RbbhrNNom2gATtg@mail.gmail.com>
	<1367102338.71133.YahooMailNeo@web171406.mail.ir2.yahoo.com>
	<CALeMXf5BYOvgnpz0BoeOs6kD-fe_NOogVuw_YBMjMd56gCkzwg@mail.gmail.com>
	<1367140336.28912.YahooMailNeo@web171405.mail.ir2.yahoo.com>
Message-ID: <CADiSq7dNegMKAYyMOv6CQf0yhVnThe0YBPDGGc7oHKBBQp4bcw@mail.gmail.com>

On 28 Apr 2013 19:16, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:
>
>
>
> > From: PJ Eby <pje at telecommunity.com>
>
> > So, can we keep inequalities (and therefore chained comparisons) out
> > of the spec then?  ;-)
>
> But it seems a reasonable expectation to have "python_version > '2.5'" in
the spec, which needs no version parsing to work, and likewise it seems not
unreasonable to also allow "'2.4' <= python_version < '2.6'". What's the
actual difficulty in allowing this? It seems a reasonable use case.
>
> > It makes a single-source version of the code tougher, because strings
> > don't have a decode() method in Python 3.
>
> I must be missing something - why not something like
>
> def ensure_unicode(s):
>     if not isinstance(s, text_type): s = s.decode('unicode_escape')
>     return s
>
> assuming you only ever pass bytestrings or Unicode in s, and text_type is
unicode on 2.x and str on 3.x?
>
> > I definitely want to move to a single source strategy ASAP, but that's
> > precisely why I'd prefer not to decode something that's already a
> > string of version-appropriate type.  ;-)
>
> Sadly, some type checking is unavoidable if you can't control what people
pass in to your code, but it seems easy enough to deal with from a
pragmatic point of view.
>
> > Mainly, I just want to keep the code size small, without opening too
> > many interop problems or backward compatibility issues.  If we outlaw
> > absolutely everything in the first version of the spec (and enforce
> > it), we don't end up with implementation-defined behavior, but can
> > still loosen restrictions later if an actual use case appears.
>
> I'm OK with restricting to ASCII (though, from having done numerous
single-source ports, the cross-platform handling of Unicode/bytes seems to
be a solved problem), but I don't think inequalities need to be removed.
(It's easy to disable in distlib's implementation, so that's not the issue
- it's whether it's a useful thing to have in the spec.) Let's see what
others say.

I say no to allowing text based ordered comparisons in environment markers,
due to the inconsistency with the version comparison rules in dependency
specifications. This also rules out chained comparisons in environment
markers.

"Might be useful" is not the relevant criterion at this point. For metadata
2.0, we're aiming more for "essential, given everything we know about
packaging based on our own experience, that of other language communities
and that of Linux distributions".

That said, I can see value in supporting version specifiers (both for
Python and installed distributions) as part of environment markers if a
suitable syntax can be found (perhaps something that looks like a function
call). It is only the use of ordinary Python string comparisons that are
inconsistent with version specifiers that I seriously object to.

Cheers,
Nick.

>
> Regards,
>
> Vinay
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130428/95dec3f2/attachment.html>

From pje at telecommunity.com  Sun Apr 28 20:46:41 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 28 Apr 2013 14:46:41 -0400
Subject: [Distutils] Are chained comparisons allowed in environment
	markers?
In-Reply-To: <1367140336.28912.YahooMailNeo@web171405.mail.ir2.yahoo.com>
References: <CALeMXf6z=3Y0KCN7zyW7+oyUnOdTuGhevjcaemNgsNs9xQmKMA@mail.gmail.com>
	<loom.20130427T112859-313@post.gmane.org>
	<CALeMXf7w89Hv+U0n0bBsQkUcReh--z1U=JfUdcpd-V5J4jYVsw@mail.gmail.com>
	<CADiSq7crB-GDTB1J0-LfT7GvG_yQuDJ6n6Yb_4thsrczx90z9Q@mail.gmail.com>
	<CAG8k2+4hhuaEJ85KvuiVVibaBXmbKqByHreeLpmJCHgyK7k4bg@mail.gmail.com>
	<loom.20130427T175412-232@post.gmane.org>
	<CALeMXf4b4=W-tC2kWuBs8qDdDRtD+Aywf9iRRt3XL4G6vQ5h0Q@mail.gmail.com>
	<1367087756.88201.YahooMailNeo@web171406.mail.ir2.yahoo.com>
	<CALeMXf5W2DO4_LpdFc79Kddcvrfb4xSeqo7RbbhrNNom2gATtg@mail.gmail.com>
	<1367102338.71133.YahooMailNeo@web171406.mail.ir2.yahoo.com>
	<CALeMXf5BYOvgnpz0BoeOs6kD-fe_NOogVuw_YBMjMd56gCkzwg@mail.gmail.com>
	<1367140336.28912.YahooMailNeo@web171405.mail.ir2.yahoo.com>
Message-ID: <CALeMXf44rDtUK-wcQhbmvmSS6t_p-_acV4jUopg4zOfTzBgt+w@mail.gmail.com>

On Sun, Apr 28, 2013 at 5:12 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>
> I must be missing something - why not something like
>
> def ensure_unicode(s):
>     if not isinstance(s, text_type): s = s.decode('unicode_escape')
>     return s
>
> assuming you only ever pass bytestrings or Unicode in s, and text_type is unicode on 2.x and str on 3.x?

Because I don't *have* unicode in 2.x, only bytes.  All the values
coming from os, platform, etc. are bytes, and they don't necessarily
have a specified encoding.  For that matter, the strings I'm parsing
are also bytes, with no specified encoding.  Going to unicode is an
invitation to platform-specific or machine-specific encoding errors.


> Sadly, some type checking is unavoidable if you can't control what people pass in to your code, but it seems easy enough to deal with from a pragmatic point of view.

It's not the type that's the problem, it's the *contents* that make a
difference here.

From setuptools at bugs.python.org  Tue Apr 30 07:49:45 2013
From: setuptools at bugs.python.org (Ian Wienand)
Date: Tue, 30 Apr 2013 05:49:45 +0000
Subject: [Distutils] [issue148] setup_requires dependency removal causes
	installation	failure
Message-ID: <1367300985.17.0.879831056887.issue148@psf.upfronthosting.co.za>


New submission from Ian Wienand:

I noticed the following issue installing the keyring package [1]

---
Downloading/unpacking keyring
  Downloading keyring-1.2.2.zip (79Kb): 79Kb downloaded
  Running setup.py egg_info for package keyring
    zip_safe flag not set; analyzing archive contents...
    
    Installed /tmp/easy_install-u0ESXQ/pytest-runner-1.2/hgtools-2.0.3-py2.6.egg
    
    Installed /root/build/keyring/pytest_runner-1.2-py2.6.egg
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/root/build/keyring/setup.py", line 114, in <module>
        setup_mod.setup(**setup_params)
      File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
        dist.run_commands()
      File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
        self.run_command(cmd)
      File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
        cmd_obj.run()
      File "<string>", line 12, in replacement_run
      File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 254, in find_sources
        mm.run()
      File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 308, in run
        self.add_defaults()
      File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 335, in add_defaults
        rcfiles = list(walk_revctrl())
      File "/usr/lib/python2.6/site-packages/setuptools/command/sdist.py", line 46, in walk_revctrl
        for item in ep.load()(dirname):
      File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load
        entry = __import__(self.module_name, globals(),globals(), ['__name__'])
    ImportError: No module named hgtools.plugins
    Complete output from command python setup.py egg_info:
    zip_safe flag not set; analyzing archive contents...
---

The dependency chain works out to be

keyring -> pytest-runner -> hgtools

hgtools is a "setup_requires" dependency for pytest-runner.  This
means the following code is run:

/usr/lib/python2.6/site-packages/setuptools/dist.py
---
class Distribution(_Distribution):
    def __init__():
     ...
        if attrs and 'setup_requires' in attrs:
            self.fetch_build_eggs(attrs.pop('setup_requires'))

    def fetch_build_eggs(self, requires):
        """Resolve pre-setup requirements"""
        from pkg_resources import working_set, parse_requirements
        for dist in working_set.resolve(
            parse_requirements(requires), installer=self.fetch_build_egg
        ):
            working_set.add(dist)
---

That goes fine, hgtools gets installed into whatever /tmp directory, is added to sys.path and pytest-runner is happy.

However, the temporary hgtools has registered some entry points with its distribution object in working_set.  It then disappears but doesn't remove itself.  Later on in sdist.py we have

---
def walk_revctrl(dirname=''):
    """Find all files under revision control"""
    for ep in pkg_resources.iter_entry_points('setuptools.file_finders'):
        for item in ep.load()(dirname):
            yield item
----

pkg_resources.iter_entry_points walks all the distribution objects in working_set, finds hgtools' entry points for this "setuptools.file_finders" stuff, tries to call it and a backtrace ensues because that temporary directory with hgtools has now gone.

This also explains why it works when run the second time around; pytest-runner is already there, so we don't need to look at its setup_requires and the problematic distribution doesn't get added.

Somehow or other, hgtools needs to get removed from the working_set in dist.py before it is removed?


[1] https://bugzilla.redhat.com/show_bug.cgi?id=924038

----------
messages: 709
nosy: iwienand
priority: bug
status: unread
title: setup_requires dependency removal causes installation failure

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue148>
_______________________________________________

From vinay_sajip at yahoo.co.uk  Tue Apr 30 12:37:16 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 30 Apr 2013 10:37:16 +0000 (UTC)
Subject: [Distutils] distlib 0.1.2 released on PyPI
Message-ID: <loom.20130430T122923-72@post.gmane.org>

I've just released version 0.1.2 of distlib on PyPI. The changes are as follows:

compat.py:

    Added BaseConfigurator backport for 2.6.

database.py:

    Return RECORD path from write_installed_files (or None if dry_run).
    Explicitly return None from write_shared_locations if dry run.

metadata.py:

    Added missing condition in todict().

scripts.py:

    Add variants and clobber flag for generation of foo/fooX/foo-X.Y.
    Added .exe manifests for Windows.

util.py:

    Regularised recording of written files.
    Added Configurator.

version.py:

    Tidyups, most suggested by Donald Stufft: Made key functions private,
    removed _Common class, removed checking for huge version numbers, made
    UnsupportedVersionError a ValueError.

wheel.py:

    Replaced absolute import with relative.
    Handle None return from write_shared_locations correctly.
    Fixed bug in Mounter for extension modules not in sub-packages.
    Made dylib-cache Python version-specific.

docs:

    Numerous documentation updates, not detailed further here.

tests:

    Numerous test refinements, not detailed further here.

other:

    Corrected setup.py to ensure that sysconfig.cfg is included. The
    omission of sysconfig.cfg from the source distribution caused
    failures on 2.6.

Please try it out!

Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Tue Apr 30 15:23:04 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 30 Apr 2013 13:23:04 +0000 (UTC)
Subject: [Distutils] distil 0.1.1 released
Message-ID: <loom.20130430T151213-32@post.gmane.org>

I've released version 0.1.1 of distil, downloadable from [1]. It's based
on distlib 0.1.2. The other changes are as follows:

* Added "distil init" to support creating an initial version of
  package.json metadata, which can then be added to during development.
* Added "distil link" to support "editable" installations, similar to
  pip install -e local_dir.
* Take into account pre-confirmation (-y) during uninstallation when dists
  that are no longer needed are found. These are now removed automatically
  when -y is specified.
* Fixed error in setting up SSL certificate verification, and adjusted PyPI
  URLs to be https:// where specified as http:// in metadata. Successful
  SSL verification is now logged.
* Added --local-dists option to allow local wheels and sdists to be used
  for installation.
* Fixed a bug in the handling of local archives (e.g. those returned
  through a configured DirectoryLocator). Local archives shouldn?t be
  deleted after unpacking.
* Added --python-tags argument to distil package when building wheels to
  configure the tags for the built wheel.
* Added --no-unpack option to distil download.
* Fixed problem with rollback on error caused by not recording SHARED and
  RECORD correctly.
* Fixed bug in writing entry points (EXPORTS) file.
* Use of 2to3 now defaults to True when run under 3.x.
* Fixed bug when run in venvs which caused e.g. dist-packages to be used
  instead of site-packages.
* Improved error message when run with Python 2.5 (not supported, but this
  is now clear from the error message).

Your feedback will be gratefully received!

Regards,

Vinay Sajip

[1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py



From p.f.moore at gmail.com  Tue Apr 30 19:46:50 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 30 Apr 2013 18:46:50 +0100
Subject: [Distutils] distil 0.1.1 released
In-Reply-To: <loom.20130430T151213-32@post.gmane.org>
References: <loom.20130430T151213-32@post.gmane.org>
Message-ID: <CACac1F8snZCT=rMwQKfPLrJnSVmTpqp78HMZ2WefmLkREQTdug@mail.gmail.com>

On 30 April 2013 14:23, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> 've released version 0.1.1 of distil, downloadable from [1]. It's based
> on distlib 0.1.2. The other changes are as follows:
>
[...]

> Your feedback will be gratefully received!
>

One thought - it would be nice to have an equivalent of pip's --target flag
(install packages into the specified directory) for distil install. I use
this fairly frequently to install packages outside of sys.path, when
creating standalone scripts bundled with their dependencies. Actually, in
that case the metadata isn't much use either, so I often delete that. An
option to not install the metadata directory would also be useful.

I know it's an unusual use-case, but I thought I might as well ask :-)

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130430/9a33ec3f/attachment.html>

From vinay_sajip at yahoo.co.uk  Tue Apr 30 22:11:29 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 30 Apr 2013 20:11:29 +0000 (UTC)
Subject: [Distutils] distil 0.1.1 released
References: <loom.20130430T151213-32@post.gmane.org>
	<CACac1F8snZCT=rMwQKfPLrJnSVmTpqp78HMZ2WefmLkREQTdug@mail.gmail.com>
Message-ID: <loom.20130430T220905-322@post.gmane.org>

Paul Moore <p.f.moore <at> gmail.com> writes:

> One thought - it would be nice to have an equivalent of pip's --target flag
> (install packages into the specified directory) for distil install. I use
> this fairly frequently to install packages outside of sys.path, when
> creating standalone scripts bundled with their dependencies. Actually, in
> that case the metadata isn't much use either, so I often delete that. An
> option to not install the metadata directory would also be useful.

What do you expect to happen with scripts, includes and non-package data? 

Regards,

Vinay Sajip