From dholth at gmail.com  Sun Jul  1 07:52:34 2012
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 1 Jul 2012 01:52:34 -0400
Subject: [Distutils] the 'wheel' binary package format
Message-ID: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>

I got tired of waiting for lxml to compile over and over again, so I
invented a binary packaging format called 'wheel' (of cheese) that
uses Metadata 1.2 and .dist-info directories instead of .egg-info,
patched pkg_resources.py to be able to load .dist-info directories,
implemented "python setup.py bdist_wheel", and patched pip to be able
to install .whl files.

The gist of the spec is that it is a zip file with the .whl extension
containing the contents of 'purelib' for a distribution, plus a
Name-1.0.dist-info/ directory with the metadata files, plus
Name-1.0.data/subdir directories for scripts, platlib, packaging's
"categories", ...

My specification so far is at
https://docs.google.com/document/d/1mWPyvoeiqCrAy4UPNnvaz7Cgrqm4s_cfaTauAeJWABI/edit
and an lxml compiled for linux-x86-64 is at
https://docs.google.com/open?id=0BxHz5bC4iN5TN0VWTFNrZGtCbWs

http://bitbucket.org/dholth/distribute
http://github.com/dholth/pip
http://pypi.python.org/pypi/wheel

Perhaps it will be useful. The implementation is still pretty rough,
for example it does not check the architecture while installing, but
it could be a handy way to speed up repeated virtualenv builds.

Daniel Holth

From chris at simplistix.co.uk  Sun Jul  1 19:15:18 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Sun, 01 Jul 2012 18:15:18 +0100
Subject: [Distutils] [Zope-dev] working with different requirements
 depending on python version
In-Reply-To: <jsnc14$t5t$1@dough.gmane.org>
References: <4FEEF703.2020001@simplistix.co.uk> <jsnc14$t5t$1@dough.gmane.org>
Message-ID: <4FF085A6.2010004@simplistix.co.uk>

On 30/06/2012 18:12, Tres Seaver wrote:
>> How have other people solved this?
>
> For repoze.sendmail, I did two fixups for Python 2.5:
>
> - - I changed 'install_requires' in setup.py based on sys.version_info
>
>    https://github.com/repoze/repoze.sendmail/blob/master/setup.py
>
> - - I pinned the verstion in tox.ini:
>
>    https://github.com/repoze/repoze.sendmail/blob/master/tox.ini

Thanks, that worked for me:

https://github.com/Simplistix/testfixtures/commit/d56035510f7a63793a82b966b8c8c273b379c7a1

https://github.com/Simplistix/checker/commit/a5f8d089a3af7da1f1e8a5dc3182ea5b5345dcf5

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk



From pje at telecommunity.com  Sun Jul  1 20:00:11 2012
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 1 Jul 2012 14:00:11 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
Message-ID: <CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>

On Sun, Jul 1, 2012 at 1:52 AM, Daniel Holth <dholth at gmail.com> wrote:

> I got tired of waiting for lxml to compile over and over again, so I
> invented a binary packaging format called 'wheel' (of cheese) that
> uses Metadata 1.2 and .dist-info directories instead of .egg-info,
> patched pkg_resources.py to be able to load .dist-info directories,
> implemented "python setup.py bdist_wheel", and patched pip to be able
> to install .whl files.
>
> The gist of the spec is that it is a zip file with the .whl extension
> containing the contents of 'purelib' for a distribution, plus a
> Name-1.0.dist-info/ directory with the metadata files, plus
> Name-1.0.data/subdir directories for scripts, platlib, packaging's
> "categories", ...
>
> My specification so far is at
>
> https://docs.google.com/document/d/1mWPyvoeiqCrAy4UPNnvaz7Cgrqm4s_cfaTauAeJWABI/edit
> and an lxml compiled for linux-x86-64 is at
> https://docs.google.com/open?id=0BxHz5bC4iN5TN0VWTFNrZGtCbWs
>
> http://bitbucket.org/dholth/distribute
> http://github.com/dholth/pip
> http://pypi.python.org/pypi/wheel
>
> Perhaps it will be useful. The implementation is still pretty rough,
> for example it does not check the architecture while installing, but
> it could be a handy way to speed up repeated virtualenv builds.
>
>
While I haven't studied the spec in detail yet, I just want to say
"hurray!" for somebody Just Doing Something About It.  From a feature
perspective, the only thing missing (IMO) would be the ability for pip to
consume .egg and win32.exe's in order to provide a migration path.
Win32.exe->egg conversion code exists in easy_install, and converting eggs
to wheels should be straightforward  as well, despite the mixed metaphor.
;-)

Btw, looking at the packaging docs cited in the spec, I'm confused about
something: it's not clear to me what resource category most data files
would be placed in, if they are actually application code.  For example, a
ZCML spec file is not editable and would thus not properly go into
"config".  ZPT files (or for that matter, most other HTML templating
language files) aren't data in any meaningful sense, either.  So what *is*
the correct category for such *code* files which are part of the library or
application, yet which are not Python modules?  The doc provides no
guidance on this point.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120701/5f96e0e0/attachment.html>

From dholth at gmail.com  Sun Jul  1 20:26:29 2012
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 1 Jul 2012 14:26:29 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
Message-ID: <2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>

Thanks.

Most of "wheel" is just implementing the specs we have, like metadata 1.2 and a lot of tarek et al work on packaging. The point of the implementation is that we can immediately benefit from a documented install format without having to convert everyone over to the same build system. The build and install phases can be decoupled as they should be.

The resource categories in .data come from the distutils2/packaging setup.cfg specification. They are mostly for including man pages, include files and so forth. If it doesn't go directly in site packages, it goes there. The wheel spec doesn't know what they are for, it only tells you where to archive them.

I'm convinced that data that the code really needs at runtime should continue to go right next to the .py files as usual. Time will tell as to the utility of .data. I think .data will make Debian maintainers happy.

The new setup command bdist_wheel does exactly the task of converting egg-info to dist-info. It just doesn't understand prebuilt eggs.

The tripartite compatibility tag python version - abi - architecture is what I need the most advice with. I want to anticipate a multi implementation world where the abi is not tied to the version of the python language used, and I want to be able to put several architectures of shared libraries in the same .whl when those shared libraries have unique file names.

Daniel Holth

On Jul 1, 2012, at 2:00 PM, PJ Eby <pje at telecommunity.com> wrote:

> On Sun, Jul 1, 2012 at 1:52 AM, Daniel Holth <dholth at gmail.com> wrote:
> I got tired of waiting for lxml to compile over and over again, so I
> invented a binary packaging format called 'wheel' (of cheese) that
> uses Metadata 1.2 and .dist-info directories instead of .egg-info,
> patched pkg_resources.py to be able to load .dist-info directories,
> implemented "python setup.py bdist_wheel", and patched pip to be able
> to install .whl files.
> 
> The gist of the spec is that it is a zip file with the .whl extension
> containing the contents of 'purelib' for a distribution, plus a
> Name-1.0.dist-info/ directory with the metadata files, plus
> Name-1.0.data/subdir directories for scripts, platlib, packaging's
> "categories", ...
> 
> My specification so far is at
> https://docs.google.com/document/d/1mWPyvoeiqCrAy4UPNnvaz7Cgrqm4s_cfaTauAeJWABI/edit
> and an lxml compiled for linux-x86-64 is at
> https://docs.google.com/open?id=0BxHz5bC4iN5TN0VWTFNrZGtCbWs
> 
> http://bitbucket.org/dholth/distribute
> http://github.com/dholth/pip
> http://pypi.python.org/pypi/wheel
> 
> Perhaps it will be useful. The implementation is still pretty rough,
> for example it does not check the architecture while installing, but
> it could be a handy way to speed up repeated virtualenv builds.
> 
> 
> While I haven't studied the spec in detail yet, I just want to say "hurray!" for somebody Just Doing Something About It.  From a feature perspective, the only thing missing (IMO) would be the ability for pip to consume .egg and win32.exe's in order to provide a migration path.  Win32.exe->egg conversion code exists in easy_install, and converting eggs to wheels should be straightforward  as well, despite the mixed metaphor.  ;-)
> 
> Btw, looking at the packaging docs cited in the spec, I'm confused about something: it's not clear to me what resource category most data files would be placed in, if they are actually application code.  For example, a ZCML spec file is not editable and would thus not properly go into "config".  ZPT files (or for that matter, most other HTML templating language files) aren't data in any meaningful sense, either.  So what *is* the correct category for such *code* files which are part of the library or application, yet which are not Python modules?  The doc provides no guidance on this point.
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120701/011983d1/attachment.html>

From robert.kern at gmail.com  Sun Jul  1 20:44:03 2012
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 01 Jul 2012 19:44:03 +0100
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
Message-ID: <jsq5pj$56k$1@dough.gmane.org>

On 7/1/12 6:52 AM, Daniel Holth wrote:
> I got tired of waiting for lxml to compile over and over again, so I
> invented a binary packaging format called 'wheel' (of cheese) that
> uses Metadata 1.2 and .dist-info directories instead of .egg-info,
> patched pkg_resources.py to be able to load .dist-info directories,
> implemented "python setup.py bdist_wheel", and patched pip to be able
> to install .whl files.
>
> The gist of the spec is that it is a zip file with the .whl extension
> containing the contents of 'purelib' for a distribution, plus a
> Name-1.0.dist-info/ directory with the metadata files, plus
> Name-1.0.data/subdir directories for scripts, platlib, packaging's
> "categories", ...
>
> My specification so far is at
> https://docs.google.com/document/d/1mWPyvoeiqCrAy4UPNnvaz7Cgrqm4s_cfaTauAeJWABI/edit
> and an lxml compiled for linux-x86-64 is at
> https://docs.google.com/open?id=0BxHz5bC4iN5TN0VWTFNrZGtCbWs
>
> http://bitbucket.org/dholth/distribute
> http://github.com/dholth/pip
> http://pypi.python.org/pypi/wheel
>
> Perhaps it will be useful. The implementation is still pretty rough,
> for example it does not check the architecture while installing, but
> it could be a handy way to speed up repeated virtualenv builds.

Very nice.

I like how much information the file names provide. Do you think it would be 
possible to add (optional) build numbers to the specification? At Enthought, we 
make a Python distribution with access to a repository of binary eggs. When 
regularly building third party eggs for downstream consumption, it's essential 
to have build numbers in the file name since we need to fix bugs in our builds 
while keeping the upstream version of the package itself the same. Linux 
distributions have the same issue and put in build numbers into their package 
file names.

I'm not sure I'd go so far as to force every .whl to have a build number in it, 
but it would be great if we could figure out a way to have a compatible 
convention that could include build numbers when needed.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco




From qwcode at gmail.com  Sun Jul  1 20:50:46 2012
From: qwcode at gmail.com (Marcus)
Date: Sun, 1 Jul 2012 11:50:46 -0700
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FE96FDA.8030206@ziade.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
Message-ID: <CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>

>
> FWIW I have started a "pypi2rpm" project that creates RPM out of PyPI
> projects, we use at Mozilla to deploy our apps,
> http://pypi.python.org/pypi/**pypi2rpm<http://pypi.python.org/pypi/pypi2rpm>
> It's just a glue script on the top of an isolated bdist_rpm command and
> distutils2.version (for sorting versions) but you can also pass your spec
> file for a given
> project and have Fedora/RHEL specific options (like adding python26-
> prefixes etc)
>
>
Hello Tarek:
if you have anything that details your rpm deployment workflow at mozilla,
i'd love to see it.
I work in a very active build environment, and repetitive"setup.py build"
time is pain point.
currently, we have a pretty fragile virtualenv caching strategy, and I've
even considered patching distutils so that "setup.py build" would somehow
work against a cache.
currently, my mindset is to migrate to a modular rpm install strategy
(managed by puppet) to prevent wasteful build time.

Marcus
P.S. Daniel's "wheel" format is another possible to answer to my concern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120701/97cfc6a8/attachment.html>

From dholth at gmail.com  Sun Jul  1 21:06:45 2012
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 1 Jul 2012 15:06:45 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <jsq5pj$56k$1@dough.gmane.org>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<jsq5pj$56k$1@dough.gmane.org>
Message-ID: <A6DDA0C2-71F9-4C4C-8B62-473462F3A351@gmail.com>


> Very nice.
> 
> I like how much information the file names provide. Do you think it would be possible to add (optional) build numbers to the specification? At Enthought, we make a Python distribution with access to a repository of binary eggs. When regularly building third party eggs for downstream consumption, it's essential to have build numbers in the file name since we need to fix bugs in our builds while keeping the upstream version of the package itself the same. Linux distributions have the same issue and put in build numbers into their package file names.

Ok. Read the python versioning spec. Perhaps it can be extended, or delimited with another character besides -

> 
> I'm not sure I'd go so far as to force every .whl to have a build number in it, but it would be great if we could figure out a way to have a compatible convention that could include build numbers when needed.
> 

From pje at telecommunity.com  Sun Jul  1 22:50:57 2012
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 1 Jul 2012 16:50:57 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
Message-ID: <CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>

On Sun, Jul 1, 2012 at 2:26 PM, Daniel Holth <dholth at gmail.com> wrote:
>
> I'm convinced that data that the code really needs at runtime should
> continue to go right next to the .py files as usual. Time will tell as to
> the utility of .data. I think .data will make Debian maintainers happy.
>
> The new setup command bdist_wheel does exactly the task of converting
> egg-info to dist-info. It just doesn't understand prebuilt eggs.
>

Well, if you do that conversion on the EGG-INFO directory of an .egg file,
then you'll have a wheel.  At least, if I understand your spec correctly.
(Since .egg contains just the code and static files.)  All you need to do
is get the egg's platform info and Python version from its filename;
everything else is in .egg-info.  The full spec is here:

   http://peak.telecommunity.com/DevCenter/EggFormats

But basically, it's just going to be just convert EGG-INFO to
Projectname.dist-info.  The actual layout of the rest of the zipfile is
essentially unchanged.

(Heck, add .whl support to pkg_resources, and wheels will have all the
advantages of eggs as well.  Though, for that to work under 3.x some of the
stub file stuff would have to be changed as well.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120701/2361ec40/attachment.html>

From dholth at gmail.com  Mon Jul  2 00:55:43 2012
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 1 Jul 2012 18:55:43 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
Message-ID: <CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>

On Sun, Jul 1, 2012 at 4:50 PM, PJ Eby <pje at telecommunity.com> wrote:
> On Sun, Jul 1, 2012 at 2:26 PM, Daniel Holth <dholth at gmail.com> wrote:
>>
>> I'm convinced that data that the code really needs at runtime should
>> continue to go right next to the .py files as usual. Time will tell as to
>> the utility of .data. I think .data will make Debian maintainers happy.
>>
>> The new setup command bdist_wheel does exactly the task of converting
>> egg-info to dist-info. It just doesn't understand prebuilt eggs.
>
>
> Well, if you do that conversion on the EGG-INFO directory of an .egg file,
> then you'll have a wheel.  At least, if I understand your spec correctly.
> (Since .egg contains just the code and static files.)  All you need to do is
> get the egg's platform info and Python version from its filename; everything
> else is in .egg-info.  The full spec is here:
>
>    http://peak.telecommunity.com/DevCenter/EggFormats
>
> But basically, it's just going to be just convert EGG-INFO to
> Projectname.dist-info.  The actual layout of the rest of the zipfile is
> essentially unchanged.

Yes. You have to use Metadata 1.2 (list requirements with
Requires-Dist:) and PKG-INFO has been renamed to METADATA by the
distutils-sig for whatever reason. The WHEEL file lets an installer
complain when the next version of the format is invented.

Once it is installed into site-packages it is not a wheel anymore, it
is just a PEP-compliant Python distribution.

> (Heck, add .whl support to pkg_resources, and wheels will have all the
> advantages of eggs as well.  Though, for that to work under 3.x some of the
> stub file stuff would have to be changed as well.)

The pkg_resources in the patched distribute this requires already
understands the unpacked format. This would be a good small project.

From dholth at gmail.com  Mon Jul  2 15:17:22 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 2 Jul 2012 09:17:22 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
Message-ID: <CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>

Shell script to build a directory full of wheels and then install
them. https://gist.github.com/3033186

From p.f.moore at gmail.com  Mon Jul  2 16:07:25 2012
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 2 Jul 2012 15:07:25 +0100
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
Message-ID: <CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>

On 2 July 2012 14:17, Daniel Holth <dholth at gmail.com> wrote:
> Shell script to build a directory full of wheels and then install
> them. https://gist.github.com/3033186

Is wheel for Python 2? I'm trying to build using 3.3a2, and I get:

>.\Scripts\pip install -e hg+https://bitbucket.org/dholth/wheel#egg=wheel -e hg+https://bitbucket.org/dholth/distribute#
egg=distribute -e git+https://github.com/dholth/pip.git#egg=pip
Obtaining wheel from hg+https://bitbucket.org/dholth/wheel#egg=wheel
  Updating d:\data\wheeltest\src\wheel clone
  Running setup.py egg_info for package wheel

Obtaining distribute from
hg+https://bitbucket.org/dholth/distribute#egg=distribute
  Cloning hg https://bitbucket.org/dholth/distribute to
d:\data\wheeltest\src\distribute
  Running setup.py egg_info for package distribute
    Traceback (most recent call last):
      File "<string>", line 3, in <module>
      File "setuptools\__init__.py", line 2, in <module>
        from setuptools.extension import Extension, Library
      File "setuptools\extension.py", line 5, in <module>
        from setuptools.dist import _get_unpatched
      File "setuptools\dist.py", line 103
        except ValueError, e:
                         ^
    SyntaxError: invalid syntax
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 3, in <module>

  File "setuptools\__init__.py", line 2, in <module>

    from setuptools.extension import Extension, Library

  File "setuptools\extension.py", line 5, in <module>

    from setuptools.dist import _get_unpatched

  File "setuptools\dist.py", line 103

    except ValueError, e:

                     ^

SyntaxError: invalid syntax

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in D:\Documents and Settings\UK03306\Application
Data\pip\pip.log

That's Python 2 syntax, so the error is correct. But why is Python 3
finding a Python 2 format file?

Paul.

From dholth at gmail.com  Mon Jul  2 16:24:42 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 2 Jul 2012 10:24:42 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
Message-ID: <CAG8k2+7F2oJxFZTZe9WCOaCEhkYVtB+bsppEGYghiNtYdSkOFg@mail.gmail.com>

On Mon, Jul 2, 2012 at 10:07 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 2 July 2012 14:17, Daniel Holth <dholth at gmail.com> wrote:
>> Shell script to build a directory full of wheels and then install
>> them. https://gist.github.com/3033186
>
> Is wheel for Python 2? I'm trying to build using 3.3a2, and I get:

At the moment, yes. I have yet to set up my tox environment to test in
both, but https://bitbucket.org/dholth/cpython contains a (slightly
older) version of it for 'packaging'.

Daniel

From dholth at gmail.com  Mon Jul  2 16:29:11 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 2 Jul 2012 10:29:11 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CAG8k2+7F2oJxFZTZe9WCOaCEhkYVtB+bsppEGYghiNtYdSkOFg@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
	<CAG8k2+7F2oJxFZTZe9WCOaCEhkYVtB+bsppEGYghiNtYdSkOFg@mail.gmail.com>
Message-ID: <CAG8k2+6hrKk9X8bfiq_M1uWM8A=yYO5igVC85288+TFTM8tZXw@mail.gmail.com>

>> Is wheel for Python 2? I'm trying to build using 3.3a2, and I get:
>
> At the moment, yes. I have yet to set up my tox environment to test in
> both, but https://bitbucket.org/dholth/cpython contains a (slightly
> older) version of it for 'packaging'.

Oops, that's wrong. It just contains some of the precursor work that
was ported over to distribute.

From p.f.moore at gmail.com  Mon Jul  2 16:36:53 2012
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 2 Jul 2012 15:36:53 +0100
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
Message-ID: <CACac1F8kS-oCG1WniAm2zdOhouHHTAkzTXFZskXWvdtqPGR_aQ@mail.gmail.com>

On 2 July 2012 15:07, Paul Moore <p.f.moore at gmail.com> wrote:
> On 2 July 2012 14:17, Daniel Holth <dholth at gmail.com> wrote:
>> Shell script to build a directory full of wheels and then install
>> them. https://gist.github.com/3033186
>
> Is wheel for Python 2? I'm trying to build using 3.3a2, and I get:

Actually, I've hit a number of other issues when I try Python 2,
largely related to (a) the fact that I'm on Windows, so you can't
install pip with itself (can't overwrite the exe), and (b) I don't
have a workable git (so I had to download the patched pip manually)
and (c) it's a develop build, so I can't delete the patched pip once
it's installed (this is mainly because I don't have any familiarity
with develop installs, so I don't know how to change the command to
get a "normal" install - the obvious approach of changing "develop" to
"install" didn't work...). Maybe the Python 3 problem wasn't python 3
as such.

I'll try to get a workable version on Windows when I have some time.
(But I'd still like to know if it's intended to work on Python 3 - if
not, I probably won't bother for now).

Paul.

From dholth at gmail.com  Mon Jul  2 16:51:33 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 2 Jul 2012 10:51:33 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CACac1F8kS-oCG1WniAm2zdOhouHHTAkzTXFZskXWvdtqPGR_aQ@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
	<CACac1F8kS-oCG1WniAm2zdOhouHHTAkzTXFZskXWvdtqPGR_aQ@mail.gmail.com>
Message-ID: <CAG8k2+6LOzJkUo7urwq5nhA0Ry7niBLMwO3TJ610iq5JUrkgug@mail.gmail.com>

> Actually, I've hit a number of other issues when I try Python 2,
> largely related to (a) the fact that I'm on Windows, so you can't
> install pip with itself (can't overwrite the exe), and (b) I don't
> have a workable git (so I had to download the patched pip manually)
> and (c) it's a develop build, so I can't delete the patched pip once
> it's installed (this is mainly because I don't have any familiarity
> with develop installs, so I don't know how to change the command to
> get a "normal" install - the obvious approach of changing "develop" to
> "install" didn't work...). Maybe the Python 3 problem wasn't python 3
> as such.

It is always a pain to use an installer to replace itself. Sorry.

> I'll try to get a workable version on Windows when I have some time.
> (But I'd still like to know if it's intended to work on Python 3 - if
> not, I probably won't bother for now).

Double oops. I have pushed bdist_wheel to my cpython fork, but that
doesn't take care of the installer. I am definitely interested in
porting the distribute version over to Python 3. I just haven't gotten
to it.

From p.f.moore at gmail.com  Mon Jul  2 17:59:51 2012
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 2 Jul 2012 16:59:51 +0100
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CAG8k2+6LOzJkUo7urwq5nhA0Ry7niBLMwO3TJ610iq5JUrkgug@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
	<CACac1F8kS-oCG1WniAm2zdOhouHHTAkzTXFZskXWvdtqPGR_aQ@mail.gmail.com>
	<CAG8k2+6LOzJkUo7urwq5nhA0Ry7niBLMwO3TJ610iq5JUrkgug@mail.gmail.com>
Message-ID: <CACac1F9LdgfBx4wkZ_CavcRGqcvNqbY55KQubUM5oxq=2oJt0A@mail.gmail.com>

On 2 July 2012 15:51, Daniel Holth <dholth at gmail.com> wrote:
> It is always a pain to use an installer to replace itself. Sorry.

Not your fault, it's just the way it is (combined with my lack of
familiarity with any aspect of pip/distribute beyond "pip install
pkg_from_pypi").

> Double oops. I have pushed bdist_wheel to my cpython fork, but that
> doesn't take care of the installer. I am definitely interested in
> porting the distribute version over to Python 3. I just haven't gotten
> to it.

Hmm, I'm a little confused as to how the bits hang together here.
Maybe some clarification would help. How close is the following?

pip - your patch for this is to recognise wheel files and use
distribute to install them
distribute - your patch for this is to add support for installing
wheel files (and to add a bdist_wheel command for building them???)
wheel - support library for the above, provides functions to work with
wheel files

Is that close?

Assuming it is, I'm not sure I follow what you say in the paragraph
above - what is the cpython patch you mention? You can't patch
distutils with bdist_wheel as it's feature-frozen, and packaging is
being taken out of 3.3 (if it hasn't already). I don't see much point
in maintaining a cpython patch that won't get accepted. And I'm not
sure what you mean by "the installer".

Based on my interpretation of what you say above, the things I see
outstanding are:

1. wheel discovery and download from PyPI  or local archives (so pip
install x will find and download x.whl before/as well as an x sdist)
2. wheel upload to PyPI (probably needs PyPI changes to support the new format)
3. converters to allow people to convert eggs and bdist_wininst
binaries to wheel format (otherwise you have a chicken-and-egg problem
in that wheels are only good for speeding up installs, rather than for
installing things you can't build for yourself)

(1) and (2) are longer-term requirements, and (3) may or may not be a
key requirement for you - in any case, others who do need the facility
could develop these independently. So it seems to me like you're
nearly there (assuming the pip and distribute maintainers accept your
patches).

Paul.

From dholth at gmail.com  Mon Jul  2 18:34:31 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 2 Jul 2012 12:34:31 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CACac1F9LdgfBx4wkZ_CavcRGqcvNqbY55KQubUM5oxq=2oJt0A@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
	<CACac1F8kS-oCG1WniAm2zdOhouHHTAkzTXFZskXWvdtqPGR_aQ@mail.gmail.com>
	<CAG8k2+6LOzJkUo7urwq5nhA0Ry7niBLMwO3TJ610iq5JUrkgug@mail.gmail.com>
	<CACac1F9LdgfBx4wkZ_CavcRGqcvNqbY55KQubUM5oxq=2oJt0A@mail.gmail.com>
Message-ID: <CAG8k2+5o6nFCjYXZq+Lo1FNMMM4B5dgnDiOqXO0v2sCMcdPtyA@mail.gmail.com>

Looks like your biggest Py3 issue is that distribute uses 2to3. "pip
install -e" of course does not. It works much better otherwise. I put
in some tiny changes to the other bits to make them work, and I can
now install Pyramid from wheels on Py3.2.

On Mon, Jul 2, 2012 at 11:59 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> Hmm, I'm a little confused as to how the bits hang together here.
> Maybe some clarification would help. How close is the following?
>
> pip - your patch for this is to recognise wheel files and use
> distribute to install them
> distribute - your patch for this is to add support for installing
> wheel files (and to add a bdist_wheel command for building them???)
> wheel - support library for the above, provides functions to work with
> wheel files
>
> Is that close?
>
> Assuming it is, I'm not sure I follow what you say in the paragraph
> above - what is the cpython patch you mention?

No longer important, just the first thing I tried. It is a bdist_wheel
for distutils2 / packaging. It wound up being much more fun to write a
setuptools/distribute plugin, so I finished there.

The most important and most complete patch is probably
pkg_resources.py (part of setuptools / distribute) to recognize the
.dist-info directories and the Provides-Extra: extension. This is just
PEP 386 and is not wheel specific. If 'markerlib' is installed it will
interpret conditional requirements; markerlib should become
standard/required.

The distribution 'wheel' is the bdist_wheel setuptools plugin. It is
also where a command line tool to manipulate wheel files would go.

pip recognizes wheels and installs them itself.
https://github.com/dholth/pip/blob/develop/pip/req.py#L759

It doesn't yet +x scripts or update .dist-info/RECORD to keep track of
what it should uninstall.

> Based on my interpretation of what you say above, the things I see
> outstanding are:
>
> 1. wheel discovery and download from PyPI  or local archives (so pip
> install x will find and download x.whl before/as well as an x sdist)
> 2. wheel upload to PyPI (probably needs PyPI changes to support the new format)

Also, we will be moving to urlsafe-b64encoded SHA256 instead of MD5 as
the hash function.

> 3. converters to allow people to convert eggs and bdist_wininst
> binaries to wheel format (otherwise you have a chicken-and-egg problem
> in that wheels are only good for speeding up installs, rather than for
> installing things you can't build for yourself)

Sure, although IMO egg2wheel is more of a roundabout way to get around
the "pip doesn't support eggs" issue. Hopefully in most cases the
maintainer or "someone with a compiler" will run bdist_wheel for you
when a new version comes out.

> (1) and (2) are longer-term requirements, and (3) may or may not be a
> key requirement for you - in any case, others who do need the facility
> could develop these independently. So it seems to me like you're
> nearly there (assuming the pip and distribute maintainers accept your
> patches).
>
> Paul.

From p.f.moore at gmail.com  Mon Jul  2 19:36:07 2012
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 2 Jul 2012 18:36:07 +0100
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CAG8k2+5o6nFCjYXZq+Lo1FNMMM4B5dgnDiOqXO0v2sCMcdPtyA@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
	<CACac1F8kS-oCG1WniAm2zdOhouHHTAkzTXFZskXWvdtqPGR_aQ@mail.gmail.com>
	<CAG8k2+6LOzJkUo7urwq5nhA0Ry7niBLMwO3TJ610iq5JUrkgug@mail.gmail.com>
	<CACac1F9LdgfBx4wkZ_CavcRGqcvNqbY55KQubUM5oxq=2oJt0A@mail.gmail.com>
	<CAG8k2+5o6nFCjYXZq+Lo1FNMMM4B5dgnDiOqXO0v2sCMcdPtyA@mail.gmail.com>
Message-ID: <CACac1F-0bEKAXg2pwn5vz3nC-a=rQgXCBjsJZRCsRa85_VMikQ@mail.gmail.com>

On 2 July 2012 17:34, Daniel Holth <dholth at gmail.com> wrote:
> Looks like your biggest Py3 issue is that distribute uses 2to3. "pip
> install -e" of course does not. It works much better otherwise. I put
> in some tiny changes to the other bits to make them work, and I can
> now install Pyramid from wheels on Py3.2.

Ah, so that's what -e is about. I can see that would be a problem,
I'll try a normal install and see how that goes.

> pip recognizes wheels and installs them itself.
> https://github.com/dholth/pip/blob/develop/pip/req.py#L759
>
> It doesn't yet +x scripts or update .dist-info/RECORD to keep track of
> what it should uninstall.

So pip remove won't work right on installed wheels? That would
probably be my #1 issue, in practice, then.

>> 1. wheel discovery and download from PyPI  or local archives (so pip
>> install x will find and download x.whl before/as well as an x sdist)
>> 2. wheel upload to PyPI (probably needs PyPI changes to support the new format)
>
> Also, we will be moving to urlsafe-b64encoded SHA256 instead of MD5 as
> the hash function.
>
>> 3. converters to allow people to convert eggs and bdist_wininst
>> binaries to wheel format (otherwise you have a chicken-and-egg problem
>> in that wheels are only good for speeding up installs, rather than for
>> installing things you can't build for yourself)
>
> Sure, although IMO egg2wheel is more of a roundabout way to get around
> the "pip doesn't support eggs" issue. Hopefully in most cases the
> maintainer or "someone with a compiler" will run bdist_wheel for you
> when a new version comes out.

Yes, if wheels become the accepted binary format. But I'd expect a
relatively long period when some developers use wheels, but a lot
don't. During that period, converters would be essential to enable end
users to start using wheels, and hence to start the process of pushing
developers to provide wheels.

But certainly converters are an interim measure. (Just an important one IMO).

Paul.

From dholth at gmail.com  Mon Jul  2 19:54:34 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 2 Jul 2012 13:54:34 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CACac1F-0bEKAXg2pwn5vz3nC-a=rQgXCBjsJZRCsRa85_VMikQ@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
	<CACac1F8kS-oCG1WniAm2zdOhouHHTAkzTXFZskXWvdtqPGR_aQ@mail.gmail.com>
	<CAG8k2+6LOzJkUo7urwq5nhA0Ry7niBLMwO3TJ610iq5JUrkgug@mail.gmail.com>
	<CACac1F9LdgfBx4wkZ_CavcRGqcvNqbY55KQubUM5oxq=2oJt0A@mail.gmail.com>
	<CAG8k2+5o6nFCjYXZq+Lo1FNMMM4B5dgnDiOqXO0v2sCMcdPtyA@mail.gmail.com>
	<CACac1F-0bEKAXg2pwn5vz3nC-a=rQgXCBjsJZRCsRa85_VMikQ@mail.gmail.com>
Message-ID: <CAG8k2+4v79uqFnS8md7pZdCFOvD3KbcYmF2wdFQXKgRzueJWRQ@mail.gmail.com>

>> It doesn't yet +x scripts or update .dist-info/RECORD to keep track of
>> what it should uninstall.
>
> So pip remove won't work right on installed wheels? That would
> probably be my #1 issue, in practice, then.

Very easy to fix pip remove though, just another elif: in pip.req
uninstall(). I also intend to define an entry point for post-install
hooks generally.

> But certainly converters are an interim measure. (Just an important one IMO).

You can use the issue tracker on http://bitbucket.org/dholth/wheel to
help build a roadmap

From dholth at gmail.com  Tue Jul  3 03:11:28 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 2 Jul 2012 21:11:28 -0400
Subject: [Distutils] the 'wheel' binary package format
In-Reply-To: <CAG8k2+4v79uqFnS8md7pZdCFOvD3KbcYmF2wdFQXKgRzueJWRQ@mail.gmail.com>
References: <CAG8k2+4YMgSDbjVsOY8J1qwkjoOdKLD28Gnp5RYDNMH8iesK5Q@mail.gmail.com>
	<CALeMXf6Mpi2VDwmwx7E-iY1VvnHb3-1cWiAT2SxXK6u4bJOMqw@mail.gmail.com>
	<2E7E9E57-F909-4390-AADE-BD6C15890EFF@gmail.com>
	<CALeMXf49yU2A5eQ+=FeV5Zu34j6z7WYJ=28s3z+8eT9HqP+nAg@mail.gmail.com>
	<CAG8k2+7N3xO4AP+yzF20kvuwh3s1jxOi09LjmyXnTh972Lw0bQ@mail.gmail.com>
	<CAG8k2+7DCf69uXQLYXDTKk__Q1OYaaTZ+Rj3iEekDdZeHsqT=A@mail.gmail.com>
	<CACac1F8fEpbezCn+FX3nbPP9mGi0o=2eYLfOob7M2DnfaQZV2w@mail.gmail.com>
	<CACac1F8kS-oCG1WniAm2zdOhouHHTAkzTXFZskXWvdtqPGR_aQ@mail.gmail.com>
	<CAG8k2+6LOzJkUo7urwq5nhA0Ry7niBLMwO3TJ610iq5JUrkgug@mail.gmail.com>
	<CACac1F9LdgfBx4wkZ_CavcRGqcvNqbY55KQubUM5oxq=2oJt0A@mail.gmail.com>
	<CAG8k2+5o6nFCjYXZq+Lo1FNMMM4B5dgnDiOqXO0v2sCMcdPtyA@mail.gmail.com>
	<CACac1F-0bEKAXg2pwn5vz3nC-a=rQgXCBjsJZRCsRa85_VMikQ@mail.gmail.com>
	<CAG8k2+4v79uqFnS8md7pZdCFOvD3KbcYmF2wdFQXKgRzueJWRQ@mail.gmail.com>
Message-ID: <CAG8k2+6zbM_cgsCSf01=EtcD-2A6=AoXNk4BCs8VOV0-hAbjxQ@mail.gmail.com>

'pip uninstall' is implemented.

On Mon, Jul 2, 2012 at 1:54 PM, Daniel Holth <dholth at gmail.com> wrote:
>>> It doesn't yet +x scripts or update .dist-info/RECORD to keep track of
>>> what it should uninstall.
>>
>> So pip remove won't work right on installed wheels? That would
>> probably be my #1 issue, in practice, then.
>
> Very easy to fix pip remove though, just another elif: in pip.req
> uninstall(). I also intend to define an entry point for post-install
> hooks generally.
>
>> But certainly converters are an interim measure. (Just an important one IMO).
>
> You can use the issue tracker on http://bitbucket.org/dholth/wheel to
> help build a roadmap

From dholth at gmail.com  Tue Jul  3 03:16:19 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 2 Jul 2012 21:16:19 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
Message-ID: <CAG8k2+4-hL-aN+AbpjW_Dg--53npGQJf5OCLhZ-SkMkvOJwZ_g@mail.gmail.com>

I would like to amend the spec. The hash column of RECORD should be

'sha256:' + urlsafe_b64encode(hashlib.sha256(data))

instead of the hopelessly obsolete md5. With a secure hash function,
you can digitally sign RECORD.

It would also make sense to allow RECORD to be omitted from RECORD.

From tarek at ziade.org  Tue Jul  3 09:37:57 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 03 Jul 2012 09:37:57 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CAG8k2+4-hL-aN+AbpjW_Dg--53npGQJf5OCLhZ-SkMkvOJwZ_g@mail.gmail.com>
References: <CAG8k2+4-hL-aN+AbpjW_Dg--53npGQJf5OCLhZ-SkMkvOJwZ_g@mail.gmail.com>
Message-ID: <4FF2A155.8040104@ziade.org>

On 7/3/12 3:16 AM, Daniel Holth wrote:
> I would like to amend the spec. The hash column of RECORD should be
>
> 'sha256:' + urlsafe_b64encode(hashlib.sha256(data))
>
> instead of the hopelessly obsolete md5. With a secure hash function,
> you can digitally sign RECORD.
The goal of the RECORD file is to make sure we know if a file was 
changed so installlers are aware of it when they want
to remove the project for instance.

It was not really intended to be some kind of security against an attack 
-- unless you have
attacks scenarri in mind ?

>
> It would also make sense to allow RECORD to be omitted from RECORD.

why ? this file is part of the installation, and as said here : 
http://www.python.org/dev/peps/pep-0376/#record

" Notice that the RECORD file can't contain a hash of itself and is just 
mentioned here"

Cheers
Tarek


> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/5fa98d54/attachment-0001.html>

From bkabrda at redhat.com  Tue Jul  3 09:42:09 2012
From: bkabrda at redhat.com (Bohuslav Kabrda)
Date: Tue, 03 Jul 2012 03:42:09 -0400 (EDT)
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CAG8k2+4-hL-aN+AbpjW_Dg--53npGQJf5OCLhZ-SkMkvOJwZ_g@mail.gmail.com>
Message-ID: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>

----- Original Message -----
> I would like to amend the spec. The hash column of RECORD should be
> 
> 'sha256:' + urlsafe_b64encode(hashlib.sha256(data))
> 
> instead of the hopelessly obsolete md5. With a secure hash function,
> you can digitally sign RECORD.
> 

Signing packages does sound interesting, but what authority would sign them? The authors of the packages themselves?

> It would also make sense to allow RECORD to be omitted from RECORD.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 

-- 
Regards,
Bohuslav "Slavek" Kabrda.

From tarek at ziade.org  Tue Jul  3 09:42:32 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 03 Jul 2012 09:42:32 +0200
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
Message-ID: <4FF2A268.8040402@ziade.org>

On 7/1/12 8:50 PM, Marcus wrote:
>
>     FWIW I have started a "pypi2rpm" project that creates RPM out of
>     PyPI projects, we use at Mozilla to deploy our apps,
>     http://pypi.python.org/pypi/pypi2rpm
>     It's just a glue script on the top of an isolated bdist_rpm
>     command and distutils2.version (for sorting versions) but you can
>     also pass your spec file for a given
>     project and have Fedora/RHEL specific options (like adding
>     python26- prefixes etc)
>
>
> Hello Tarek:
> if you have anything that details your rpm deployment workflow at 
> mozilla, i'd love to see it.
> I work in a very active build environment, and repetitive"setup.py 
> build" time is pain point.
> currently, we have a pretty fragile virtualenv caching strategy, and 
> I've even considered patching distutils so that "setup.py build" would 
> somehow work against a cache.
> currently, my mindset is to migrate to a modular rpm install strategy 
> (managed by puppet) to prevent wasteful build time.
>
> Marcus
> P.S. Daniel's "wheel" format is another possible to answer to my concern
>
>
To speed up deployments, we use pip's --download-cache option, so all 
builds on the box share the downloaded tarballs.

We also use our own local mirror, and an extra apache directory for 
packages that are not on pypi.

I have also added a proxy that returns a 404 when pip tries to hit the 
net, to simulate the --allow-hosts option easy_install has.

Everything is driven from a Makefile. Example: 
https://github.com/mozilla-services/tokenserver/blob/master/Makefile
The script to create RPMs is at 
https://github.com/mozilla-services/mopytools

HTH
Tarek



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/6d94e429/attachment.html>

From donald.stufft at gmail.com  Tue Jul  3 09:43:07 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 3 Jul 2012 03:43:07 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
Message-ID: <C37E41528C8645E3AF4DA5CB0EA05AFD@gmail.com>

Ideally the authors would sign them with GPG imo. Which is already 
possible.


On Tuesday, July 3, 2012 at 3:42 AM, Bohuslav Kabrda wrote:

> ----- Original Message -----
> > I would like to amend the spec. The hash column of RECORD should be
> > 
> > 'sha256:' + urlsafe_b64encode(hashlib.sha256(data))
> > 
> > instead of the hopelessly obsolete md5. With a secure hash function,
> > you can digitally sign RECORD.
> > 
> 
> 
> Signing packages does sound interesting, but what authority would sign them? The authors of the packages themselves?
> 
> > It would also make sense to allow RECORD to be omitted from RECORD.
> > _______________________________________________
> > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org)
> > http://mail.python.org/mailman/listinfo/distutils-sig
> > 
> 
> 
> -- 
> Regards,
> Bohuslav "Slavek" Kabrda.
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org)
> http://mail.python.org/mailman/listinfo/distutils-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/eea556e5/attachment.html>

From donald.stufft at gmail.com  Tue Jul  3 09:44:35 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 3 Jul 2012 03:44:35 -0400
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2A268.8040402@ziade.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
Message-ID: <7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>

On Tuesday, July 3, 2012 at 3:42 AM, Tarek Ziad? wrote:
> I have also added a proxy that returns a 404 when pip tries to hit the net, to simulate the --allow-hosts option easy_install has.
I hate this particular misfeature of python packaging. http://restricted.crate.io/ gets rid of it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/6b452e39/attachment.html>

From tarek at ziade.org  Tue Jul  3 09:45:20 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 03 Jul 2012 09:45:20 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
Message-ID: <4FF2A310.1070800@ziade.org>

On 7/3/12 9:42 AM, Bohuslav Kabrda wrote:
> ----- Original Message -----
>> I would like to amend the spec. The hash column of RECORD should be
>>
>> 'sha256:' + urlsafe_b64encode(hashlib.sha256(data))
>>
>> instead of the hopelessly obsolete md5. With a secure hash function,
>> you can digitally sign RECORD.
>>
> Signing packages does sound interesting, but what authority would sign them? The authors of the packages themselves?

Notice that there's already a --sign feature in Distutils, using gpg.

Hash in the RECORD file have nothing to do with making sure the package 
is originated from developer X.
Its only purpose is to know if a file on the system was changed


Cheers
Tarek

From donald.stufft at gmail.com  Tue Jul  3 09:48:05 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 3 Jul 2012 03:48:05 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF2A310.1070800@ziade.org>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
Message-ID: <C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>

On Tuesday, July 3, 2012 at 3:45 AM, Tarek Ziad? wrote:
>  
> Hash in the RECORD file have nothing to do with making sure the package  
> is originated from developer X.
> Its only purpose is to know if a file on the system was changed
>  
Using sha256 would enable preventing someone from maliciously changing the
file. Similar to how IDS systems capture hashes of binaries to compare against.
Of course someone using the system like this would need to protect the filesystem
storing the RECORD files accordingly.

I also think that switching to sha256 is pretty low cost with minimal (no?) downsides
with some possible upsides. Is there a reason to stay with md5?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/9f1f9b46/attachment.html>

From tarek at ziade.org  Tue Jul  3 09:48:41 2012
From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=)
Date: Tue, 03 Jul 2012 09:48:41 +0200
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
Message-ID: <4FF2A3D9.2080200@ziade.org>

On 7/3/12 9:44 AM, Donald Stufft wrote:
> On Tuesday, July 3, 2012 at 3:42 AM, Tarek Ziad? wrote:
>> I have also added a proxy that returns a 404 when pip tries to hit 
>> the net, to simulate the --allow-hosts option easy_install has.
> I hate this particular misfeature of python packaging. 
> http://restricted.crate.io/ gets rid of it.

Not sure how http://restricted.crate.io/ qorks (unreachable for me right 
now - 'internal server error')

But the feature I need is to be able to do a full offline deployment 
without touching the net *but* my local index+directories, and by 
cutting all the crawling pip or easy+install do when
following links.

That's something possible in packaging's crawler, and that should be a 
hard requirement imho for whatever tool is built

Cheers
Tarek


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/7b390269/attachment.html>

From bkabrda at redhat.com  Tue Jul  3 09:49:59 2012
From: bkabrda at redhat.com (Bohuslav Kabrda)
Date: Tue, 03 Jul 2012 03:49:59 -0400 (EDT)
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF2A310.1070800@ziade.org>
Message-ID: <ef44a919-a184-4ea4-ac37-7212374e0253@zmail15.collab.prod.int.phx2.redhat.com>

----- Original Message -----
> On 7/3/12 9:42 AM, Bohuslav Kabrda wrote:
> > ----- Original Message -----
> >> I would like to amend the spec. The hash column of RECORD should
> >> be
> >>
> >> 'sha256:' + urlsafe_b64encode(hashlib.sha256(data))
> >>
> >> instead of the hopelessly obsolete md5. With a secure hash
> >> function,
> >> you can digitally sign RECORD.
> >>
> > Signing packages does sound interesting, but what authority would
> > sign them? The authors of the packages themselves?
> 
> Notice that there's already a --sign feature in Distutils, using gpg.
> 

Ah, I didn't know about that.

> Hash in the RECORD file have nothing to do with making sure the
> package
> is originated from developer X.
> Its only purpose is to know if a file on the system was changed
> 

Well, since there is the --sign feature, I totally agree that md5 is sufficient for making checksums.

> 
> Cheers
> Tarek
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 

-- 
Regards,
Bohuslav "Slavek" Kabrda.

From donald.stufft at gmail.com  Tue Jul  3 09:51:25 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 3 Jul 2012 03:51:25 -0400
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2A3D9.2080200@ziade.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
	<4FF2A3D9.2080200@ziade.org>
Message-ID: <FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>

On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziad? wrote:
> Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error')
Hmm that's odd, it works for me now and I don't see anything logged. All it does is it doesn't list
any external links on the simple pages so pip, easy_install etc never find any links to spider ;)

That's all being refactored currently to work a lot nicer though on Crate's end. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/4e588f54/attachment.html>

From tarek at ziade.org  Tue Jul  3 09:52:41 2012
From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=)
Date: Tue, 03 Jul 2012 09:52:41 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
Message-ID: <4FF2A4C9.3050508@ziade.org>

On 7/3/12 9:48 AM, Donald Stufft wrote:
> On Tuesday, July 3, 2012 at 3:45 AM, Tarek Ziad? wrote:
>>
>> Hash in the RECORD file have nothing to do with making sure the package
>> is originated from developer X.
>> Its only purpose is to know if a file on the system was changed
>>
> Using sha256 would enable preventing someone from maliciously changing the
> file.

If someone has access to that file, it means that he can also change the 
RECORD file
so you have no way of trusting RECORD either.


> Similar to how IDS systems capture hashes of binaries to compare against.
> Of course someone using the system like this would need to protect the 
> filesystem
> storing the RECORD files accordingly.

I think that's the main issue - where are you going to put the RECORD file ?

>
> I also think that switching to sha256 is pretty low cost with minimal 
> (no?) downsides
> with some possible upsides. Is there a reason to stay with md5?

The file is two times smaller and faster to create, and md5 does its job 
at providing
a hash for a file. I still fail to see a use case for stronger hashes


Cheers
Tarek


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/c1f689fc/attachment-0001.html>

From tarek at ziade.org  Tue Jul  3 09:55:06 2012
From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=)
Date: Tue, 03 Jul 2012 09:55:06 +0200
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
	<4FF2A3D9.2080200@ziade.org>
	<FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>
Message-ID: <4FF2A55A.9040208@ziade.org>

On 7/3/12 9:51 AM, Donald Stufft wrote:
> On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziad? wrote:
>> Not sure how http://restricted.crate.io/ qorks (unreachable for me 
>> right now - 'internal server error')
> Hmm that's odd, it works for me now and I don't see anything logged.

The 3rd hit worked.

> All it does is it doesn't list
> any external links on the simple pages so pip, easy_install etc never 
> find any links to spider ;)
>
> That's all being refactored currently to work a lot nicer though on 
> Crate's end.

But IIRC when the package is downloaded, the metadata are built 
in-place, and the installer may visit the home url for example that's in 
setup.py, no ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/58dac35f/attachment.html>

From donald.stufft at gmail.com  Tue Jul  3 09:58:10 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 3 Jul 2012 03:58:10 -0400
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2A55A.9040208@ziade.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
	<4FF2A3D9.2080200@ziade.org>
	<FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>
	<4FF2A55A.9040208@ziade.org>
Message-ID: <554347CF3229402DBE3F30B1CC1F6363@gmail.com>

On Tuesday, July 3, 2012 at 3:55 AM, Tarek Ziad? wrote:
> But IIRC when the package is downloaded, the metadata are built in-place, and the installer may visit the home url for example that's in setup.py, no ?  
pip at least, and I assume easy_install doesn't use the home url etc links from
inside of the setup.py. PyPI takes that data and places it on the simple index
page and pip looks at that simple index page and finds all the urls it can to spider.
So removing urls from the simple index page effectively removes external linking.

The only time that's not true is if there a dependency_links inside of setup.py but
that is rarely used.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/93bdc777/attachment.html>

From tarek at ziade.org  Tue Jul  3 10:08:07 2012
From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=)
Date: Tue, 03 Jul 2012 10:08:07 +0200
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2A7E3.6020604@notmyidea.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
	<4FF2A3D9.2080200@ziade.org>
	<FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>
	<4FF2A55A.9040208@ziade.org> <4FF2A7E3.6020604@notmyidea.org>
Message-ID: <4FF2A867.7050203@ziade.org>

On 7/3/12 10:05 AM, Alexis M?taireau wrote:
> Le mar. 03 juil. 2012 09:55:06 CEST, Tarek Ziad? a ?crit :
>> On 7/3/12 9:51 AM, Donald Stufft wrote:
>>> On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziad? wrote:
>>>> Not sure how http://restricted.crate.io/ qorks (unreachable for me
>>>> right now - 'internal server error')
>>> Hmm that's odd, it works for me now and I don't see anything logged.
>>
>> The 3rd hit worked.
>>
>>> All it does is it doesn't list
>>> any external links on the simple pages so pip, easy_install etc never
>>> find any links to spider ;)
>>>
>>> That's all being refactored currently to work a lot nicer though on
>>> Crate's end.
>>
>> But IIRC when the package is downloaded, the metadata are built
>> in-place, and the installer may visit the home url for example that's
>> in setup.py, no ?
>
> the crawler follows the links present in the simple index before 
> determining which distribution to download.
>
> However, that's true that the metadata is buit from the information 
> contained in the setup.py file, and reuses the url present in setup.py 
> as the "url" metadata. In any case, that's *after* following the links 
> in the index.

Yeah so even a 'cleaned up' mirror will not do the trick.  For pip and 
easy_install, we need a lower-level barrier to avoid hitting the net





From alexis at notmyidea.org  Tue Jul  3 10:05:55 2012
From: alexis at notmyidea.org (=?UTF-8?B?QWxleGlzIE3DqXRhaXJlYXU=?=)
Date: Tue, 03 Jul 2012 10:05:55 +0200
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2A55A.9040208@ziade.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
	<4FF2A3D9.2080200@ziade.org>
	<FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>
	<4FF2A55A.9040208@ziade.org>
Message-ID: <4FF2A7E3.6020604@notmyidea.org>

Le mar. 03 juil. 2012 09:55:06 CEST, Tarek Ziad? a ?crit :
> On 7/3/12 9:51 AM, Donald Stufft wrote:
>> On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziad? wrote:
>>> Not sure how http://restricted.crate.io/ qorks (unreachable for me
>>> right now - 'internal server error')
>> Hmm that's odd, it works for me now and I don't see anything logged.
>
> The 3rd hit worked.
>
>> All it does is it doesn't list
>> any external links on the simple pages so pip, easy_install etc never
>> find any links to spider ;)
>>
>> That's all being refactored currently to work a lot nicer though on
>> Crate's end.
>
> But IIRC when the package is downloaded, the metadata are built
> in-place, and the installer may visit the home url for example that's
> in setup.py, no ?

the crawler follows the links present in the simple index before 
determining which distribution to download.

However, that's true that the metadata is buit from the information 
contained in the setup.py file, and reuses the url present in setup.py 
as the "url" metadata. In any case, that's *after* following the links 
in the index.

From alexis at notmyidea.org  Tue Jul  3 10:12:02 2012
From: alexis at notmyidea.org (=?UTF-8?B?QWxleGlzIE3DqXRhaXJlYXU=?=)
Date: Tue, 03 Jul 2012 10:12:02 +0200
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2A867.7050203@ziade.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
	<4FF2A3D9.2080200@ziade.org>
	<FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>
	<4FF2A55A.9040208@ziade.org> <4FF2A7E3.6020604@notmyidea.org>
	<4FF2A867.7050203@ziade.org>
Message-ID: <4FF2A952.8090106@notmyidea.org>

Le mar. 03 juil. 2012 10:08:07 CEST, Tarek Ziad? a ?crit :
> Yeah so even a 'cleaned up' mirror will not do the trick.  For pip and
> easy_install, we need a lower-level barrier to avoid hitting the net

Well, if links are removed from the indexes, then the crawler will not 
try to follow the links, which sounds enough, isn't it? In which cases 
would an installer hit the external links if it doesn't have a way to 
discover them?

Additionally, distutils2 do have a "follow_links" options that can be 
set to False (I'm even wondering if thats not the default option)

Alexis


From tarek at ziade.org  Tue Jul  3 10:15:42 2012
From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=)
Date: Tue, 03 Jul 2012 10:15:42 +0200
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2A952.8090106@notmyidea.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
	<4FF2A3D9.2080200@ziade.org>
	<FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>
	<4FF2A55A.9040208@ziade.org> <4FF2A7E3.6020604@notmyidea.org>
	<4FF2A867.7050203@ziade.org> <4FF2A952.8090106@notmyidea.org>
Message-ID: <4FF2AA2E.3020509@ziade.org>

On 7/3/12 10:12 AM, Alexis M?taireau wrote:
> Le mar. 03 juil. 2012 10:08:07 CEST, Tarek Ziad? a ?crit :
>> Yeah so even a 'cleaned up' mirror will not do the trick.  For pip and
>> easy_install, we need a lower-level barrier to avoid hitting the net
>
> Well, if links are removed from the indexes, then the crawler will not 
> try to follow the links, which sounds enough, isn't it? In which cases 
> would an installer hit the external links if it doesn't have a way to 
> discover them?

The scenario I had in mind was:

1. installer downloads Foo-1.2.tgz
2. installer runs Foo's setup.py
3. Foo's metadata mentions http://example.com
4. installer follow that link

But I did not check the code yet

>
> Additionally, distutils2 do have a "follow_links" options that can be 
> set to False (I'm even wondering if thats not the default option)
>
> Alexis
>



From donald.stufft at gmail.com  Tue Jul  3 11:30:54 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 3 Jul 2012 05:30:54 -0400
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2AA2E.3020509@ziade.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
	<7AD6B2ED01EB4010BC1A7486AD620B00@gmail.com>
	<4FF2A3D9.2080200@ziade.org>
	<FDB86E15CADC407E8BBA06AA33A0D3CE@gmail.com>
	<4FF2A55A.9040208@ziade.org> <4FF2A7E3.6020604@notmyidea.org>
	<4FF2A867.7050203@ziade.org> <4FF2A952.8090106@notmyidea.org>
	<4FF2AA2E.3020509@ziade.org>
Message-ID: <A1E249FE266B40A2BDC1FC78A255DF54@gmail.com>

On Tuesday, July 3, 2012 at 4:15 AM, Tarek Ziad? wrote:
> The scenario I had in mind was:
>  
> 1. installer downloads Foo-1.2.tgz
> 2. installer runs Foo's setup.py
> 3. Foo's metadata mentions http://example.com
> 4. installer follow that link
>  
> But I did not check the code yet
The only time this would happen is if the metadata includes dependency  
links which isn't a very common bit of metadata.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/3ef9762e/attachment.html>

From dholth at gmail.com  Tue Jul  3 13:14:43 2012
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 3 Jul 2012 07:14:43 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF2A4C9.3050508@ziade.org>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
Message-ID: <CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>

It's embarrassing to see md5 used for any reason. You go to pypi, and
every download link has an md5 sum of the package, and you think "what
is this archaic system that gives me a useless hash, implicated in
such fine situations as the Flame malware and ever-improving attacks
against md5?" It is irrelevant that it is "probably good enough for
this limited use". You might as well use CRC32; it is much shorter.

By re-using RECORD to include a secure hash of every file in an
archive, you can sign all the files in the archive by signing RECORD,
similar to how jars are signed.  The digital signature is right there
inside the archive, and if you decide you would rather have a .tar.xz
instead of a .zip the signature is still valid.

From jeroen at dekkers.ch  Tue Jul  3 14:48:35 2012
From: jeroen at dekkers.ch (Jeroen Dekkers)
Date: Tue, 03 Jul 2012 14:48:35 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
Message-ID: <87wr2lau3g.wl%jeroen@dekkers.ch>

At Tue, 3 Jul 2012 07:14:43 -0400,
Daniel Holth wrote:
> It's embarrassing to see md5 used for any reason. You go to pypi, and
> every download link has an md5 sum of the package, and you think "what
> is this archaic system that gives me a useless hash, implicated in
> such fine situations as the Flame malware and ever-improving attacks
> against md5?" It is irrelevant that it is "probably good enough for
> this limited use". You might as well use CRC32; it is much shorter.

Yes, you're right, pypi could as well use CRC32. From a security
perspective nothing would change, nor if we would switch to sha512,
because there is no way to know whether the hash is correct. Without a
trust path the hash is pretty useless except for verifying that the
download isn't corrupted.

And even if we would have trust paths, the md5 attacks are collision
attacks, not preimage attacks. That means the security threat you're
worrying about is that a developer uploads something to pypi with the
intention of replacing that by something else with the same hash
without anyone noticing. And although it is worthwhile to protect
against such kind of things, you should also ask the question why
you're running code from such a developer.

And yes, attacks on md5 will only get better, so we should migrate to
better hashes in the future. But if there is something to be
embarrassed about, it's not the use of md5, but the lack of proper
code signing and trust paths between developers.


Kind regards,

Jeroen Dekkers

From dholth at gmail.com  Tue Jul  3 15:54:20 2012
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 3 Jul 2012 09:54:20 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <87wr2lau3g.wl%jeroen@dekkers.ch>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
Message-ID: <CAG8k2+7wWfEnqO2hbg+vP_+F0e+MeZMuU7sj1e=c2+7qc48EPw@mail.gmail.com>

> And yes, attacks on md5 will only get better, so we should migrate to
> better hashes in the future. But if there is something to be
> embarrassed about, it's not the use of md5, but the lack of proper
> code signing and trust paths between developers.

I'm going to implement this except I will replace the sha256: with a
sha256= There is simply no realistic drawback. Strong hashing is a
prerequisite for a trust path, and you avoid the need to even think
about why it is OK in this specific circumstance that a weak hash is
being used.

From pje at telecommunity.com  Tue Jul  3 16:32:43 2012
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 3 Jul 2012 10:32:43 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <87wr2lau3g.wl%jeroen@dekkers.ch>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
Message-ID: <CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>

On Tue, Jul 3, 2012 at 8:48 AM, Jeroen Dekkers <jeroen at dekkers.ch> wrote:

> And yes, attacks on md5 will only get better, so we should migrate to
> better hashes in the future.


No, because that's not what the RECORD hashes are for.  It's not an
intrusion detection system, it's an installer conflict and "oops I edited
the wrong file" checker.

People who are upset because md5 is low security are correctly
understanding that this system *provides no security*.  We are not
promising ANY security, so *not* using a secure hash is actually
preferable.  The goal is data integrity against accidental overwrite by
dumb installer tools (e.g. distutils) and accidental edits, not security
against malicious tampering.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/7c0aadb4/attachment.html>

From tarek at ziade.org  Tue Jul  3 16:53:03 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 03 Jul 2012 16:53:03 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
	<CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
Message-ID: <4FF3074F.3080406@ziade.org>

On 7/3/12 4:32 PM, PJ Eby wrote:
> On Tue, Jul 3, 2012 at 8:48 AM, Jeroen Dekkers <jeroen at dekkers.ch 
> <mailto:jeroen at dekkers.ch>> wrote:
>
>     And yes, attacks on md5 will only get better, so we should migrate to
>     better hashes in the future. 
>
>
> No, because that's not what the RECORD hashes are for.  It's not an 
> intrusion detection system, it's an installer conflict and "oops I 
> edited the wrong file" checker.
>
> People who are upset because md5 is low security are correctly 
> understanding that this system *provides no security*.  We are not 
> promising ANY security, so *not* using a secure hash is actually 
> preferable.  The goal is data integrity against accidental overwrite 
> by dumb installer tools (e.g. distutils) and accidental edits, not 
> security against malicious tampering.

Yeah I don't really understand this debate over md5 hashes here. I 
suggest that we emphasis in PEP 376 the fact that the sole purpose is to 
have a checksum.



>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/d1b31cf8/attachment.html>

From tarek at ziade.org  Tue Jul  3 17:00:50 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 03 Jul 2012 17:00:50 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CAG8k2+7wWfEnqO2hbg+vP_+F0e+MeZMuU7sj1e=c2+7qc48EPw@mail.gmail.com>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
	<CAG8k2+7wWfEnqO2hbg+vP_+F0e+MeZMuU7sj1e=c2+7qc48EPw@mail.gmail.com>
Message-ID: <4FF30922.5060407@ziade.org>

On 7/3/12 3:54 PM, Daniel Holth wrote:

> I'm going to implement this except I will replace the sha256: with a
> sha256= There is simply no realistic drawback.

I am -1000 for any change to the RECORD file hashes in PEP 376 unless 
there's a clear use case.

>   Strong hashing is a
> prerequisite for a trust path, and you avoid the need to even think
> about why it is OK in this specific circumstance that a weak hash is
> being used.
Sorry but I don't understand your use case.

What "strong", "weak" or "trust" means here ?

The use case we have is:  we need a check sum for every file, that's all.

If you want to build a system where you can verify the origin of the files,
you need something like a public/private key system. Which is what --sign
is for.

Otherwise you're just going to make hashes longer for no apparent reason.

Cheers
Tarek

From qwcode at gmail.com  Tue Jul  3 17:33:41 2012
From: qwcode at gmail.com (Marcus)
Date: Tue, 3 Jul 2012 08:33:41 -0700
Subject: [Distutils] Self Introduction and Getting Hands On
In-Reply-To: <4FF2A268.8040402@ziade.org>
References: <3b2aaf04-5f6d-4cc3-86b4-64234b50610c@zmail15.collab.prod.int.phx2.redhat.com>
	<4FE96FDA.8030206@ziade.org>
	<CAPYWazpZW--u26w-5eH86638S-QOF_TMqQcw-4h09FgcU5EAMw@mail.gmail.com>
	<4FF2A268.8040402@ziade.org>
Message-ID: <CAPYWazrUxi2+7fL2xDbCzhmKS3gH1E0Dyf4QvcvdKeN4L2BAoQ@mail.gmail.com>

>
> To speed up deployments, we use pip's --download-cache option, so all
> builds on the box share the downloaded tarballs.
>
> We also use our own local mirror, and an extra apache directory for
> packages that are not on pypi.
>
> I have also added a proxy that returns a 404 when pip tries to hit the
> net, to simulate the --allow-hosts option easy_install has.
>

> Everything is driven from a Makefile. Example:
> https://github.com/mozilla-services/tokenserver/blob/master/Makefile
> The script to create RPMs is at
> https://github.com/mozilla-services/mopytools
>

thanks for the link to the makefile.
we also use a download cache and a local mirror (although we're migrating
to findlink dirs instead)

for us, that still leaves the issue of wasteful "setup.py build/install"
time (for many dependencies) in our repetitive test/build environment.

that's my interest in modular binary build solutions like rpm.

the idea would be to stop rebuilding a large virtualenv for every test
build.

but rather just rebuild a dev rpm the specific package(s) that's changing
and a new dev master rpm that references those changes.

*then* go install the new master rpm on the test box, which will find most
of it's rpm dependencies already satisfied.

but I'll admit I'm new to thinking about binary build solutions.

thanks
Marcus









>
>
> HTH
> Tarek
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/f53e6037/attachment.html>

From merwok at netwok.org  Tue Jul  3 17:10:41 2012
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Tue, 03 Jul 2012 11:10:41 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF3074F.3080406@ziade.org>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
	<CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
	<4FF3074F.3080406@ziade.org>
Message-ID: <4FF30B71.3080900@netwok.org>

Le 03/07/2012 10:53, Tarek Ziad? a ?crit :
> On 7/3/12 4:32 PM, PJ Eby wrote:
>> No, because that's not what the RECORD hashes are for.  It's not an
>> intrusion detection system, it's an installer conflict and "oops I
>> edited the wrong file" checker.
>>
>> People who are upset because md5 is low security are correctly
>> understanding that this system *provides no security*.  We are not
>> promising ANY security, so *not* using a secure hash is actually
>> preferable.  The goal is data integrity against accidental overwrite
>> by dumb installer tools (e.g. distutils) and accidental edits, not
>> security against malicious tampering.

Exactly.  Promises of false security do not help users.

> Yeah I don't really understand this debate over md5 hashes here. I
> suggest that we emphasis in PEP 376 the fact that the sole purpose is to
> have a checksum.

Putting that on my list of editions for the PEPs!

Cheers

From jeroen at dekkers.ch  Tue Jul  3 17:57:18 2012
From: jeroen at dekkers.ch (Jeroen Dekkers)
Date: Tue, 03 Jul 2012 17:57:18 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
	<CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
Message-ID: <87txxobzxd.wl%jeroen@dekkers.ch>

At Tue, 3 Jul 2012 10:32:43 -0400,
PJ Eby wrote:
> 
> On Tue, Jul 3, 2012 at 8:48 AM, Jeroen Dekkers <jeroen at dekkers.ch> wrote:
> 
> > And yes, attacks on md5 will only get better, so we should migrate to
> > better hashes in the future.
> 
> 
> No, because that's not what the RECORD hashes are for.  It's not an
> intrusion detection system, it's an installer conflict and "oops I edited
> the wrong file" checker.

Sorry for not being clear, but I totally agree. I was replying to the
md5 on PyPI are embarrassing part and meant that we should migrate to
use better hashes on PyPI in the future.


Jeroen Dekkers

From dholth at gmail.com  Tue Jul  3 18:05:44 2012
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 3 Jul 2012 12:05:44 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF30B71.3080900@netwok.org>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
	<CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
	<4FF3074F.3080406@ziade.org> <4FF30B71.3080900@netwok.org>
Message-ID: <9118A0AC-88AA-4124-99F4-33B29434A3D2@gmail.com>

I am just re-using record in wheel files so I can implement a verify function someday. Pay no attention to this backward-compatible change. You can use the checksum you prefer, and if it does not begin with hashfunc= then you know it's an md5.

No discussion about adding provides-extra and the reserved extra names for python setup.py test? How about that the environment markers spec says you can use == but (naked version number) (4.0) is the only example given for "exactly this version"?

And why is pkg-info called metadata now anyway?

Daniel Holth

On Jul 3, 2012, at 11:10 AM, ?ric Araujo <merwok at netwok.org> wrote:

> Le 03/07/2012 10:53, Tarek Ziad? a ?crit :
>> On 7/3/12 4:32 PM, PJ Eby wrote:
>>> No, because that's not what the RECORD hashes are for.  It's not an
>>> intrusion detection system, it's an installer conflict and "oops I
>>> edited the wrong file" checker.
>>> 
>>> People who are upset because md5 is low security are correctly
>>> understanding that this system *provides no security*.  We are not
>>> promising ANY security, so *not* using a secure hash is actually
>>> preferable.  The goal is data integrity against accidental overwrite
>>> by dumb installer tools (e.g. distutils) and accidental edits, not
>>> security against malicious tampering.
> 
> Exactly.  Promises of false security do not help users.
> 
>> Yeah I don't really understand this debate over md5 hashes here. I
>> suggest that we emphasis in PEP 376 the fact that the sole purpose is to
>> have a checksum.
> 
> Putting that on my list of editions for the PEPs!
> 
> Cheers
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

From donald.stufft at gmail.com  Tue Jul  3 18:09:56 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 3 Jul 2012 12:09:56 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <87txxobzxd.wl%jeroen@dekkers.ch>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
	<CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
	<87txxobzxd.wl%jeroen@dekkers.ch>
Message-ID: <9A80691EE58741FDAB2BC9850595FFD9@gmail.com>

On Tuesday, July 3, 2012 at 11:57 AM, Jeroen Dekkers wrote:
> Sorry for not being clear, but I totally agree. I was replying to the
> md5 on PyPI are embarrassing part and meant that we should migrate to
> use better hashes on PyPI in the future.
> 
> 

Off topic for this list, but I would agree, and have attempted to have this
happen but have been told that those hashes are not for security but only
for a checksum and will not be changed.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/1c2e0331/attachment.html>

From merwok at netwok.org  Tue Jul  3 18:50:00 2012
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Tue, 03 Jul 2012 12:50:00 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <9118A0AC-88AA-4124-99F4-33B29434A3D2@gmail.com>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
	<CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
	<4FF3074F.3080406@ziade.org> <4FF30B71.3080900@netwok.org>
	<9118A0AC-88AA-4124-99F4-33B29434A3D2@gmail.com>
Message-ID: <4FF322B8.5070206@netwok.org>

> No discussion about adding provides-extra and the reserved extra names
> for python setup.py test?  How about that the environment markers spec
> says you can use == but (naked version number)
It is a bit hard to comment on a very short proposal which proposes an 
implementation but not a specification part, and does not summarize the 
previous discussions on the topic (on distutils-sig and 
the-packaging-of-fellowship Google group).  I know it?s a boring task to 
go through the archives to see the various proposals, but someone has to 
do it.  On the principle I am +1 for build dependencies and +1 for test 
dependencies.  I don?t know what Provides-Extra is about, nor what the 
last thing you talk about is.

> And why is pkg-info called metadata now anyway?
I thought this was explained in PEP 376: pkg-info was renamed to 
dist-info to respect Python terminology (see glossary in the PEP or the 
distutils docs, and please follow it), and the file became a directory 
with metadata and other stuff in various files, hence the METADATA file 
name.

Regards

From dholth at gmail.com  Tue Jul  3 22:46:55 2012
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 3 Jul 2012 16:46:55 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF322B8.5070206@netwok.org>
References: <5fa880bf-8a41-4495-8afb-227bc331a635@zmail15.collab.prod.int.phx2.redhat.com>
	<4FF2A310.1070800@ziade.org>
	<C7E6130467E5471DA6A5DD58C6FBE6C3@gmail.com>
	<4FF2A4C9.3050508@ziade.org>
	<CAG8k2+5Fo79P3YB5hs_U5vkq3f82NNM-tMJdRs3qYAmOMfhGZQ@mail.gmail.com>
	<87wr2lau3g.wl%jeroen@dekkers.ch>
	<CALeMXf76twE0jkKeyjCKA21_ZJYmSC7FXPM5=9ikAtMV=RbSNA@mail.gmail.com>
	<4FF3074F.3080406@ziade.org> <4FF30B71.3080900@netwok.org>
	<9118A0AC-88AA-4124-99F4-33B29434A3D2@gmail.com>
	<4FF322B8.5070206@netwok.org>
Message-ID: <CAG8k2+7kvsRsnaLLXe0zA6kDRNg8O_C1tjex4yUQE4FGGzj2-w@mail.gmail.com>

Fair enough. I should write up provides-extra as an actual patch to the
pep. It is just a way to express together with requires-dist what pip
install pyramid[test] installs above just pip install pyramid, it is needed
for setup tools to generate dist info without losing that information, and
my distribute fork has an implementation. Are there really other proposals
for including that package[extras] feature in the static metadata?

I think that is the only part of my packaging work that needs to amend an
existing spec, unless you want to opine on putting a build serial number in
the versioning pep.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/0e8ad428/attachment.html>

From jared.jennings.ctr at eglin.af.mil  Wed Jul  4 01:33:08 2012
From: jared.jennings.ctr at eglin.af.mil (Jennings,
	Jared L CTR USAF AFMC 46 SK/CCI)
Date: Tue, 3 Jul 2012 18:33:08 -0500
Subject: [Distutils]  Proposal: drop md5 for sha256
Message-ID: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>

On hosts configured for compliance with U.S. Federal Information
Processing Standard (FIPS) 140-2
<http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>, like
those in some banks and, yes, the U.S. Department of Defense,
cryptographic modules (such as OpenSSL, which underlies hashlib) are not
allowed to calculate MD5 digests, because MD5 is no longer a FIPS
Approved digest algorithm.

I know no one is trying here to lean on MD5 for security, but the
standard says nothing about the reason why you're using MD5: just that
you can't.

No one expects a digest algorithm to fail, and Python 2.x may not have
been fixed to check for that before being frozen
<https://bugzilla.redhat.com/show_bug.cgi?id=746118#c3>, so if you run
an MD5 checksum on a FIPS-compliant system with an unpatched Python 2.x,
the Python interpreter will segfault. (Ruby, too, had this problem and
was itself only recently fixed,
<http://bugs.ruby-lang.org/issues/4944>.)

I have to configure hosts in accordance with FIPS 140-2, so the more
places I can get rid of MD5, the less headaches I have.

 -- Jared Jennings, RHCE, Network Admin, SURVICE Engineering Co.

From pje at telecommunity.com  Wed Jul  4 02:50:07 2012
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 3 Jul 2012 20:50:07 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
Message-ID: <CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>

On Tue, Jul 3, 2012 at 7:33 PM, Jennings, Jared L CTR USAF AFMC 46 SK/CCI <
jared.jennings.ctr at eglin.af.mil> wrote:

> On hosts configured for compliance with U.S. Federal Information
> Processing Standard (FIPS) 140-2
> <http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>, like
> those in some banks and, yes, the U.S. Department of Defense,
> cryptographic modules (such as OpenSSL, which underlies hashlib) are not
> allowed to calculate MD5 digests, because MD5 is no longer a FIPS
> Approved digest algorithm.
>

So if it's not a cryptographic module, it's okay?  ;-)

I know no one is trying here to lean on MD5 for security, but the
> standard says nothing about the reason why you're using MD5: just that
> you can't.
>
> No one expects a digest algorithm to fail, and Python 2.x may not have
> been fixed to check for that before being frozen
> <https://bugzilla.redhat.com/show_bug.cgi?id=746118#c3>, so if you run
> an MD5 checksum on a FIPS-compliant system with an unpatched Python 2.x,
> the Python interpreter will segfault. (Ruby, too, had this problem and
> was itself only recently fixed,
> <http://bugs.ruby-lang.org/issues/4944>.)
>
> I have to configure hosts in accordance with FIPS 140-2, so the more
> places I can get rid of MD5, the less headaches I have.
>

If we replace it with something else, then I suggest we replace it with
something that's even MORE braindead than md5 so that nobody will mistake
it for a secure hash.  Otherwise, we will have this exact same problem all
over again when the replacement "secure" hash is disabled by a newer
version of FIPS.

The other option is simply to forego a checksum altogether and assume same
size = same file.  Honestly, I don't remember why we cared about detecting
such modifications in the first place: neither PEP 376 nor 262 explain why,
and 376 doesn't explain why it went with md5 instead of sha1 (as in PEP
262).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/d204dbb8/attachment.html>

From glyph at twistedmatrix.com  Wed Jul  4 03:29:19 2012
From: glyph at twistedmatrix.com (Glyph)
Date: Tue, 3 Jul 2012 18:29:19 -0700
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
	<CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
Message-ID: <4E25F1BA-9F3D-4AC9-9697-293A93543AFA@twistedmatrix.com>

On Jul 3, 2012, at 5:50 PM, PJ Eby <pje at telecommunity.com> wrote:

> Otherwise, we will have this exact same problem all over again when the replacement "secure" hash is disabled by a newer version of FIPS.

Or, you know, somebody could maintain the dang software and automate the process of producing these hashes.  I am slightly baffled by the tone of this thread, like the hash algorithm needs to be set in stone forever.  There's a reason that most software treats hashes as pluggable: new algorithms come out every few years, you have to expect that your choice will be obsoleted for some reason (not necessarily just security!) in the future.  Granted, there's no real security in this case, but why not use a hash algorithm with less probability of collision?

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/77d6ccc9/attachment.html>

From setuptools at bugs.python.org  Wed Jul  4 06:11:41 2012
From: setuptools at bugs.python.org (R. Andrew Ohana)
Date: Wed, 04 Jul 2012 04:11:41 +0000
Subject: [Distutils] [issue140] [PATCH] Make 'develop --egg-path' respect
	specified path	in easy_install.pth
Message-ID: <1341375101.14.0.600862816848.issue140@psf.upfronthosting.co.za>


New submission from R. Andrew Ohana <andrew.ohana at gmail.com>:

Currently easy_install.pth will either contain paths relative to the site-packages directory, or the normalized_path. This is because the framework has been setup to use normalized paths everywhere. The patch should properly extend the current framework to make the argument to the egg-path option actually respected.

----------
files: respect-egg-path.patch
messages: 663
nosy: ohanar
priority: feature
status: unread
title: [PATCH] Make 'develop --egg-path' respect specified path in easy_install.pth
Added file: http://bugs.python.org/setuptools/file82/respect-egg-path.patch

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue140>
_______________________________________________
-------------- next part --------------
diff --git a/pkg_resources.py b/pkg_resources.py
index 79db00b..5eacb69 100644
--- a/pkg_resources.py
+++ b/pkg_resources.py
@@ -1650,13 +1650,13 @@ def register_finder(importer_type, distribution_finder):
     _distribution_finders[importer_type] = distribution_finder
 
 
-def find_distributions(path_item, only=False):
+def find_distributions(path_item, only=False, basedir=None):
     """Yield distributions accessible via `path_item`"""
     importer = get_importer(path_item)
     finder = _find_adapter(_distribution_finders, importer)
-    return finder(importer, path_item, only)
+    return finder(importer, path_item, only, basedir)
 
-def find_in_zip(importer, path_item, only=False):
+def find_in_zip(importer, path_item, only=False, basedir=None):
     metadata = EggMetadata(importer)
     if metadata.has_metadata('PKG-INFO'):
         yield Distribution.from_filename(path_item, metadata=metadata)
@@ -1679,13 +1679,15 @@ def StringIO(*args, **kw):
         from StringIO import StringIO
     return StringIO(*args,**kw)
 
-def find_nothing(importer, path_item, only=False):
+def find_nothing(importer, path_item, only=False, basedir=None):
     return ()
 register_finder(object,find_nothing)
 
-def find_on_path(importer, path_item, only=False):
+def find_on_path(importer, path_item, only=False, basedir=None):
     """Yield distributions accessible on a sys.path directory"""
-    path_item = _normalize_cached(path_item)
+    normal_path_item = _normalize_cached(path_item, basedir=basedir)
+    if basedir is None or path_item.startswith(basedir):
+        path_item = normal_path_item
 
     if os.path.isdir(path_item) and os.access(path_item, os.R_OK):
         if path_item.lower().endswith('.egg'):
@@ -1824,15 +1826,24 @@ def null_ns_handler(importer, path_item, packageName, module):
 register_namespace_handler(object,null_ns_handler)
 
 
-def normalize_path(filename):
+def normalize_path(filename,basedir=None):
     """Normalize a file/dir name for comparison purposes"""
+    if basedir is not None and not os.path.isabs(filename):
+        filename = os.path.join(basedir, filename)
     return os.path.normcase(os.path.realpath(filename))
 
-def _normalize_cached(filename,_cache={}):
+def _normalize_cached(filename,_cache={},basedir=None):
     try:
-        return _cache[filename]
+        if basedir is None:
+            return _cache[filename]
+        else:
+            return _cache[(filename,basedir)]
     except KeyError:
-        _cache[filename] = result = normalize_path(filename)
+        result = normalize_path(filename,basedir)
+        if basedir is None:
+            _cache[filename] = result
+        else:
+            _cache[(filename,basedir)] = result
         return result
 
 def _set_parent_ns(packageName):
diff --git a/setuptools/command/develop.py b/setuptools/command/develop.py
index f128b80..fb1c05e 100755
--- a/setuptools/command/develop.py
+++ b/setuptools/command/develop.py
@@ -53,10 +53,15 @@ class develop(easy_install):
 
         self.egg_link = os.path.join(self.install_dir, ei.egg_name+'.egg-link')
         self.egg_base = ei.egg_base
+
+        target = normalize_path(self.egg_base)
+
         if self.egg_path is None:
             self.egg_path = os.path.abspath(ei.egg_base)
+            dist_location = target
+        else:
+            dist_location = self.egg_path
 
-        target = normalize_path(self.egg_base)
         if normalize_path(os.path.join(self.install_dir, self.egg_path)) != target:
             raise DistutilsOptionError(
                 "--egg-path must be a relative path from the install"
@@ -65,7 +70,7 @@ class develop(easy_install):
         
         # Make a distribution for the package's source
         self.dist = Distribution(
-            target,
+            dist_location,
             PathMetadata(target, os.path.abspath(ei.egg_info)),
             project_name = ei.egg_name
         )
diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
index af4e349..f1c9ff9 100755
--- a/setuptools/command/easy_install.py
+++ b/setuptools/command/easy_install.py
@@ -465,7 +465,7 @@ Please make the appropriate changes for your system and try again.
             # at this point, we know it's a local .egg, we just don't know if
             # it's already installed.
             for dist in self.local_index[spec.project_name]:
-                if dist.location==download:
+                if not self.pth_file.cmp_paths(dist.location, download):
                     break
             else:
                 install_needed = True   # it's not in the local index
@@ -946,18 +946,26 @@ See the setuptools documentation for the "develop" command for more info.
             return
 
         for d in self.pth_file[dist.key]:    # drop old entries
-            if self.multi_version or d.location != dist.location:
+            if (self.multi_version or
+                    self.pth_file.cmp_paths(d.location, dist.location)):
                 log.info("Removing %s from easy-install.pth file", d)
                 self.pth_file.remove(d)
                 if d.location in self.shadow_path:
                     self.shadow_path.remove(d.location)
 
         if not self.multi_version:
-            if dist.location in self.pth_file.paths:
-                log.info(
-                    "%s is already the active version in easy-install.pth",
-                    dist
-                )
+            if dist in self.pth_file:
+                updated = self.pth_file.update(dist)
+                if updated:
+                    log.info(
+                        "Updated %s to the active version in easy-install.pth",
+                        dist
+                    )
+                else:
+                    log.info(
+                        "%s is already the active version in easy-install.pth",
+                        dist
+                    )
             else:
                 log.info("Adding %s to easy-install.pth file", dist)
                 self.pth_file.add(dist) # add new entry
@@ -1320,12 +1328,19 @@ class PthDistributions(Environment):
         self.basedir = normalize_path(os.path.dirname(self.filename))
         self._load(); Environment.__init__(self, [], None, None)
         for path in yield_lines(self.paths):
-            map(self.add, find_distributions(path, True))
+            map(self.add, find_distributions(path, True, self.basedir))
+
+    def cmp_paths(self,left,right):
+        """Return 0 iff left and right refer to the same location
+        relative to self.basedir
+        """
+        return int(normalize_path(left, self.basedir) != \
+                normalize_path(right, self.basedir))
 
     def _load(self):
         self.paths = []
+        self.normalized_paths = dict.fromkeys(self.sitedirs)
         saw_import = False
-        seen = dict.fromkeys(self.sitedirs)
         if os.path.isfile(self.filename):
             for line in open(self.filename,'rt'):
                 if line.startswith('import'):
@@ -1337,14 +1352,14 @@ class PthDistributions(Environment):
                     continue
                 # skip non-existent paths, in case somebody deleted a package
                 # manually, and duplicate paths as well
-                path = self.paths[-1] = normalize_path(
-                    os.path.join(self.basedir,path)
-                )
-                if not os.path.exists(path) or path in seen:
+                real_path = normalize_path(path, self.basedir)
+                if not os.path.exists(real_path) or real_path in self.normalized_paths:
                     self.paths.pop()    # skip it
                     self.dirty = True   # we cleaned up, so we're dirty now :)
                     continue
-                seen[path] = 1
+                if real_path.startswith(self.basedir):
+                    path = self.paths[-1] = real_path
+                self.normalized_paths[real_path] = path
 
         if self.paths and not saw_import:
             self.dirty = True   # ensure anything we touch has import wrappers
@@ -1379,21 +1394,40 @@ class PthDistributions(Environment):
 
         self.dirty = False
 
+    def __contains__(self,dist):
+        """Return True iff `dist` is already in the distribution map"""
+        return normalize_path(dist.location, self.basedir) in self.normalized_paths
+
     def add(self,dist):
         """Add `dist` to the distribution map"""
-        if dist.location not in self.paths and dist.location not in self.sitedirs:
+        path = normalize_path(dist.location, self.basedir)
+        if path not in self.normalized_paths and path not in self.sitedirs:
+            self.normalized_paths[path] = dist.location
             self.paths.append(dist.location); self.dirty = True
         Environment.add(self,dist)
 
     def remove(self,dist):
         """Remove `dist` from the distribution map"""
-        while dist.location in self.paths:
-            self.paths.remove(dist.location); self.dirty = True
+        path = normalize_path(dist.location, self.basedir)
+        if path in self.normalized_paths:
+            self.paths.remove(self.normalized_paths[path])
+            del self.normalized_paths[path]; self.dirty = True
         Environment.remove(self,dist)
 
+    def update(self,dist):
+        """Update `dist` in the distribution map"""
+        path = normalize_path(dist.location, self.basedir)
+        if path in self.normalized_paths and dist.location not in self.paths:
+            self.paths.remove(self.normalized_paths[path])
+            self.paths.append(dist.location)
+            self.normalized_paths[path] = dist.location; self.dirty = True
+            return True
+        return False
 
     def make_relative(self,path):
-        npath, last = os.path.split(normalize_path(path))
+        if not path.startswith(self.basedir):
+            return path
+        npath, last = os.path.split(normalize_path(path, self.basedir))
         baselen = len(self.basedir)
         parts = [last]
         sep = os.altsep=='/' and '/' or os.sep

From donald.stufft at gmail.com  Wed Jul  4 07:39:24 2012
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 4 Jul 2012 01:39:24 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4E25F1BA-9F3D-4AC9-9697-293A93543AFA@twistedmatrix.com>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
	<CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
	<4E25F1BA-9F3D-4AC9-9697-293A93543AFA@twistedmatrix.com>
Message-ID: <E583C15F04924042B3357B38EB9BB984@gmail.com>

On Tuesday, July 3, 2012 at 9:29 PM, Glyph wrote:
> 
> Or, you know, somebody could maintain the dang software and automate the process of producing these hashes.  I am slightly baffled by the tone of this thread, like the hash algorithm needs to be set in stone forever.  There's a reason that most software treats hashes as pluggable: new algorithms come out every few years, you have to expect that your choice will be obsoleted for some reason (not necessarily just security!) in the future.  Granted, there's no real security in this case, but why not use a hash algorithm with less probability of collision?
> 
I tend to agree wrt to hashes and I have an outstanding pull request against pip to make it treat hashes as pluggable at least ;) 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120704/3a365301/attachment.html>

From a.badger at gmail.com  Wed Jul  4 08:45:22 2012
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Tue, 3 Jul 2012 23:45:22 -0700
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
Message-ID: <20120704064522.GB10315@unaka.lan>

On Tue, Jul 03, 2012 at 06:33:08PM -0500, Jennings, Jared L CTR USAF AFMC 46 SK/CCI wrote:
> On hosts configured for compliance with U.S. Federal Information
> Processing Standard (FIPS) 140-2
> <http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>, like
> those in some banks and, yes, the U.S. Department of Defense,
> cryptographic modules (such as OpenSSL, which underlies hashlib) are not
> allowed to calculate MD5 digests, because MD5 is no longer a FIPS
> Approved digest algorithm.
> 
> I know no one is trying here to lean on MD5 for security, but the
> standard says nothing about the reason why you're using MD5: just that
> you can't.
> 
> No one expects a digest algorithm to fail, and Python 2.x may not have
> been fixed to check for that before being frozen
> <https://bugzilla.redhat.com/show_bug.cgi?id=746118#c3>, so if you run
> an MD5 checksum on a FIPS-compliant system with an unpatched Python 2.x,
> the Python interpreter will segfault. (Ruby, too, had this problem and
> was itself only recently fixed,
> <http://bugs.ruby-lang.org/issues/4944>.)
> 
> I have to configure hosts in accordance with FIPS 140-2, so the more
> places I can get rid of MD5, the less headaches I have.
> 
I've just had to look into this for a bug in a package on Fedora and it's
not all bad but also not all good.  I believe that in current python2 and
python3 (including soon to be released python-3.3),  if it's compiled
against openssl, the md5 hash constructor will SIGABRT when in FIPS mode.
If it's compiled against the internal md5 code, it will ignore FIPS mode.
Dave Malcolm has a patch in the tracker that hasn't yet been approved and
merged that allows one to pass a flag to the hash constructor that says that
the call is not being used for cryptographic purposes and then the
constructor will work even in FIPS mode.  I've seen no indication in the
tracker that this would be applied to future python-2.7.x releases, but it
could be backported by individual distributors of python2 (for instance,
Linux distributions).

A version of the patch is presently applied to the Fedora Linux 17 versions
of python2 and python3 if someone is curious.

Note that openssl itself allows the use of MD5 in FIPS mode under a similar
strategy.  So I'm not entirely certain that the standard forbids use of MD5
for non-cryptographic purposes.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120703/2902f23b/attachment.pgp>

From tarek at ziade.org  Wed Jul  4 09:35:06 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 04 Jul 2012 09:35:06 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
	<CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
Message-ID: <4FF3F22A.7010708@ziade.org>

On 7/4/12 2:50 AM, PJ Eby wrote:
> On Tue, Jul 3, 2012 at 7:33 PM, Jennings, Jared L CTR USAF AFMC 46 
> SK/CCI <jared.jennings.ctr at eglin.af.mil 
> <mailto:jared.jennings.ctr at eglin.af.mil>> wrote:
>
>     On hosts configured for compliance with U.S. Federal Information
>     Processing Standard (FIPS) 140-2
>     <http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>, like
>     those in some banks and, yes, the U.S. Department of Defense,
>     cryptographic modules (such as OpenSSL, which underlies hashlib)
>     are not
>     allowed to calculate MD5 digests, because MD5 is no longer a FIPS
>     Approved digest algorithm.
>
>
> So if it's not a cryptographic module, it's okay?  ;-)
>
>     I know no one is trying here to lean on MD5 for security, but the
>     standard says nothing about the reason why you're using MD5: just that
>     you can't.
>
>     No one expects a digest algorithm to fail, and Python 2.x may not have
>     been fixed to check for that before being frozen
>     <https://bugzilla.redhat.com/show_bug.cgi?id=746118#c3>, so if you run
>     an MD5 checksum on a FIPS-compliant system with an unpatched
>     Python 2.x,
>     the Python interpreter will segfault. (Ruby, too, had this problem and
>     was itself only recently fixed,
>     <http://bugs.ruby-lang.org/issues/4944>.)
>
>     I have to configure hosts in accordance with FIPS 140-2, so the more
>     places I can get rid of MD5, the less headaches I have.
>
>
> If we replace it with something else, then I suggest we replace it 
> with something that's even MORE braindead than md5 so that nobody will 
> mistake it for a secure hash.  Otherwise, we will have this exact same 
> problem all over again when the replacement "secure" hash is disabled 
> by a newer version of FIPS.
>
> The other option is simply to forego a checksum altogether and assume 
> same size = same file.  Honestly, I don't remember why we cared about 
> detecting such modifications in the first place: neither PEP 376 nor 
> 262 explain why, and 376 doesn't explain why it went with md5 instead 
> of sha1 (as in PEP 262).

I wanted to be able to offer a way for installers to detect that a file 
was changed to avoid deleting it for instance, and issue a warning to 
the user -- or maybe give a chance to the installer to save a copy of 
the file somewhere.

I picked md5 because I wanted it brain dead and could not imagine that 
would  be an issue somehow. Maybe zlib.crc32 would be a better choice.

If we remove the hash, oh well. no big deal I guess. If an installer 
wants to add this feature it can maintain hashes itself.


>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120704/2af27806/attachment.html>

From tarek at ziade.org  Wed Jul  4 09:47:32 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 04 Jul 2012 09:47:32 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF3F22A.7010708@ziade.org>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
	<CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
	<4FF3F22A.7010708@ziade.org>
Message-ID: <4FF3F514.8090004@ziade.org>

On 7/4/12 9:35 AM, Tarek Ziad? wrote:
> On 7/4/12 2:50 AM, PJ Eby wrote:
>> On Tue, Jul 3, 2012 at 7:33 PM, Jennings, Jared L CTR USAF AFMC 46 
>> SK/CCI <jared.jennings.ctr at eglin.af.mil 
>> <mailto:jared.jennings.ctr at eglin.af.mil>> wrote:
>>
>>     On hosts configured for compliance with U.S. Federal Information
>>     Processing Standard (FIPS) 140-2
>>     <http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>, like
>>     those in some banks and, yes, the U.S. Department of Defense,
>>     cryptographic modules (such as OpenSSL, which underlies hashlib)
>>     are not
>>     allowed to calculate MD5 digests, because MD5 is no longer a FIPS
>>     Approved digest algorithm.
>>
>>
>> So if it's not a cryptographic module, it's okay?  ;-)
>>
>>     I know no one is trying here to lean on MD5 for security, but the
>>     standard says nothing about the reason why you're using MD5: just
>>     that
>>     you can't.
>>
>>     No one expects a digest algorithm to fail, and Python 2.x may not
>>     have
>>     been fixed to check for that before being frozen
>>     <https://bugzilla.redhat.com/show_bug.cgi?id=746118#c3>, so if
>>     you run
>>     an MD5 checksum on a FIPS-compliant system with an unpatched
>>     Python 2.x,
>>     the Python interpreter will segfault. (Ruby, too, had this
>>     problem and
>>     was itself only recently fixed,
>>     <http://bugs.ruby-lang.org/issues/4944>.)
>>
>>     I have to configure hosts in accordance with FIPS 140-2, so the more
>>     places I can get rid of MD5, the less headaches I have.
>>
>>
>> If we replace it with something else, then I suggest we replace it 
>> with something that's even MORE braindead than md5 so that nobody 
>> will mistake it for a secure hash.  Otherwise, we will have this 
>> exact same problem all over again when the replacement "secure" hash 
>> is disabled by a newer version of FIPS.
>>
>> The other option is simply to forego a checksum altogether and assume 
>> same size = same file.  Honestly, I don't remember why we cared about 
>> detecting such modifications in the first place: neither PEP 376 nor 
>> 262 explain why, and 376 doesn't explain why it went with md5 instead 
>> of sha1 (as in PEP 262).
>
> I wanted to be able to offer a way for installers to detect that a 
> file was changed to avoid deleting it for instance, and issue a 
> warning to the user -- or maybe give a chance to the installer to save 
> a copy of the file somewhere.
>
> I picked md5 because I wanted it brain dead and could not imagine that 
> would  be an issue somehow. Maybe zlib.crc32 would be a better choice.

Oh let's do a fletcher checksum ! This one should be universally 
authorized by any system

grabbed on the web:

def fletcher(path, modulus=255):

     with open(path) as f:
         data = f.read()
     numbers = map(ord, data)
     a = b = 0
     for number in numbers:
         a += number
         b += a
     a %= modulus
     b %= modulus
     return hex((a << 8) | b)[2:].upper().zfill(4)



>
> If we remove the hash, oh well. no big deal I guess. If an installer 
> wants to add this feature it can maintain hashes itself.
>
>
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120704/527cf018/attachment-0001.html>

From tarek at ziade.org  Wed Jul  4 11:07:12 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 04 Jul 2012 11:07:12 +0200
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF3F514.8090004@ziade.org>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
	<CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
	<4FF3F22A.7010708@ziade.org> <4FF3F514.8090004@ziade.org>
Message-ID: <4FF407C0.2060604@ziade.org>

On 7/4/12 9:47 AM, Tarek Ziad? wrote:

> Oh let's do a fletcher checksum ! This one should be universally 
> authorized by any system
>
Better version:

http://tarek.pastebin.mozilla.org/1690480

takes 4 seconds on my MBA for a 40 mb file, so it seems fast enough 
since our PyPI limit is 8 Mb.



From dholth at gmail.com  Wed Jul  4 13:40:09 2012
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 4 Jul 2012 06:40:09 -0500
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
Message-ID: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>

Hopefully this will be a more useful spec.

This begs for parenthesis in environment markers, something that is
easy to do by compiling markers to Python functions using the ast as
`markerlib` does.


Provides-Extra (multiple use)
:::::::::::::::::::::::::::::
A string containing the name of an optional feature. Must be a valid Python
identifier. May be used to make a dependency conditional on whether the
optional feature has been requested.
Example::
    Provides-Extra: pdf
    Requires-Dist: reportlab; extra == 'pdf'

...

Full changeset at
https://bitbucket.org/dholth/python-peps/changeset/4dc72bbf9ed2

From pje at telecommunity.com  Wed Jul  4 15:21:51 2012
From: pje at telecommunity.com (PJ Eby)
Date: Wed, 4 Jul 2012 09:21:51 -0400
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF407C0.2060604@ziade.org>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
	<CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
	<4FF3F22A.7010708@ziade.org> <4FF3F514.8090004@ziade.org>
	<4FF407C0.2060604@ziade.org>
Message-ID: <CALeMXf7eWiEXGYW63LQ_=1Vg190VqCdjbOmxH8he7FuQ8c-27w@mail.gmail.com>

On Wed, Jul 4, 2012 at 5:07 AM, Tarek Ziad? <tarek at ziade.org> wrote:

> On 7/4/12 9:47 AM, Tarek Ziad? wrote:
>
>  Oh let's do a fletcher checksum ! This one should be universally
>> authorized by any system
>>
>>  Better version:
>
> http://tarek.pastebin.mozilla.**org/1690480<http://tarek.pastebin.mozilla.org/1690480>
>
> takes 4 seconds on my MBA for a 40 mb file, so it seems fast enough since
> our PyPI limit is 8 Mb.


It's too bad Python's built-in hash() isn't guaranteed consistent across
versions and implementations, otherwise we could just use that!  (Or more
precisely, a spec for how to combine the Python hashes of specified-size
blocks.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120704/dd244da5/attachment.html>

From pje at telecommunity.com  Wed Jul  4 15:24:02 2012
From: pje at telecommunity.com (PJ Eby)
Date: Wed, 4 Jul 2012 09:24:02 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
Message-ID: <CALeMXf7W9S4UmuBtxxqRbBwvj6KVUyYn-JxEN-_ZspgY1=6Lzw@mail.gmail.com>

On Wed, Jul 4, 2012 at 7:40 AM, Daniel Holth <dholth at gmail.com> wrote:

>
> Provides-Extra (multiple use)
> :::::::::::::::::::::::::::::
> A string containing the name of an optional feature. Must be a valid Python
> identifier. May be used to make a dependency conditional on whether the
> optional feature has been requested.
> Example::
>     Provides-Extra: pdf
>     Requires-Dist: reportlab; extra == 'pdf'
>
>
+1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120704/5cde6f58/attachment.html>

From dholth at gmail.com  Wed Jul  4 20:51:39 2012
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 4 Jul 2012 13:51:39 -0500
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <4FF407C0.2060604@ziade.org>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
	<CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
	<4FF3F22A.7010708@ziade.org> <4FF3F514.8090004@ziade.org>
	<4FF407C0.2060604@ziade.org>
Message-ID: <CAG8k2+4cDptvWrf0bVfy1S0ivvetuqu=Eg1rHB9z8uOXwB5ExA@mail.gmail.com>

Every hash function is fast enough. Sha256: > 100 megabytes per second on a
single core. Size of one of my normal virtualenv: < 50. The proposal for
record just makes the hash pluggable, so if you have a slow machine and a
very fast disk and verifying distributions is taking too long then you can
do something about it.

I think the skein hash is even faster than md5 while also being modern, but
Jared surely can't use it unless it becomes SHA-3.

But if you really want to save some actual time, use binary packages :-)

Pip install lxml - 1m 51s
Pip install -f file:///temp/wheels lxml - 27s

I am not sure why pip is so slow for me. The lxml binary package install
could take as little as 0.1 seconds if pip wasn't consulting the net.

RPM hashes installed files. It is mostly to avoid accidentally deleting
edited configs, but you can "rpm verify" for other reasons if you want.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120704/891a3b8b/attachment.html>

From carl at oddbird.net  Thu Jul  5 18:46:39 2012
From: carl at oddbird.net (Carl Meyer)
Date: Thu, 05 Jul 2012 10:46:39 -0600
Subject: [Distutils] Proposal: drop md5 for sha256
In-Reply-To: <CAG8k2+4cDptvWrf0bVfy1S0ivvetuqu=Eg1rHB9z8uOXwB5ExA@mail.gmail.com>
References: <09B8D4A01E10D14CA028A4D0CD5779D90294FCCD@VFEGMLEG03.Enterprise.afmc.ds.af.mil>
	<CALeMXf7RG100Ld2gchuJWTkRQCaJbUh7TPRtLCcU1rNnVRcMsg@mail.gmail.com>
	<4FF3F22A.7010708@ziade.org> <4FF3F514.8090004@ziade.org>
	<4FF407C0.2060604@ziade.org>
	<CAG8k2+4cDptvWrf0bVfy1S0ivvetuqu=Eg1rHB9z8uOXwB5ExA@mail.gmail.com>
Message-ID: <4FF5C4EF.9080604@oddbird.net>

On 07/04/2012 12:51 PM, Daniel Holth wrote:
> Pip install lxml - 1m 51s
> Pip install -f file:///temp/wheels lxml - 27s
> 
> I am not sure why pip is so slow for me. The lxml binary package install
> could take as little as 0.1 seconds if pip wasn't consulting the net.

If you don't want pip to consult the network, use the --no-index flag
along with --find-links.

Carl

From dholth at gmail.com  Fri Jul  6 16:32:51 2012
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 6 Jul 2012 10:32:51 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CALeMXf7W9S4UmuBtxxqRbBwvj6KVUyYn-JxEN-_ZspgY1=6Lzw@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<CALeMXf7W9S4UmuBtxxqRbBwvj6KVUyYn-JxEN-_ZspgY1=6Lzw@mail.gmail.com>
Message-ID: <CAG8k2+5VgRxD+LKku-NZFN2uquexFKbCRY68TtgZ-vfQrzdP1w@mail.gmail.com>

>> Add Provides-Extra: to PEP 345 and clarify Requires-Dist: name (1) is the same as Requires-Dist: name (==1)

> +1

Great. How do we get https://bitbucket.org/dholth/python-peps merged
into hg.python.org/peps/ ?

From tseaver at palladion.com  Fri Jul  6 16:40:59 2012
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 06 Jul 2012 10:40:59 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+5VgRxD+LKku-NZFN2uquexFKbCRY68TtgZ-vfQrzdP1w@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<CALeMXf7W9S4UmuBtxxqRbBwvj6KVUyYn-JxEN-_ZspgY1=6Lzw@mail.gmail.com>
	<CAG8k2+5VgRxD+LKku-NZFN2uquexFKbCRY68TtgZ-vfQrzdP1w@mail.gmail.com>
Message-ID: <jt6tdq$4sl$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/06/2012 10:32 AM, Daniel Holth wrote:
>>> Add Provides-Extra: to PEP 345 and clarify Requires-Dist: name (1)
>>> is the same as Requires-Dist: name (==1)
> 
>> +1
> 
> Great. How do we get https://bitbucket.org/dholth/python-peps merged 
> into hg.python.org/peps/ ?

Given that PEP 345 is already in 'Accepted' state, I'm not sure we can
just merge it in.  We might need to create a new PEP which extends PEP
345 and adds the feature (it likely needs to bump the 'Metadata-Version',
too).


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/2+PQACgkQ+gerLs4ltQ57KwCfWR5W+gL0bgLOLOcyyOlXCI7e
KTIAoLDLUTjNQjw9VA9Hx8R4rYbW5XqZ
=pl62
-----END PGP SIGNATURE-----


From merwok at netwok.org  Fri Jul  6 17:01:33 2012
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Fri, 06 Jul 2012 11:01:33 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
Message-ID: <4FF6FDCD.8010000@netwok.org>

Hi,

> Hopefully this will be a more useful spec.
Thanks for the proposal, but it is not really a spec.  You?re suggesting 
name and syntax for a new field, but you don?t describe what it means 
and what it is used for.  From what I understand, the goal is similar to 
build dependencies and optional dependencies in systems like Debian?s 
dpkg; for the build dependencies, there were multiple threads about 
adding Setup-Requires-Dist (like setuptools? setup_requires) and 
Test-Requires-Dist (like tests_require, or whatever the spelling is?I 
always get it wrong :), for which we were near to agreement.  See 
https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-packaging/setup-requires/the-fellowship-of-the-packaging/KVsDQawfEno/M-ODKx9Ne5YJ 
and 
https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-packaging/setup-requires/the-fellowship-of-the-packaging/yTsRLtl6eoE/fVGb4t2zm68J

> This begs for parenthesis in environment markers, something that is
> easy to do by compiling markers to Python functions using the ast as
> `markerlib` does.
distutils2 supports Python 2.5, which has _ast but not ast IIRC.  If you 
have other suggestions for improvements like this please open bug 
reports though!  Feedback and patches are welcome.

Regards

From dholth at gmail.com  Fri Jul  6 17:07:42 2012
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 6 Jul 2012 11:07:42 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <jt6tdq$4sl$1@dough.gmane.org>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<CALeMXf7W9S4UmuBtxxqRbBwvj6KVUyYn-JxEN-_ZspgY1=6Lzw@mail.gmail.com>
	<CAG8k2+5VgRxD+LKku-NZFN2uquexFKbCRY68TtgZ-vfQrzdP1w@mail.gmail.com>
	<jt6tdq$4sl$1@dough.gmane.org>
Message-ID: <CAG8k2+5LrAb3XtbZoFDV5gBicp3fX+jHpYcRg5WpNi4quguo0A@mail.gmail.com>

> Given that PEP 345 is already in 'Accepted' state, I'm not sure we can
> just merge it in.  We might need to create a new PEP which extends PEP
> 345 and adds the feature (it likely needs to bump the 'Metadata-Version',
> too).

I would have thought so too, but then I downloaded 16,984 of the
newest sdists hosted on pypi, parsed their PKG-INFO or METADATA files,
and discovered adoption is quite low. Also, it is not Final.

 'Metadata-Version: 1.0': 15384,
 'Metadata-Version: 1.02': 1,
 'Metadata-Version: 1.1': 1383,
 'Metadata-Version: 1.2': 7,

From dholth at gmail.com  Fri Jul  6 17:20:35 2012
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 6 Jul 2012 11:20:35 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <4FF6FDCD.8010000@netwok.org>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
Message-ID: <CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>

On Fri, Jul 6, 2012 at 11:01 AM, ?ric Araujo <merwok at netwok.org> wrote:
>> Hopefully this will be a more useful spec.
>
> Thanks for the proposal, but it is not really a spec.  You?re suggesting
> name and syntax for a new field, but you don?t describe what it means and
> what it is used for.  From what I understand, the goal is similar to build
> dependencies and optional dependencies in systems like Debian?s dpkg; for
> the build dependencies, there were multiple threads about adding
> Setup-Requires-Dist (like setuptools? setup_requires) and Test-Requires-Dist
> (like tests_require, or whatever the spelling is?I always get it wrong :),
> for which we were near to agreement.  See
> https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-packaging/setup-requires/the-fellowship-of-the-packaging/KVsDQawfEno/M-ODKx9Ne5YJ
> and
> https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-packaging/setup-requires/the-fellowship-of-the-packaging/yTsRLtl6eoE/fVGb4t2zm68J

It is exactly the same as setuptools' "extras" feature, and I didn't
quote the entire edit on-list. Also, there is a reference
implementation at https://bitbucket.org/dholth/distribute/ in the
DistInfoDistribution class.

The 'test' and 'doc' reserved names are not an essential part of the
Provides-Extra feature. Provides-Extra: is primarily for optional
run-time (not build-time) dependencies.

>> This begs for parenthesis in environment markers, something that is
>> easy to do by compiling markers to Python functions using the ast as
>> `markerlib` does.
>
> distutils2 supports Python 2.5, which has _ast but not ast IIRC.  If you
> have other suggestions for improvements like this please open bug reports
> though!  Feedback and patches are welcome.

markerlib is not required for Provides-Extra: It was just a convenient
way to add the additional 'extra' variable name to environment
markers.

markerlib uses a few helpers from ast.py. Python 2.5 support would
mean copying those helpers into markerlib.

From brian at vanguardistas.net  Fri Jul  6 17:45:25 2012
From: brian at vanguardistas.net (Brian Sutherland)
Date: Fri, 6 Jul 2012 17:45:25 +0200
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <4FF6FDCD.8010000@netwok.org>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
Message-ID: <20120706154525.GA55169@minimac.lan>

On Fri, Jul 06, 2012 at 11:01:33AM -0400, ?ric Araujo wrote:
> Hi,
> 
> >Hopefully this will be a more useful spec.
> Thanks for the proposal, but it is not really a spec.  You?re
> suggesting name and syntax for a new field, but you don?t describe
> what it means and what it is used for.  From what I understand, the
> goal is similar to build dependencies and optional dependencies in
> systems like Debian?s dpkg;

You can also think of extras as similar to Debian's metapackages. For
example, looking at the zope.component packaging:

    http://packages.debian.org/source/sid/zope.component

The zope.component [zcml] extra is translated to metapackage
(python-zope.component-zcml) that depends on python-zope.component and
python-zope.configuration.

-- 
Brian Sutherland

From dholth at gmail.com  Fri Jul  6 17:45:15 2012
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 6 Jul 2012 11:45:15 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
Message-ID: <CAG8k2+7gWpag-tWved-jTVS1+oPgQRXjFncx8RA+8Fq_9+irJg@mail.gmail.com>

You're right, the example could be better.

It works like so:

beaglevote's METADATA:

Provides-Extra: quux
Provides-Extra: pdf
Requires-Dist: doghouse
Requires-Dist: sphinx; extra == 'quux'
Requires-Dist: reportlab; extra == 'pdf'

"install beaglevote" -> evaluate Requires-Dist: with extra == None: ->
requires doghouse

"install beaglevote[pdf]" -> set(Requires-Dist: with extra == None) +
set(Requires-Dist: with extra == 'pdf') -> requires doghouse,
reportlab

"install beaglevote[quux]" -> set(Requires-Dist: with extra == None) +
set(Requires-Dist: with extra == 'quux') -> requires doghouse, sphinx

"install beaglevote[pearl]" -> error, 'pearl' is not mentioned in
Provides-Extra:


Extras are only supposed to add requirements. To make things as
predictable as possible, extra is a string, not a list, requirements
are always evaluated with extra == None first, and the requirements
are re-evaluated for each requested extra individually. This is to
avoid mischief like

Requires-Dist: something; 'foo' in extra and not 'bar' in extra

Requires-Dist: another; extra != 'quux'


Parenthesis would be desirable so you can say:

Requires-Dist: splort; (python_version < '2.6' or sys.platform ==
'win32') and extra == 'blurble'

From dholth at gmail.com  Fri Jul  6 21:33:08 2012
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 6 Jul 2012 15:33:08 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+7gWpag-tWved-jTVS1+oPgQRXjFncx8RA+8Fq_9+irJg@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAG8k2+7gWpag-tWved-jTVS1+oPgQRXjFncx8RA+8Fq_9+irJg@mail.gmail.com>
Message-ID: <CAG8k2+72fcu3q5OtOoW7oVVY7dZAQ3gjeotKif3_RwUSk_xgpg@mail.gmail.com>

And some stats: of the 16k sdists I surveyed, 9484 would need to use
'Requires-Dist:', and 1624 define what would be a Provides-Extra:
line.

I found 483 unique extra names.

So Provides-Extra: really has nothing to do with tests or docs. I'm
sorry I included that distracting remark in the specification.
Provides-Extra: is all about requiring dependencies that enable an
optional feature of a distribution.

Some of the most popular extra names:

[(5, 'kid'),
 (5, 'lxml'),
 (5, 'oracle'),
 (5, 'pylons'),
 (5, 'simplejson'),
 (5, 'wsgi'),
 (5, 'xml'),
 (6, 'all'),
 (6, 'django'),
 (6, 'jinja2'),
 (6, 'nose'),
 (7, 'sphinx'),
 (8, 'mysql'),
 (9, 'doc'),
 (10, 'zcml'),
 (12, 'app'),
 (12, 'zopesupport'),
 (13, 'dev'),
 (20, 'genshi'),
 (22, 'mako'),
 (41, 'testing'),
 (60, 'docs'),
 (305, 'tests'),
 (1053, 'test')]

From setuptools at bugs.python.org  Sat Jul  7 17:27:09 2012
From: setuptools at bugs.python.org (fonnesbeck)
Date: Sat, 07 Jul 2012 15:27:09 +0000
Subject: [Distutils] [issue141] Setuptools looks for dependencies even when
	installed
Message-ID: <1341674829.95.0.687130546139.issue141@psf.upfronthosting.co.za>


New submission from fonnesbeck <fonnesbeck at gmail.com>:

When installing a package using setuptools, the installer looks for and tries to install dependencies even when they are already satisfied. For example, a package that requires scipy (no version requirement specified) looks for a version online even though I have a very current version already installed:


Processing dependencies for mcex==0.1
Searching for scipy
Reading http://pypi.python.org/simple/scipy/
Reading http://www.scipy.org
Reading http://sourceforge.net/project/showfiles.php?group_id=27747&package_id=19531
Reading http://new.scipy.org/Wiki/Download
Best match: scipy 0.10.1

----------
messages: 664
nosy: fonnesbeck
priority: bug
status: unread
title: Setuptools looks for dependencies even when installed

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue141>
_______________________________________________

From setuptools at bugs.python.org  Mon Jul  9 17:13:45 2012
From: setuptools at bugs.python.org (Tres Seaver)
Date: Mon, 09 Jul 2012 15:13:45 +0000
Subject: [Distutils] [issue142] easy_install ignores 'headers' passed to
	setup()
Message-ID: <1341846825.8.0.0531672503538.issue142@psf.upfronthosting.co.za>


New submission from Tres Seaver <tseaver at palladion.com>:

A bare distutils install of a package with 'headers' passed to 'setup()'
dispatches to 'install_headers', which copies the exported header files
into the system include path.

'easy_install' bypasses the distutils 'install()' when doing a normal
egg install, and does not re-implement the 'install_headers' dance.

Furthermore, the 'bdist_egg' it creates, even though it contains the
header files, has no metadata which would allow the installer to
discover that they should be copied.

----------
messages: 669
nosy: tseaver
priority: bug
status: unread
title: easy_install ignores 'headers' passed to setup()

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue142>
_______________________________________________

From tseaver at palladion.com  Mon Jul  9 18:11:23 2012
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 09 Jul 2012 12:11:23 -0400
Subject: [Distutils] [issue142] easy_install ignores 'headers' passed to
	setup()
In-Reply-To: <1341846825.8.0.0531672503538.issue142@psf.upfronthosting.co.za>
References: <1341846825.8.0.0531672503538.issue142@psf.upfronthosting.co.za>
Message-ID: <jtevr9$t7h$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/09/2012 11:13 AM, Tres Seaver wrote:
> 
> New submission from Tres Seaver <tseaver at palladion.com>:
> 
> A bare distutils install of a package with 'headers' passed to
> 'setup()' dispatches to 'install_headers', which copies the exported
> header files into the system include path.
> 
> 'easy_install' bypasses the distutils 'install()' when doing a normal 
> egg install, and does not re-implement the 'install_headers' dance.
> 
> Furthermore, the 'bdist_egg' it creates, even though it contains the 
> header files, has no metadata which would allow the installer to 
> discover that they should be copied.

Note that distribute has the same bug, which I reported here:

 https://bitbucket.org/tarek/distribute/issue/295


- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/7AqQACgkQ+gerLs4ltQ4/kwCfWEBYSY+dlEfD0b/QNxXnPNxA
lPcAoIj0XsQCu2AbPuVWYLEGmWpfKL4Y
=VNY3
-----END PGP SIGNATURE-----


From erik.m.bray at gmail.com  Mon Jul  9 18:57:50 2012
From: erik.m.bray at gmail.com (Erik Bray)
Date: Mon, 9 Jul 2012 12:57:50 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
Message-ID: <CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>

On Fri, Jul 6, 2012 at 11:20 AM, Daniel Holth <dholth at gmail.com> wrote:
> It is exactly the same as setuptools' "extras" feature, and I didn't
> quote the entire edit on-list. Also, there is a reference
> implementation at https://bitbucket.org/dholth/distribute/ in the
> DistInfoDistribution class.
>
> The 'test' and 'doc' reserved names are not an essential part of the
> Provides-Extra feature. Provides-Extra: is primarily for optional
> run-time (not build-time) dependencies.

Though in the interest of not having too many different ways to do a
similar thing, could this not also be used for tests/docs?  Perhaps
even "Provides-Extra: tests", and "Provides-Extra: docs" should be
reserved names as you suggested.  I envision something like:


Provides-Extra: tests
Requires-Dist: nose; extra == 'tests'
Provides-Extra: docs
Requires-Dist: sphinx; extra == 'docs'


$ pysetup test
Tests require the 'nose' package. Please run `pysetup install
mypackage[tests]` to install the test dependencies.

Ditto for building the docs.  In a sense, being able to run the tests,
or build the docs could be considered 'extra' features.  One could
even have:

Provides-Extra: setup
Requires-Dist: mysetuphooks; extra == 'setup'

for build-time dependencies.  Although in that case, pysetup (or any
other installer) would explicitly have to check for this in the
metadata before doing anything else.  So maybe this example stretches
this syntax too far?  Plus build-time dependencies are not exactly
"extras".  It's very much required to do anything with the package.

Erik

From dholth at gmail.com  Mon Jul  9 19:27:17 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 9 Jul 2012 13:27:17 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
Message-ID: <CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>

> Though in the interest of not having too many different ways to do a
> similar thing, could this not also be used for tests/docs?  Perhaps
> even "Provides-Extra: tests", and "Provides-Extra: docs" should be
> reserved names as you suggested.  I envision something like:

...

> Provides-Extra: setup
> Requires-Dist: mysetuphooks; extra == 'setup'
>
> for build-time dependencies.  Although in that case, pysetup (or any
> other installer) would explicitly have to check for this in the
> metadata before doing anything else.  So maybe this example stretches
> this syntax too far?  Plus build-time dependencies are not exactly
> "extras".  It's very much required to do anything with the package.

Pre-built or binary packages won't want to install the build-time
dependencies. The check for a particular Provides-Extra is trivial,
just a list membership test.

I expect to see a lot of bugs in packages that require some of their
build-time dependencies at run time.

From erik.m.bray at gmail.com  Mon Jul  9 21:35:43 2012
From: erik.m.bray at gmail.com (Erik Bray)
Date: Mon, 9 Jul 2012 15:35:43 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
Message-ID: <CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>

On Mon, Jul 9, 2012 at 1:27 PM, Daniel Holth <dholth at gmail.com> wrote:
>> Though in the interest of not having too many different ways to do a
>> similar thing, could this not also be used for tests/docs?  Perhaps
>> even "Provides-Extra: tests", and "Provides-Extra: docs" should be
>> reserved names as you suggested.  I envision something like:
>
> ...
>
>> Provides-Extra: setup
>> Requires-Dist: mysetuphooks; extra == 'setup'
>>
>> for build-time dependencies.  Although in that case, pysetup (or any
>> other installer) would explicitly have to check for this in the
>> metadata before doing anything else.  So maybe this example stretches
>> this syntax too far?  Plus build-time dependencies are not exactly
>> "extras".  It's very much required to do anything with the package.
>
> Pre-built or binary packages won't want to install the build-time
> dependencies. The check for a particular Provides-Extra is trivial,
> just a list membership test.
>
> I expect to see a lot of bugs in packages that require some of their
> build-time dependencies at run time.

Maybe, but wouldn't that be the developer's problem?  setup_requires
has the same issue--it does not install the setup requirements into
site-packages.  Instead it just does an egg install into the source
directory and activates it on the path.  Outside of future setup.py
runs, the setup_requires distributions are not available (you would
have to add them to install_requires too if you need them at runtime).

Under a scheme like this one would have to list that dependency under
Requires-Dist twice: with and without the 'extra' marker.  I might
still prefer extending the metadata format to add a
Setup-Requires-Dist or the like.

Erik

From dholth at gmail.com  Mon Jul  9 22:37:42 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 9 Jul 2012 16:37:42 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
	<CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
Message-ID: <CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>

>> I expect to see a lot of bugs in packages that require some of their
>> build-time dependencies at run time.
>
> Maybe, but wouldn't that be the developer's problem?  setup_requires
> has the same issue--it does not install the setup requirements into
> site-packages.  Instead it just does an egg install into the source
> directory and activates it on the path.  Outside of future setup.py
> runs, the setup_requires distributions are not available (you would
> have to add them to install_requires too if you need them at runtime).

OK, it sounds like this "bug" has been mostly anticipated and taken
care of already.

> Under a scheme like this one would have to list that dependency under
> Requires-Dist twice: with and without the 'extra' marker.  I might
> still prefer extending the metadata format to add a
> Setup-Requires-Dist or the like.

I don't follow. Does anyone build/install packages without also first
installing the run-time requirements? Surely Setup-Requires-Dist would
have exactly the same problem. It seems like two ways to say exactly
the same thing.

Let's just try to focus on getting Metadata 1.2 out the door and
implemented before its 8th birthday, preferably during Julython... The
only thing I need is to be able to represent the wildly popular
"extras" feature in METADATA to benefit from PEP 345 and 376 without
rewriting thousands of Python packages. That is all. Save the debate
about reserved "extra" names for setup / build / test / doc
dependencies for Metadata 1.3.

From erik.m.bray at gmail.com  Tue Jul 10 00:14:19 2012
From: erik.m.bray at gmail.com (Erik Bray)
Date: Mon, 9 Jul 2012 18:14:19 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
	<CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
	<CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>
Message-ID: <CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>

On Mon, Jul 9, 2012 at 4:37 PM, Daniel Holth <dholth at gmail.com> wrote:
>>> I expect to see a lot of bugs in packages that require some of their
>>> build-time dependencies at run time.
>>
>> Maybe, but wouldn't that be the developer's problem?  setup_requires
>> has the same issue--it does not install the setup requirements into
>> site-packages.  Instead it just does an egg install into the source
>> directory and activates it on the path.  Outside of future setup.py
>> runs, the setup_requires distributions are not available (you would
>> have to add them to install_requires too if you need them at runtime).
>
> OK, it sounds like this "bug" has been mostly anticipated and taken
> care of already.
>
>> Under a scheme like this one would have to list that dependency under
>> Requires-Dist twice: with and without the 'extra' marker.  I might
>> still prefer extending the metadata format to add a
>> Setup-Requires-Dist or the like.
>
> I don't follow. Does anyone build/install packages without also first
> installing the run-time requirements? Surely Setup-Requires-Dist would
> have exactly the same problem. It seems like two ways to say exactly
> the same thing.

The difference is that setup/build dependencies generally should be
downloaded, extracted, and added to the path before another commands
are run, because the packaging framework allows for the behavior of
built-in commands themselves to be modified, as well as certain
metadata.  The usefulness of this has been discussed in past threads.
See

http://mail.python.org/pipermail/distutils-sig/2012-May/018627.html

and the thread Eric Araujo pointed out at:

https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-packaging/setup-requires/the-fellowship-of-the-packaging/KVsDQawfEno/M-ODKx9Ne5YJ

> Let's just try to focus on getting Metadata 1.2 out the door and
> implemented before its 8th birthday, preferably during Julython... The
> only thing I need is to be able to represent the wildly popular
> "extras" feature in METADATA to benefit from PEP 345 and 376 without
> rewriting thousands of Python packages. That is all. Save the debate
> about reserved "extra" names for setup / build / test / doc
> dependencies for Metadata 1.3.

I don't have data on the "popularity" of setup_requires--is there some
easy way to find that out without downloading an entire PyPI mirror?

That said, I'm anticipating that it would be very useful for
packaging.  As it is, setup.py files tend to contain a lot of
repetitive boilerplate, especially for non-pure-Python packages.  With
the elimination of setup.py, more packages are going to require setup
hook-script modules to be shipped with their code, and I could see
some of these being highly reusable.

I'm certain this is a useful feature--just not so certain whether it's
desirable needed to modify the Metadata spec to account for it.

In the worst case, I could imagine writing some kind of "bootstrap"
module that can be dropped into a project and used via a setup-hook
that is then possible of downloading other build-time dependencies.
But that's exactly the sort of thing I'm trying to get away from--if I
have to maintain a bootstrap script (or another module containing
setup-hooks, custom commands, etc.) I have to update every project
that uses it every time those scripts are updated.

Erik

From dholth at gmail.com  Tue Jul 10 01:38:42 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 9 Jul 2012 19:38:42 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
	<CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
	<CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>
	<CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>
Message-ID: <CAG8k2+59NX6LHMbsf+CW4NrkuvyWKktg7WW0SGg5FVNU_dyA3A@mail.gmail.com>

>>> Under a scheme like this one would have to list that dependency under
>>> Requires-Dist twice: with and without the 'extra' marker.  I might
>>> still prefer extending the metadata format to add a
>>> Setup-Requires-Dist or the like.

> The difference is that setup/build dependencies generally should be
> downloaded, extracted, and added to the path before another commands
> are run, because the packaging framework allows for the behavior of
> built-in commands themselves to be modified, as well as certain
> metadata.  The usefulness of this has been discussed in past threads.

Setup requirements are certainly necessary.

I see what you mean now. 'Setup-Requires-Dist' is a different feature
than 'Provides-Extra', since with 'Provides-Extra' there isn't an
elegant way to figure out what "just the setup requirements are".
Provides-Extra is only useful if you want to install the mandatory
runtime requirements at the same time (which is exactly what you want
to do for 'extras').

You could make the argument that you only need setup_requires_dist in
setup.cfg... since it will generate METADATA the first time (when
downloading from revision control) anyway. But it can also go in
METADATA, you will get no complaint from me.

Of course I did download PyPI, that's the way to tell. Just under 80%
of the newest sdists for each package on PyPI contain PKG-INFO.

How many phases does a build + install have? Is this about right?

1. Install setup requirements "Setup-Requires-Dist" from .dist-info/METADATA
2. Run setup "dist_info" (pip always runs egg_info) to get a
.dist-info directory with METADATA inside
3. Read up-to-date runtime requirements "Requires-Dist"
4. Install those
5. Install package setup "install"

From dholth at gmail.com  Tue Jul 10 04:28:34 2012
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 9 Jul 2012 22:28:34 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
	<CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
	<CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>
	<CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>
Message-ID: <CAG8k2+50d58u51fu+1=QTgYZoKPwvizjxD7ivUV0xj1KUPshog@mail.gmail.com>

New draft up at https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068

Adds Provides-Extra: (with longer explanation), Setup-Requires-Dist:
(full text below)

Setup-Requires-Dist lives because it makes sense to list packages only
needed for setup.py to run before install is possible.

The reserved extra names test and doc live. No one should mind
installing a distribution and its dependencies in order to run the
unit tests or generate the documentation.


Setup-Requires-Dist (multiple use)
::::::::::::::::::::::::::::::::::

Like Requires-Dist, but names dependencies needed while the
distributions's distutils / packaging `setup.py` / `setup.cfg` is run.
Commonly used to generate a manifest from version control.

Examples::

    Setup-Requires-Dist: custom_setup_command

Dependencies mentioned in `Setup-Requires-Dist` may be installed
exclusively for setup and are not guaranteed to be available at run
time.

From dholth at gmail.com  Wed Jul 11 17:44:43 2012
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 11 Jul 2012 11:44:43 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+50d58u51fu+1=QTgYZoKPwvizjxD7ivUV0xj1KUPshog@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
	<CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
	<CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>
	<CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>
	<CAG8k2+50d58u51fu+1=QTgYZoKPwvizjxD7ivUV0xj1KUPshog@mail.gmail.com>
Message-ID: <CAG8k2+4JMAhtuZ2+0mtvFK1BRpx_KtBrUpYGbjEozGKQRHvRww@mail.gmail.com>

On Mon, Jul 9, 2012 at 10:28 PM, Daniel Holth <dholth at gmail.com> wrote:
> New draft up at https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068
>
> Adds Provides-Extra: (with longer explanation), Setup-Requires-Dist:

To review, Setup-Requires-Dist (only installed for setup.py to run,
called build requirements in some systems) is a completely different
feature than Provides-Extra (for example when code uses 'try: import
somepackage; except: don't offer a feature; called optional_features
in CPAN; similar to Debian 'Suggests' dependencies).

From mj at zopatista.com  Thu Jul 12 13:07:01 2012
From: mj at zopatista.com (Martijn Pieters)
Date: Thu, 12 Jul 2012 13:07:01 +0200
Subject: [Distutils] Request: new zc.buildout release
Message-ID: <1D559E4A-043A-43D5-A026-8D34927DDAD3@zopatista.com>

Could we have a new release of zc.buildout please? 1.5.2 was released nearly 2 years ago, and trunk now contains some interesting bug fixes and new features.

If there is something blocking a 1.6 release, could we see about branching off the 1.5.x line to at least push out the bug fixes?

-- 
Martijn Pieters
www.zopatista.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120712/5f511b88/attachment.html>

From jim at zope.com  Thu Jul 12 15:40:37 2012
From: jim at zope.com (Jim Fulton)
Date: Thu, 12 Jul 2012 09:40:37 -0400
Subject: [Distutils] Request: new zc.buildout release
In-Reply-To: <1D559E4A-043A-43D5-A026-8D34927DDAD3@zopatista.com>
References: <1D559E4A-043A-43D5-A026-8D34927DDAD3@zopatista.com>
Message-ID: <CAPDm-FgjYnfrCq4hupU2Gt+-5hkxkgffk17pXhxNk5D5t7DOLA@mail.gmail.com>

On Thu, Jul 12, 2012 at 7:07 AM, Martijn Pieters <mj at zopatista.com> wrote:
> Could we have a new release of zc.buildout please? 1.5.2 was released nearly
> 2 years ago, and trunk now contains some interesting bug fixes and new
> features.
>
> If there is something blocking a 1.6 release,

Other than time, no.

> could we see about branching
> off the 1.5.x line to at least push out the bug fixes?

If someone wants to work on either, that's cool. I think 1.6 would be easier.

I'm working on buildout 2:

  https://github.com/buildout/buildout

  (I've gotten blocked a bit dealing with Python 3.3. I should
   probably set that aside and get 2a3 out sooner.)

And don't have time for buildout 1.

Jim
-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm

From mj at zopatista.com  Thu Jul 12 15:51:16 2012
From: mj at zopatista.com (Martijn Pieters)
Date: Thu, 12 Jul 2012 15:51:16 +0200
Subject: [Distutils] Request: new zc.buildout release
In-Reply-To: <CAPDm-FgjYnfrCq4hupU2Gt+-5hkxkgffk17pXhxNk5D5t7DOLA@mail.gmail.com>
References: <1D559E4A-043A-43D5-A026-8D34927DDAD3@zopatista.com>
	<CAPDm-FgjYnfrCq4hupU2Gt+-5hkxkgffk17pXhxNk5D5t7DOLA@mail.gmail.com>
Message-ID: <E8DBE4DB-2FD3-4718-B97B-08375E4771C2@zopatista.com>

On Jul 12, 2012, at 15:40 , Jim Fulton wrote:
> I'm working on buildout 2:
> 
>  https://github.com/buildout/buildout
> 
>  (I've gotten blocked a bit dealing with Python 3.3. I should
>   probably set that aside and get 2a3 out sooner.)

Ah, I merged a fix I made on trunk into the svn '2' branch, but I now guess that it won't make it to your git repository. Perhaps the '2' branch should be nixed from SVN and replaced with a 'moved to github' sign?

> And don't have time for buildout 1.


Can we just release it as is then? Who with access on pypi can do this?

-- 
Martijn Pieters
www.zopatista.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120712/65ef3762/attachment.html>

From jim at zope.com  Thu Jul 12 15:55:20 2012
From: jim at zope.com (Jim Fulton)
Date: Thu, 12 Jul 2012 09:55:20 -0400
Subject: [Distutils] Request: new zc.buildout release
In-Reply-To: <E8DBE4DB-2FD3-4718-B97B-08375E4771C2@zopatista.com>
References: <1D559E4A-043A-43D5-A026-8D34927DDAD3@zopatista.com>
	<CAPDm-FgjYnfrCq4hupU2Gt+-5hkxkgffk17pXhxNk5D5t7DOLA@mail.gmail.com>
	<E8DBE4DB-2FD3-4718-B97B-08375E4771C2@zopatista.com>
Message-ID: <CAPDm-FjwQNXRCK__EYktcPOJeyM6kWUmNsk8AHGfQ=mX3J9Q0g@mail.gmail.com>

On Thu, Jul 12, 2012 at 9:51 AM, Martijn Pieters <mj at zopatista.com> wrote:
> On Jul 12, 2012, at 15:40 , Jim Fulton wrote:
>
> I'm working on buildout 2:
>
>  https://github.com/buildout/buildout
>
>  (I've gotten blocked a bit dealing with Python 3.3. I should
>   probably set that aside and get 2a3 out sooner.)
>
>
> Ah, I merged a fix I made on trunk into the svn '2' branch, but I now guess
> that it won't make it to your git repository. Perhaps the '2' branch should
> be nixed from SVN and replaced with a 'moved to github' sign?

Perhaps.

>
> And don't have time for buildout 1.
>
>
> Can we just release it as is then? Who with access on pypi can do this?

Well, "just" releasing it means testing it on at least windows, mac and ubuntu.
Fixing any (non-intermittent) test failures and releasing.

If someone is willing to do all of that (dealing with the hassle of setting up a
dev environment with clean Python's), then I'll authorize them to make
a release.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm

From erik.m.bray at gmail.com  Thu Jul 12 17:08:21 2012
From: erik.m.bray at gmail.com (Erik Bray)
Date: Thu, 12 Jul 2012 11:08:21 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+50d58u51fu+1=QTgYZoKPwvizjxD7ivUV0xj1KUPshog@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
	<CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
	<CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>
	<CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>
	<CAG8k2+50d58u51fu+1=QTgYZoKPwvizjxD7ivUV0xj1KUPshog@mail.gmail.com>
Message-ID: <CAOTD34bf+AATw_PqnURKrYZZzH0pt7YSfgkEPzjiX_N++9gd2A@mail.gmail.com>

On Mon, Jul 9, 2012 at 10:28 PM, Daniel Holth <dholth at gmail.com> wrote:
> New draft up at https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068
>
> Adds Provides-Extra: (with longer explanation), Setup-Requires-Dist:
> (full text below)
>
> Setup-Requires-Dist lives because it makes sense to list packages only
> needed for setup.py to run before install is possible.
>
> The reserved extra names test and doc live. No one should mind
> installing a distribution and its dependencies in order to run the
> unit tests or generate the documentation.
>
>
> Setup-Requires-Dist (multiple use)
> ::::::::::::::::::::::::::::::::::
>
> Like Requires-Dist, but names dependencies needed while the
> distributions's distutils / packaging `setup.py` / `setup.cfg` is run.
> Commonly used to generate a manifest from version control.
>
> Examples::
>
>     Setup-Requires-Dist: custom_setup_command
>
> Dependencies mentioned in `Setup-Requires-Dist` may be installed
> exclusively for setup and are not guaranteed to be available at run
> time.

+1 To this.  I agree Setup-Requires and Provides-Extra are different
use cases and should both be added to the spec.  I think that filling
in these two missing features will benefit packaging in the long run.
If this idea gains some amount of acceptance I'm happy to work on an
implementation of the setup-requires end of things for packaging.

Erik

From dholth at gmail.com  Thu Jul 12 18:03:49 2012
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 12 Jul 2012 12:03:49 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAOTD34bf+AATw_PqnURKrYZZzH0pt7YSfgkEPzjiX_N++9gd2A@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
	<CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
	<CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>
	<CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>
	<CAG8k2+50d58u51fu+1=QTgYZoKPwvizjxD7ivUV0xj1KUPshog@mail.gmail.com>
	<CAOTD34bf+AATw_PqnURKrYZZzH0pt7YSfgkEPzjiX_N++9gd2A@mail.gmail.com>
Message-ID: <CAG8k2+6vAfRKWswHHBmpUKO-HxNFAaTazAH3g5gG0-xEA7XdYA@mail.gmail.com>

> +1 To this.  I agree Setup-Requires and Provides-Extra are different
> use cases and should both be added to the spec.  I think that filling
> in these two missing features will benefit packaging in the long run.
> If this idea gains some amount of acceptance I'm happy to work on an
> implementation of the setup-requires end of things for packaging.

One last addition, explicitly allow X- headers. IMO the technical
purpose of each tag is solid. I would like to get this implemented by
the end of July so that it can be used.

After that I would like to open PEP 346 "Metadata 1.3". Metadata 1.3 could add:

Parenthesis in environment markers (make official "a subset of Python
without most of the good stuff?")

The ~> operator for version specs "~> 1.2" is sugar for ">= 1.2, <
2.0". (Not allowed in environment markers). Thanks dstufft.

Metadata-Is-Static: true / false (name to be determined; means "does
Requires-Dist reliably use environment markers, or does it depend on
Python code in setup.py)

From aclark at aclark.net  Thu Jul 12 21:59:34 2012
From: aclark at aclark.net (Alex Clark)
Date: Thu, 12 Jul 2012 15:59:34 -0400
Subject: [Distutils] Request: new zc.buildout release
In-Reply-To: <1D559E4A-043A-43D5-A026-8D34927DDAD3@zopatista.com>
References: <1D559E4A-043A-43D5-A026-8D34927DDAD3@zopatista.com>
Message-ID: <jtnab6$4c1$1@dough.gmane.org>

On 7/12/12 7:07 AM, Martijn Pieters wrote:
> Could we have a new release of zc.buildout please? 1.5.2 was released
> nearly 2 years ago, and trunk now contains some interesting bug fixes
> and new features.


Such as? Just curious?


>
> If there is something blocking a 1.6 release, could we see about
> branching off the 1.5.x line to at least push out the bug fixes?
>
> --
> Martijn Pieters
> www.zopatista.com <http://www.zopatista.com>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>


-- 
Alex Clark ? http://pythonpackages.com/ONE_CLICK_RELEASE




From bayespokerguy at gmail.com  Fri Jul 13 19:13:09 2012
From: bayespokerguy at gmail.com (Fred G)
Date: Fri, 13 Jul 2012 13:13:09 -0400
Subject: [Distutils] easy_install issues
Message-ID: <CAMg+7mZrcSE+wep6QSuk_OUDO3UrJt_BYs-msYoEKHX7cHGfYw@mail.gmail.com>

Hi--

I was having some difficulties installing easy_install.  I was following a
tutorial that had me on the command prompt typing
C:/easy_tutorial\ez_setup.py

but in the tutorial this did something else then what my computer did.  For
me, a notepad file with a python script opened when I did this, instead of
actually running anything.  I've spent a good 2-3 hours looking at the
easy_install documentation, downloading a free software to open .tar.gz
files, opening various egg files and .py files, and haven't gotten anywhere
really.

Is there a very basic step-by-step guide for how to install it that doesn't
come up in a google search for this? Or a good description that can explain
things in an easy way? I have python 2.7.3 and would really like to get
this module installed.

Thanks in advance for any help you can provide.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120713/7cbe7d31/attachment.html>

From pje at telecommunity.com  Fri Jul 13 21:38:59 2012
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 13 Jul 2012 15:38:59 -0400
Subject: [Distutils] easy_install issues
In-Reply-To: <CAMg+7mZrcSE+wep6QSuk_OUDO3UrJt_BYs-msYoEKHX7cHGfYw@mail.gmail.com>
References: <CAMg+7mZrcSE+wep6QSuk_OUDO3UrJt_BYs-msYoEKHX7cHGfYw@mail.gmail.com>
Message-ID: <CALeMXf4wRirXb9GQsK0gSJ3hs-K+=YYSgsmN0UOEuSz9iX=x7g@mail.gmail.com>

On Fri, Jul 13, 2012 at 1:13 PM, Fred G <bayespokerguy at gmail.com> wrote:
> Hi--
>
> I was having some difficulties installing easy_install.  I was following a
> tutorial that had me on the command prompt typing
> C:/easy_tutorial\ez_setup.py
>
> but in the tutorial this did something else then what my computer did.  For
> me, a notepad file with a python script opened when I did this, instead of
> actually running anything.

Apparently, your computer's python installation is configured to open
.py files in notepad, instead of in Python.  Try typing:

    python ez_setup.py

Instead of just ez_setup.py.

From jim at zope.com  Fri Jul 13 22:51:31 2012
From: jim at zope.com (Jim Fulton)
Date: Fri, 13 Jul 2012 16:51:31 -0400
Subject: [Distutils] Request: new zc.buildout release
In-Reply-To: <jtnab6$4c1$1@dough.gmane.org>
References: <1D559E4A-043A-43D5-A026-8D34927DDAD3@zopatista.com>
	<jtnab6$4c1$1@dough.gmane.org>
Message-ID: <CAPDm-Fi0-vJb1x1F6Y=6yjgKM9KDEdVyNza+RXC29TCNumA8qg@mail.gmail.com>

On Thu, Jul 12, 2012 at 3:59 PM, Alex Clark <aclark at aclark.net> wrote:
> On 7/12/12 7:07 AM, Martijn Pieters wrote:
>>
>> Could we have a new release of zc.buildout please? 1.5.2 was released
>> nearly 2 years ago, and trunk now contains some interesting bug fixes
>> and new features.
>
>
>
> Such as? Just curious?

See the changes for 1.6:

http://svn.zope.org/zc.buildout/trunk/CHANGES.txt?view=auto

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm

From jim at zope.com  Sat Jul 14 20:30:00 2012
From: jim at zope.com (Jim Fulton)
Date: Sat, 14 Jul 2012 14:30:00 -0400
Subject: [Distutils] distribute broken Python 3.3.3
In-Reply-To: <4FE80ACF.6030604@ziade.org>
References: <CAPDm-Fifc=zCCvxok1amKok_cZR-Tm0mg6O0NiV-sMhPLTdgEQ@mail.gmail.com>
	<4FE80ACF.6030604@ziade.org>
Message-ID: <CAPDm-Fgfx0f2fH1O5_jBdmu6163V-iGpAxF=CMqXcWL1NApMzA@mail.gmail.com>

On Mon, Jun 25, 2012 at 2:53 AM, Tarek Ziad? <tarek at ziade.org> wrote:
> On 6/25/12 1:11 AM, Jim Fulton wrote:
>>
>>
>> https://bitbucket.org/tarek/distribute/issue/289/distribute-broken-with-python-33
>>
>> I'm gonna try to work around it in buildout 2 by monkey-patching
>> distribute.
>>
>> Jim
>>
> If you write the fix, maybe you can do it against Distribute ? we can
> release it right after.

Here it is:

  https://bitbucket.org/tarek/distribute/pull-request/12/issue-283-bdist_egg-issues-with-python

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm

From bayespokerguy at gmail.com  Sat Jul 14 21:47:29 2012
From: bayespokerguy at gmail.com (Fred G)
Date: Sat, 14 Jul 2012 15:47:29 -0400
Subject: [Distutils] easy_install issues
In-Reply-To: <CALeMXf4wRirXb9GQsK0gSJ3hs-K+=YYSgsmN0UOEuSz9iX=x7g@mail.gmail.com>
References: <CAMg+7mZrcSE+wep6QSuk_OUDO3UrJt_BYs-msYoEKHX7cHGfYw@mail.gmail.com>
	<CALeMXf4wRirXb9GQsK0gSJ3hs-K+=YYSgsmN0UOEuSz9iX=x7g@mail.gmail.com>
Message-ID: <CAMg+7ma7JVzXpdGYxdMexascesxC-FyoJFyUPMC4cT9_nhKR9Q@mail.gmail.com>

Hi PJ,

Great-- thanks! That was a problem and the fact that when I tried to run it
afterwards, I had to go to the Python\Scripts directory, easy_install
loaded into that.

Thanks for the help!

On Fri, Jul 13, 2012 at 3:38 PM, PJ Eby <pje at telecommunity.com> wrote:

> On Fri, Jul 13, 2012 at 1:13 PM, Fred G <bayespokerguy at gmail.com> wrote:
> > Hi--
> >
> > I was having some difficulties installing easy_install.  I was following
> a
> > tutorial that had me on the command prompt typing
> > C:/easy_tutorial\ez_setup.py
> >
> > but in the tutorial this did something else then what my computer did.
>  For
> > me, a notepad file with a python script opened when I did this, instead
> of
> > actually running anything.
>
> Apparently, your computer's python installation is configured to open
> .py files in notepad, instead of in Python.  Try typing:
>
>     python ez_setup.py
>
> Instead of just ez_setup.py.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120714/df2cb3ef/attachment.html>

From dholth at gmail.com  Wed Jul 18 19:53:53 2012
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 18 Jul 2012 13:53:53 -0400
Subject: [Distutils] strong hashing in RECORD and signed wheel files
Message-ID: <CAG8k2+40C3WXcRcoz-NvvLsYcKCZ3oXptBb6r-i_uzgTv=KMfQ@mail.gmail.com>

One of the main design goals for the wheel built package format is
that a wheel archive extracted to sys.path is a PEP-376 compliant
installation. I also want most wheel files to be cryptographically
signed. The idea is to include a < 256 byte JSON Web Signature of
RECORD, "RECORD.jws", which will only take a couple of milliseconds to
generate, in the .dist-info directory. This is only meaningful if
RECORD has strong hashes, or the installer would have to always
rewrite RECORD on install just to include md5 sums to follow the spec.

PyPi already lets you sign whole archives with GPG, but this scheme is
more flexible because you can re-compress the .zip archive without
invalidating signatures, you could use HTTP Range (partial content)
requests to download and verify only the metadata of an archive
without having to download the whole thing (this is one reason why
.whl is a zip archive), and the whole system will be a bit more
relaxed than GPG so that bdist_wheel can just generate signing keys
automatically and an end user can decide whether to trust or
distribute a particular key later -- perhaps a particular key would be
valid per maintainer, or per distribution, or for only certain
versions of a distribution.

That's why I would like PEP 376 to understand hashes-besides-md5 in RECORD.

From pje at telecommunity.com  Wed Jul 18 23:44:59 2012
From: pje at telecommunity.com (PJ Eby)
Date: Wed, 18 Jul 2012 17:44:59 -0400
Subject: [Distutils] strong hashing in RECORD and signed wheel files
In-Reply-To: <CAG8k2+40C3WXcRcoz-NvvLsYcKCZ3oXptBb6r-i_uzgTv=KMfQ@mail.gmail.com>
References: <CAG8k2+40C3WXcRcoz-NvvLsYcKCZ3oXptBb6r-i_uzgTv=KMfQ@mail.gmail.com>
Message-ID: <CALeMXf6ys85mMEO1CNxGYGRf=c+Ho0AOQSxp3sPE1QXAmMcgbQ@mail.gmail.com>

On Wed, Jul 18, 2012 at 1:53 PM, Daniel Holth <dholth at gmail.com> wrote:
> One of the main design goals for the wheel built package format is
> that a wheel archive extracted to sys.path is a PEP-376 compliant
> installation. I also want most wheel files to be cryptographically
> signed. The idea is to include a < 256 byte JSON Web Signature of
> RECORD, "RECORD.jws", which will only take a couple of milliseconds to
> generate, in the .dist-info directory. This is only meaningful if
> RECORD has strong hashes, or the installer would have to always
> rewrite RECORD on install just to include md5 sums to follow the spec.

If you're including another file anyway, why not just put the
signatures in there, then?

From dholth at gmail.com  Thu Jul 19 00:54:40 2012
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 18 Jul 2012 18:54:40 -0400
Subject: [Distutils] strong hashing in RECORD and signed wheel files
In-Reply-To: <CALeMXf6ys85mMEO1CNxGYGRf=c+Ho0AOQSxp3sPE1QXAmMcgbQ@mail.gmail.com>
References: <CAG8k2+40C3WXcRcoz-NvvLsYcKCZ3oXptBb6r-i_uzgTv=KMfQ@mail.gmail.com>
	<CALeMXf6ys85mMEO1CNxGYGRf=c+Ho0AOQSxp3sPE1QXAmMcgbQ@mail.gmail.com>
Message-ID: <E99A99C8-3AA4-4C8F-B971-B8DB79608430@gmail.com>

On Jul 18, 2012, at 5:44 PM, PJ Eby <pje at telecommunity.com> wrote:

> On Wed, Jul 18, 2012 at 1:53 PM, Daniel Holth <dholth at gmail.com> wrote:
>> 
>> generate, in the .dist-info directory. This is only meaningful if
>> RECORD has strong hashes, or the installer would have to always
>> rewrite RECORD on install just to include md5 sums to follow the spec.
> 
> If you're including another file anyway, why not just put the
> signatures in there, then?

Of course that would work too. Jarsigner does that. Record is just tantalizingly close to what's needed.

From menghsiu.lee at gmail.com  Sun Jul 22 09:32:46 2012
From: menghsiu.lee at gmail.com (Menghsiu Lee)
Date: Sun, 22 Jul 2012 15:32:46 +0800
Subject: [Distutils] i have run the exe file, but it does not seem to work
Message-ID: <CAKD8P62GnofsrS6impQdHNkFMt_vNdMg1-vYYJhUddgb-6+k=w@mail.gmail.com>

To whom it may concern:
I read the response on the link below:
http://www.mail-archive.com/distutils-sig at python.org/msg13767.html

And I follow your instruction, but then the exe file has idled.
The attachment is my screen cut.
The complete file link is in below:
http://dl.dropbox.com/u/63928380/blackjack.rar

Best regards,

menghsiu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120722/48027775/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ScreenHunter_69 Jul. 22 15.02.jpg
Type: image/jpeg
Size: 27460 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120722/48027775/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: setup.py
Type: application/octet-stream
Size: 386 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120722/48027775/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: blackjack.py
Type: application/octet-stream
Size: 33139 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120722/48027775/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: game2exe.py
Type: application/octet-stream
Size: 6060 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120722/48027775/attachment-0005.obj>

From oscar.j.benjamin at gmail.com  Mon Jul 23 00:11:30 2012
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sun, 22 Jul 2012 23:11:30 +0100
Subject: [Distutils] i have run the exe file,
	but it does not seem to work
In-Reply-To: <CAKD8P62GnofsrS6impQdHNkFMt_vNdMg1-vYYJhUddgb-6+k=w@mail.gmail.com>
References: <CAKD8P62GnofsrS6impQdHNkFMt_vNdMg1-vYYJhUddgb-6+k=w@mail.gmail.com>
Message-ID: <CAHVvXxSyJ356buCFtRVSK-AhPUMux2sV_6Tk8hSMQz4Wh83MbQ@mail.gmail.com>

On 22 July 2012 08:32, Menghsiu Lee <menghsiu.lee at gmail.com> wrote:

> To whom it may concern:
> I read the response on the link below:
> http://www.mail-archive.com/distutils-sig at python.org/msg13767.html
>
> And I follow your instruction, but then the exe file has idled.
> The attachment is my screen cut.
> The complete file link is in below:
> http://dl.dropbox.com/u/63928380/blackjack.rar
>

Please don't post links to .rar files. The reason you got such an
unfriendly response on python-list is that instead of explaining your
problem you were expecting others to download a .rar file from an untrusted
source. This is what someone might do if they were trying to spread viruses.

You've attached your code this time, which is better, but you can't expect
someone to simply download your code files look through them, guess what
you're trying to do and tell you how to make it work. You need to explain:
1) What you are trying to do
2) What steps you took
3) What is not working

Also, I think your problem is with py2exe, in which case you might have
better luck using the py2exe-users mailing list:
https://lists.sourceforge.net/lists/listinfo/py2exe-users

Oscar.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120722/eb78c361/attachment.html>

From oscar.j.benjamin at gmail.com  Mon Jul 23 11:25:20 2012
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 23 Jul 2012 10:25:20 +0100
Subject: [Distutils] i have run the exe file,
	but it does not seem to work
In-Reply-To: <CAKD8P62_gwrzKWudSWQQEnA5MPKvdqbxS8ki=neiGFaATsDXHQ@mail.gmail.com>
References: <CAKD8P62GnofsrS6impQdHNkFMt_vNdMg1-vYYJhUddgb-6+k=w@mail.gmail.com>
	<CAHVvXxSyJ356buCFtRVSK-AhPUMux2sV_6Tk8hSMQz4Wh83MbQ@mail.gmail.com>
	<CAKD8P62_gwrzKWudSWQQEnA5MPKvdqbxS8ki=neiGFaATsDXHQ@mail.gmail.com>
Message-ID: <CAHVvXxQuy9xRT3hWPmZNkx+8PrKYrO5KZpHmLYorH_4az4B-Fg@mail.gmail.com>

Send your response to the mailing list, not to me.

On Jul 23, 2012 7:08 AM, "Menghsiu Lee" <menghsiu.lee at gmail.com> wrote:
>
> Dear Oscar:
> Frankly speaking, your response is the most friendly one. Thanks for
teaching me how to ask questions. As you can imagine, I am totally new in
python or in programming. I put the whole thing in a rar file because I am
afraid that people can not know what I wanna say and I am trying to open up
everything to everyone who helps me. Hence, I didn't know that rar file may
be used as a virus tool, cause I never have such idea.
> In response to the following questions, here are my answers.
> 1) What you are trying to do
> I am trying to make a game based on python. This game is a poker game. I
want to make people can put their face on the cards. Friends can play this
game via internet
> or invite each other via FB account. Also, like playing with friends, I
hope that people can place the bids on the table. The winner can take the
bid, but these bids are not cashable. They can only be used as a record for
fun. I do not know if I can make this or not, but I will try.
>
>
> 2) What steps you took
> This is the first step of the program. I am still working on sketch the
draft of the program.
> The basic functions are done, but I think that I may add more functions
to complete this program. This is just in the beginning. Things are fixed
yet.
>
> 3) What is not working
> Since I have change the codes and make some basic functions, I would like
to share this program with my friend. I would like to know what they think
about this program and would like to know if I can still make changes(eg,
functions, ideas) This is why I want to make the file an exe file.
>
>
> I have make pygame2exe.py file. This is what appears in the end.
>
> So, is this the correct way to ask questions?

No. There is no way that anyone can help you from that description. You
need to explain specifically what you want help with not provide a general
description of what you're trying to do.

>
>
> menghsiu
>
>
>
>
>
>
>
> On Mon, Jul 23, 2012 at 6:11 AM, Oscar Benjamin <
oscar.j.benjamin at gmail.com> wrote:
>>
>> On 22 July 2012 08:32, Menghsiu Lee <menghsiu.lee at gmail.com> wrote:
>>>
>>> To whom it may concern:
>>> I read the response on the link below:
>>> http://www.mail-archive.com/distutils-sig at python.org/msg13767.html
>>>
>>> And I follow your instruction, but then the exe file has idled.
>>> The attachment is my screen cut.
>>> The complete file link is in below:
>>> http://dl.dropbox.com/u/63928380/blackjack.rar
>>
>>
>> Please don't post links to .rar files. The reason you got such an
unfriendly response on python-list is that instead of explaining your
problem you were expecting others to download a .rar file from an untrusted
source. This is what someone might do if they were trying to spread viruses.
>>
>> You've attached your code this time, which is better, but you can't
expect someone to simply download your code files look through them, guess
what you're trying to do and tell you how to make it work. You need to
explain:
>> 1) What you are trying to do
>> 2) What steps you took
>> 3) What is not working
>>
>> Also, I think your problem is with py2exe, in which case you might have
better luck using the py2exe-users mailing list:
>> https://lists.sourceforge.net/lists/listinfo/py2exe-users
>>
>> Oscar.
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120723/f4b278af/attachment.html>

From menghsiu.lee at gmail.com  Mon Jul 23 15:41:53 2012
From: menghsiu.lee at gmail.com (Menghsiu Lee)
Date: Mon, 23 Jul 2012 21:41:53 +0800
Subject: [Distutils] following modules appear to be missing:
Message-ID: <CAKD8P61rZtQTes8-H9j6cRsySi8Lp2-czwF_Ne1GYk2Ev_Lqvg@mail.gmail.com>

Hi, I have tried to compiled a python file with pygame module into a exe
file. I have written game2exe.py and I read some articles online which told
me to include the "msvcp71.dll", "dwmapi.dll" files to my python:

def isSystemDLL(pathname):
        if os.path.basename(pathname).lower() in ("msvcp71.dll",
"dwmapi.dll"):
                return 0
                return origIsSystemDLL(pathname)

Then, I got the error messages like
The following modules appear to be missing:
AppKit, Foundation, ICCProfile, Numeric, OpenGL.GL, __imaging_gif, scproxy,
copyreg, dummy.process, numpy, pkg_resorces, quitueue, winreg,
pygame-sdlmain_osx

I tried to include:
self.extra_datas = ["kaiu.ttf",
                            "ARIALUNI.TTF",
                            'images/cards',
                            'images',
                            'sounds',
                            'pygame'
                            'Manifests'
                            ]

These files contains both modules and resource of my program, but in the
end, when I click on the blackjack.exe, it replies me a blank screen with
nothing going on.
In the Manifests folder, I have include:
Microsoft.VC90.CRT.manifest
msvcm90.dll
msvcp90.dll
msvcr90.dll


I also tried to compiled setup.py with py2exe:
from distutils.core import setup

import py2exe,sys,os
import pygame

setup(console=['blackjack.py'])



origIsSystemDLL = py2exe.build_exe.isSystemDLL
def isSystemDLL(pathname):
        if os.path.basename(pathname).lower() in ("msvcp71.dll",
"dwmapi.dll"):
                return 0
        return origIsSystemDLL(pathname)
py2exe.build_exe.isSystemDLL = isSystemDLL


It works fine until the end, but when I click the blackjack.exe, it returns
to me:
file "pygame\___init__pyc", line 263 in moduleattributeError:imageext



Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120723/b28e5757/attachment.html>

From pje at telecommunity.com  Mon Jul 23 19:08:23 2012
From: pje at telecommunity.com (PJ Eby)
Date: Mon, 23 Jul 2012 13:08:23 -0400
Subject: [Distutils] following modules appear to be missing:
In-Reply-To: <CAKD8P61rZtQTes8-H9j6cRsySi8Lp2-czwF_Ne1GYk2Ev_Lqvg@mail.gmail.com>
References: <CAKD8P61rZtQTes8-H9j6cRsySi8Lp2-czwF_Ne1GYk2Ev_Lqvg@mail.gmail.com>
Message-ID: <CALeMXf6K2+CK=aZTajpTSpFE8-ToFFxca1oYnMaPeLk8VQboAw@mail.gmail.com>

This sounds like people familiar with py2exe will be better able to
help you.  As far as I can tell there is nothing about your problem
that relates to the Distutils per se.  py2exe is an add-on to the
distutils, and has its own mailing list, which you can find here:

https://lists.sourceforge.net/lists/listinfo/py2exe-users

On Mon, Jul 23, 2012 at 9:41 AM, Menghsiu Lee <menghsiu.lee at gmail.com> wrote:
> Hi, I have tried to compiled a python file with pygame module into a exe
> file. I have written game2exe.py and I read some articles online which told
> me to include the "msvcp71.dll", "dwmapi.dll" files to my python:
>
> def isSystemDLL(pathname):
>         if os.path.basename(pathname).lower() in ("msvcp71.dll",
> "dwmapi.dll"):
>                 return 0
>                 return origIsSystemDLL(pathname)
>
> Then, I got the error messages like
> The following modules appear to be missing:
> AppKit, Foundation, ICCProfile, Numeric, OpenGL.GL, __imaging_gif, scproxy,
> copyreg, dummy.process, numpy, pkg_resorces, quitueue, winreg,
> pygame-sdlmain_osx
>
> I tried to include:
> self.extra_datas = ["kaiu.ttf",
>                             "ARIALUNI.TTF",
>                             'images/cards',
>                             'images',
>                             'sounds',
>                             'pygame'
>                             'Manifests'
>                             ]
>
> These files contains both modules and resource of my program, but in the
> end, when I click on the blackjack.exe, it replies me a blank screen with
> nothing going on.
> In the Manifests folder, I have include:
> Microsoft.VC90.CRT.manifest
> msvcm90.dll
> msvcp90.dll
> msvcr90.dll
>
>
> I also tried to compiled setup.py with py2exe:
> from distutils.core import setup
>
> import py2exe,sys,os
> import pygame
>
> setup(console=['blackjack.py'])
>
>
>
> origIsSystemDLL = py2exe.build_exe.isSystemDLL
> def isSystemDLL(pathname):
>         if os.path.basename(pathname).lower() in ("msvcp71.dll",
> "dwmapi.dll"):
>                 return 0
>         return origIsSystemDLL(pathname)
> py2exe.build_exe.isSystemDLL = isSystemDLL
>
>
> It works fine until the end, but when I click the blackjack.exe, it returns
> to me:
> file "pygame\___init__pyc", line 263 in moduleattributeError:imageext
>
>
>
> Thanks.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>

From dholth at gmail.com  Wed Jul 25 15:33:48 2012
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 25 Jul 2012 09:33:48 -0400
Subject: [Distutils] 'markerlib' (environment markers) is ready
Message-ID: <CAG8k2+5HkTDratuVfiju-8CoUPB1x=uWXWYqmMZ=mUwWZNjdHw@mail.gmail.com>

I'm happy with markerlib, my implementation of environment markers. In
markerlib, a marker is a function

def marker(environment=None, override=None)

where environment is the environment and override updates the
environment; if environment is None then it's the default environment.
So marker(override={'python_version':'2.6'}) evaluates the marker with
the python version changed to 2.6. The API is two functions compile()
(recommended) and interpret() (compatible with packaging API).

The implementation takes advantage of the fact that variables can
contain "." at the interpreter level, and then implements markers as
Python with only (Compare, BoolOp, Attribute, Name, Load, Str, cmpop,
boolop) AST nodes allowed.

On CPython 2.7 it is a little bit slower than the distutils2
implementation for a single compilation+evaluation of a statement like
"a == 'a' or b == 'b'" but is much faster for repeated evaluation. It
works for Python >= 2.5. It has 100% code coverage.

Would you be interested in folding this into packaging, or distribute
(yes, the setuptools fork) since markers are needed to properly
support .dist-info, perhaps as a different top-level name 'markers' or
'envmarkers'?

From tarek at ziade.org  Wed Jul 25 15:49:49 2012
From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 25 Jul 2012 15:49:49 +0200
Subject: [Distutils] 'markerlib' (environment markers) is ready
In-Reply-To: <CAG8k2+5HkTDratuVfiju-8CoUPB1x=uWXWYqmMZ=mUwWZNjdHw@mail.gmail.com>
References: <CAG8k2+5HkTDratuVfiju-8CoUPB1x=uWXWYqmMZ=mUwWZNjdHw@mail.gmail.com>
Message-ID: <500FF97D.2060902@ziade.org>

On 7/25/12 3:33 PM, Daniel Holth wrote:
> I'm happy with markerlib, my implementation of environment markers. In
> markerlib, a marker is a function
>
> def marker(environment=None, override=None)
>
> where environment is the environment and override updates the
> environment; if environment is None then it's the default environment.
> So marker(override={'python_version':'2.6'}) evaluates the marker with
> the python version changed to 2.6. The API is two functions compile()
> (recommended) and interpret() (compatible with packaging API).
>
> The implementation takes advantage of the fact that variables can
> contain "." at the interpreter level, and then implements markers as
> Python with only (Compare, BoolOp, Attribute, Name, Load, Str, cmpop,
> boolop) AST nodes allowed.
>
> On CPython 2.7 it is a little bit slower than the distutils2
> implementation for a single compilation+evaluation of a statement like
> "a == 'a' or b == 'b'" but is much faster for repeated evaluation. It
> works for Python >= 2.5. It has 100% code coverage.
>
> Would you be interested in folding this into packaging, or distribute
> (yes, the setuptools fork) since markers are needed to properly
> support .dist-info, perhaps as a different top-level name 'markers' or
> 'envmarkers'?
> _______________________________________________

cool !

+1


> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig



From dholth at gmail.com  Thu Jul 26 17:59:36 2012
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 26 Jul 2012 11:59:36 -0400
Subject: [Distutils] Adding Provides-Extra as an edit to PEP 345
In-Reply-To: <CAG8k2+6vAfRKWswHHBmpUKO-HxNFAaTazAH3g5gG0-xEA7XdYA@mail.gmail.com>
References: <CAG8k2+4BirpGeYOUky07O2Ye95mhm6KvVoubmiOCrRuOq0_pHQ@mail.gmail.com>
	<4FF6FDCD.8010000@netwok.org>
	<CAG8k2+6AXLA1J+PrwkCCo+kTAtcFmhecuJkRGH=m4cYfVjT2Zw@mail.gmail.com>
	<CAOTD34a8q1mvEkdDAU7aHeTH2BOA8khojcLmo-GnRer-10mgEw@mail.gmail.com>
	<CAG8k2+6Dqb5RwkjDWqBAPX697WVCk45i7B56s91gb8gkvFTy-Q@mail.gmail.com>
	<CAOTD34Yz5G3vz5aBhpq-3Ctj8he14tV0BuZJDokNsGSfDjWxiA@mail.gmail.com>
	<CAG8k2+6+VhBftZnVQA8s=Gx9ZqfzNE9-9R-osMpMtDsdFe4qPw@mail.gmail.com>
	<CAOTD34aTgS_Yrmw346DzSnAbT4DdaVXcK7u9xumvoXxXU6Z4hw@mail.gmail.com>
	<CAG8k2+50d58u51fu+1=QTgYZoKPwvizjxD7ivUV0xj1KUPshog@mail.gmail.com>
	<CAOTD34bf+AATw_PqnURKrYZZzH0pt7YSfgkEPzjiX_N++9gd2A@mail.gmail.com>
	<CAG8k2+6vAfRKWswHHBmpUKO-HxNFAaTazAH3g5gG0-xEA7XdYA@mail.gmail.com>
Message-ID: <CAG8k2+7t580xPLmisHhKz0CAKb-1WA=CBbhi-jA74vaNSGLKag@mail.gmail.com>

On Thu, Jul 12, 2012 at 12:03 PM, Daniel Holth <dholth at gmail.com> wrote:
>> +1 To this.  I agree Setup-Requires and Provides-Extra are different
>> use cases and should both be added to the spec.  I think that filling
>> in these two missing features will benefit packaging in the long run.
>> If this idea gains some amount of acceptance I'm happy to work on an
>> implementation of the setup-requires end of things for packaging.
>
> One last addition, explicitly allow X- headers. IMO the technical
> purpose of each tag is solid. I would like to get this implemented by
> the end of July so that it can be used.

If there are no further comments, I will ask Benjamin Peterson to
merge this into python-peps this weekend.

To avoid NameError, distutils2 should define extra == None by default
until it supports extras (simply not installing any extra
dependencies), or markerlib should just be folded in. For markerlib's
Python 2.4 problem a fallback that doesn't require _ast is either the
current distutils2 marker implementation, or to evaluate empty markers
as True, and non-empty markers as False, so only installing
unconditional dependencies; I don't suppose many Python 2.4 users are
clamoring to adopt .dist-info & Metadata 1.2...

Thanks,

Daniel Holth

From torsten at irlaender.de  Fri Jul 27 10:10:40 2012
From: torsten at irlaender.de (torsten at irlaender.de)
Date: Fri, 27 Jul 2012 10:10:40 +0200 (CEST)
Subject: [Distutils] Generic revision support to tag eggs
Message-ID: <fadb08306a90ba185e1522008d7d2a69.squirrel@webmail.kontent.com>

Hi all,

I was wondering if there is any support for adding a revision tag on eggs
other than SVN. I found Issue42 [0] in the bugtracker, which deals with this
but there is no progress since a long time.

So i want to reraise this question here. I did not look any further in the
current suggested solution to solve the issue but my first idea was: Why
so complicated with logic for finding revision tagger etc.? Because i am
really new to distutils i maybe/likeky i did not get the point. But why not
provide a simple option to the egg_info command to define a call of a command
or script which returns the revision number?

Example:

[egg_info]
tag_build = .dev
tag_command = hg tip| head -n 1|cut -d " " -f4|cut -d: -f1

This adds a very simple way of adding hg support but in general to tag
generic
revisions.

I have already written a patch which implements this to get a quick fix for
this. So i ask myself if this might be usefull for anyone else, or if my
solution is way too simple and naiv?

Please do not wonder if i do not respond on the list until the 8. August
as i am on holiday.

best wishes
Torsten Irl?nder

[0] http://bugs.python.org/setuptools/issue42