From ncoghlan at gmail.com  Sat Feb  1 03:14:20 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 1 Feb 2014 12:14:20 +1000
Subject: [Distutils] Letting the Python implementation produce the
 Python ABI tag
In-Reply-To: <52EA24C8.9080003@gmx.com>
References: <52EA24C8.9080003@gmx.com>
Message-ID: <CADiSq7dupCQgoVkvGi=ihhTtM3+YgQZu3_FjUJ0_0s6wAJQzMA@mail.gmail.com>

On 30 January 2014 20:09, Squeaky <squeaky_pl at gmx.com> wrote:
> Hi!
>
> This is my first message to this mailing list. I saw a discussion about
> construction of SOABI and I thought that it might be tiresome to keep code
> to produce that for every possible Python implementation in wheel
> generators. After all it's the Python implementation that knows better in
> which version and why they broke binary ABI.
>
> Let's say that PyPy 2.2 has this tag set to pp22 but they might have
> released PyPy 2.3 without breaking the ABI and prefer to advertise PyPy2.3
> as pp22 compatible thus reducing the number of wheels a python package
> maintainer has to produce. The ABI is separate from releases.
>
> Also this would benefit if there were new Python implementations coming into
> ecosystem and they could have a function in their stdlib that advertises
> their abi level. This would allow these new implementations to work with
> wheels instantly.
>
> Does it make sense or it's an utopia?

It's not utopia, it's the way Python 3.2+ works on POSIX systems (due
to http://www.python.org/dev/peps/pep-3149/). The compatibility
problem is mainly posed by the python.org Windows builds and any
versions prior to 3.2, as those *don't* implement
"sysconfig.get_config_var('SOABI')" correctly, so we're going to have
to come up with a different solution (hopefully one that will also
sensibly handle other implementations that don't yet provide the SOABI
config var).

PEP 425 also defines some specific abbreviations for CPython, PyPy,
IronPython and Jython to help avoid filenames getting excessively long
in the typical case, but the underlying concept is still based on the
SOABI tag introduced in PEP 3149.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Feb  1 03:33:29 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 1 Feb 2014 12:33:29 +1000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <CAAmsm4gB0f9YNx05tvO2=L+vQy8Lk6RNvpTqsbWuzVOkzGqfyA@mail.gmail.com>
References: <CAAmsm4gB0f9YNx05tvO2=L+vQy8Lk6RNvpTqsbWuzVOkzGqfyA@mail.gmail.com>
Message-ID: <CADiSq7fPbNBQOAS3oWHigerzgXVOKFqhoEsCAVO+ukC_YLv9Dw@mail.gmail.com>

On 1 February 2014 05:31, Brian Wickman <wickman at gmail.com> wrote:
> This is in response to Vinay's thread but since I wasn't subscribed to
> distutils-sig, I couldn't easily respond directly to it.
>
> Vinay's right, the technology here isn't revolutionary but what's notable is
> that we've been using it in production for almost 3 years at Twitter.  It's
> also been open-sourced for a couple years at
> https://github.com/twitter/commons/tree/master/src/python/twitter/common/python
> but not widely announced (it is, after all, just a small subdirectory in a
> fairly large mono-repo, and was only recently published independently to
> PyPI as twitter.common.python.)
>
> PEX files are just executable zip files with hashbangs containing a
> carefully constructed __main__.py and a PEX-INFO, which is json-encoded
> dictionary describing how to scrub and bootstrap sys.path and the like.
> They work equally well unpacked into a standalone directory.
>
> In practice PEX files are simultaneously our replacement for virtualenv and
> also our way of distributing Python applications to production.  Now we
> could use virtualenv to do this but it's hard to argue with a deployment
> process that is literally "cp".  Furthermore, most of our machines don't
> have compiler toolchains or external network access, so hermetically sealing
> all dependencies once at build time (possibly for multiple platforms since
> all developers use Macs) has huge appeal.  This is even more important at
> Twitter where it's common to run a dozen different Python applications on
> the same box at the same time, some using 2.6, some 2.7, some PyPy, but all
> with varying versions of underlying dependencies.

Ah, very interesting - this is exactly the kind of thing we were
trying to enable with the executable directory/zip file support in
Python 2.6, and then we went and forgot to cover it in the original
"What's New in Python 2.6?" doc, so an awful lot of people never
learned about the existence of the feature :P

As Daniel noted, PEP 441 is at least as much about letting people know
"Hey, Python has supported direct execution of directories and zip
archives since Python 2.6!" as it is about actually providing some
tools that better support doing things that way :)

> Speaking to recent distutils-sig threads, we used to go way out of our way
> to never hit disk (going so far as building our own .egg packager and pure
> python recursive zipimport implementation so that we could import from eggs
> within zips, write ephemeral .so's to dlopen and unlink) but we've since
> moved away from that position for simplicity's sake.  For practical reasons
> we've always needed "not zip-safe" PEX files where all code is written to
> disk prior to execution (ex: legacy Django applications that have
> __file__-relative business logic) so we decided to just throw away the
> magical zipimport stuff and embrace using disk as a pure cache.  This seems
> more compatible philosophically with the direction wheels are going for
> example.
>
> Since there's been more movement in the PEP space recently, we've been
> evolving PEX in order to be as standards-compliant as possible, which is why
> I've been more visible recently re: talks, .whl support and the like.  I'd
> also love to chat about more about PEX and how it relates to things like PEP
> 441 and/or other attempts like pyzzer.

I think it's very interesting and relevant indeed :)

The design in PEP 441 just involves providing some very basic
infrastructure around making use of the direct execution support,
mostly in order to make the feature more discoverable in the first
place, but it sounds like Twitter have a much richer and
battle-hardened approach in PEX.

It may still prove to be more appropriate to keep the stdlib
infrastructure very basic (i.e. more at the PEP 441) level, leaving
something like PEX free to provide cross-version consistency. I'll
wait and see how the public documentation for PEX evolves before
forming a firm opinion one way or the other, but one possible outcome
would be a situation akin to the pyvenv vs virtualenv distinction,
where pyvenv has the benefit of always being available even in
environments where it is difficult to get additional third party tools
approved, but also has the inherent downside of Python version
dependent differences in available features and behaviour.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Sat Feb  1 09:23:42 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 08:23:42 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
Message-ID: <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>

On Fri, 31/1/14, Brian Wickman <wickman at gmail.com> wrote:

> There are myriad other practical reasons. ?Here are some:

Thanks for taking the time to respond with the details - they are good data points
to think about! 
 
> Lastly, there are social reasons. It's just hard to convince most engineers
> to use things like pkg_resources or pkgutil to manipulate resources
> when for them the status quo is just using __file__. ?Bizarrely the social
> challenges are just as hard as the abovementioned technical challenges.

I agree it's bizarre, but sadly it's not surprising. People get used to certain ways
of doing things, and a certain kind of collective myopia develops when it
comes to looking at different ways of doing things. Having worked with fairly
diverse systems in my time, ISTM that sections of the Python community have
this myopia too. For example, the Java hatred and PEP 8 zealotry that you see
here and there.

One of the things that's puzzled me, for example, is why people think it's reasonable
or even necessary to have copies of pip and setuptools in every virtual environment
- often the same people who will tell you that your code isn't DRY enough! It's
certainly not a technical requirement, yet one of the reasons why PEP 405 venvs
aren't that popular is that pip and setuptools aren't automatically put in there. It's a
social issue - it's been decided that rather than exploring a technical approach to
addressing any issue with installing into venvs, it's better to bundle pip and setuptools
with Python 3.4, since that will seemingly be easier for people to swallow :-)

Regards,

Vinay Sajip

From ncoghlan at gmail.com  Sat Feb  1 09:43:25 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 1 Feb 2014 18:43:25 +1000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
Message-ID: <CADiSq7d33vS-Z3OMLG5BH0XH7ZLWADbbV_4v9hv_EAP9FNKPWA@mail.gmail.com>

On 1 February 2014 18:23, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> On Fri, 31/1/14, Brian Wickman <wickman at gmail.com> wrote:
>
>> There are myriad other practical reasons.  Here are some:
>
> Thanks for taking the time to respond with the details - they are good data points
> to think about!
>
>> Lastly, there are social reasons. It's just hard to convince most engineers
>> to use things like pkg_resources or pkgutil to manipulate resources
>> when for them the status quo is just using __file__.  Bizarrely the social
>> challenges are just as hard as the abovementioned technical challenges.
>
> I agree it's bizarre, but sadly it's not surprising. People get used to certain ways
> of doing things, and a certain kind of collective myopia develops when it
> comes to looking at different ways of doing things. Having worked with fairly
> diverse systems in my time, ISTM that sections of the Python community have
> this myopia too. For example, the Java hatred and PEP 8 zealotry that you see
> here and there.
>
> One of the things that's puzzled me, for example, is why people think it's reasonable
> or even necessary to have copies of pip and setuptools in every virtual environment
> - often the same people who will tell you that your code isn't DRY enough! It's
> certainly not a technical requirement, yet one of the reasons why PEP 405 venvs
> aren't that popular is that pip and setuptools aren't automatically put in there. It's a
> social issue - it's been decided that rather than exploring a technical approach to
> addressing any issue with installing into venvs, it's better to bundle pip and setuptools
> with Python 3.4, since that will seemingly be easier for people to swallow :-)

FWIW, installing into a venv from outside it works fine (that's how
ensurepip works in 3.4). However, it's substantially *harder* to
explain to people how to use it correctly that way. In theory you
could change activation so that it also affected the default install
locations, but the advantage of just having them installed per venv is
that you're relying more on the builtin Python path machinery rather
than adding something new. So while it's wasteful of disk space and
means needing to upgrade them in every virtualenv, it does actually
categorically eliminate many potential sources of bugs. Doing things
the way pip and virtualenv do them also meant there was a whole pile
of design work that *didn't need to be done* to get a functional
system up and running. Avoiding work by leveraging existing
capabilities is a time honoured engineering tradition, even when the
simple way isn't the most elegant way. Consider also the fact that we
had full virtual machines long before we have usable Linux containers:
full isolation is actually *easier* than partial isolation, because
there are fewer places for things to go wrong, and less integration
work to do in the first place.

That said, something I mentioned to the OpenStack folks a while ago
(and I think on this list, but potentially not), is that I have now
realised the much-reviled (for good reason) *.pth files actually have
a legitimate use case in allowing API compatible versions of packages
to be shared between multiple virtual environments - you can trade
reduced isolation for easier upgrades on systems containing multiple
virtual environments by adding a suitable *.pth file to the venv
rather than the package itself. While there's currently no convenient
tooling around that, they're a feature CPython has supported for as
long as I can remember, so tools built on that idea would comfortably
work on all commonly supported Python versions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From noah at coderanger.net  Sat Feb  1 09:50:56 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sat, 1 Feb 2014 00:50:56 -0800
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CADiSq7d33vS-Z3OMLG5BH0XH7ZLWADbbV_4v9hv_EAP9FNKPWA@mail.gmail.com>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CADiSq7d33vS-Z3OMLG5BH0XH7ZLWADbbV_4v9hv_EAP9FNKPWA@mail.gmail.com>
Message-ID: <9CA7B887-F6C9-495E-9B70-11E6878AD313@coderanger.net>


On Feb 1, 2014, at 12:43 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 1 February 2014 18:23, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> On Fri, 31/1/14, Brian Wickman <wickman at gmail.com> wrote:
>> 
>>> There are myriad other practical reasons.  Here are some:
>> 
>> Thanks for taking the time to respond with the details - they are good data points
>> to think about!
>> 
>>> Lastly, there are social reasons. It's just hard to convince most engineers
>>> to use things like pkg_resources or pkgutil to manipulate resources
>>> when for them the status quo is just using __file__.  Bizarrely the social
>>> challenges are just as hard as the abovementioned technical challenges.
>> 
>> I agree it's bizarre, but sadly it's not surprising. People get used to certain ways
>> of doing things, and a certain kind of collective myopia develops when it
>> comes to looking at different ways of doing things. Having worked with fairly
>> diverse systems in my time, ISTM that sections of the Python community have
>> this myopia too. For example, the Java hatred and PEP 8 zealotry that you see
>> here and there.
>> 
>> One of the things that's puzzled me, for example, is why people think it's reasonable
>> or even necessary to have copies of pip and setuptools in every virtual environment
>> - often the same people who will tell you that your code isn't DRY enough! It's
>> certainly not a technical requirement, yet one of the reasons why PEP 405 venvs
>> aren't that popular is that pip and setuptools aren't automatically put in there. It's a
>> social issue - it's been decided that rather than exploring a technical approach to
>> addressing any issue with installing into venvs, it's better to bundle pip and setuptools
>> with Python 3.4, since that will seemingly be easier for people to swallow :-)
> 
> FWIW, installing into a venv from outside it works fine (that's how
> ensurepip works in 3.4). However, it's substantially *harder* to
> explain to people how to use it correctly that way. In theory you
> could change activation so that it also affected the default install
> locations, but the advantage of just having them installed per venv is
> that you're relying more on the builtin Python path machinery rather
> than adding something new. So while it's wasteful of disk space and
> means needing to upgrade them in every virtualenv, it does actually
> categorically eliminate many potential sources of bugs. Doing things
> the way pip and virtualenv do them also meant there was a whole pile
> of design work that *didn't need to be done* to get a functional
> system up and running. Avoiding work by leveraging existing
> capabilities is a time honoured engineering tradition, even when the
> simple way isn't the most elegant way. Consider also the fact that we
> had full virtual machines long before we have usable Linux containers:
> full isolation is actually *easier* than partial isolation, because
> there are fewer places for things to go wrong, and less integration
> work to do in the first place.
> 
> That said, something I mentioned to the OpenStack folks a while ago
> (and I think on this list, but potentially not), is that I have now
> realised the much-reviled (for good reason) *.pth files actually have
> a legitimate use case in allowing API compatible versions of packages
> to be shared between multiple virtual environments - you can trade
> reduced isolation for easier upgrades on systems containing multiple
> virtual environments by adding a suitable *.pth file to the venv
> rather than the package itself. While there's currently no convenient
> tooling around that, they're a feature CPython has supported for as
> long as I can remember, so tools built on that idea would comfortably
> work on all commonly supported Python versions.

In all but a tiny number of cases, you could use a symlink for this. Much less magic :-)

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/4520fc26/attachment-0001.sig>

From vinay_sajip at yahoo.co.uk  Sat Feb  1 10:29:50 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 09:29:50 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CADiSq7d33vS-Z3OMLG5BH0XH7ZLWADbbV_4v9hv_EAP9FNKPWA@mail.gmail.com>
Message-ID: <1391246990.89390.YahooMailBasic@web172401.mail.ir2.yahoo.com>

On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> FWIW, installing into a venv from outside it works fine

That's what I'm saying :-)

> However, it's substantially *harder* to explain to people how
> to use it correctly that way.

But distil installs into venvs (both PEP 405 and virtualenv)
without needing to be in there. It's as simple as

distil -e path/to/venv install XYZ

So I'm not sure what you mean by "use correctly" or what
exactly is hard to explain. Distil doesn't do anything special -
after all, installing a distribution just means moving
certain files into certain locations. If you know that you're
installing into one of (a) a specific venv, or (b) user
site-packages, or (c) system site-packages, then you know
what the installation locations are without doing anything
especially clever, don't you?

Of course, building and installation are commingled in
distutils and setuptools - that's "build-from-source-is-
all-you need" myopia right there - but I don't see how that
prevents installation-from-without or forces the usage of
installation-from-within, at least at a fundamental level.

> In theory you could change activation so that it also
> affected the default install locations, but the advantage
> of just having them installed per venv is that you're relying
> more on the builtin Python path machinery rather
> than adding something new.

But virtualenv solves an analogous problem when creating
venvs with different interpreters by reinvoking itself with a
specific interpreter. Distil does something similar, but that's
more to do with the fact that code in setup.py sometimes
takes different paths depending on which version of Python
is running it.

> So while it's wasteful of disk space and

Disk space is cheap, so that's not the issue, it's the
lack of elegance.

> Avoiding work by leveraging existing capabilities is a time
> honoured engineering tradition, even when the simple way
> isn't the most elegant way.

I'm pretty pragmatic, so I agree with this point in general,
but in this case I think you're side-stepping the issue of
technical debt. We're talking about file-system manipulations,
after all, not anything more esoteric than that. Any approach
which perpetuates the "python setup.py install" way of doing
things is, to me, not a good thing, which I think has been
recognised from the good old days when Tarek started
taking a fresh look at packaging.

> I have now realised the much-reviled (for good reason) *.pth
> files actually have a legitimate use case in allowing API
> compatible versions of packages to be shared between
> multiple virtual environments

Do you mean ref files, as discussed on import-sig?

Sure, but that's a tangential issue in my view. Worthwhile
pursuing, certainly, but an incremental improvement over what
we have now (and actually less necessary in the parallel
universe where wheels are more like jars and not reviled for
that just because of "Java - ewww").

Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Sat Feb  1 10:36:20 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 09:36:20 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <9CA7B887-F6C9-495E-9B70-11E6878AD313@coderanger.net>
Message-ID: <1391247380.86406.YahooMailBasic@web172406.mail.ir2.yahoo.com>

On Sat, 1/2/14, Noah Kantrowitz <noah at coderanger.net> wrote:

> In all but a tiny number of cases, you could use a symlink for this.
> Much less magic :-)

That's "POSIX is all there is" myopia, right there. While recent versions
of Windows have symlinks more like POSIX symlinks, XP only has a
stunted version called "reparse points" or "junction points" which are
not really fit for purpose. I think you'll find that XP environments are found
in rather more than "a tiny number of cases", and even though Microsoft
has end-of-lifed XP in terms of support, I fear it'll be around for a while
yet.

Regards,

Vinay Sajip

From ncoghlan at gmail.com  Sat Feb  1 10:38:47 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 1 Feb 2014 19:38:47 +1000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <9CA7B887-F6C9-495E-9B70-11E6878AD313@coderanger.net>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CADiSq7d33vS-Z3OMLG5BH0XH7ZLWADbbV_4v9hv_EAP9FNKPWA@mail.gmail.com>
 <9CA7B887-F6C9-495E-9B70-11E6878AD313@coderanger.net>
Message-ID: <CADiSq7cO8vqcPz97=RfbJi2MA+gSCmY55VCOLgEkmLDzEVtYnw@mail.gmail.com>

On 1 February 2014 18:50, Noah Kantrowitz <noah at coderanger.net> wrote:
> On Feb 1, 2014, at 12:43 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> That said, something I mentioned to the OpenStack folks a while ago
>> (and I think on this list, but potentially not), is that I have now
>> realised the much-reviled (for good reason) *.pth files actually have
>> a legitimate use case in allowing API compatible versions of packages
>> to be shared between multiple virtual environments - you can trade
>> reduced isolation for easier upgrades on systems containing multiple
>> virtual environments by adding a suitable *.pth file to the venv
>> rather than the package itself. While there's currently no convenient
>> tooling around that, they're a feature CPython has supported for as
>> long as I can remember, so tools built on that idea would comfortably
>> work on all commonly supported Python versions.
>
> In all but a tiny number of cases, you could use a symlink for this. Much less magic :-)

Unfortunately, that has two problems:

- the PEP 376 installation database directories are versioned so you
would need at least *two* symlinks (and potentially more for packages
that include multiple top level entries) for the installation metadata
to be properly available, and the metadata one would still have to be
updated in every virtual environment whenever the shared version
changed
- Windows symlinks are still broken and hard to use compared to POSIX symlinks

If it wasn't for the "support execution of arbitrary Python code
during interpreter startup" "feature" *.pth files likely wouldn't
worry so many people, but we're stuck with that now for backwards
compatibility reasons :P (Well, and because nobody thought to suggest
reducing its power to just adding new sys.path entries in Python 3...)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Feb  1 10:47:18 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 1 Feb 2014 19:47:18 +1000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391246990.89390.YahooMailBasic@web172401.mail.ir2.yahoo.com>
References: <CADiSq7d33vS-Z3OMLG5BH0XH7ZLWADbbV_4v9hv_EAP9FNKPWA@mail.gmail.com>
 <1391246990.89390.YahooMailBasic@web172401.mail.ir2.yahoo.com>
Message-ID: <CADiSq7fFEG9x97pD7HhvXiXsCftCp3M+1T-Ud1q70wFQ9eNVTg@mail.gmail.com>

On 1 February 2014 19:29, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> FWIW, installing into a venv from outside it works fine
>
> That's what I'm saying :-)
>
>> However, it's substantially *harder* to explain to people how
>> to use it correctly that way.
>
> But distil installs into venvs (both PEP 405 and virtualenv)
> without needing to be in there. It's as simple as
>
> distil -e path/to/venv install XYZ
>
> So I'm not sure what you mean by "use correctly" or what
> exactly is hard to explain. Distil doesn't do anything special -
> after all, installing a distribution just means moving
> certain files into certain locations. If you know that you're
> installing into one of (a) a specific venv, or (b) user
> site-packages, or (c) system site-packages, then you know
> what the installation locations are without doing anything
> especially clever, don't you?

I'm talking about easing the cognitive burden on developers. The
pip/virtualenv model is that once an environment is activated,
developers don't need to think about it anymore - they're in that
environment until they explicitly switch it off.

By installing pip *into* the environment, then it is automatically
affected by the same settings as all other Python components, so "pip
install library" inside an activated virtual environment *just does
the right thing*, and it does it in a way that doesn't require any
additional custom machinery or testing. Getting a substantial
usability improvement *almost for free* (since requirements.txt makes
it easy to discard and rebuild virtual environments as needed,
reducing the maintainability burden) is a big win.

Making any installer automatically respect an activated virtual
environment without installing it into that environment is no doubt
*feasible*, but just installing the installer *into* the environment
has the same effect with requiring any additional engineering or
testing effort.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dayanandasaraswati at gmail.com  Sat Feb  1 07:42:15 2014
From: dayanandasaraswati at gmail.com (Sanath Kumar)
Date: Fri, 31 Jan 2014 22:42:15 -0800
Subject: [Distutils] Issue with method _darwin_compiler_fixup in
	distutils/unixcompiler.py
Message-ID: <CABU+LKPqL-s34X6FZ7SGjaQ3mpLcg-x9mn=Ya37Sa75rSret9Q@mail.gmail.com>

Hello all,

When compiling MySQL-python module on Mac OSX Mavericks, I executed the
command "python setup.py build" and got an error when compiling the C
extension. In the output from build command, I see that it has tried
running the C compiler with the following command:

/ A p p l i c a t i o n s / X c o d e . a p p / C o n t e n t s / D e v e l
o p e r / T o o l c h a i n s / X c o d e D e f a u l t . x c t o o l c h a
i n / u s r / b i n / c l a n g   - f n o - s t r i c t - a l i a s i n g
- f n o - c o m m o n   - d y n a m i c   - a r c h   x 8 6 _ 6 4   - a r c
h   i 3 8 6   - g   - O s   - p i p e   - f n o - c o m m o n   - f n o - s
t r i c t - a l i a s i n g   - f w r a p v   - m n o - f u s e d - m a d d
  - D E N A B L E _ D T R A C E   - D M A C O S X   - D N D E B U G   - W a
l l   - W s t r i c t - p r o t o t y p e s   - W s h o r t e n - 6 4 - t o
- 3 2   - D N D E B U G   - g   - f w r a p v   - O s   - W a l l   - W s t
r i c t - p r o t o t y p e s   - D E N A B L E _ D T R A C E   - a r c h
x 8 6 _ 6 4   - a r c h   i 3 8 6   - p i p e
-Dversion_info=(1,2,4,'beta',4) -D__version__=1.2.4b4
-I/usr/local/Cellar/mysql/5.6.13/include
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-c _mysql.c -o build/temp.macosx-10.9-intel-2.7/_mysql.o -Os -g
-fno-strict-aliasing


This command is obviously malformed. I tracked down the root cause to be
the following line of code in _darwin_compiler_fixup method in
distutils/unixcompiler.py function:

       compiler_so = list(compiler_so)

Reading the code, this function expects the input parameter 'compiler_so'
to be a list, but the actual input in my case was a string. Therefore
instead of splitting the words of this sentence, it created a list with
each character in the sentence. Changing to the following fixes the problem:

      compiler_so = compiler_so.split()


This seems like a legit bug to me. Any thoughts?


Thanks,

Sanath
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140131/f6f7f238/attachment-0001.html>

From vinay_sajip at yahoo.co.uk  Sat Feb  1 11:49:15 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 10:49:15 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CADiSq7fFEG9x97pD7HhvXiXsCftCp3M+1T-Ud1q70wFQ9eNVTg@mail.gmail.com>
Message-ID: <1391251755.40393.YahooMailBasic@web172405.mail.ir2.yahoo.com>

On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:


> I'm talking about easing the cognitive burden on developers.
> The pip/virtualenv model is that once an environment is
> activated, developers don't need to think about it anymore -
> they're in that environment until they explicitly switch it off.

But distil deals with this::

$ pyvenv-3.3 /tmp/venv
$ source /tmp/venv/bin/activate
(venv) $ distil --ve
[...]
Python: 3.3
[...]

So, distil is using the venv's Python. Let's deactivate and try
again:

(venv) $ deactivate
$ distil --ve
[...]
Python: 2.7
[...]

So we're now using the system python - so far, so good.
 
> By installing pip *into* the environment, then it is automatically
> affected by the same settings as all other Python components,
> so "pip install library" inside an activated virtual environment
> *just does the right thing*, and it does it in a way that doesn't
> require any additional custom machinery or testing.

Likewise, with distil:

$ source /tmp/venv/bin/activate
(venv) $ distil list
(venv) $ distil install sarge
Checking requirements for sarge (0.1.3) ... done.
The following new packages will be downloaded and installed:
    sarge (0.1.3)
Downloading sarge-0.1.3.tar.gz to /tmp/tmpn01ls2
[...]
Building sarge (0.1.3) ...
[...]
Installing sarge (0.1.3) ...
[...]
    Installation completed.
(venv) $ distil list
sarge 0.1.3
(venv) $

So, I am at a loss to see your point.

> Making any installer automatically respect an activated
> virtual environment without installing it into that environment
> is no doubt *feasible*, but just installing the installer *into* the
> environment has the same effect with requiring any additional
> engineering or testing effort.

Not only feasible, there needing to be no doubt about it ;-), but
also not rocket science, and demonstrated above. Distil isn't
doing anything especially clever, so I don't understand what you
mean by "any additional engineering or testing effort". ISTM
you're making things seem more scary than they are. Of course,
if you find some specific *installation* scenario that distil doesn't
cover, I'll be all ears.

I emphasised *installation* to distinguish from building - distil
eschews running setup.py during installation, since if it did that
then it would be no different from pip in that respect, so there'd
be no point to it. So, "distil install" can't deal with cases of
substantive logic in setup.py - but that's not to say that distil
can't deal with setup.py at all. Instead, that functionality is
provided in other distil commands which deal with building
wheels from sdists. They could be relatively easily merged
into an install command that works like pip, but I thought the
point was to separate building from installation, so that's why
distil is like it is at the moment.

Regards,

Vinay Sajip

From p.f.moore at gmail.com  Sat Feb  1 11:53:52 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 1 Feb 2014 10:53:52 +0000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <9CA7B887-F6C9-495E-9B70-11E6878AD313@coderanger.net>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CADiSq7d33vS-Z3OMLG5BH0XH7ZLWADbbV_4v9hv_EAP9FNKPWA@mail.gmail.com>
 <9CA7B887-F6C9-495E-9B70-11E6878AD313@coderanger.net>
Message-ID: <CACac1F9ico_GDXh6H=AdYD0FtwVD_w65wtBdSvxjWTre8S2eYg@mail.gmail.com>

 1 February 2014 08:50, Noah Kantrowitz <noah at coderanger.net> wrote:
> In all but a tiny number of cases, you could use a symlink for this. Much less magic :-)

Windows isn't a "tiny number" of cases, unfortunately :-)
Paul

From p.f.moore at gmail.com  Sat Feb  1 11:57:41 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 1 Feb 2014 10:57:41 +0000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391247380.86406.YahooMailBasic@web172406.mail.ir2.yahoo.com>
References: <9CA7B887-F6C9-495E-9B70-11E6878AD313@coderanger.net>
 <1391247380.86406.YahooMailBasic@web172406.mail.ir2.yahoo.com>
Message-ID: <CACac1F_vZHZzSapc==JA+UFQdpm6=VwLhs-6Dw4GQbbj6L4eXg@mail.gmail.com>

On 1 February 2014 09:36, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> While recent versions
> of Windows have symlinks more like POSIX symlinks, XP only has a
> stunted version called "reparse points" or "junction points" which are
> not really fit for purpose. I think you'll find that XP environments are found
> in rather more than "a tiny number of cases", and even though Microsoft
> has end-of-lifed XP in terms of support, I fear it'll be around for a while
> yet.

One crippling thing about even recent Windows symlink support is that
to create a symlink requires elevation. There is a user right that can
be assigned to allow creation of symling without elevation, but
bizarrely (and, I gather, deliberately!) Windows strips that right
from administrative accounts - so you have a situation where a normal
user with the right can create symlinks, but an administrator with the
same right still needs elevation. And a major proportion of Windows
accounts are administrators.

Sigh. One day they might get it right.
Paul

From nad at acm.org  Sat Feb  1 11:58:41 2014
From: nad at acm.org (Ned Deily)
Date: Sat, 01 Feb 2014 02:58:41 -0800
Subject: [Distutils] Issue with method _darwin_compiler_fixup in
	distutils/unixcompiler.py
References: <CABU+LKPqL-s34X6FZ7SGjaQ3mpLcg-x9mn=Ya37Sa75rSret9Q@mail.gmail.com>
Message-ID: <nad-471AC6.02584101022014@news.gmane.org>

In article 
<CABU+LKPqL-s34X6FZ7SGjaQ3mpLcg-x9mn=Ya37Sa75rSret9Q at mail.gmail.com>,
 Sanath Kumar <dayanandasaraswati at gmail.com> wrote:
> When compiling MySQL-python module on Mac OSX Mavericks, I executed the
> command "python setup.py build" and got an error when compiling the C
> extension. In the output from build command, I see that it has tried
> running the C compiler with the following command:
> 
> / A p p l i c a t i o n s / X c o d e . a p p / C o n t e n t s / D e v e l
> o p e r / T o o l c h a i n s / X c o d e D e f a u l t . x c t o o l c h a
> i n / u s r / b i n / c l a n g   - f n o - s t r i c t - a l i a s i n g
> - f n o - c o m m o n   - d y n a m i c   - a r c h   x 8 6 _ 6 4   - a r c
> h   i 3 8 6   - g   - O s   - p i p e   - f n o - c o m m o n   - f n o - s
[...]

This appears to be an instance of Issue18071 which was fixed in Python 
2.7.6.  A workaround is to ensure you have installed the Xcode Command 
Line Tools; on OS X 10.9, run "xcodebuild --install".

http://bugs.python.org/issue18071

-- 
 Ned Deily,
 nad at acm.org


From p.f.moore at gmail.com  Sat Feb  1 12:05:42 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 1 Feb 2014 11:05:42 +0000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391246990.89390.YahooMailBasic@web172401.mail.ir2.yahoo.com>
References: <CADiSq7d33vS-Z3OMLG5BH0XH7ZLWADbbV_4v9hv_EAP9FNKPWA@mail.gmail.com>
 <1391246990.89390.YahooMailBasic@web172401.mail.ir2.yahoo.com>
Message-ID: <CACac1F-dxJd3P9ATX-DAnMYtaKhPAOwY8+9G28yx48JyMA6vzQ@mail.gmail.com>

On 1 February 2014 09:29, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> However, it's substantially *harder* to explain to people how
>> to use it correctly that way.
>
> But distil installs into venvs (both PEP 405 and virtualenv)
> without needing to be in there. It's as simple as
>
> distil -e path/to/venv install XYZ

I would like it if pip worked the same way (even though the tide of
installing pip into each virtualenv may be unstoppable by now). Would
you be interested in writing a patch to do this? I don't have the time
to commit to it and nobody else seems particularly interested in doing
it, which is one reason it's not happened, I suspect. It's certainly
not a fundamental limitation of pip.

If you'd rather have this as a selling point for distil, then that's
fair. But the vast weight of existing pip users and documentation,
plus the core support for pip as the preferred default installer
encapsulated in PEP 453, is going to make this much less useful to
people in practice.

Paul.

From ncoghlan at gmail.com  Sat Feb  1 12:31:49 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 1 Feb 2014 21:31:49 +1000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391251755.40393.YahooMailBasic@web172405.mail.ir2.yahoo.com>
References: <CADiSq7fFEG9x97pD7HhvXiXsCftCp3M+1T-Ud1q70wFQ9eNVTg@mail.gmail.com>
 <1391251755.40393.YahooMailBasic@web172405.mail.ir2.yahoo.com>
Message-ID: <CADiSq7eQjcD0SQYtsOkBaXPQzneg3UB8Y6XNmWd74EUTLQf-VQ@mail.gmail.com>

On 1 February 2014 20:49, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> So, I am at a loss to see your point.

My point is that doing it the way virtualenv/pip did avoided a bunch
of design work and associated testing, and reduced the opportunities
for bugs - when you're trying to get things done with limited
resources, that's a sensible engineering trade-off to make.

As Paul noted in his reply, there's nothing that *inherently* limits
pip to working that way, it just hasn't been a priority to change.

That said (and this is a point that hadn't occurred to me earlier),
it's also worth noting that not only does the bootstrapping approach
work well enough in most cases, but it also solves the problem of
being able to easily have a newer (or older!) pip in virtual
environments than is provided by the distro package manager on Linux
systems, *without* needing to do a global install outside the package
manager's control.

On that last point, note these two recommendations that PEP 453 makes
to downstream redistributors:

* ``pip install --upgrade pip`` in a global installation should not
affect any already created virtual environments (but is permitted to
affect future virtual environments, even though it will not do so when
using the standard implementation of ensurepip).

* ``pip install --upgrade pip`` in a virtual environment should not
affect the global installation.

If pip maintained perfect backwards compatibility across every
release, those recommendations wouldn't be necessary. However, we're
still in the process of actively hardening the security model, and
that *does* mean we're going to break things at times. Having the
version of pip in use be virtual environment dependent rather than
Python installation dependent helps deal with that more gracefully.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Sat Feb  1 13:03:34 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 12:03:34 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CACac1F-dxJd3P9ATX-DAnMYtaKhPAOwY8+9G28yx48JyMA6vzQ@mail.gmail.com>
Message-ID: <1391256214.60131.YahooMailBasic@web172404.mail.ir2.yahoo.com>

On Sat, 1/2/14, Paul Moore <p.f.moore at gmail.com> wrote:

> I would like it if pip worked the same way (even though the
> tide of installing pip into each virtualenv may be unstoppable
> by now). Would you be interested in writing a patch to do this?
> I don't have the time to commit to it and nobody else seems
> particularly interested in doing it, which is one reason it's not
> happened, I suspect. It's certainly not a fundamental limitation
> of pip.

It's not, and virtualenv does it with -p, right? Here's the distil code
that does it:

ARG_RE = re.compile(r'-[ep](=?.*)?')
# Use encode rather than br'' to avoid syntax error in 2.5, so we can
# print a more helpful message.
PYTHON_RE = re.compile(r'Python \d(\.\d+)+'.encode('utf-8'))

def find_python(env_or_python, is_env):
    result = None
    if not is_env:
        result = env_or_python
    else:
        env = os.path.abspath(env_or_python)
        if os.path.isdir(env):
            from distutils import sysconfig
            exe = 'python%s' % sysconfig.get_config_var('EXE')
            for subdir in ('bin', 'Scripts'):
                p = os.path.join(env, subdir, exe)
                if os.path.isfile(p) and os.access(p, os.X_OK):
                    result = p
                    break
    if result:
        import subprocess

        try:
            cmd = [result, '--version']
            p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                                 stderr=subprocess.STDOUT)
            out, _ = p.communicate()
            if p.returncode:
                logger.error('Command %r returned non-zero status: %d',
                             ' '.join(cmd), p.returncode)
                result = None
            elif not PYTHON_RE.match(out):
                logger.error('Command %r - unexpected output: %r',
                             ' '.join(cmd), out)
                result = None
        except Exception as e:
            logger.exception('Command %r - unexpected exception: %r',
                         ' '.join(cmd), e)
            result = None
    return result

def find_env_or_python(args=None):
    result = None
    args = (args or sys.argv)[1:]
    nargs = len(args)
    for i, arg in enumerate(args):
        m = ARG_RE.match(arg)
        if m:
            value = m.groups()[0]
            start = i + 1
            if value:
                if value[0] in '=:':
                    argval = value[1:]
                else:
                    argval = value
                end = i + 2
            elif i < (nargs - 1):
                argval = args[i + 1]
                end = i + 3
            # if result is set, then start and end indicate the slice to lose
            # from sys.argv. First, validate the env.
            result = find_python(argval, arg[1] =='e')
            if result:
                del sys.argv[start:end]
            else:
                if arg[1] == 'e':
                    msg = 'No such environment: %r' % argval
                else:
                    msg = 'No such Python interpreter: %r' % argval
                raise ValueError(msg)
    return result

def main(args=None):
    python = find_env_or_python()
    if python:
        cmd = [python] + sys.argv
        import subprocess
        return subprocess.call(cmd)
    # the rest of the main() logic goes here as normal

> If you'd rather have this as a selling point for distil, then that's
> fair. But the vast weight of existing pip users and
> documentation, plus the core support for pip as the preferred
> default installer encapsulated in PEP 453, is going to make
> this much less useful to people in practice.

I'm not trying to "sell" distil versus pip, as I've said before, though I do
want it to be better than pip, by trying out different ideas on how to
do things. In my admittedly biased opinion it already is better in numerous
respects, but if people won't see for lack of looking, that's not my loss :-)

The reasons I wouldn't volunteer to add this to pip are:

1. I don't believe I really need to, since the code above is pretty self-
    contained and shouldn't take too long to adapt. So, any obstacles to
    adding it would surely not be technical. Anyone can feel free to add
    the above to pip.

2. I find working on pip painful because of how it works. I have no
    animus against pip; I have overcome this pain in the past and
    did a fair chunk of the work in getting it to work on 2.x and 3.x in
    a single code-base. But that was when I thought it was important
    for 3.x adoption to get pip and virtualenv working under 3.x in a
    low-maintenance fashion (i.e. no 2to3). Of course, I now have the
    use of distil, so I seldom need to use pip :-)

3. I try to avoid "politics" and "religion" in software development when
    I can, and I feel their presence more in the pip project than I'm
    comfortable with. I don't really want to fight non-technical battles.

Re. PEP 453, I voiced my reservations at the time, but if "that ship
has sailed", then so be it - "nobody ever got fired for recommending
pip". The Python community will get the packaging infrastructure that
it deserves, just like a populace does with elections :-)

Regards,

Vinay Sajip

From vinay_sajip at yahoo.co.uk  Sat Feb  1 13:20:55 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 12:20:55 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CADiSq7eQjcD0SQYtsOkBaXPQzneg3UB8Y6XNmWd74EUTLQf-VQ@mail.gmail.com>
Message-ID: <1391257255.53980.YahooMailBasic@web172402.mail.ir2.yahoo.com>


On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> My point is that doing it the way virtualenv/pip did avoided a bunch
> of design work and associated testing, and reduced the
> opportunities for bugs - when you're trying to get things done with
> limited resources, that's a sensible engineering trade-off to make.

A "bunch of design work" makes it seem a lot more complicated
than it really is. Your suggestion comes across like an ex post facto
rationalisation of a decision which was perhaps more truly based
on social concerns than technical ones.

Note that I've developed distil as part of my volunteer activities - it
doesn't pay any of my bills - and on my own. And you're telling me
about how to get the best out of "limited resources"? :-)

> That said (and this is a point that hadn't occurred to me earlier),
> it's also worth noting that not only does the bootstrapping
> approach work well enough in most cases, but it also solves the
> problem of being able to easily have a newer (or older!) pip in
> virtual environments than is provided by the distro package
> manager on Linux systems

Eh? The problem of having a newer or older pip in a venv
only exists because pip needs to be in a venv ... so I don't see
the relevance of this to our earlier discussion. Since distil
doesn't occupy a space in venvs,. the concern of a system
version being older or newer than that in a venv doesn't arise.

Regards,

Vinay Sajip

From ncoghlan at gmail.com  Sat Feb  1 13:51:28 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 1 Feb 2014 22:51:28 +1000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391257255.53980.YahooMailBasic@web172402.mail.ir2.yahoo.com>
References: <CADiSq7eQjcD0SQYtsOkBaXPQzneg3UB8Y6XNmWd74EUTLQf-VQ@mail.gmail.com>
 <1391257255.53980.YahooMailBasic@web172402.mail.ir2.yahoo.com>
Message-ID: <CADiSq7f7O_Agv3YtgCVYbf1p8a-RYnZOFY7fvrdARn=y06zBXQ@mail.gmail.com>

On 1 February 2014 22:20, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>
> On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> My point is that doing it the way virtualenv/pip did avoided a bunch
>> of design work and associated testing, and reduced the
>> opportunities for bugs - when you're trying to get things done with
>> limited resources, that's a sensible engineering trade-off to make.
>
> A "bunch of design work" makes it seem a lot more complicated
> than it really is. Your suggestion comes across like an ex post facto
> rationalisation of a decision which was perhaps more truly based
> on social concerns than technical ones.

I have no idea how most of the internal design decisions on pip were
made (and I neither need nor particularly want to know - that would be
rather missing the point of collaborative development). However,
you're trying to make out that the distil approach is so clearly
superior that you don't understand why anyone would ever have
implemented it differently, and I'm trying to point out that there's
an entirely plausible answer to that question that doesn't involve
assuming poor engineering trade-offs on the part of the pip
development team.

> Note that I've developed distil as part of my volunteer activities - it
> doesn't pay any of my bills - and on my own. And you're telling me
> about how to get the best out of "limited resources"? :-)

No, things we do in our free time are the *best* opportunities to
investigate things like that, because they're not necessarily goal
oriented - they're more about satisfying our curiousity. Goal oriented
development is a *terrible* environment for exploration, because we
have to ruthlessly prune activities that don't obviously move us
closer to our goal.

Consider the following two engineering design options:

A. We install an independent copy of the installer in every virtual
environment, guaranteeing full isolation from the system Python and
permitting the use of different installer versions for different
projects, all using the exact same isolation mechanism as is used for
any Python component

B. We design a separate mechanism that allows a system installation of
the installer to easily be used to install software into virtual
environments, but requiring that the system and all virtual
environments be using the same version of the installer, and needing
to come up with a distinct targeting mechanism separate from the
activation mechanism already provided by virtual environments

Now *a priori*, you don't know how complex B is going to be. On the
other hand, you already *know* A will work, because you already have
working virtual environments.

So, from a scope of isolation point of view, A has the advantage: the
installer itself is isolated in addition to the installed components.

And from a design uncertainty point of view, A also has the advantage:
there is *no* design uncertainty, because you're just using the
already-known-to-work virtual environment mechanism.

So in a goal-oriented "provide isolation of Python applications"
context, there's no *reason* to even explore B: it has more design
uncertainty, and offers a lower level of isolation (because the
installer is still shared).

Now, you *have* had time to explore option B in distil, and have
discovered that it actually wasn't that hard after all. Cool, that's
great and would be a nice feature to have in pip as well. However, the
fact that *having already done it*, you discovered it wasn't all that
difficult, doesn't change the fact that the level of effort involved
was still greater than the nonexistent amount of work needed to get
the existing solution working once virtual environments are available
(and I noticed that you cut the part of the email pointing out the
concrete technical benefits of being able to isolate the installer in
addition to the installed components).

It's one thing to disagree with a design decision after we have made
the absolutely best possible case that we can for why that decision
was the right choice, and still failing to convince ourselves. It's
something else entirely to dismiss someone else's use case as invalid
and *then* argue that an arguably simpler design (that doesn't handle
the dismissed use case and was still only hypothetical at the time the
decision was made) would have been a better choice.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Sat Feb  1 15:04:53 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 14:04:53 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CADiSq7f7O_Agv3YtgCVYbf1p8a-RYnZOFY7fvrdARn=y06zBXQ@mail.gmail.com>
Message-ID: <1391263493.60250.YahooMailBasic@web172404.mail.ir2.yahoo.com>

On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I have no idea how most of the internal design decisions on
> pip were made (and I neither need nor particularly want to
> know - that would be rather missing the point of collaborative
> development). However, you're trying to make out that the
> distil approach is so clearly superior that you don't

Not "superior" in terms of "clever". I've made a point of saying
that distil *doesn't do anything clever* in this area. If you're
saying "a bunch of design work", making "a bunch" sound like
"a lot", without any detailed knowledge of the design internals,
then how do you know if it was a hard engineering issue or a
trivial one? If I say "I didn't find it hard, and I'm not claiming to
be especially clever" then at least I'm speaking from a
knowledge of what it takes to achieve something, rather than
perhaps taking someone else's word for it. Decisions should
be properly informed, in my view.

Oh, and collaborative development sometimes involves
challenging prior work and assumptions. That's one way
in which things get better - we build on the past, but
don't remain enslaved to it.

> understand why anyone would ever have implemented it
> differently, and I'm trying to point out that there's an entirely
> plausible answer to that question that doesn't involve
> assuming poor engineering trade-offs on the part of the pip
> development team.

I didn't (at least, not intentionally) say anything about poor
engineering trade-offs made by anyone - what would be the
point of that? I do say that better engineering approaches might
exist, and suggest that the way distil does it one such approach.
If you can suggest technical flaws in the approach distil takes,
that's one thing, and I will be keen to look at any points you make.
If you have no time to examine distil and make pronouncements,
that's fine too - we're all busy people. But defending the status
quo seemingly on nebulous technical grounds like "a bunch", while
avowing no knowledge of the technical details, seems wrong.
How many milli-hodges are there in a bunch? ;-)

> No, things we do in our free time are the *best* opportunities
> to investigate things like that, because they're not necessarily
> goal oriented - they're more about satisfying our curiousity.

I don't disagree - my clumsily made point  was to to say
"volunteer" in the sense of "part-time", and therefore to
strengthen the fact that I well understand "limited resources".

> Consider the following two engineering design options:
[details snipped]
> Now *a priori*, you don't know how complex B is going to be.
> On the other hand, you already *know* A will work, because you
> already have working virtual environments.

That is fine as an *a priori* position, and it's the position I was in
when contemplating the development of distil. But we don't stay 
stuck in *a priori* forever: as I developed distil, it became clear
that alternative approaches can work. Of course there were
uncertainties at the beginning, but by doing the development
work I was able to understand better the nature of those, and
I could convert them into certainties of what would and wouldn't
work.

> And from a design uncertainty point of view, A also has the
> advantage: there is *no* design uncertainty, because you're
> just using the already-known-to-work virtual environment
> mechanism.

So you're saying PEP 405 was a mistake? We should just have
stuck with the tried-and-tested virtualenv, which required changing
every time a new Python version came out, because it used (of
necessity) what might be uncharitable to call "hacks"?
 
> Now, you *have* had time to explore option B in distil, and
> have discovered that it actually wasn't that hard after all.
> Cool, that's great and would be a nice feature to have in pip
> as well. However, the fact that *having already done it*, you
> discovered it wasn't all that difficult, doesn't change the fact
> that the level of effort involved was still greater than the
> nonexistent amount of work needed to get the existing solution
> working

Yes, but it's not as if I'm just presenting this aspect of distil *now*.
I certainly highlighted the salient features on this list *before* the
PEP 453 ship set sail and distil is pretty well documented on
readthedocs.org.

> and I noticed that you cut the part of the email pointing out the
> concrete technical benefits of being able to isolate the installer
> in addition to the installed components.

I wasn't trying to gloss over anything. If an installer maintains
backward compatibility, then isolating the installer is not
especially valuable. Even if the installer doesn't get it right,
using different versions of the installer is perfectly possible
(and easy with distil, as it's a single-file executable), and in
my experience it's not been unheard of to have had to
"downgrade" virtualenv or pip because a new release broke
something.
 
> It's one thing to disagree with a design decision after we
> have made the absolutely best possible case that we can
> for why that decision was the right choice, and still failing
> to convince ourselves. It's something else entirely to
> dismiss someone else's use case as invalid and *then*
> argue that an arguably simpler design (that doesn't handle
> the dismissed use case and was still only hypothetical at
> the time the decision was made)

Is this some revisionist view of history? I'm confused :-) The
original announcement I made about distil was on 26 March,
2013:

http://thread.gmane.org/gmane.comp.python.distutils.devel/17336

The relevant features were fully documented in the version 0.1.0
documentation for distil, which was tagged on 22 March, 2013:

https://bitbucket.org/vinay.sajip/docs-distil/commits/04ac4badcab941e12ff6ecc0a3b6784fdc977e9b

This was on readthedocs.org and pythonhosted.org at the time
of the announcement.

The date on PEP 453 is 10 August, 2013 - at least four months
later:

http://www.python.org/dev/peps/pep-0453/

So, if no one who participated in the PEP 453 decision had time
to look at distil in that four month period, fair enough. It doesn't
mean that I'm somehow trying to bring something *new* to the
discussion *now*. I certainly made my reservations about the pip-
bundling route at the time it was being discussed here, but if
nobody was willing to look at alternatives, it's not something I
could help.

Of course, once the decision is made, what can you do but
defend it? I understand your position, and I'm not trying to
change anything. If anyone wants to provide specific, technical
feedback about any problems with distil's approach in any
area, I'll welcome that feedback - but I've already expressed my
view on non-technical battles.

Regards,

Vinay Sajip

From donald at stufft.io  Sat Feb  1 16:45:03 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 1 Feb 2014 10:45:03 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <1391263493.60250.YahooMailBasic@web172404.mail.ir2.yahoo.com>
References: <1391263493.60250.YahooMailBasic@web172404.mail.ir2.yahoo.com>
Message-ID: <B1DE6FD2-A953-49B9-923A-1D518690281A@stufft.io>

It predates my involvement with pip, but pip used to have the feature where it
did not require being installed into the virtualenv. It had the -E path/to/venv
flag, and also had a PIP_RESPECT_VIRTUALENV env var that would have it
automatically install into the activated virtualenv.

This was removed in the 1.1 release which was released on 2012-02-16. Again
this predates my involvement with pip, however the changelog states that::

    * Removed ``-E``/``--environment`` option and ``PIP_RESPECT_VIRTUALENV``;
      both use a restart-in-venv mechanism that's broken, and neither one is
      useful since every virtualenv now has pip inside it. Replace ``pip -E
      path/to/venv install Foo`` with ``virtualenv path/to/venv &&
      path/to/venv/pip install Foo``.

It was removed in this commit: https://github.com/pypa/pip/commit/87dd157.

So perhaps it isn't that pip is lacking a useful feature, it's that pip tried
it, as you did with distil, and due to pip's much larger user base it got
a much more thorough real world experience with it and it turned out to be
ultimately more trouble than it was worth? It could also have been a problem
with how pip wrote it of course, and perhaps you've discovered a better way
of writing it. However at the time it's obvious that the pain it was causing
was enough to cause the pip developers to break backwards compatibility to
remove that feature.

Why exactly was it removed? Well I can't find the reasoning beyond that commit
so we'd need to get Carl or perhaps Jannis to come and tell us if they still
remember. However it's clearly not that pip didn't just try or think of this,
it did and it ultimately removed it.

On Feb 1, 2014, at 9:04 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
>> I have no idea how most of the internal design decisions on
>> pip were made (and I neither need nor particularly want to
>> know - that would be rather missing the point of collaborative
>> development). However, you're trying to make out that the
>> distil approach is so clearly superior that you don't
> 
> Not "superior" in terms of "clever". I've made a point of saying
> that distil *doesn't do anything clever* in this area. If you're
> saying "a bunch of design work", making "a bunch" sound like
> "a lot", without any detailed knowledge of the design internals,
> then how do you know if it was a hard engineering issue or a
> trivial one? If I say "I didn't find it hard, and I'm not claiming to
> be especially clever" then at least I'm speaking from a
> knowledge of what it takes to achieve something, rather than
> perhaps taking someone else's word for it. Decisions should
> be properly informed, in my view.
> 
> Oh, and collaborative development sometimes involves
> challenging prior work and assumptions. That's one way
> in which things get better - we build on the past, but
> don't remain enslaved to it.
> 
>> understand why anyone would ever have implemented it
>> differently, and I'm trying to point out that there's an entirely
>> plausible answer to that question that doesn't involve
>> assuming poor engineering trade-offs on the part of the pip
>> development team.
> 
> I didn't (at least, not intentionally) say anything about poor
> engineering trade-offs made by anyone - what would be the
> point of that? I do say that better engineering approaches might
> exist, and suggest that the way distil does it one such approach.
> If you can suggest technical flaws in the approach distil takes,
> that's one thing, and I will be keen to look at any points you make.
> If you have no time to examine distil and make pronouncements,
> that's fine too - we're all busy people. But defending the status
> quo seemingly on nebulous technical grounds like "a bunch", while
> avowing no knowledge of the technical details, seems wrong.
> How many milli-hodges are there in a bunch? ;-)
> 
>> No, things we do in our free time are the *best* opportunities
>> to investigate things like that, because they're not necessarily
>> goal oriented - they're more about satisfying our curiousity.
> 
> I don't disagree - my clumsily made point  was to to say
> "volunteer" in the sense of "part-time", and therefore to
> strengthen the fact that I well understand "limited resources".
> 
>> Consider the following two engineering design options:
> [details snipped]
>> Now *a priori*, you don't know how complex B is going to be.
>> On the other hand, you already *know* A will work, because you
>> already have working virtual environments.
> 
> That is fine as an *a priori* position, and it's the position I was in
> when contemplating the development of distil. But we don't stay 
> stuck in *a priori* forever: as I developed distil, it became clear
> that alternative approaches can work. Of course there were
> uncertainties at the beginning, but by doing the development
> work I was able to understand better the nature of those, and
> I could convert them into certainties of what would and wouldn't
> work.
> 
>> And from a design uncertainty point of view, A also has the
>> advantage: there is *no* design uncertainty, because you're
>> just using the already-known-to-work virtual environment
>> mechanism.
> 
> So you're saying PEP 405 was a mistake? We should just have
> stuck with the tried-and-tested virtualenv, which required changing
> every time a new Python version came out, because it used (of
> necessity) what might be uncharitable to call "hacks"?
> 
>> Now, you *have* had time to explore option B in distil, and
>> have discovered that it actually wasn't that hard after all.
>> Cool, that's great and would be a nice feature to have in pip
>> as well. However, the fact that *having already done it*, you
>> discovered it wasn't all that difficult, doesn't change the fact
>> that the level of effort involved was still greater than the
>> nonexistent amount of work needed to get the existing solution
>> working
> 
> Yes, but it's not as if I'm just presenting this aspect of distil *now*.
> I certainly highlighted the salient features on this list *before* the
> PEP 453 ship set sail and distil is pretty well documented on
> readthedocs.org.
> 
>> and I noticed that you cut the part of the email pointing out the
>> concrete technical benefits of being able to isolate the installer
>> in addition to the installed components.
> 
> I wasn't trying to gloss over anything. If an installer maintains
> backward compatibility, then isolating the installer is not
> especially valuable. Even if the installer doesn't get it right,
> using different versions of the installer is perfectly possible
> (and easy with distil, as it's a single-file executable), and in
> my experience it's not been unheard of to have had to
> "downgrade" virtualenv or pip because a new release broke
> something.
> 
>> It's one thing to disagree with a design decision after we
>> have made the absolutely best possible case that we can
>> for why that decision was the right choice, and still failing
>> to convince ourselves. It's something else entirely to
>> dismiss someone else's use case as invalid and *then*
>> argue that an arguably simpler design (that doesn't handle
>> the dismissed use case and was still only hypothetical at
>> the time the decision was made)
> 
> Is this some revisionist view of history? I'm confused :-) The
> original announcement I made about distil was on 26 March,
> 2013:
> 
> http://thread.gmane.org/gmane.comp.python.distutils.devel/17336
> 
> The relevant features were fully documented in the version 0.1.0
> documentation for distil, which was tagged on 22 March, 2013:
> 
> https://bitbucket.org/vinay.sajip/docs-distil/commits/04ac4badcab941e12ff6ecc0a3b6784fdc977e9b
> 
> This was on readthedocs.org and pythonhosted.org at the time
> of the announcement.
> 
> The date on PEP 453 is 10 August, 2013 - at least four months
> later:
> 
> http://www.python.org/dev/peps/pep-0453/
> 
> So, if no one who participated in the PEP 453 decision had time
> to look at distil in that four month period, fair enough. It doesn't
> mean that I'm somehow trying to bring something *new* to the
> discussion *now*. I certainly made my reservations about the pip-
> bundling route at the time it was being discussed here, but if
> nobody was willing to look at alternatives, it's not something I
> could help.
> 
> Of course, once the decision is made, what can you do but
> defend it? I understand your position, and I'm not trying to
> change anything. If anyone wants to provide specific, technical
> feedback about any problems with distil's approach in any
> area, I'll welcome that feedback - but I've already expressed my
> view on non-technical battles.
> 
> Regards,
> 
> Vinay Sajip
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/3fc3408b/attachment-0001.sig>

From donald at stufft.io  Sat Feb  1 17:19:07 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 1 Feb 2014 11:19:07 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <B1DE6FD2-A953-49B9-923A-1D518690281A@stufft.io>
References: <1391263493.60250.YahooMailBasic@web172404.mail.ir2.yahoo.com>
 <B1DE6FD2-A953-49B9-923A-1D518690281A@stufft.io>
Message-ID: <3B94B8AB-139E-4A38-97CB-A4AD39A02188@stufft.io>


On Feb 1, 2014, at 10:45 AM, Donald Stufft <donald at stufft.io> wrote:

> It predates my involvement with pip, but pip used to have the feature where it
> did not require being installed into the virtualenv. It had the -E path/to/venv
> flag, and also had a PIP_RESPECT_VIRTUALENV env var that would have it
> automatically install into the activated virtualenv.
> 
> This was removed in the 1.1 release which was released on 2012-02-16. Again
> this predates my involvement with pip, however the changelog states that::
> 
>    * Removed ``-E``/``--environment`` option and ``PIP_RESPECT_VIRTUALENV``;
>      both use a restart-in-venv mechanism that's broken, and neither one is
>      useful since every virtualenv now has pip inside it. Replace ``pip -E
>      path/to/venv install Foo`` with ``virtualenv path/to/venv &&
>      path/to/venv/pip install Foo``.
> 
> It was removed in this commit: https://github.com/pypa/pip/commit/87dd157.
> 
> So perhaps it isn't that pip is lacking a useful feature, it's that pip tried
> it, as you did with distil, and due to pip's much larger user base it got
> a much more thorough real world experience with it and it turned out to be
> ultimately more trouble than it was worth? It could also have been a problem
> with how pip wrote it of course, and perhaps you've discovered a better way
> of writing it. However at the time it's obvious that the pain it was causing
> was enough to cause the pip developers to break backwards compatibility to
> remove that feature.
> 
> Why exactly was it removed? Well I can't find the reasoning beyond that commit
> so we'd need to get Carl or perhaps Jannis to come and tell us if they still
> remember. However it's clearly not that pip didn't just try or think of this,
> it did and it ultimately removed it.
> 
> On Feb 1, 2014, at 9:04 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> 
>> On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> 
>>> I have no idea how most of the internal design decisions on
>>> pip were made (and I neither need nor particularly want to
>>> know - that would be rather missing the point of collaborative
>>> development). However, you're trying to make out that the
>>> distil approach is so clearly superior that you don't
>> 
>> Not "superior" in terms of "clever". I've made a point of saying
>> that distil *doesn't do anything clever* in this area. If you're
>> saying "a bunch of design work", making "a bunch" sound like
>> "a lot", without any detailed knowledge of the design internals,
>> then how do you know if it was a hard engineering issue or a
>> trivial one? If I say "I didn't find it hard, and I'm not claiming to
>> be especially clever" then at least I'm speaking from a
>> knowledge of what it takes to achieve something, rather than
>> perhaps taking someone else's word for it. Decisions should
>> be properly informed, in my view.
>> 
>> Oh, and collaborative development sometimes involves
>> challenging prior work and assumptions. That's one way
>> in which things get better - we build on the past, but
>> don't remain enslaved to it.
>> 
>>> understand why anyone would ever have implemented it
>>> differently, and I'm trying to point out that there's an entirely
>>> plausible answer to that question that doesn't involve
>>> assuming poor engineering trade-offs on the part of the pip
>>> development team.
>> 
>> I didn't (at least, not intentionally) say anything about poor
>> engineering trade-offs made by anyone - what would be the
>> point of that? I do say that better engineering approaches might
>> exist, and suggest that the way distil does it one such approach.
>> If you can suggest technical flaws in the approach distil takes,
>> that's one thing, and I will be keen to look at any points you make.
>> If you have no time to examine distil and make pronouncements,
>> that's fine too - we're all busy people. But defending the status
>> quo seemingly on nebulous technical grounds like "a bunch", while
>> avowing no knowledge of the technical details, seems wrong.
>> How many milli-hodges are there in a bunch? ;-)
>> 
>>> No, things we do in our free time are the *best* opportunities
>>> to investigate things like that, because they're not necessarily
>>> goal oriented - they're more about satisfying our curiousity.
>> 
>> I don't disagree - my clumsily made point  was to to say
>> "volunteer" in the sense of "part-time", and therefore to
>> strengthen the fact that I well understand "limited resources".
>> 
>>> Consider the following two engineering design options:
>> [details snipped]
>>> Now *a priori*, you don't know how complex B is going to be.
>>> On the other hand, you already *know* A will work, because you
>>> already have working virtual environments.
>> 
>> That is fine as an *a priori* position, and it's the position I was in
>> when contemplating the development of distil. But we don't stay 
>> stuck in *a priori* forever: as I developed distil, it became clear
>> that alternative approaches can work. Of course there were
>> uncertainties at the beginning, but by doing the development
>> work I was able to understand better the nature of those, and
>> I could convert them into certainties of what would and wouldn't
>> work.
>> 
>>> And from a design uncertainty point of view, A also has the
>>> advantage: there is *no* design uncertainty, because you're
>>> just using the already-known-to-work virtual environment
>>> mechanism.
>> 
>> So you're saying PEP 405 was a mistake? We should just have
>> stuck with the tried-and-tested virtualenv, which required changing
>> every time a new Python version came out, because it used (of
>> necessity) what might be uncharitable to call "hacks"?
>> 
>>> Now, you *have* had time to explore option B in distil, and
>>> have discovered that it actually wasn't that hard after all.
>>> Cool, that's great and would be a nice feature to have in pip
>>> as well. However, the fact that *having already done it*, you
>>> discovered it wasn't all that difficult, doesn't change the fact
>>> that the level of effort involved was still greater than the
>>> nonexistent amount of work needed to get the existing solution
>>> working
>> 
>> Yes, but it's not as if I'm just presenting this aspect of distil *now*.
>> I certainly highlighted the salient features on this list *before* the
>> PEP 453 ship set sail and distil is pretty well documented on
>> readthedocs.org.
>> 
>>> and I noticed that you cut the part of the email pointing out the
>>> concrete technical benefits of being able to isolate the installer
>>> in addition to the installed components.
>> 
>> I wasn't trying to gloss over anything. If an installer maintains
>> backward compatibility, then isolating the installer is not
>> especially valuable. Even if the installer doesn't get it right,
>> using different versions of the installer is perfectly possible
>> (and easy with distil, as it's a single-file executable), and in
>> my experience it's not been unheard of to have had to
>> "downgrade" virtualenv or pip because a new release broke
>> something.
>> 
>>> It's one thing to disagree with a design decision after we
>>> have made the absolutely best possible case that we can
>>> for why that decision was the right choice, and still failing
>>> to convince ourselves. It's something else entirely to
>>> dismiss someone else's use case as invalid and *then*
>>> argue that an arguably simpler design (that doesn't handle
>>> the dismissed use case and was still only hypothetical at
>>> the time the decision was made)
>> 
>> Is this some revisionist view of history? I'm confused :-) The
>> original announcement I made about distil was on 26 March,
>> 2013:
>> 
>> http://thread.gmane.org/gmane.comp.python.distutils.devel/17336
>> 
>> The relevant features were fully documented in the version 0.1.0
>> documentation for distil, which was tagged on 22 March, 2013:
>> 
>> https://bitbucket.org/vinay.sajip/docs-distil/commits/04ac4badcab941e12ff6ecc0a3b6784fdc977e9b
>> 
>> This was on readthedocs.org and pythonhosted.org at the time
>> of the announcement.
>> 
>> The date on PEP 453 is 10 August, 2013 - at least four months
>> later:
>> 
>> http://www.python.org/dev/peps/pep-0453/
>> 
>> So, if no one who participated in the PEP 453 decision had time
>> to look at distil in that four month period, fair enough. It doesn't
>> mean that I'm somehow trying to bring something *new* to the
>> discussion *now*. I certainly made my reservations about the pip-
>> bundling route at the time it was being discussed here, but if
>> nobody was willing to look at alternatives, it's not something I
>> could help.
>> 
>> Of course, once the decision is made, what can you do but
>> defend it? I understand your position, and I'm not trying to
>> change anything. If anyone wants to provide specific, technical
>> feedback about any problems with distil's approach in any
>> area, I'll welcome that feedback - but I've already expressed my
>> view on non-technical battles.
>> 
>> Regards,
>> 
>> Vinay Sajip
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> 
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Additionally I'd point out that, looking at how you've implemented it, it seems
to depend on the fact that distil is a single file and thus doesn't need
installed into site-packages. Pip however is not a single file and to use the
same technique as you did would require installing pip into each virtualenv or
doing some gross hacks to add pip to the sys.path dynamically.

Now it's possible that pip *could* be modified to fully support running as a
single file, however that would also entail getting our dependencies to support
that too. Some of them may work currently by accident but I would be
uncomfortable relying on support by accident and would want to have them
committed to running like that.

Finally I haven't touched distil because you've been less then forthcoming
about it's own code. I realize it's just using zip tech to bundle it and thus
I can unpack it and read it but:

  A) That is a gigantic pain in the ass to do just because for reasons unknown
     you've been unwilling to publish it's source.
  B) I actually *did* that once, and found that there is no OSS license
     attached to the code and instead it's "All Rights Reserved" at which point
     I trashed the files.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/19f70df4/attachment.sig>

From vinay_sajip at yahoo.co.uk  Sat Feb  1 17:25:57 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 16:25:57 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <B1DE6FD2-A953-49B9-923A-1D518690281A@stufft.io>
Message-ID: <1391271957.29649.YahooMailBasic@web172404.mail.ir2.yahoo.com>

On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:

> So perhaps it isn't that pip is lacking a useful feature, it's that
> pip tried it, as you did with distil, and due to pip's much larger
> user base it got a much more thorough real world experience
> with it and it turned out to be ultimately more trouble than it was
> worth? It could also have been a problem with how pip wrote it
> of course, and perhaps you've discovered a better way of
> writing it. However at the time it's obvious that the pain it was
> causing was enough to cause the pip developers to break
> backwards compatibility to remove that feature.

> Why exactly was it removed? Well I can't find the reasoning
> beyond that commit so we'd need to get Carl or perhaps Jannis
> to come and tell us if they still remember. However it's clearly not
> that pip didn't just try or think of this, it did and it ultimately removed it.

Sure, and if they did find some problem which is also present in
distil (i.e. due to a fundamental drawback of the approach) then I'll put
my thinking cap back on. It'll be worth having that problem on record
to avoid any repetition of work in the future.

Regards,

Vinay Sajip

From vinay_sajip at yahoo.co.uk  Sat Feb  1 17:50:31 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 16:50:31 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <3B94B8AB-139E-4A38-97CB-A4AD39A02188@stufft.io>
Message-ID: <1391273431.69796.YahooMailBasic@web172405.mail.ir2.yahoo.com>

On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:

> Additionally I'd point out that, looking at how you've
> implemented it, it seems to depend on the fact that distil
> is a single file and thus doesn't need installed into
> site-packages.

No, I don't believe that's the case - I developed (and continue to
develop) distil without running it as a single-file executable, since
most of its code is in a zip when deployed as a single file, so not
readily debuggable. The deployment of distil as a single file, though
seen by some as controversial (because unusual), is merely
meant as a convenience. Any other deployment mechanism would
mean using pip to install it, which kind of defeats the purpose :-)

> Finally I haven't touched distil because you've been less then
> forthcoming about it's own code. I realize it's just using zip tech
>  to bundle it and thus I can unpack it and read it but:

Well, it takes less commitment to try it out as a black box and report
failures or usability drawbacks than to dig into its source code, but
there hasn't been a lot of feedback, so I really doubt if that's a big
barrier to people trying it out. I don't believe pip users spend a lot of
time looking into pip source code.

As you say, the code is inspectable, though not as readily as it
would be if it were on GitHub, say. I haven't given particular thought
to the license since I haven't published it, but there's no conspiracy
to keep anything secret or proprietary, and nothing is obfuscated.

I've been looking for usability feedback and some of the features are
still experimental (not so much on the installation side, more on the
build side), so it's premature to open it up fully: I'm not looking for code
contributions, as distil is still a test-bed for distlib and some ideas
about how to improve the user experience. If that means some people
lose interest, I'm OK with that: I've contributed plenty to open source and
I expect to continue to do so. I just don't subscribe to the idea that
somehow everything magically gets better when it's put on GitHub.

Note that distlib is licensed under a valid OSS license, but it hasn't
made much difference in terms of the amount of feedback I've
received.

Regards,

Vinay Sajip

From donald at stufft.io  Sat Feb  1 18:06:02 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 1 Feb 2014 12:06:02 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <1391273431.69796.YahooMailBasic@web172405.mail.ir2.yahoo.com>
References: <1391273431.69796.YahooMailBasic@web172405.mail.ir2.yahoo.com>
Message-ID: <759472D1-1EFF-49B3-8A11-51AB11846FA0@stufft.io>


On Feb 1, 2014, at 11:50 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:
> 
>> Additionally I'd point out that, looking at how you've
>> implemented it, it seems to depend on the fact that distil
>> is a single file and thus doesn't need installed into
>> site-packages.
> 
> No, I don't believe that's the case - I developed (and continue to
> develop) distil without running it as a single-file executable, since
> most of its code is in a zip when deployed as a single file, so not
> readily debuggable. The deployment of distil as a single file, though
> seen by some as controversial (because unusual), is merely
> meant as a convenience. Any other deployment mechanism would
> mean using pip to install it, which kind of defeats the purpose :-)
> 
>> Finally I haven't touched distil because you've been less then
>> forthcoming about it's own code. I realize it's just using zip tech
>> to bundle it and thus I can unpack it and read it but:
> 
> Well, it takes less commitment to try it out as a black box and report
> failures or usability drawbacks than to dig into its source code, but
> there hasn't been a lot of feedback, so I really doubt if that's a big
> barrier to people trying it out. I don't believe pip users spend a lot of
> time looking into pip source code.
> 
> As you say, the code is inspectable, though not as readily as it
> would be if it were on GitHub, say. I haven't given particular thought
> to the license since I haven't published it, but there's no conspiracy
> to keep anything secret or proprietary, and nothing is obfuscated.
> 
> I've been looking for usability feedback and some of the features are
> still experimental (not so much on the installation side, more on the
> build side), so it's premature to open it up fully: I'm not looking for code
> contributions, as distil is still a test-bed for distlib and some ideas
> about how to improve the user experience. If that means some people
> lose interest, I'm OK with that: I've contributed plenty to open source and
> I expect to continue to do so. I just don't subscribe to the idea that
> somehow everything magically gets better when it's put on GitHub.

I never said anything gets magically better on GitHub, I told you the
reason why *I* haven?t bothered to try it. You can do what you want
with that information, ignore it or otherwise. I have a ton of things to
do in my packaging time from pip to PyPI to PEPs and most people
tend to have limited time. If you want people to try your thing out
the onus is on you to make it attractive for them to try. As it is right now
it?s not attractive for me as I can?t look to see how it?s doing something
and then go and try to add that to pip or even improve distil itself.

> 
> Note that distlib is licensed under a valid OSS license, but it hasn't
> made much difference in terms of the amount of feedback I've
> received.

Distlib is a significantly different case, it?s a library that is going to be
directly useful for a very small population of people. The vast majority
of people do not have any need or desire to directly work with distlib.

I mean think about it, it?s basically a library that is going to be useful
primarily to people writing an installer, so you?ve got pip, zc.buildout,
easy_install, distil, and maybe curdling as the bulk of your entire
demographic. Of those pip has roughly 3 active developers at the
moment, No idea on zc.buildout, easy_install has 1, distil has 1, and
curdling has I have no idea. So your main demographic can probably
be counted on your fingers alone. Of course you haven?t received a ton
of feedback. I can guarantee you that if distlib wasn?t released under an
OSS license that pip wouldn?t have used it, and then you likely wouldn?t
have gotten at least half of the issues filed on the bug tracker which came
from a member of the pip developers.

> 
> Regards,
> 
> Vinay Sajip


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/07cc86e0/attachment.sig>

From Steve.Dower at microsoft.com  Sat Feb  1 18:24:57 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 1 Feb 2014 17:24:57 +0000
Subject: [Distutils] distlib.mount API design (was: wheels on
	sys.path	clarification (reboot))
In-Reply-To: <CACac1F_jZcNRuSqOS3ytUoNBuvMM0ztjDjsDxEtbh3NvpkBV0Q@mail.gmail.com>
References: <CADiSq7cuKv1u5M8DzfeNaGFSX=voSj1evy1HjizqhXKw+wnOeg@mail.gmail.com>
 <CACac1F-MDhC8pOG2mDEkTFgXid6Z=G2JdMW1JQzJQweL3fbK4g@mail.gmail.com>
 <CADiSq7d7wTTK7W6gm25Dkz2t6wVDb4gBRh8YUUVidbo08YxFqg@mail.gmail.com>,
 <CACac1F_jZcNRuSqOS3ytUoNBuvMM0ztjDjsDxEtbh3NvpkBV0Q@mail.gmail.com>
Message-ID: <bda0116d40fb44018cfbb7bbfcb08ae3@BLUPR03MB389.namprd03.prod.outlook.com>

FWIW, Windows (by default) has a regular maintenance task that will clean up old files in the TEMP directory. I think the default settings will delete files older than 30 days and more aggressively if disk space is running low.

I'd say pick a consistent/static subfolder ('wheel_mount_35_amd64' or something), autogenerate whatever is needed within there, and leave it behind. Users who are concerned can rm -rf whenever they like and everyone else can let the OS handle it.

Cheers,
Steve

Top-posted from my Windows Phone
________________________________
From: Paul Moore<mailto:p.f.moore at gmail.com>
Sent: ?1/?30/?2014 14:52
To: Nick Coghlan<mailto:ncoghlan at gmail.com>
Cc: DistUtils mailing list<mailto:distutils-sig at python.org>
Subject: Re: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))

On 30 January 2014 22:38, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The advantage of wheels over plain zipfiles for this use case is the
> structured metadata. distlib.mount doesn't try to guess the package
> structure for the extensions, you have to provide an EXTENSIONS file in the
> metadata that explains what C extensions are present and how they should map
> into the module namespace.

OK, I think I get the idea now.

I'm still not comfortable with the temp directory clutter that
unpacking leaves (in particular on Windows where deletion isn't even
possible in an atexit routine) but I'll survive.

I *would* like to see the various technical issues and implications in
the API documentation, though. The implications and limitations, and
in particular the manual cache management requirements, need to be
made explicit. (I thought I'd seen docs somewhere, but they definitely
aren't in the API reference for the distlib.wheel module).

Paul
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/f01f2899/attachment.html>

From brett at python.org  Sat Feb  1 18:26:44 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 1 Feb 2014 12:26:44 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
Message-ID: <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>

On Sat, Feb 1, 2014 at 3:23 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> On Fri, 31/1/14, Brian Wickman <wickman at gmail.com> wrote:
>
> > There are myriad other practical reasons.  Here are some:
>
> Thanks for taking the time to respond with the details - they are good
> data points
> to think about!
>
> > Lastly, there are social reasons. It's just hard to convince most
> engineers
> > to use things like pkg_resources or pkgutil to manipulate resources
> > when for them the status quo is just using __file__.  Bizarrely the
> social
> > challenges are just as hard as the abovementioned technical challenges.
>
> I agree it's bizarre, but sadly it's not surprising. People get used to
> certain ways
> of doing things, and a certain kind of collective myopia develops when it
> comes to looking at different ways of doing things. Having worked with
> fairly
> diverse systems in my time, ISTM that sections of the Python community have
> this myopia too. For example, the Java hatred and PEP 8 zealotry that you
> see
> here and there.
>

PEP 302 tried to unify this with get_data() and set_data() on loaders, but
prior to Python 3.3 you just didn't have any guarantee that __loader__
would even be set, let alone have a loader with those methods. Paul can
tell me if my hunch is off, but I assume the dream was that people would do
`__loader__.get_data('asset.txt')` instead of
`os.path.join(os.path.dirname(__file__), 'asset.txt')` to read something
bundled with your package. But as Brian has pointed out, people just have
habits at this point of assuming unpacked files and working off of __file__.

I know Daniel has said he wanted some concept of a listdir() on loaders so
that they could basically act as a virtual file system for packages, but it
would also require locking down what relative vs. absolute paths meant in
get_data/set_data (i.e. is it relative to the loader in the package or the
package itself? My pref is the latter for the case of reused loaders) and
really pushing people to use the loader APIs for reading intra-package
"files".


>
> One of the things that's puzzled me, for example, is why people think it's
> reasonable
> or even necessary to have copies of pip and setuptools in every virtual
> environment
> - often the same people who will tell you that your code isn't DRY enough!
> It's
> certainly not a technical requirement, yet one of the reasons why PEP 405
> venvs
> aren't that popular is that pip and setuptools aren't automatically put in
> there. It's a
> social issue - it's been decided that rather than exploring a technical
> approach to
> addressing any issue with installing into venvs, it's better to bundle pip
> and setuptools
> with Python 3.4, since that will seemingly be easier for people to swallow
> :-)
>

I suspect it's also ignorance and differences in deployment strategies.
Some people have such small deployments that hitting PyPI every time works,
for others like Brian it can't be more than a cp w/ an unzip. Maybe the
Packaging Users Guide could have a Recommended Deployment Strategies
section or something. I doubt there are that many common needs beyond "pull
from PyPI every time", "pull from your own wheel repo", "copy everything
over in a single wheel/zip you unpack" so that people at least have a
starting point to work from (especially if the instructions work on all
platforms, i.e. no symlink discussions).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/cdfb4b78/attachment-0001.html>

From donald at stufft.io  Sat Feb  1 18:34:34 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 01 Feb 2014 12:34:34 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
Message-ID: <1391276074.26266.78087269.317ED389@webmail.messagingengine.com>

On Sat, Feb 1, 2014, at 12:26 PM, Brett Cannon wrote:




On Sat, Feb 1, 2014 at 3:23 AM, Vinay Sajip
<[1]vinay_sajip at yahoo.co.uk> wrote:

On Fri, 31/1/14, Brian Wickman <[2]wickman at gmail.com> wrote:



> There are myriad other practical reasons.  Here are some:



Thanks for taking the time to respond with the details - they are good
data points

to think about!





> Lastly, there are social reasons. It's just hard to convince most
engineers

> to use things like pkg_resources or pkgutil to manipulate resources

> when for them the status quo is just using __file__.  Bizarrely the
social

> challenges are just as hard as the abovementioned technical
challenges.



I agree it's bizarre, but sadly it's not surprising. People get used to
certain ways

of doing things, and a certain kind of collective myopia develops when
it

comes to looking at different ways of doing things. Having worked with
fairly

diverse systems in my time, ISTM that sections of the Python community
have

this myopia too. For example, the Java hatred and PEP 8 zealotry that
you see

here and there.


PEP 302 tried to unify this with get_data() and set_data() on loaders,
but prior to Python 3.3 you just didn't have any guarantee that
__loader__ would even be set, let alone have a loader with those
methods. Paul can tell me if my hunch is off, but I assume the dream
was that people would do `__loader__.get_data('asset.txt')` instead of
`os.path.join(os.path.dirname(__file__), 'asset.txt')` to read
something bundled with your package. But as Brian has pointed out,
people just have habits at this point of assuming unpacked files and
working off of __file__.

I know Daniel has said he wanted some concept of a listdir() on loaders
so that they could basically act as a virtual file system for packages,
but it would also require locking down what relative vs. absolute paths
meant in get_data/set_data (i.e. is it relative to the loader in the
package or the package itself? My pref is the latter for the case of
reused loaders) and really pushing people to use the loader APIs for
reading intra-package "files".


There's also the problem when you need to give a file that you have
packaged as part of your thing to an API that only accepts filenames.
The standard ssl module is one of these that i've run into recently.




One of the things that's puzzled me, for example, is why people think
it's reasonable

or even necessary to have copies of pip and setuptools in every virtual
environment

- often the same people who will tell you that your code isn't DRY
enough! It's

certainly not a technical requirement, yet one of the reasons why PEP
405 venvs

aren't that popular is that pip and setuptools aren't automatically put
in there. It's a

social issue - it's been decided that rather than exploring a technical
approach to

addressing any issue with installing into venvs, it's better to bundle
pip and setuptools

with Python 3.4, since that will seemingly be easier for people to
swallow :-)


I suspect it's also ignorance and differences in deployment strategies.
Some people have such small deployments that hitting PyPI every time
works, for others like Brian it can't be more than a cp w/ an unzip.
Maybe the Packaging Users Guide could have a Recommended Deployment
Strategies section or something. I doubt there are that many common
needs beyond "pull from PyPI every time", "pull from your own wheel
repo", "copy everything over in a single wheel/zip you unpack" so that
people at least have a starting point to work from (especially if the
instructions work on all platforms, i.e. no symlink discussions).

_______________________________________________

Distutils-SIG maillist  -  [3]Distutils-SIG at python.org

[4]https://mail.python.org/mailman/listinfo/distutils-sig

References

1. mailto:vinay_sajip at yahoo.co.uk
2. mailto:wickman at gmail.com
3. mailto:Distutils-SIG at python.org
4. https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/88ea178f/attachment.html>

From vinay_sajip at yahoo.co.uk  Sat Feb  1 18:40:35 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 17:40:35 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <759472D1-1EFF-49B3-8A11-51AB11846FA0@stufft.io>
Message-ID: <1391276435.78453.YahooMailBasic@web172402.mail.ir2.yahoo.com>

On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:

> I never said anything gets magically better on GitHub, I
> told you the reason why *I* haven?t bothered to try it.

And I'm OK with that.
 
> Distlib is a significantly different case, it?s a library that is going to be
> directly useful for a very small population of people. The vast majority
> of people do not have any need or desire to directly work with distlib.

And your point is? This is distutils-sig, not python-list: it's the only
reasonable place to try and solicit feedback for something like distlib.
I'm not *complaining* that I haven't had much feedback - just stating it.
To state the bleedin' obvious, I don't expect to have any say in how
other people spend their time, and I hope I didn't give any contrary
expectation.

> I can guarantee you that if distlib wasn?t released under an
> OSS license that pip wouldn?t have used it

Sure, but I assume it wasn't used *just* because of the license. It
has to be useful in some way too.

Regards,

Vinay Sajip

From vinay_sajip at yahoo.co.uk  Sat Feb  1 18:53:03 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 17:53:03 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
Message-ID: <1391277183.72995.YahooMailBasic@web172401.mail.ir2.yahoo.com>

On Sat, 1/2/14, Brett Cannon <brett at python.org> wrote:

> PEP 302 tried to unify this with get_data() and set_data()
> on loaders, but prior to Python 3.3 you just didn't have any
> guarantee that __loader__ would even be set, let alone
> have a loader with those methods. Paul can tell me if my
> hunch is off, but I assume the dream was that people would
> do `__loader__.get_data('asset.txt')` instead of
> `os.path.join(os.path.dirname(__file__), 'asset.txt')` to read
> something bundled with your package. But as Brian has
> pointed out, people just have habits at this point of assuming
> unpacked files and working off of __file__.

Right, and one question is whether we need to educate people
away from this way of doing things. In the distlib.resources module
I've tried to work around things like the absence of __loader__
that you mention, and to provide a uniform API for accessing
package resources whether in file system or zip, using the
PEP 302 machinery whenever possible, and filling in some of
the gaps that pkgutil has and which pkg_resources attempts to
fill.

> My pref is [...] and really pushing people to use the loader APIs
> for reading intra-package "files".

+1
?
> Maybe the Packaging Users Guide could have a
> Recommended Deployment Strategies section
[...] 
> (especially if the instructions work on all platforms, i.e. no
> symlink discussions)

+1 again.

Regards,

Vinay Sajip
 
 
 

From p.f.moore at gmail.com  Sat Feb  1 18:59:27 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 1 Feb 2014 17:59:27 +0000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
Message-ID: <CACac1F8PcWiOAFhkDD15DbAW+8=d7VZmj-8ojhCQs4y5y9vjHg@mail.gmail.com>

On 1 February 2014 17:26, Brett Cannon <brett at python.org> wrote:
> PEP 302 tried to unify this with get_data() and set_data() on loaders, but
> prior to Python 3.3 you just didn't have any guarantee that __loader__ would
> even be set, let alone have a loader with those methods. Paul can tell me if
> my hunch is off, but I assume the dream was that people would do
> `__loader__.get_data('asset.txt')` instead of
> `os.path.join(os.path.dirname(__file__), 'asset.txt')` to read something
> bundled with your package. But as Brian has pointed out, people just have
> habits at this point of assuming unpacked files and working off of __file__.

That's right. Unfortunately, (a) people have habits, as you say, (b)
the loader API was too limited - for good reason, we couldn't add any
more without breaking more than we gained at the time, and (c) things
like ssl that need files as Donald points out are always a problem.

Setuptools (and distlib more recently) tried to address this with
unpack-on-demand tricks, but honestly non-filesystem loaders are still
such an uncommon use case that most people don't bother. It's just a
very, very slow process of change.

> I know Daniel has said he wanted some concept of a listdir() on loaders so
> that they could basically act as a virtual file system for packages, but it
> would also require locking down what relative vs. absolute paths meant in
> get_data/set_data (i.e. is it relative to the loader in the package or the
> package itself? My pref is the latter for the case of reused loaders) and
> really pushing people to use the loader APIs for reading intra-package
> "files".

That is probably the most annoying omission from PEP 302. IIRC, we did
want to add it, but couldn't. But I can't recall the reason now. Maybe
it could be remedied - it really is a pain.

Paul

From vinay_sajip at yahoo.co.uk  Sat Feb  1 18:56:43 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 17:56:43 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <1391276074.26266.78087269.317ED389@webmail.messagingengine.com>
Message-ID: <1391277403.43881.YahooMailBasic@web172401.mail.ir2.yahoo.com>

On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:

> There's also the problem when you need to give a file that you
> have packaged as part of your thing to an API that only accepts
> filenames. The standard ssl module is one of these that i've run
> into recently.

Yes, this is one of the pkgutil gaps I referred to in my reply to Brett,
which pkg_resources and distlib.resources both address.

Regards,

Vinay Sajip




From p.f.moore at gmail.com  Sat Feb  1 19:00:05 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 1 Feb 2014 18:00:05 +0000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391277183.72995.YahooMailBasic@web172401.mail.ir2.yahoo.com>
References: <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
 <1391277183.72995.YahooMailBasic@web172401.mail.ir2.yahoo.com>
Message-ID: <CACac1F9ZEVM59mHRQgr7y2cQmhT_SHw-Kiz-Wbhcby+t39Jimw@mail.gmail.com>

On 1 February 2014 17:53, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> My pref is [...] and really pushing people to use the loader APIs
>> for reading intra-package "files".
>
> +1
>
>> Maybe the Packaging Users Guide could have a
>> Recommended Deployment Strategies section
> [...]
>> (especially if the instructions work on all platforms, i.e. no
>> symlink discussions)
>
> +1 again.

+1 to both of these.
Paul

From p.f.moore at gmail.com  Sat Feb  1 19:02:13 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 1 Feb 2014 18:02:13 +0000
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391276435.78453.YahooMailBasic@web172402.mail.ir2.yahoo.com>
References: <759472D1-1EFF-49B3-8A11-51AB11846FA0@stufft.io>
 <1391276435.78453.YahooMailBasic@web172402.mail.ir2.yahoo.com>
Message-ID: <CACac1F-NR1BQboQd8k0tzRpT4fZ06yL9hJdJdwPW7Y=N6WfmBg@mail.gmail.com>

On 1 February 2014 17:40, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> I never said anything gets magically better on GitHub, I
>> told you the reason why *I* haven't bothered to try it.
>
> And I'm OK with that.

FWIW, the difficulty of seeing the source of distlil is one of the
reasons I've not done more with it. Not the only reason, I know, but
certainly one of them for me.

Just another data point for you.
Paul

From donald at stufft.io  Sat Feb  1 19:02:58 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 01 Feb 2014 13:02:58 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391276435.78453.YahooMailBasic@web172402.mail.ir2.yahoo.com>
References: <1391276435.78453.YahooMailBasic@web172402.mail.ir2.yahoo.com>
Message-ID: <1391277778.1370.78092933.1E7101C6@webmail.messagingengine.com>

On Sat, Feb 1, 2014, at 12:40 PM, Vinay Sajip wrote:
> On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:
> 
> > I never said anything gets magically better on GitHub, I
> > told you the reason why *I* haven?t bothered to try it.
> 
> And I'm OK with that.
>  
> > Distlib is a significantly different case, it?s a library that is going to be
> > directly useful for a very small population of people. The vast majority
> > of people do not have any need or desire to directly work with distlib.
> 
> And your point is? This is distutils-sig, not python-list: it's the only
> reasonable place to try and solicit feedback for something like distlib.
> I'm not *complaining* that I haven't had much feedback - just stating it.
> To state the bleedin' obvious, I don't expect to have any say in how
> other people spend their time, and I hope I didn't give any contrary
> expectation.
> 
> > I can guarantee you that if distlib wasn?t released under an
> > OSS license that pip wouldn?t have used it
> 
> Sure, but I assume it wasn't used *just* because of the license. It
> has to be useful in some way too.

Sure, a useless tool/experiment is not made useful by a reasonable
license, but a useful tool/experiment can be made useless by an
unreasonable license.

distil isn't licensed so that I can even legally *use* it as far as I
understand
the law.

> 
> Regards,
> 
> Vinay Sajip

From brett at python.org  Sat Feb  1 19:06:19 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 1 Feb 2014 13:06:19 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391276074.26266.78087269.317ED389@webmail.messagingengine.com>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
 <1391276074.26266.78087269.317ED389@webmail.messagingengine.com>
Message-ID: <CAP1=2W7VjfvX3CJajHcQun+TsFgFBh3vkZcSXgd375_9uf9Skw@mail.gmail.com>

On Sat, Feb 1, 2014 at 12:34 PM, Donald Stufft <donald at stufft.io> wrote:

>  On Sat, Feb 1, 2014, at 12:26 PM, Brett Cannon wrote:
>
>
>
>
> On Sat, Feb 1, 2014 at 3:23 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote:
>
>
> On Fri, 31/1/14, Brian Wickman <wickman at gmail.com> wrote:
>
>  > There are myriad other practical reasons.  Here are some:
>
> Thanks for taking the time to respond with the details - they are good
> data points
>  to think about!
>
>
>  > Lastly, there are social reasons. It's just hard to convince most
> engineers
>  > to use things like pkg_resources or pkgutil to manipulate resources
>  > when for them the status quo is just using __file__.  Bizarrely the
> social
>  > challenges are just as hard as the abovementioned technical challenges.
>
> I agree it's bizarre, but sadly it's not surprising. People get used to
> certain ways
>  of doing things, and a certain kind of collective myopia develops when it
>  comes to looking at different ways of doing things. Having worked with
> fairly
>  diverse systems in my time, ISTM that sections of the Python community
> have
>  this myopia too. For example, the Java hatred and PEP 8 zealotry that you
> see
>  here and there.
>
>
> PEP 302 tried to unify this with get_data() and set_data() on loaders, but
> prior to Python 3.3 you just didn't have any guarantee that __loader__
> would even be set, let alone have a loader with those methods. Paul can
> tell me if my hunch is off, but I assume the dream was that people would do
> `__loader__.get_data('asset.txt')` instead of
> `os.path.join(os.path.dirname(__file__), 'asset.txt')` to read something
> bundled with your package. But as Brian has pointed out, people just have
> habits at this point of assuming unpacked files and working off of __file__.
>
> I know Daniel has said he wanted some concept of a listdir() on loaders so
> that they could basically act as a virtual file system for packages, but it
> would also require locking down what relative vs. absolute paths meant in
> get_data/set_data (i.e. is it relative to the loader in the package or the
> package itself? My pref is the latter for the case of reused loaders) and
> really pushing people to use the loader APIs for reading intra-package
> "files".
>
>
> There's also the problem when you need to give a file that you have
> packaged as part of your thing to an API that only accepts filenames. The
> standard ssl module is one of these that i've run into recently.
>

Yes, that is definitely a design flaw in the ssl module that should get
remedied. Did you file a bug to add a new API (whether new function or new
parameters) to accept a file-like object or string instead?

-Brett


>
>
>
>
>
>  One of the things that's puzzled me, for example, is why people think
> it's reasonable
>  or even necessary to have copies of pip and setuptools in every virtual
> environment
>  - often the same people who will tell you that your code isn't DRY
> enough! It's
>  certainly not a technical requirement, yet one of the reasons why PEP 405
> venvs
>  aren't that popular is that pip and setuptools aren't automatically put
> in there. It's a
>  social issue - it's been decided that rather than exploring a technical
> approach to
>  addressing any issue with installing into venvs, it's better to bundle
> pip and setuptools
>  with Python 3.4, since that will seemingly be easier for people to
> swallow :-)
>
>
> I suspect it's also ignorance and differences in deployment strategies.
> Some people have such small deployments that hitting PyPI every time works,
> for others like Brian it can't be more than a cp w/ an unzip. Maybe the
> Packaging Users Guide could have a Recommended Deployment Strategies
> section or something. I doubt there are that many common needs beyond "pull
> from PyPI every time", "pull from your own wheel repo", "copy everything
> over in a single wheel/zip you unpack" so that people at least have a
> starting point to work from (especially if the instructions work on all
> platforms, i.e. no symlink discussions).
>   *_______________________________________________*
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/d6ef1fa5/attachment-0001.html>

From donald at stufft.io  Sat Feb  1 19:12:53 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 01 Feb 2014 13:12:53 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <CAP1=2W7VjfvX3CJajHcQun+TsFgFBh3vkZcSXgd375_9uf9Skw@mail.gmail.com>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
 <1391276074.26266.78087269.317ED389@webmail.messagingengine.com>
 <CAP1=2W7VjfvX3CJajHcQun+TsFgFBh3vkZcSXgd375_9uf9Skw@mail.gmail.com>
Message-ID: <1391278373.3314.78096009.4F188CF4@webmail.messagingengine.com>





--
Donald Stufft
donald at stufft.io





On Sat, Feb 1, 2014, at 01:06 PM, Brett Cannon wrote:




On Sat, Feb 1, 2014 at 12:34 PM, Donald Stufft <[1]donald at stufft.io>
wrote:

On Sat, Feb 1, 2014, at 12:26 PM, Brett Cannon wrote:




On Sat, Feb 1, 2014 at 3:23 AM, Vinay Sajip
<[2]vinay_sajip at yahoo.co.uk> wrote:

On Fri, 31/1/14, Brian Wickman <[3]wickman at gmail.com> wrote:


> There are myriad other practical reasons.  Here are some:


Thanks for taking the time to respond with the details - they are good
data points

to think about!



> Lastly, there are social reasons. It's just hard to convince most
engineers

> to use things like pkg_resources or pkgutil to manipulate resources

> when for them the status quo is just using __file__.  Bizarrely the
social

> challenges are just as hard as the abovementioned technical
challenges.


I agree it's bizarre, but sadly it's not surprising. People get used to
certain ways

of doing things, and a certain kind of collective myopia develops when
it

comes to looking at different ways of doing things. Having worked with
fairly

diverse systems in my time, ISTM that sections of the Python community
have

this myopia too. For example, the Java hatred and PEP 8 zealotry that
you see

here and there.


PEP 302 tried to unify this with get_data() and set_data() on loaders,
but prior to Python 3.3 you just didn't have any guarantee that
__loader__ would even be set, let alone have a loader with those
methods. Paul can tell me if my hunch is off, but I assume the dream
was that people would do `__loader__.get_data('asset.txt')` instead of
`os.path.join(os.path.dirname(__file__), 'asset.txt')` to read
something bundled with your package. But as Brian has pointed out,
people just have habits at this point of assuming unpacked files and
working off of __file__.

I know Daniel has said he wanted some concept of a listdir() on loaders
so that they could basically act as a virtual file system for packages,
but it would also require locking down what relative vs. absolute paths
meant in get_data/set_data (i.e. is it relative to the loader in the
package or the package itself? My pref is the latter for the case of
reused loaders) and really pushing people to use the loader APIs for
reading intra-package "files".


There's also the problem when you need to give a file that you have
packaged as part of your thing to an API that only accepts filenames.
The standard ssl module is one of these that i've run into recently.


Yes, that is definitely a design flaw in the ssl module that should get
remedied. Did you file a bug to add a new API (whether new function or
new parameters) to accept a file-like object or string instead?

-Brett



I don't recall if I did or not, I know there's one open for allowing in
memory client side certs, no idea for a ca bundle.


It's easy enough to work around of course, but it requires the author
of a tool that uses that to expect to need to work around it, and many
authors simply don't care to support zipped usage.







One of the things that's puzzled me, for example, is why people think
it's reasonable

or even necessary to have copies of pip and setuptools in every virtual
environment

- often the same people who will tell you that your code isn't DRY
enough! It's

certainly not a technical requirement, yet one of the reasons why PEP
405 venvs

aren't that popular is that pip and setuptools aren't automatically put
in there. It's a

social issue - it's been decided that rather than exploring a technical
approach to

addressing any issue with installing into venvs, it's better to bundle
pip and setuptools

with Python 3.4, since that will seemingly be easier for people to
swallow :-)


I suspect it's also ignorance and differences in deployment strategies.
Some people have such small deployments that hitting PyPI every time
works, for others like Brian it can't be more than a cp w/ an unzip.
Maybe the Packaging Users Guide could have a Recommended Deployment
Strategies section or something. I doubt there are that many common
needs beyond "pull from PyPI every time", "pull from your own wheel
repo", "copy everything over in a single wheel/zip you unpack" so that
people at least have a starting point to work from (especially if the
instructions work on all platforms, i.e. no symlink discussions).

_______________________________________________
Distutils-SIG maillist  -  [4]Distutils-SIG at python.org
[5]https://mail.python.org/mailman/listinfo/distutils-sig



_______________________________________________

Distutils-SIG maillist  -  [6]Distutils-SIG at python.org

[7]https://mail.python.org/mailman/listinfo/distutils-sig

References

1. mailto:donald at stufft.io
2. mailto:vinay_sajip at yahoo.co.uk
3. mailto:wickman at gmail.com
4. mailto:Distutils-SIG at python.org
5. https://mail.python.org/mailman/listinfo/distutils-sig
6. mailto:Distutils-SIG at python.org
7. https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/9962892b/attachment.html>

From brett at python.org  Sat Feb  1 19:16:18 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 1 Feb 2014 13:16:18 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <CACac1F9ZEVM59mHRQgr7y2cQmhT_SHw-Kiz-Wbhcby+t39Jimw@mail.gmail.com>
References: <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
 <1391277183.72995.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CACac1F9ZEVM59mHRQgr7y2cQmhT_SHw-Kiz-Wbhcby+t39Jimw@mail.gmail.com>
Message-ID: <CAP1=2W72eTFQ5UeD6hOrddDVKMof2u_kjsiTnsJZ4H1VC0_zZw@mail.gmail.com>

On Sat, Feb 1, 2014 at 1:00 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 1 February 2014 17:53, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> >> My pref is [...] and really pushing people to use the loader APIs
> >> for reading intra-package "files".
> >
> > +1
> >
> >> Maybe the Packaging Users Guide could have a
> >> Recommended Deployment Strategies section
> > [...]
> >> (especially if the instructions work on all platforms, i.e. no
> >> symlink discussions)
> >
> > +1 again.
>
> +1 to both of these.
>

While I'm not in a good position to launch the latter suggestion, the
former one I can help with. =)  I'll start a thread over on import-sig
about how to handle get_data/set_data and anything missing so that
intra-package data reading can be covered by loaders.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/acd6b4b8/attachment.html>

From noah at coderanger.net  Sat Feb  1 19:20:49 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sat, 1 Feb 2014 10:20:49 -0800
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <1391247380.86406.YahooMailBasic@web172406.mail.ir2.yahoo.com>
References: <1391247380.86406.YahooMailBasic@web172406.mail.ir2.yahoo.com>
Message-ID: <B029D0A6-CCCC-409E-82E9-B0ADDCD2A7BA@coderanger.net>


On Feb 1, 2014, at 1:36 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> On Sat, 1/2/14, Noah Kantrowitz <noah at coderanger.net> wrote:
> 
>> In all but a tiny number of cases, you could use a symlink for this.
>> Much less magic :-)
> 
> That's "POSIX is all there is" myopia, right there. While recent versions
> of Windows have symlinks more like POSIX symlinks, XP only has a
> stunted version called "reparse points" or "junction points" which are
> not really fit for purpose. I think you'll find that XP environments are found
> in rather more than "a tiny number of cases", and even though Microsoft
> has end-of-lifed XP in terms of support, I fear it'll be around for a while
> yet.

Junctions on Windows are actually more flexible than POSIX symlinks, and I have in fact used them for Python packages before when doing Django dev on Windows XP.

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/37d5d8fc/attachment.sig>

From brett at python.org  Sat Feb  1 19:28:13 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 1 Feb 2014 13:28:13 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <CAP1=2W72eTFQ5UeD6hOrddDVKMof2u_kjsiTnsJZ4H1VC0_zZw@mail.gmail.com>
References: <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
 <1391277183.72995.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CACac1F9ZEVM59mHRQgr7y2cQmhT_SHw-Kiz-Wbhcby+t39Jimw@mail.gmail.com>
 <CAP1=2W72eTFQ5UeD6hOrddDVKMof2u_kjsiTnsJZ4H1VC0_zZw@mail.gmail.com>
Message-ID: <CAP1=2W4EDfcPNT7aOA5CLZdyr8uikkP8QZ3arhTWgTgG-_dFPA@mail.gmail.com>

On Sat, Feb 1, 2014 at 1:16 PM, Brett Cannon <brett at python.org> wrote:

>
>
>
> On Sat, Feb 1, 2014 at 1:00 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 1 February 2014 17:53, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> >> My pref is [...] and really pushing people to use the loader APIs
>> >> for reading intra-package "files".
>> >
>> > +1
>> >
>> >> Maybe the Packaging Users Guide could have a
>> >> Recommended Deployment Strategies section
>> > [...]
>> >> (especially if the instructions work on all platforms, i.e. no
>> >> symlink discussions)
>> >
>> > +1 again.
>>
>> +1 to both of these.
>>
>
> While I'm not in a good position to launch the latter suggestion, the
> former one I can help with. =)  I'll start a thread over on import-sig
> about how to handle get_data/set_data and anything missing so that
> intra-package data reading can be covered by loaders.
>

But I can file bugs =)

https://github.com/pypa/python-packaging-user-guide/issues/24
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/0a956256/attachment-0001.html>

From brett at python.org  Sat Feb  1 19:45:11 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 1 Feb 2014 13:45:11 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <CAP1=2W72eTFQ5UeD6hOrddDVKMof2u_kjsiTnsJZ4H1VC0_zZw@mail.gmail.com>
References: <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
 <1391277183.72995.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CACac1F9ZEVM59mHRQgr7y2cQmhT_SHw-Kiz-Wbhcby+t39Jimw@mail.gmail.com>
 <CAP1=2W72eTFQ5UeD6hOrddDVKMof2u_kjsiTnsJZ4H1VC0_zZw@mail.gmail.com>
Message-ID: <CAP1=2W5HJY7ZxbpwwVL-Ke7yxa==J-vi2iyseD=X_0=ABFx=fg@mail.gmail.com>

On Sat, Feb 1, 2014 at 1:16 PM, Brett Cannon <brett at python.org> wrote:

>
>
>
> On Sat, Feb 1, 2014 at 1:00 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 1 February 2014 17:53, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> >> My pref is [...] and really pushing people to use the loader APIs
>> >> for reading intra-package "files".
>> >
>> > +1
>> >
>> >> Maybe the Packaging Users Guide could have a
>> >> Recommended Deployment Strategies section
>> > [...]
>> >> (especially if the instructions work on all platforms, i.e. no
>> >> symlink discussions)
>> >
>> > +1 again.
>>
>> +1 to both of these.
>>
>
> While I'm not in a good position to launch the latter suggestion, the
> former one I can help with. =)  I'll start a thread over on import-sig
> about how to handle get_data/set_data and anything missing so that
> intra-package data reading can be covered by loaders.
>

https://mail.python.org/pipermail/import-sig/2014-February/000770.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/5ab5d126/attachment.html>

From vinay_sajip at yahoo.co.uk  Sat Feb  1 19:52:57 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 18:52:57 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <B029D0A6-CCCC-409E-82E9-B0ADDCD2A7BA@coderanger.net>
Message-ID: <1391280777.70291.YahooMailBasic@web172401.mail.ir2.yahoo.com>

On Sat, 1/2/14, Noah Kantrowitz <noah at coderanger.net> wrote:

> Junctions on Windows are actually more flexible than POSIX symlinks,
> and I have in fact used them for Python packages before when doing
> Django dev on Windows XP.

Why do you say they're more flexible? They work as directories/mount
points and AFAIK you can't use them for files. Plus, what Paul said about
the elevation issues (not a factor for XP, I know, but would need to be
taken into account for any solution across multiple Windows versions).

Regards,

Vinay Sajip

From vinay_sajip at yahoo.co.uk  Sat Feb  1 20:00:16 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 19:00:16 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CAP1=2W7VjfvX3CJajHcQun+TsFgFBh3vkZcSXgd375_9uf9Skw@mail.gmail.com>
Message-ID: <1391281216.10104.YahooMailBasic@web172403.mail.ir2.yahoo.com>

On Sat, 1/2/14, Brett Cannon <brett at python.org> wrote:

> Yes, that is definitely a design flaw in the ssl module that should
> get remedied. Did you file a bug to add a new API (whether new
> function or new parameters) to accept a file-like object or string
> instead?
 
While that might improve flexibility in how you use the ssl module, it
sadly won't solve the problem in general. Some third party APIs
will expect file paths, whereas others will require passing OS-level
handles, rather than file-like objects - thus effectively requiring a
file which you can open with os.open().

There are other Python APIs which need a pathname - I know
imp.load_dynamic() will be familiar to you :-)

Regards,

Vinay Sajip

From vinay_sajip at yahoo.co.uk  Sat Feb  1 20:07:35 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 19:07:35 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <1391277778.1370.78092933.1E7101C6@webmail.messagingengine.com>
Message-ID: <1391281655.83742.YahooMailBasic@web172406.mail.ir2.yahoo.com>

On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:

> distil isn't licensed so that I can even legally *use* it as far as I
> understand the law.

Oh really? I hadn't realised. I had thought my announcement in March
inviting people to try it out would suffice, but IANAL. Having a single-file
deployment means having a comprehensive license statement is
tricky, but I will think on it.

Not that it makes a difference to you, of course, but thanks for pointing
it out.

Regards,

Vinay Sajip

From vinay_sajip at yahoo.co.uk  Sat Feb  1 20:23:28 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 19:23:28 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <1391277778.1370.78092933.1E7101C6@webmail.messagingengine.com>
Message-ID: <1391282608.11170.YahooMailBasic@web172401.mail.ir2.yahoo.com>

On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:

> distil isn't licensed so that I can even legally *use* it as
> far as I understand the law.

>From a quick Google search, I got the following from Daniel J. Bernstein's website at http://cr.yp.to/softwarelaw.html - the page entitled "Software user's rights":

    Free software
    -------------------
    What does all this mean for the free software world? Once
    you've legally downloaded a program, you can compile it.
    You can run it. You can modify it. You can distribute your
    patches for other people to use. If you think you need a
    license from the copyright holder, you've been bamboozled
    by Microsoft. As long as you're not distributing the software,
    you have nothing to worry about.

I don't see anything illegal about downloading from the BitBucket
page I linked to in my original post.

So, it doesn't seem all that clear cut ... I suppose it depends on the
jurisdiction, too.

Regards,

Vinay Sajip


From christian at python.org  Sat Feb  1 20:41:42 2014
From: christian at python.org (Christian Heimes)
Date: Sat, 01 Feb 2014 20:41:42 +0100
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <CAP1=2W7VjfvX3CJajHcQun+TsFgFBh3vkZcSXgd375_9uf9Skw@mail.gmail.com>
References: <CAAmsm4hjKRczyE5ExnmtZ1DDPpcZ5DnDQ9H=Go9VLyneMfH5RA@mail.gmail.com>
 <1391243022.51658.YahooMailBasic@web172401.mail.ir2.yahoo.com>
 <CAP1=2W6UmYgA_t-Y42gQVxjCksZywa23OOa5T_-BwagKoqgWbA@mail.gmail.com>
 <1391276074.26266.78087269.317ED389@webmail.messagingengine.com>
 <CAP1=2W7VjfvX3CJajHcQun+TsFgFBh3vkZcSXgd375_9uf9Skw@mail.gmail.com>
Message-ID: <52ED4DF6.5030400@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 01.02.2014 19:06, Brett Cannon wrote:
> Yes, that is definitely a design flaw in the ssl module that should
> get remedied. Did you file a bug to add a new API (whether new
> function or new parameters) to accept a file-like object or string
> instead?

Python 3.4 will allow to load CA certs from a string (PEM) or binary
buffer (DER). I haven't fixed load_cert_chain() yet.

http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.load_verify_locations
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJS7U3wAAoJEMeIxMHUVQ1FETAQAJd6STF1KOG+YMLbuWFnMOen
d+wm8H/abbyVdwg/OL3KGvRJa+qR/FoW3srMJhaVgD7pkiW+2FxRy3N4FWIJbsfM
KxyNKkvnYIIdnZdsv9oz7lVHhdibrUJKvIyRMfaN/wPFcYlB6Y57wh5Z+rkZCRzf
gkw7kSuinceNXtGTsBFYzMKkN/Je6RmNtP96RdtG5zzxuS8V07LjWpcw7VlYoMml
kq2lxcH1B9yR7lDdnrkm5fdUMUzWzRiAi7ZlBYxdIR4PP+UcVaP6AESURy/GF3mn
gbQc0UGz3pI+afS5V/Ajqd/K4w5vqC1rQ48M935XVWvduyT1+Dg5a38lHEThcHOp
gWdY1YOqXwMsxO34/juABjeJkaTJk1bd76CZFJHl2lE5aHJfk49iTIR6Gcq8fOUe
Cb6t0HykYhAF/hBr1ILrN1wkDMq+6Lq2QunKLhn4N4IYP4+WTTxduYuVn9CNJHW3
NCY2RYp0yHvCKw6D86bNr/5ssDIvqfBr4u58Rs93QmkIuYO/2c113Ty2ET3f0+u8
PnfjIGSqDFjctNETPpCygjxMh0IQ+VlKiCV8XlHGA9cWNho+jMLzZOKlKo/notxY
oW7gm2XTxs9O42fO8P9cl1+1I85o3VdDaN5ey16qkptkCSvlSsh1JsBcL+TtP/We
f0Wg5gtO5y+q/3c6JcFx
=FGVA
-----END PGP SIGNATURE-----

From brett at python.org  Sat Feb  1 20:43:48 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 1 Feb 2014 14:43:48 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391281216.10104.YahooMailBasic@web172403.mail.ir2.yahoo.com>
References: <CAP1=2W7VjfvX3CJajHcQun+TsFgFBh3vkZcSXgd375_9uf9Skw@mail.gmail.com>
 <1391281216.10104.YahooMailBasic@web172403.mail.ir2.yahoo.com>
Message-ID: <CAP1=2W7YSDBc2P7hsT788-BBts_L3hUcT2m5r20BvMsNkDxKSQ@mail.gmail.com>

On Sat, Feb 1, 2014 at 2:00 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> On Sat, 1/2/14, Brett Cannon <brett at python.org> wrote:
>
> > Yes, that is definitely a design flaw in the ssl module that should
> > get remedied. Did you file a bug to add a new API (whether new
> > function or new parameters) to accept a file-like object or string
> > instead?
>
> While that might improve flexibility in how you use the ssl module, it
> sadly won't solve the problem in general. Some third party APIs
> will expect file paths, whereas others will require passing OS-level
> handles, rather than file-like objects - thus effectively requiring a
> file which you can open with os.open().
>

No one is talking about solving this problem overnight. But if we don't
want to bother putting the effort in now then the zipimport module might as
well get tossed as this is a constant issue that people are not designing
APIs to support. So if people are going to implicitly not support it
because no one is asking for them to care then we shouldn't try to support
it ourselves and make our lives easier (gods know the zipimport module is a
pain to keep working).


>
> There are other Python APIs which need a pathname - I know
> imp.load_dynamic() will be familiar to you :-)
>

Sure, but that's for technical reasons so it gets a pass (plus I didn't
design that API =).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/095f12e5/attachment.html>

From vinay_sajip at yahoo.co.uk  Sat Feb  1 21:00:53 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 1 Feb 2014 20:00:53 +0000 (GMT)
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
	executable archive format for Python)
In-Reply-To: <CAP1=2W7YSDBc2P7hsT788-BBts_L3hUcT2m5r20BvMsNkDxKSQ@mail.gmail.com>
Message-ID: <1391284853.62532.YahooMailBasic@web172406.mail.ir2.yahoo.com>

On Sat, 1/2/14, Brett Cannon <brett at python.org> wrote:

> No one is talking about solving this problem overnight.
> But if we don't want to bother putting the effort in now
> then the zipimport module might as well get tossed as
> this is a constant issue that people are not designing APIs
> to support.

Whoa - nobody said that in-Python APIs of this nature shouldn't
be improved (as Christian says he's already done with this one).
I was pointing out that there are also third-party APIs that we
don't control which people will want to use, meaning that the
need [for an API] to copy in-zip resources to the file system will
be around for quite some time.

Regards,

Vinay Sajip

From brett at python.org  Mon Feb  3 18:43:37 2014
From: brett at python.org (Brett Cannon)
Date: Mon, 3 Feb 2014 12:43:37 -0500
Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform
 executable archive format for Python)
In-Reply-To: <1391284853.62532.YahooMailBasic@web172406.mail.ir2.yahoo.com>
References: <CAP1=2W7YSDBc2P7hsT788-BBts_L3hUcT2m5r20BvMsNkDxKSQ@mail.gmail.com>
 <1391284853.62532.YahooMailBasic@web172406.mail.ir2.yahoo.com>
Message-ID: <CAP1=2W5t9F3qqs_FSHRkq5zoCLFQj0U0VxY=9_+9b0+u7B0J8w@mail.gmail.com>

On Sat, Feb 1, 2014 at 3:00 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> On Sat, 1/2/14, Brett Cannon <brett at python.org> wrote:
>
> > No one is talking about solving this problem overnight.
> > But if we don't want to bother putting the effort in now
> > then the zipimport module might as well get tossed as
> > this is a constant issue that people are not designing APIs
> > to support.
>
> Whoa - nobody said that in-Python APIs of this nature shouldn't
> be improved (as Christian says he's already done with this one).
> I was pointing out that there are also third-party APIs that we
> don't control which people will want to use, meaning that the
> need [for an API] to copy in-zip resources to the file system will
> be around for quite some time.
>

Sorry for the misunderstanding. Obviously read your text in a way you
didn't mean for it to be read. =)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140203/d1974960/attachment.html>

From carl at oddbird.net  Tue Feb  4 01:44:38 2014
From: carl at oddbird.net (Carl Meyer)
Date: Mon, 03 Feb 2014 17:44:38 -0700
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <1391271957.29649.YahooMailBasic@web172404.mail.ir2.yahoo.com>
References: <1391271957.29649.YahooMailBasic@web172404.mail.ir2.yahoo.com>
Message-ID: <52F037F6.8040000@oddbird.net>

Hi Vinay,

On 02/01/2014 09:25 AM, Vinay Sajip wrote:
> On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:
>> So perhaps it isn't that pip is lacking a useful feature, it's that
>> pip tried it, as you did with distil, and due to pip's much larger
>> user base it got a much more thorough real world experience
>> with it and it turned out to be ultimately more trouble than it was
>> worth? It could also have been a problem with how pip wrote it
>> of course, and perhaps you've discovered a better way of
>> writing it. However at the time it's obvious that the pain it was
>> causing was enough to cause the pip developers to break
>> backwards compatibility to remove that feature.
> 
>> Why exactly was it removed? Well I can't find the reasoning
>> beyond that commit so we'd need to get Carl or perhaps Jannis
>> to come and tell us if they still remember. However it's clearly not
>> that pip didn't just try or think of this, it did and it ultimately removed it.
> 
> Sure, and if they did find some problem which is also present in
> distil (i.e. due to a fundamental drawback of the approach) then I'll put
> my thinking cap back on. It'll be worth having that problem on record
> to avoid any repetition of work in the future.

Sadly I can't offer that specific problem, because AFAIK we never
tracked it down. It was a recurring problem in IRC where people would
report using -E and pip would fail to take its action within the
indicated virtualenv, instead impacting their global environment. After
trying and failing a number of times to isolate and reproduce that
problem, I decided not to continue spending time on that, and that it
would be more robust to rely on the natural isolation provided by just
using pip inside the env, which required no additional code to debug and
maintain. (Pip had already been installed inside every virtualenv for
quite a while by then.)

I regret the way I handled that particular backwards-incompatibility
(not providing a deprecation path), but I think the technical choice was
a good one. Identifying code that is a bug-report-magnet and choosing a
viable approach where that code can be removed entirely is a sound
technical choice, not a matter of "politics" or "religion" (I had no
political or religious motivations in removing pip -E).

As far as I can see, you haven't presented any technical issues with the
current approach, just a feeling that it's inelegant (is this perhaps a
religious feeling? :P). My feeling differs; I find it elegant to rely
only on the virtualenv's built-in isolation, and inelegant to require
extra restart-in-env code.

Carl

From vinay_sajip at yahoo.co.uk  Tue Feb  4 10:23:37 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 09:23:37 +0000 (GMT)
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
	Twitter's multi-platform executable archive format for Python))
In-Reply-To: <52F037F6.8040000@oddbird.net>
Message-ID: <1391505817.7601.YahooMailBasic@web172406.mail.ir2.yahoo.com>

On Tue, 4/2/14, Carl Meyer <carl at oddbird.net> wrote:

> Sadly I can't offer that specific problem, because AFAIK we never
> tracked it down. It was a recurring problem in IRC where people
> would report using -E and pip would fail to take its action within
> the indicated virtualenv, instead impacting their global environment.
> After trying and failing a number of times to isolate and reproduce
> that problem, I decided not to continue spending time on that,
> and that it would be more robust to rely on the natural isolation
> provided by just using pip inside the env, which required no
> additional code to debug and maintain. (Pip had already been
> installed inside every virtualenv for quite a while by then.)
 
> I regret the way I handled that particular backwards-incompatibility
> (not providing a deprecation path), but I think the technical choice
> was a good one. Identifying code that is a bug-report-magnet and
> choosing a viable approach where that code can be removed
> entirely is a sound technical choice, not a matter of "politics" or
> "religion" (I had no political or religious motivations in removing
> pip -E).

I haven't made any comment about pip -E and how / why it was
removed - until Donald's post about it, I didn't even know about it.

To me, the correct solution would have been to isolate the bug and
remove it, but I completely understand the pragmatic approach you
took. But that leaves a "dark corner" where people might be afraid
to experiment in this area with pip, because it's somehow "scary".
This illustrates in part why working *on* pip is painful for me - it's
not exactly a large code-base, but the way it and its tests are
structured makes debugging it needlessly hard. In an environment
where this is all time-constrained volunteer activity, it is not at all
surprising that you took the action you did.

> As far as I can see, you haven't presented any technical issues with
> the current approach, just a feeling that it's inelegant (is this perhaps
> a religious feeling? :P). My feeling differs; I find it elegant to rely
> only on the virtualenv's built-in isolation, and inelegant to require
> extra restart-in-env code.

I didn't say that the current approach didn't work - it's just that IMO
it's inelegant to require something to be installed into every
virtual environment, just to be able to install packages into it. Seemingly
we've gotten so used to it, like "python setup.py install", that it seems
like part of the natural order of the universe :-)

It's not especially elegant to have to restart in a venv, but IMO it's the
lesser of two "evils". It is something that we currently have to do
(virtualenv does it too, with -p) but if a more elegant approach were
found, I'd certainly use it. The other inelegance (install pip in every venv)
is not essential, as I've shown.

I don't know exactly what approach pip took to implement -E - was it
similar to the approach in my snippet? Certainly, if anyone were to
point out any drawbacks with the logic in that snippet, I'd be interested.
But if pip took a different approach to implement -E, it's possible that
there was a problem with the details of that particular approach, and
the problems with pip -E don't provide any evidence to invalidate the
restart approach in general.

I agree that tastes differ, and technical arguments can perfectly
validly be about aesthetics in my view, but I don't regard these sorts
of arguments as "religious". If "religion" is to be mentioned in connection
with pip, then perhaps it's fair to say that pip is the "sacred cow" of Python
packaging ;-)

Regards,

Vinay Sajip

From vinay_sajip at yahoo.co.uk  Tue Feb  4 10:52:50 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 09:52:50 +0000 (UTC)
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
	Twitter's multi-platform executable archive format for Python))
References: <1391271957.29649.YahooMailBasic@web172404.mail.ir2.yahoo.com>
 <52F037F6.8040000@oddbird.net>
Message-ID: <loom.20140204T103612-550@post.gmane.org>

Carl Meyer <carl <at> oddbird.net> writes:

> Sadly I can't offer that specific problem, because AFAIK we never
> tracked it down. It was a recurring problem in IRC where people would
> report using -E and pip would fail to take its action within the

After posting my first response to your post, I remembered that Donald
had provided a link to the commit that removed the -E. From a quick look at
that commit, ISTM that distil's approach to restarting is somewhat easier
to reason about and debug than the approach taken by pip to implement -E:
it takes place before any of the rest of the logic in the restarted program
is run, so a question like "why didn't restart_in_venv get called, when it
should have been?" doesn't arise. There are no rabbit-holes to go down. The
approach in that snippet could apply to any script - it's not especially
distil-specific.

Regards,

Vinay Sajip


From ncoghlan at gmail.com  Tue Feb  4 12:04:05 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 4 Feb 2014 21:04:05 +1000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <1391505817.7601.YahooMailBasic@web172406.mail.ir2.yahoo.com>
References: <52F037F6.8040000@oddbird.net>
 <1391505817.7601.YahooMailBasic@web172406.mail.ir2.yahoo.com>
Message-ID: <CADiSq7cbCWE2Pp_k51R_ZzYatim=1fgAeeLxn8NB9i=k-CfygA@mail.gmail.com>

On 4 February 2014 19:23, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>
> It's not especially elegant to have to restart in a venv, but IMO it's the
> lesser of two "evils". It is something that we currently have to do
> (virtualenv does it too, with -p) but if a more elegant approach were
> found, I'd certainly use it. The other inelegance (install pip in every venv)
> is not essential, as I've shown.

I've pointed out the technical problem with trying to rely solely on a
global install on Linux: it makes it hard to use a newer version of
pip than the OS provides. Installing pip into the virtual environments
avoids that problem without conflicting with the OS package manager
when it comes to the system Python installation.

That decoupling from the distro version is a worthwhile technical
benefit of the current approach. While it does mean that multiple
virtual environments may need to be updated when a new version of pip
is released, it *also* makes testing with different versions
straightforward.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Tue Feb  4 13:01:26 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 12:01:26 +0000 (GMT)
Subject: [Distutils] restart-in-venv
In-Reply-To: <CADiSq7cbCWE2Pp_k51R_ZzYatim=1fgAeeLxn8NB9i=k-CfygA@mail.gmail.com>
Message-ID: <1391515286.41253.YahooMailBasic@web172402.mail.ir2.yahoo.com>

On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I've pointed out the technical problem with trying to rely solely
> on a global install on Linux: it makes it hard to use a newer version
> of pip than the OS provides. Installing pip into the virtual
> environments avoids that problem without conflicting with the
> OS package manager when it comes to the system Python
> installation.

> That decoupling from the distro version is a worthwhile technical
> benefit of the current approach. While it does mean that multiple
> virtual environments may need to be updated when a new version of pip
> is released, it *also* makes testing with different versions
> straightforward.
 
Well, it's not a bad thing IMO that distil leaves no trace of itself in a
venv, so no updating of any venv is required, were a new release of
distil to come out.

In terms of distro version isolation, I'm still not sure I get the practical
problem you're talking about, so let's get into specifics. In a parallel
universe, let's assume there's a tool that can install packages into
venvs, without needing to be installed into them first. (It seems a
desirable feature, which prompted the development of pip -E in the
first place). Let's say that this tool is shipped as part of a distro and
generally kept up to date by the distro's package manager.

Let's say that a new version of this tool is available upstream, but
not yet integrated into the distro, and that someone wants to use it
instead of the distro-supplied version. Where's the problem exactly?
It could just be a single file to download and run, and you could have
any number of different versions of it co-existing without any particular
problem. It needn't be any worse than any other type of tool where a
more-recent-than-distro version is desired.

I'm probably being dense, so perhaps you could outline any specific
problem scenarios along the lines of "a user wants to do X, but can't,
because of Y" where X is some aspect of installation / upgrading /
uninstallation of packages in a venv and Y is something which
intrinsically arises from the installation tool not being installed in the
venv. That would certainly help me understand where you're coming
from.

Regards,

Vinay Sajip

From saschpe at gmx.de  Tue Feb  4 12:43:40 2014
From: saschpe at gmx.de (Sascha Peilicke)
Date: Tue, 04 Feb 2014 12:43:40 +0100
Subject: [Distutils] PEP-459 feedback from openSUSE packaging
Message-ID: <2249682.syZXsYgalI@bort.suse.de>

Hi guys,

a colleague of mine hinted me to Nick's plea for feedback on various PEPs from 
other distros perspectives. 

I will provide some general remarks first and latter comment at least PEP-459. 
So for openSUSE (and SLES), we automate Python package generation as much as 
possible. For that we parse the metadata as found on PyPI and maul the source 
distribution (yeah, the tarball) for every usable bit. The latter is necessary 
to properly install files such as README, AUTHORS, LICENSE(.txt,...). We also 
use it to grep for '*test*' files to decide if it'S worth generating a RPM 
%check section.

Some modules do add 'package_data'. That's usually helpful since it goes 
straight into %python_sitelib. For the files mentioned above, we have 
'data_files'. Not only is it seldomly used, it's almost universally done 
wrong. Simply because every distro differs on where to put these files to.

A different cause of woe are install_requires / requires vs. setup_requires 
vs. test_require. Some people use 'requires', which is mostly documentation 
and lots of people put _everything_ in install_requires. From a distribution 
viewpoint, you have different sets of requirements:

 - build-time
   + optionally doc-requires
   + optionally test-requires
 - run-time

So setup_requires / test_require can be used to generate semi-accurate 
BuildRequires: $bla RPM spec tags. But as said, few people use them and less 
do it correct. Maybe because 'setup_requires' doesn't specificy build-time 
reqs but 'setuptools-invocation-time' reqs (which is sth. different). Also, we 
simply use 'install_requires' as both 'Requires:' (runtime) and 
'BuildRequires:' (build-time). But that's a cludge. For example, projects 
include 'Sphinx' in install_requires. What they meant is "if you want to build 
docs, use Sphinx". What they specified is "you always need it". Thankfully, 
the advent of pep allows us to check requirements.txt and test-
requirements.txt. The latter are usually build-time (for the RPM %check 
section). I guess I have to dig into the other PEPs first to see if this 
really changed before being able to comment on that any further.

In general, the other metadata is good enough (except 'license', see below), 
'name', 'version', 'upstream_url' and 'description' are used for their 
respective RPM spec counterparts. 'long_description' is used for '%description 
$PKG_NAME'. The tarball download URL is used as 'Source0:'. All other metadata 
tags are ignored because we don't need them to build a RPM.


The 'license' metadata tag is causing the most issues for us. A perceived 50% 
just put "GPL" in there. Which GPL version? GPL-only or actually LGPL?. We 
have a legal crawler that tries to match the version from the source code but 
often it's becomes a manual task or needs a check with upstream. This tag is 
probably the least interesting for an upstream developer but the most 
important one for any distro that has a corporate legal entity somewhere in 
behind (I should say, sue-able entitity :-). So with regards to PEP-459 
specifically, I have specific recommendations for the license tag. Instead of 

        "This field SHOULD contain fewer than 512 characters and MUST contain 
fewer 
        than 2048.

        This field SHOULD NOT contain any line breaks."

I would propose:

        "This filed SHOULD contain a standardized license identifier as
         published by spdx.org."


SPDX-sytle license identifiers are short (less than 20 chars) and can be 
parsed automatically. They are meant to be unambiguous and cross-distro.
SPDX.org license tags are used extensively inside openSUSE and SLES and (to my 
knowledge) for Fedora and Debian too. That would be the single most 
interesting change I'd be interest in.

I hope this was helpful, feel free to bug me if you need further detail.
-- 
Mit freundlichen Gr??en,
Sascha Peilicke

From ncoghlan at gmail.com  Tue Feb  4 14:37:27 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 4 Feb 2014 23:37:27 +1000
Subject: [Distutils] restart-in-venv
In-Reply-To: <1391515286.41253.YahooMailBasic@web172402.mail.ir2.yahoo.com>
References: <CADiSq7cbCWE2Pp_k51R_ZzYatim=1fgAeeLxn8NB9i=k-CfygA@mail.gmail.com>
 <1391515286.41253.YahooMailBasic@web172402.mail.ir2.yahoo.com>
Message-ID: <CADiSq7dKv5ABgiP6CUqhrp_xVxZNp_mEHDWi+OWwOhq8QfRm=Q@mail.gmail.com>

Fedora 19 still ships pip 1.3.1. Fedora 20 ships pip 1.4.1. Fedora 21
will probably ship 1.5.2 (although that hasn't actually happened yet,
so Rawhide still has 1.4.1).

The relative alignment of the release cycles is such that the pip
version system Python will always be about 6 months behind upstream.

At the moment, this doesn't really matter, because people can just
upgrade pip in their virtual environment(s) instead. And the way of
handling that is completely consistent cross-distro, cross-platform
and cross-version.

It's a higher level of isolation than that offered by *not* installing
pip into the virtual environments - the two approaches are *not*
equivalent, and one is *not* clearly superior to the other, but rather
there are pros and cons to both.

Cheers,
Nick.

From marius at pov.lt  Tue Feb  4 14:42:45 2014
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 4 Feb 2014 15:42:45 +0200
Subject: [Distutils] PEP-459 feedback from openSUSE packaging
In-Reply-To: <2249682.syZXsYgalI@bort.suse.de>
References: <2249682.syZXsYgalI@bort.suse.de>
Message-ID: <20140204134245.GA10361@fridge.pov.lt>

On Tue, Feb 04, 2014 at 12:43:40PM +0100, Sascha Peilicke wrote:
> The 'license' metadata tag is causing the most issues for us. A perceived 50% 
> just put "GPL" in there. Which GPL version? GPL-only or actually LGPL?. 

FWIW you can sometimes get more detailed information from the classifiers:
https://pypi.python.org/pypi?%3Aaction=list_classifiers

E.g.

  setup(...
      licence='GPL',
      classifiers=[
          ...,
          'License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)',
          ...
      ])

Marius Gedminas
-- 
Unix gives you enough rope to shoot yourself in the foot.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140204/d2d3a9f9/attachment.sig>

From p.f.moore at gmail.com  Tue Feb  4 14:58:33 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 4 Feb 2014 13:58:33 +0000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <1391505817.7601.YahooMailBasic@web172406.mail.ir2.yahoo.com>
References: <52F037F6.8040000@oddbird.net>
 <1391505817.7601.YahooMailBasic@web172406.mail.ir2.yahoo.com>
Message-ID: <CACac1F8guknVxNCEcEYHYCh8MLaOzcX1Qd1viJY5AbhVphQUJg@mail.gmail.com>

On 4 February 2014 09:23, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> To me, the correct solution would have been to isolate the bug and
> remove it, but I completely understand the pragmatic approach you
> took. But that leaves a "dark corner" where people might be afraid
> to experiment in this area with pip, because it's somehow "scary"

The big problem with this type of thing is that pip tends to be used
on lots of systems with sometimes *extremely* obscure and odd setups -
what some of the Linux distros do to a standard Python install amazes
me (I'm sure they have good reasons, of course...). So it's not so
much a case of "scary dark corners" as lots of unanticipated and
extremely difficult to reproduce use cases that we have to support but
where the only knowledge of what workarounds are needed is
encapsulated in the code.

It's a classic maintain vs reimplement issue, unfortunately...

Paul

From ncoghlan at gmail.com  Tue Feb  4 15:02:08 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 00:02:08 +1000
Subject: [Distutils] PEP-459 feedback from openSUSE packaging
In-Reply-To: <2249682.syZXsYgalI@bort.suse.de>
References: <2249682.syZXsYgalI@bort.suse.de>
Message-ID: <CADiSq7feKC28rVFeM6PY=Gp+wzbeGNJF=sX=2s3oC2eKttcX3w@mail.gmail.com>

On 4 February 2014 21:43, Sascha Peilicke <saschpe at gmx.de> wrote:
> Hi guys,
>
> a colleague of mine hinted me to Nick's plea for feedback on various PEPs from
> other distros perspectives.

Thanks! Happy to see my linux.conf.au presentation bearing fruit :)

> I will provide some general remarks first and latter comment at least PEP-459.
> So for openSUSE (and SLES), we automate Python package generation as much as
> possible. For that we parse the metadata as found on PyPI and maul the source
> distribution (yeah, the tarball) for every usable bit. The latter is necessary
> to properly install files such as README, AUTHORS, LICENSE(.txt,...). We also
> use it to grep for '*test*' files to decide if it'S worth generating a RPM
> %check section.
>
> Some modules do add 'package_data'. That's usually helpful since it goes
> straight into %python_sitelib. For the files mentioned above, we have
> 'data_files'. Not only is it seldomly used, it's almost universally done
> wrong. Simply because every distro differs on where to put these files to.

PEP 426/459 will hopefully help with some of these, but I suspect only
up to a point - pypi.python.org is always going to maintain a lower
barrier to entry than the Linux distros that try to hammer their
packages into a more integrated whole.

> A different cause of woe are install_requires / requires vs. setup_requires
> vs. test_require. Some people use 'requires', which is mostly documentation
> and lots of people put _everything_ in install_requires. From a distribution
> viewpoint, you have different sets of requirements:
>
>  - build-time
>    + optionally doc-requires
>    + optionally test-requires
>  - run-time
>
> So setup_requires / test_require can be used to generate semi-accurate
> BuildRequires: $bla RPM spec tags. But as said, few people use them and less
> do it correct. Maybe because 'setup_requires' doesn't specificy build-time
> reqs but 'setuptools-invocation-time' reqs (which is sth. different). Also, we
> simply use 'install_requires' as both 'Requires:' (runtime) and
> 'BuildRequires:' (build-time). But that's a cludge. For example, projects
> include 'Sphinx' in install_requires. What they meant is "if you want to build
> docs, use Sphinx". What they specified is "you always need it". Thankfully,
> the advent of pep allows us to check requirements.txt and test-
> requirements.txt. The latter are usually build-time (for the RPM %check
> section). I guess I have to dig into the other PEPs first to see if this
> really changed before being able to comment on that any further.

Yes, this aspect of the current system is a bit of a mess. One of the
things we're aiming for with the wheel format is to clarify that even
in the existing metadata, "install_requires" should refer to things
needed to create a wheel, while "requires" should refer to things
needed to actuall run the software after unpacking the wheel.

The proposed PEP 426 dependency tags are probably best summarised in
this section:
http://www.python.org/dev/peps/pep-0426/#mapping-dependencies-to-development-and-distribution-activities

Where a documentation dependency like Sphinx ends up would depend on
the project. If the project has no bundled documentation (e.g. online
docs only), then Sphinx would just be a "dev_requires" dependency.
However, if it *did* ship with generated documentation that needed to
be installed on the target system (e.g. man pages), then Sphinx would
instead be a "build_requires" dependency.

I expect many upstream projects will still need help from the distros
to get this right, but the ultimate aim for metadata 2.0 is to make it
easier for distro repackagers to submit such patches upstream and *get
them accepted as non-controversial changes*.

> In general, the other metadata is good enough (except 'license', see below),
> 'name', 'version', 'upstream_url' and 'description' are used for their
> respective RPM spec counterparts. 'long_description' is used for '%description
> $PKG_NAME'. The tarball download URL is used as 'Source0:'. All other metadata
> tags are ignored because we don't need them to build a RPM.

>From a name and version point of view, I'm not at all familiar with
the openSUSE policies, so I'd be interested in knowing if the
restrictions in http://www.python.org/dev/peps/pep-0426/#name would
also meet openSUSE naming guidelines.

The version numbering PEP (http://www.python.org/dev/peps/pep-0440/)
is annoyingly complicated, but it unfortunately needs to be in order
to tolerate the diversity of versioning schems already in use on PyPI
:(

> The 'license' metadata tag is causing the most issues for us. A perceived 50%
> just put "GPL" in there. Which GPL version? GPL-only or actually LGPL?. We
> have a legal crawler that tries to match the version from the source code but
> often it's becomes a manual task or needs a check with upstream. This tag is
> probably the least interesting for an upstream developer but the most
> important one for any distro that has a corporate legal entity somewhere in
> behind (I should say, sue-able entitity :-). So with regards to PEP-459
> specifically, I have specific recommendations for the license tag. Instead of
>
>         "This field SHOULD contain fewer than 512 characters and MUST contain
> fewer
>         than 2048.
>
>         This field SHOULD NOT contain any line breaks."
>
> I would propose:
>
>         "This filed SHOULD contain a standardized license identifier as
>          published by spdx.org."
>
>
> SPDX-sytle license identifiers are short (less than 20 chars) and can be
> parsed automatically. They are meant to be unambiguous and cross-distro.
> SPDX.org license tags are used extensively inside openSUSE and SLES and (to my
> knowledge) for Fedora and Debian too. That would be the single most
> interesting change I'd be interest in.

Oh, I hadn't seen SPDX before - very interesting. I'm wondering if it
may be a better fit for the PyPI Trove classifiers though - then it
wouldn't even need to wait for metadata 2.0, we could just add them to
the list of supported classifiers
(https://pypi.python.org/pypi?%3Aaction=list_classifiers) and projects
could start listing them in their current metadata.

Something like:

    License :: SPDX :: <tag>

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Tue Feb  4 15:00:13 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 14:00:13 +0000 (GMT)
Subject: [Distutils] restart-in-venv
In-Reply-To: <CADiSq7dKv5ABgiP6CUqhrp_xVxZNp_mEHDWi+OWwOhq8QfRm=Q@mail.gmail.com>
Message-ID: <1391522413.20185.YahooMailBasic@web172401.mail.ir2.yahoo.com>

On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> It's a higher level of isolation than that offered by *not* installing
> pip into the virtual environments - the two approaches are
> *not* equivalent, and one is *not* clearly superior to the other,
> but rather there are pros and cons to both.

Obviously the two approaches aren't equivalent,  or there wouldn't
be anything to discuss. What you haven't yet done is answered
my question about what the cons are of the not-in-venv approach
in the practical "you can't do X, because of Y", sense. What
*practical* problems are caused by a lower level of isolation
exactly, which make the higher level of isolation necessary?

ISTM the pip developers tried to implement -E because it
was (independently of me) seen as a desirable feature, and
Paul said as much, too. If it didn't work for pip, that could well
be because of the specific implementation approach used,
as I've said in my response to Carl.

Rather than talk in abstractions like "higher levels of isolation",
which have an implied sense of goodness, it would be helpful
to spell out specific scenarios (in the X/Y sense I mentioned
before) that illustrate that goodness versus the badness of
other approaches.

Regards,

Vinay Sajip


From dholth at gmail.com  Tue Feb  4 15:05:38 2014
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 4 Feb 2014 09:05:38 -0500
Subject: [Distutils] PEP-459 feedback from openSUSE packaging
In-Reply-To: <CADiSq7feKC28rVFeM6PY=Gp+wzbeGNJF=sX=2s3oC2eKttcX3w@mail.gmail.com>
References: <2249682.syZXsYgalI@bort.suse.de>
 <CADiSq7feKC28rVFeM6PY=Gp+wzbeGNJF=sX=2s3oC2eKttcX3w@mail.gmail.com>
Message-ID: <CAG8k2+4Jj4Ch1BRUWzybyUej7ROkd8=R30dd1igXwp7d1kNkiQ@mail.gmail.com>

On Tue, Feb 4, 2014 at 9:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 4 February 2014 21:43, Sascha Peilicke <saschpe at gmx.de> wrote:
>> Hi guys,
>>
>> a colleague of mine hinted me to Nick's plea for feedback on various PEPs from
>> other distros perspectives.
>
> Thanks! Happy to see my linux.conf.au presentation bearing fruit :)
>
>> I will provide some general remarks first and latter comment at least PEP-459.
>> So for openSUSE (and SLES), we automate Python package generation as much as
>> possible. For that we parse the metadata as found on PyPI and maul the source
>> distribution (yeah, the tarball) for every usable bit. The latter is necessary
>> to properly install files such as README, AUTHORS, LICENSE(.txt,...). We also
>> use it to grep for '*test*' files to decide if it'S worth generating a RPM
>> %check section.
>>
>> Some modules do add 'package_data'. That's usually helpful since it goes
>> straight into %python_sitelib. For the files mentioned above, we have
>> 'data_files'. Not only is it seldomly used, it's almost universally done
>> wrong. Simply because every distro differs on where to put these files to.
>
> PEP 426/459 will hopefully help with some of these, but I suspect only
> up to a point - pypi.python.org is always going to maintain a lower
> barrier to entry than the Linux distros that try to hammer their
> packages into a more integrated whole.
>
>> A different cause of woe are install_requires / requires vs. setup_requires
>> vs. test_require. Some people use 'requires', which is mostly documentation
>> and lots of people put _everything_ in install_requires. From a distribution
>> viewpoint, you have different sets of requirements:
>>
>>  - build-time
>>    + optionally doc-requires
>>    + optionally test-requires
>>  - run-time
>>
>> So setup_requires / test_require can be used to generate semi-accurate
>> BuildRequires: $bla RPM spec tags. But as said, few people use them and less
>> do it correct. Maybe because 'setup_requires' doesn't specificy build-time
>> reqs but 'setuptools-invocation-time' reqs (which is sth. different). Also, we
>> simply use 'install_requires' as both 'Requires:' (runtime) and
>> 'BuildRequires:' (build-time). But that's a cludge. For example, projects
>> include 'Sphinx' in install_requires. What they meant is "if you want to build
>> docs, use Sphinx". What they specified is "you always need it". Thankfully,
>> the advent of pep allows us to check requirements.txt and test-
>> requirements.txt. The latter are usually build-time (for the RPM %check
>> section). I guess I have to dig into the other PEPs first to see if this
>> really changed before being able to comment on that any further.
>
> Yes, this aspect of the current system is a bit of a mess. One of the
> things we're aiming for with the wheel format is to clarify that even
> in the existing metadata, "install_requires" should refer to things
> needed to create a wheel, while "requires" should refer to things
> needed to actuall run the software after unpacking the wheel.
>
> The proposed PEP 426 dependency tags are probably best summarised in
> this section:
> http://www.python.org/dev/peps/pep-0426/#mapping-dependencies-to-development-and-distribution-activities
>
> Where a documentation dependency like Sphinx ends up would depend on
> the project. If the project has no bundled documentation (e.g. online
> docs only), then Sphinx would just be a "dev_requires" dependency.
> However, if it *did* ship with generated documentation that needed to
> be installed on the target system (e.g. man pages), then Sphinx would
> instead be a "build_requires" dependency.
>
> I expect many upstream projects will still need help from the distros
> to get this right, but the ultimate aim for metadata 2.0 is to make it
> easier for distro repackagers to submit such patches upstream and *get
> them accepted as non-controversial changes*.
>
>> In general, the other metadata is good enough (except 'license', see below),
>> 'name', 'version', 'upstream_url' and 'description' are used for their
>> respective RPM spec counterparts. 'long_description' is used for '%description
>> $PKG_NAME'. The tarball download URL is used as 'Source0:'. All other metadata
>> tags are ignored because we don't need them to build a RPM.
>
> From a name and version point of view, I'm not at all familiar with
> the openSUSE policies, so I'd be interested in knowing if the
> restrictions in http://www.python.org/dev/peps/pep-0426/#name would
> also meet openSUSE naming guidelines.
>
> The version numbering PEP (http://www.python.org/dev/peps/pep-0440/)
> is annoyingly complicated, but it unfortunately needs to be in order
> to tolerate the diversity of versioning schems already in use on PyPI
> :(
>
>> The 'license' metadata tag is causing the most issues for us. A perceived 50%
>> just put "GPL" in there. Which GPL version? GPL-only or actually LGPL?. We
>> have a legal crawler that tries to match the version from the source code but
>> often it's becomes a manual task or needs a check with upstream. This tag is
>> probably the least interesting for an upstream developer but the most
>> important one for any distro that has a corporate legal entity somewhere in
>> behind (I should say, sue-able entitity :-). So with regards to PEP-459
>> specifically, I have specific recommendations for the license tag. Instead of
>>
>>         "This field SHOULD contain fewer than 512 characters and MUST contain
>> fewer
>>         than 2048.
>>
>>         This field SHOULD NOT contain any line breaks."
>>
>> I would propose:
>>
>>         "This filed SHOULD contain a standardized license identifier as
>>          published by spdx.org."
>>
>>
>> SPDX-sytle license identifiers are short (less than 20 chars) and can be
>> parsed automatically. They are meant to be unambiguous and cross-distro.
>> SPDX.org license tags are used extensively inside openSUSE and SLES and (to my
>> knowledge) for Fedora and Debian too. That would be the single most
>> interesting change I'd be interest in.
>
> Oh, I hadn't seen SPDX before - very interesting. I'm wondering if it
> may be a better fit for the PyPI Trove classifiers though - then it
> wouldn't even need to wait for metadata 2.0, we could just add them to
> the list of supported classifiers
> (https://pypi.python.org/pypi?%3Aaction=list_classifiers) and projects
> could start listing them in their current metadata.
>
> Something like:
>
>     License :: SPDX :: <tag>
>
> Cheers,
> Nick.

Those SPDX are great. They could certainly go into either license or
into a trove classifier, the difference being the trove classifiers
are checked against a static list.

From ncoghlan at gmail.com  Tue Feb  4 15:18:41 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 00:18:41 +1000
Subject: [Distutils] PEP-459 feedback from openSUSE packaging
In-Reply-To: <CAG8k2+4Jj4Ch1BRUWzybyUej7ROkd8=R30dd1igXwp7d1kNkiQ@mail.gmail.com>
References: <2249682.syZXsYgalI@bort.suse.de>
 <CADiSq7feKC28rVFeM6PY=Gp+wzbeGNJF=sX=2s3oC2eKttcX3w@mail.gmail.com>
 <CAG8k2+4Jj4Ch1BRUWzybyUej7ROkd8=R30dd1igXwp7d1kNkiQ@mail.gmail.com>
Message-ID: <CADiSq7f4278Brirjrqmm2MpvOwoAEQJtirWDsKa_ZDFLb8_1hg@mail.gmail.com>

On 5 February 2014 00:05, Daniel Holth <dholth at gmail.com> wrote:
> On Tue, Feb 4, 2014 at 9:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Oh, I hadn't seen SPDX before - very interesting. I'm wondering if it
>> may be a better fit for the PyPI Trove classifiers though - then it
>> wouldn't even need to wait for metadata 2.0, we could just add them to
>> the list of supported classifiers
>> (https://pypi.python.org/pypi?%3Aaction=list_classifiers) and projects
>> could start listing them in their current metadata.
>>
>> Something like:
>>
>>     License :: SPDX :: <tag>
>>
>> Cheers,
>> Nick.
>
> Those SPDX are great. They could certainly go into either license or
> into a trove classifier, the difference being the trove classifiers
> are checked against a static list.

The main advantage I can see to going the classifier route is that it
means not having to wait for metadata 2.0 to promote them - folks can
start using them as soon as they're registered on PyPI.

It also avoids compatibility issues when attempting to convert the
many current projects with unclear license terms to metadata 2.0,
while still making it easy for distro repackagers to offer upstream
patches or bug reports to request license clarifications.

However, I do like the idea of having metadata 2.0 encourage the use
of OSI approved SPDX tags in the license field. I just don't think we
can upgrade that from a SHOULD to a MUST without breaking too many
packages :(

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Feb  4 15:31:03 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 00:31:03 +1000
Subject: [Distutils] restart-in-venv
In-Reply-To: <1391522413.20185.YahooMailBasic@web172401.mail.ir2.yahoo.com>
References: <CADiSq7dKv5ABgiP6CUqhrp_xVxZNp_mEHDWi+OWwOhq8QfRm=Q@mail.gmail.com>
 <1391522413.20185.YahooMailBasic@web172401.mail.ir2.yahoo.com>
Message-ID: <CADiSq7cZms=ogOyG=hC9MGEX7_k9bTqej=KGgbUi7Uo4+owcpQ@mail.gmail.com>

On 5 February 2014 00:00, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> It's a higher level of isolation than that offered by *not* installing
>> pip into the virtual environments - the two approaches are
>> *not* equivalent, and one is *not* clearly superior to the other,
>> but rather there are pros and cons to both.
>
> Obviously the two approaches aren't equivalent,  or there wouldn't
> be anything to discuss. What you haven't yet done is answered
> my question about what the cons are of the not-in-venv approach
> in the practical "you can't do X, because of Y", sense. What
> *practical* problems are caused by a lower level of isolation
> exactly, which make the higher level of isolation necessary?
>
> ISTM the pip developers tried to implement -E because it
> was (independently of me) seen as a desirable feature, and
> Paul said as much, too. If it didn't work for pip, that could well
> be because of the specific implementation approach used,
> as I've said in my response to Carl.
>
> Rather than talk in abstractions like "higher levels of isolation",
> which have an implied sense of goodness, it would be helpful
> to spell out specific scenarios (in the X/Y sense I mentioned
> before) that illustrate that goodness versus the badness of
> other approaches.

I'm not sure how I can explain it more clearly, but I'll try.

Scenario A: one tool to rule them all. Not versioned per virtualenv.
If you want multiple versions on a system, you must manage that
independently of both virtualenv and the system package manager. You
may end up manipulating a virtualenv with version X.Y one day and
version X.Z the next day, and then go back to version X.Y.

Scenario B: the installation tool is installed into the virtual
environment when it is created. No external event will impact the
behaviour of installation commands in that virtual environment - you
must upgrade the tool within the environment. There are only two
packaging systems in use: the system package manager and virtualenv.
There is no third mechanism needed to version the package management
tool, because that is handled using virtualenv. If one client is using
pip 1.3, and the other pip 1.5 (with it's much stricter default
security settings), then having separate virtual environments for
separate projects will deal with that automatically.

I'm not saying what you want to do *can't* be done (it obviously can).
I'm saying it adds additional complexity, because you're adding a
second isolation mechanism into the mix, with a completelely
independent set of ways of interacting with it.

For an example of that consider that "pip install" and "python -m pip
install" are essentially equivalent commands. The status quo preserves
that equivalence even inside a virtual environment, because there is
no separate mechanism to be considered. If the system Python
site-packages is disabled in an active virtual environment, you don't
need to carefully ensure that doesn't affect your install tool,
because it's right there in the virtual environment with you.

Is it possible that a tool could be written that *did* adequately
address all of these cases, with the test suite to prove it? Yes, I'm
sure that's possible. What I'm saying is that I *don't really see the
point*, since it doesn't achieve anything that couldn't be achieved
with a utility script that simply iterates over all defined virtual
environments and execute "pip install --upgrade" in each of them (with
the wheel cache, every venv after the first should be quick).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Tue Feb  4 15:33:55 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 14:33:55 +0000 (GMT)
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
	Twitter's multi-platform executable archive format for Python))
In-Reply-To: <CACac1F8guknVxNCEcEYHYCh8MLaOzcX1Qd1viJY5AbhVphQUJg@mail.gmail.com>
Message-ID: <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>

On Tue, 4/2/14, Paul Moore <p.f.moore at gmail.com> wrote:

> The big problem with this type of thing is that pip tends to be used
> on lots of systems with sometimes *extremely* obscure and odd
> setups - what some of the Linux distros do to a standard Python
> install amazes me (I'm sure they have good reasons, of course...).
> So it's not so much a case of "scary dark corners" as lots of
> unanticipated and extremely difficult to reproduce use cases that
> we have to support but where the only knowledge of what
> workarounds are needed is encapsulated in the code.

You've just restated "scary dark corners" in a different way which
sounds like there's more to it technically. But there' s still no hard
data on the table!

Surely, with all the convolutions that these distros go through, can one
still assume that pip-the-program-that-they-all-run assumes control via
some notional main program, which is passed some arguments? If
we can't assume this, please tell me why not. If we can, then it would
be like this:

def main():
    # Do stuff with sys.argv

Now, distil has a similar entry point (as innumerable other scripts do)
and all I did to restart in venv was to look for a -e argument and process
it before the main event:

def main():
    if '-e' found in sys.argv:
        validate the passed parameter, prepare a new command line
        using the interpreter from the venv and the sys.argv passed in,
        but minus the -e bit, since we've dealt with that here
        call subprocess with that command line (the restart) 
        return - our work here is done
    # Do stuff with sys.argv

It doesn't get much simpler. Of course there could be bugs in the
code called in the if suite, but it's not doing a complicated job so it
should be easy to identify and eradicate them. Now, are you saying
that the bit of code represented by that if suite, which only has to
process sys.argv, look for an identifiable venv in the file system and
check that it seems to have a runnable Python interpreter, can be
bamboozled by something distros do to their Python installations?
I'm having trouble buying that as an argument, unless there is more
hard data on the table. What "unanticipated and extremely difficult
to reproduce use cases" could interfere with the operations in that
if suite? It's hard to argue against voodoo, but anyone on this list
could put me right with a particular circumstance that I haven't
thought of, but that they have run into, which undermines my
argument. It's not uncommon to see the "well ... we're not quite
sure what's happening in there ... better play it safe", but that
seems out of place in an engineering context.

Regards,

Vinay Sajip

From ncoghlan at gmail.com  Tue Feb  4 16:02:52 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 01:02:52 +1000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
References: <CACac1F8guknVxNCEcEYHYCh8MLaOzcX1Qd1viJY5AbhVphQUJg@mail.gmail.com>
 <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
Message-ID: <CADiSq7ebrU8U6ZG17h4nmj7j64W-kwC9GLhqeAz6vcAc+er8WQ@mail.gmail.com>

On 5 February 2014 00:33, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> On Tue, 4/2/14, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> The big problem with this type of thing is that pip tends to be used
>> on lots of systems with sometimes *extremely* obscure and odd
>> setups - what some of the Linux distros do to a standard Python
>> install amazes me (I'm sure they have good reasons, of course...).
>> So it's not so much a case of "scary dark corners" as lots of
>> unanticipated and extremely difficult to reproduce use cases that
>> we have to support but where the only knowledge of what
>> workarounds are needed is encapsulated in the code.
>
> You've just restated "scary dark corners" in a different way which
> sounds like there's more to it technically. But there' s still no hard
> data on the table!

http://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html

Vinay, please let it drop. Installing pip into every virtualenv works,
and you haven't provided a concrete end user *benefit* for removing
that layer of isolation - for upgrading pip after an upstream update
to be burdensome, a user would need to both have a lot of virtual
environments *and* fail to implement a script that automated the task
of iterating over them all and upgrading pip.

You accuse us of FUD, yet have presented no evidence that your
approach works flawlessly across multiple versions of Fedora, Debian,
openSUSE, RHEL, CentOS, Debian, Ubuntu, Windows, Mac OS X, etc,
despite the fact that many distros are known to ship old versions of
pip (and would likely end up shipping old system version of any new
tool that performed a similar role, if they shipped it natively at
all), and that pip used to have this feature, but it was removed due
to the creation of hard to reproduce and debug scenarios.

You yourself stated that end users would need to manage parallel
installs of the virtualenv independent tool, since it would be
designed not to be installed directly inside each virtualenv. And if
it *was* intended to be installed inside virtualenv for parallel
install support, then users would need to deal with the fact that
sometimes it would be in the virtualenv and sometimes not.

The status quo is simple, consistent, and comprehensible for
developers and end users alike. If the other participants in
*distutils-sig* think your suggested approach sounds confusing to use,
what chance is there for end users?

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Tue Feb  4 16:00:50 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 15:00:50 +0000 (GMT)
Subject: [Distutils] restart-in-venv
In-Reply-To: <CADiSq7cZms=ogOyG=hC9MGEX7_k9bTqej=KGgbUi7Uo4+owcpQ@mail.gmail.com>
Message-ID: <1391526050.14942.YahooMailBasic@web172401.mail.ir2.yahoo.com>

On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I'm not sure how I can explain it more clearly, but I'll try.
[snip]

I understood the scenarios you outlined.

> I'm not saying what you want to do *can't* be done (it obviously can).
> I'm saying it adds additional complexity, because you're adding a
> second isolation mechanism into the mix, with a completelely
> independent set of ways of interacting with it.

There is no separate isolation mechanism in my approach - distil
runs completely in the context of the virtualenv, so that's all the
isolation mechanism there is, in both cases. The difference is that
distil doesn't need to be installed in the venv - but otherwise, its
sys.path is the same as for any other tool in the venv, as it is running
*with the venv's interpreter*, just as the pip in a venv is. Perhaps
I haven't made that explicitly clear (I assumed it was implicit in
"restart in venv"). For example, the distil run with -e has no access
to system site-packages if the venv itself is isolated from system
site-packages.

> For an example of that consider that "pip install" and "python -m
>  pip install" are essentially equivalent commands. The status quo
> preserves that equivalence even inside a virtual environment,
> because there is no separate mechanism to be considered.

Well, distil doesn't have a __main__.py and is not designed to
be called with -m, but I don't see an essential difference, because
the main vehicle of isolation is *the venv machinery* in *both*
pip and distil cases.

> If the system Python site-packages is disabled in an active
> virtual environment, you don't need to carefully ensure that
> doesn't affect your install tool, because it's right there in the
> virtual environment with you.

As I said above, distil respects venv isolation: if a venv is
isolated from system site-packages, then distil code run with -e
can't see system site-packages, automatically, without any
"careful ensuring" being necessary. So from what I can see, your
argument is based on an imperfect understanding of how distil
works. If you'd like to understand it better I'll provide the support,
but if you don't want to or can't do that - well, that's a shame, but
such is life :-)

Regards,

Vinay Sajip

From ncoghlan at gmail.com  Tue Feb  4 16:05:25 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 01:05:25 +1000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
References: <CACac1F8guknVxNCEcEYHYCh8MLaOzcX1Qd1viJY5AbhVphQUJg@mail.gmail.com>
 <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
Message-ID: <CADiSq7eUex8Nh6ESzAnW_ajW-2UJ=FR6DYSi1aqsurmFrvonzQ@mail.gmail.com>

On 5 February 2014 00:33, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> def main():
>     if '-e' found in sys.argv:
>         validate the passed parameter, prepare a new command line
>         using the interpreter from the venv and the sys.argv passed in,
>         but minus the -e bit, since we've dealt with that here
>         call subprocess with that command line (the restart)
>         return - our work here is done
>     # Do stuff with sys.argv

That only works if the system site packages is configured to be
visible inside the virtual environment - it usually isn't these days.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Feb  4 16:10:14 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 01:10:14 +1000
Subject: [Distutils] restart-in-venv
In-Reply-To: <1391526050.14942.YahooMailBasic@web172401.mail.ir2.yahoo.com>
References: <CADiSq7cZms=ogOyG=hC9MGEX7_k9bTqej=KGgbUi7Uo4+owcpQ@mail.gmail.com>
 <1391526050.14942.YahooMailBasic@web172401.mail.ir2.yahoo.com>
Message-ID: <CADiSq7dsnucZwyJqBYHeLOEa99Wz59C6xxbycer=ird3vDdqLw@mail.gmail.com>

On 5 February 2014 01:00, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> As I said above, distil respects venv isolation: if a venv is
> isolated from system site-packages, then distil code run with -e
> can't see system site-packages, automatically, without any
> "careful ensuring" being necessary. So from what I can see, your
> argument is based on an imperfect understanding of how distil
> works. If you'd like to understand it better I'll provide the support,
> but if you don't want to or can't do that - well, that's a shame, but
> such is life :-)

So now we're back to the point Daniel (I believe) made of "hey, if you
restrict yourself to a single file script or a zip archive with a
__main__.py, you don't need to install it before running it"? Well
yes, that's how Python works.

So what you're really arguing for is for pip to become a single file
executable like distil, and deal with the consequences of cert file
unpacking, etc. Somehow, I don't see that getting anywhere near the
top of anyone's priority list any time soon.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From saschpe at gmx.de  Tue Feb  4 15:32:32 2014
From: saschpe at gmx.de (Sascha Peilicke)
Date: Tue, 04 Feb 2014 15:32:32 +0100
Subject: [Distutils] PEP-459 feedback from openSUSE packaging
In-Reply-To: <CADiSq7f4278Brirjrqmm2MpvOwoAEQJtirWDsKa_ZDFLb8_1hg@mail.gmail.com>
References: <2249682.syZXsYgalI@bort.suse.de>
 <CAG8k2+4Jj4Ch1BRUWzybyUej7ROkd8=R30dd1igXwp7d1kNkiQ@mail.gmail.com>
 <CADiSq7f4278Brirjrqmm2MpvOwoAEQJtirWDsKa_ZDFLb8_1hg@mail.gmail.com>
Message-ID: <23993478.SBI9QOKbDZ@bort.suse.de>

On Wednesday 05 February 2014 00:18:41 Nick Coghlan wrote:
> On 5 February 2014 00:05, Daniel Holth <dholth at gmail.com> wrote:
> > On Tue, Feb 4, 2014 at 9:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> Oh, I hadn't seen SPDX before - very interesting. I'm wondering if it
> >> may be a better fit for the PyPI Trove classifiers though - then it
> >> wouldn't even need to wait for metadata 2.0, we could just add them to
> >> the list of supported classifiers
> >> (https://pypi.python.org/pypi?%3Aaction=list_classifiers) and projects
> >> could start listing them in their current metadata.
> >> 
> >> Something like:
> >>     License :: SPDX :: <tag>
> >> 
> >> Cheers,
> >> Nick.
> > 
> > Those SPDX are great. They could certainly go into either license or
> > into a trove classifier, the difference being the trove classifiers
> > are checked against a static list.
> 
> The main advantage I can see to going the classifier route is that it
> means not having to wait for metadata 2.0 to promote them - folks can
> start using them as soon as they're registered on PyPI.
> 
> It also avoids compatibility issues when attempting to convert the
> many current projects with unclear license terms to metadata 2.0,
> while still making it easy for distro repackagers to offer upstream
> patches or bug reports to request license clarifications.
> 
> However, I do like the idea of having metadata 2.0 encourage the use
> of OSI approved SPDX tags in the license field. I just don't think we
> can upgrade that from a SHOULD to a MUST without breaking too many
> packages :(

I agree, it has to be a SHOULD of some sort.
-- 
Mit freundlichen Gr??en,
Sascha Peilicke

From saschpe at gmx.de  Tue Feb  4 15:31:41 2014
From: saschpe at gmx.de (Sascha Peilicke)
Date: Tue, 04 Feb 2014 15:31:41 +0100
Subject: [Distutils] PEP-459 feedback from openSUSE packaging
In-Reply-To: <20140204134245.GA10361@fridge.pov.lt>
References: <2249682.syZXsYgalI@bort.suse.de>
 <20140204134245.GA10361@fridge.pov.lt>
Message-ID: <1715378.fp8K2uT5cR@bort.suse.de>

On Tuesday 04 February 2014 15:42:45 Marius Gedminas wrote:
> On Tue, Feb 04, 2014 at 12:43:40PM +0100, Sascha Peilicke wrote:
> > The 'license' metadata tag is causing the most issues for us. A perceived
> > 50% just put "GPL" in there. Which GPL version? GPL-only or actually
> > LGPL?.
> FWIW you can sometimes get more detailed information from the classifiers:
> https://pypi.python.org/pypi?%3Aaction=list_classifiers
> 
> E.g.
> 
>   setup(...
>       licence='GPL',
>       classifiers=[
>           ...,
>           'License :: OSI Approved :: GNU General Public License v2 or later
> (GPLv2+)', ...
>       ])
> 

True, that may help in manual review. But often classifiers aren't enough. 
Please see my more detailed reply elsewhere in this thread.
-- 
Mit freundlichen Gr??en,
Sascha Peilicke

From saschpe at gmx.de  Tue Feb  4 15:58:51 2014
From: saschpe at gmx.de (Sascha Peilicke)
Date: Tue, 04 Feb 2014 15:58:51 +0100
Subject: [Distutils] Fwd: Re:  PEP-459 feedback from openSUSE packaging
Message-ID: <2179340.ksyf7d7AaB@bort.suse.de>

Err, forgot reply-all.
-- 
Mit freundlichen Gr??en,
Sascha Peilicke
-------------- next part --------------
An embedded message was scrubbed...
From: Sascha Peilicke <saschpe at gmx.de>
Subject: Re: [Distutils] PEP-459 feedback from openSUSE packaging
Date: Tue, 04 Feb 2014 15:31:41 +0100
Size: 1451
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140204/bbb09830/attachment-0001.mht>

From vinay_sajip at yahoo.co.uk  Tue Feb  4 16:48:35 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 15:48:35 +0000 (GMT)
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
	Twitter's multi-platform executable archive format for Python))
In-Reply-To: <CADiSq7eUex8Nh6ESzAnW_ajW-2UJ=FR6DYSi1aqsurmFrvonzQ@mail.gmail.com>
Message-ID: <1391528915.72519.YahooMailBasic@web172405.mail.ir2.yahoo.com>

On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> That only works if the system site packages is configured to
> be visible inside the virtual environment - it usually isn't
> these days.

If that were true, distil wouldn't work in such venvs, but it seems
to work fine. Can you show otherwise?

Regards,

Vinay Sajip

From p.f.moore at gmail.com  Tue Feb  4 16:53:42 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 4 Feb 2014 15:53:42 +0000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
References: <CACac1F8guknVxNCEcEYHYCh8MLaOzcX1Qd1viJY5AbhVphQUJg@mail.gmail.com>
 <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
Message-ID: <CACac1F-TFymuYZOg42CQbeFGhqzWuXPtLmGZd_3+RrkONkJJEw@mail.gmail.com>

On 4 February 2014 14:33, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> You've just restated "scary dark corners" in a different way which
> sounds like there's more to it technically. But there' s still no hard
> data on the table!

But I specifically stated that the hard data has been *lost* and the
only remaining evidence is how pip works. I'm not presenting that as a
good thing, just as "how life is". There's at least one article
somewhere on the internet discussing "why you shouldn't rewrite" in
these terms - I think it might have been Joel on Software or maybe
Coding Horror - but I can't find it right now.

> Surely, with all the convolutions that these distros go through, can one
> still assume that pip-the-program-that-they-all-run assumes control via
> some notional main program, which is passed some arguments? If
> we can't assume this, please tell me why not. If we can, then it would
> be like this:

On at least one system (OSX?) there's some arrangement where the
system Python is not actually the executable, but redirects through a
shell script in some manner. I know this has caused bug reports in the
past, but I do not have details, nor do I intend to go looking and see
if it's applicable here.

*If* you were suggesting that pip implement "re-exec in venv" the way
you suggest (and I understand that you aren't, even if you are saying
it should be simple to do so) then the onus is on you to confirm that
doing so will not cause regressions or bugs in this and similar known
risk areas. Otherwise, the assessment of the suggestion will remain
"interesting, but insufficient benefit to justify the risk".

Anyway, never mind. I agree the technique is interesting, and we have
no demonstrated bugs in it. But I don't imagine anyone will take the
risk on a widely-used project like pip any time in the near future.

Paul

From ncoghlan at gmail.com  Tue Feb  4 16:57:01 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 01:57:01 +1000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <1391528915.72519.YahooMailBasic@web172405.mail.ir2.yahoo.com>
References: <CADiSq7eUex8Nh6ESzAnW_ajW-2UJ=FR6DYSi1aqsurmFrvonzQ@mail.gmail.com>
 <1391528915.72519.YahooMailBasic@web172405.mail.ir2.yahoo.com>
Message-ID: <CADiSq7fbjgzayDty8R2L1RjGuvT+aBxbx916VYgO0u+5Qe03dA@mail.gmail.com>

On 5 February 2014 01:48, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> That only works if the system site packages is configured to
>> be visible inside the virtual environment - it usually isn't
>> these days.
>
> If that were true, distil wouldn't work in such venvs, but it seems
> to work fine. Can you show otherwise?

I wrote that before you reminded me that distil avoids most of the
tricky isolation problems by collapsing everything into a single file
to provide natural isolation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Tue Feb  4 16:59:25 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 15:59:25 +0000 (GMT)
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
	Twitter's multi-platform executable archive format for Python))
In-Reply-To: <CADiSq7ebrU8U6ZG17h4nmj7j64W-kwC9GLhqeAz6vcAc+er8WQ@mail.gmail.com>
Message-ID: <1391529565.58279.YahooMailBasic@web172403.mail.ir2.yahoo.com>

On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Vinay, please let it drop.

> You accuse us of FUD, yet have presented no evidence that
> your approach works flawlessly across multiple versions of
> Fedora, Debian, openSUSE, RHEL, CentOS, Debian, Ubuntu,
> Windows, Mac OS X, etc,

I merely asked for a hearing, and made no claims I couldn't
substantiate, and with actual code in places. But you seem
determined not to give me that hearing, so I'll drop it. It's a
shame you have to resort to an argument from authority.

Regards,

Vinay Sajip

From barry at python.org  Tue Feb  4 17:25:26 2014
From: barry at python.org (Barry Warsaw)
Date: Tue, 4 Feb 2014 11:25:26 -0500
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
References: <CACac1F8guknVxNCEcEYHYCh8MLaOzcX1Qd1viJY5AbhVphQUJg@mail.gmail.com>
 <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
 <CADiSq7eUex8Nh6ESzAnW_ajW-2UJ=FR6DYSi1aqsurmFrvonzQ@mail.gmail.com>
Message-ID: <20140204112526.52851853@anarchist.wooz.org>

On Feb 05, 2014, at 01:05 AM, Nick Coghlan wrote:

>That only works if the system site packages is configured to be
>visible inside the virtual environment - it usually isn't these days.

Really?  I do this all the time.  It prevents downloading gobs of stuff from
PyPI that my system already provides in reasonably up-to-date versions.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140204/a638210e/attachment.sig>

From ncoghlan at gmail.com  Tue Feb  4 17:38:19 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 02:38:19 +1000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <1391529565.58279.YahooMailBasic@web172403.mail.ir2.yahoo.com>
References: <CADiSq7ebrU8U6ZG17h4nmj7j64W-kwC9GLhqeAz6vcAc+er8WQ@mail.gmail.com>
 <1391529565.58279.YahooMailBasic@web172403.mail.ir2.yahoo.com>
Message-ID: <CADiSq7cS8uRUEKEESgT9g8vV_q-o30qUc17rSYRqeg8xxX=PsA@mail.gmail.com>

On 5 February 2014 01:59, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> Vinay, please let it drop.
>
>> You accuse us of FUD, yet have presented no evidence that
>> your approach works flawlessly across multiple versions of
>> Fedora, Debian, openSUSE, RHEL, CentOS, Debian, Ubuntu,
>> Windows, Mac OS X, etc,
>
> I merely asked for a hearing, and made no claims I couldn't
> substantiate, and with actual code in places. But you seem
> determined not to give me that hearing, so I'll drop it. It's a
> shame you have to resort to an argument from authority.

No hearing? You haven't answered *any* of our technical objections,
instead dismissing them all as irrelevant.

- doesn't work for packages that require installation (which I had
forgotten until you reminded me)
- doesn't address the distro isolation problem (except by ducking it
entirely and saying "don't install it", which isn't a good end user
experience, especially on Windows - hence the whole PEP 453 thing so
Windows users didn't need to download and run get-pip.py)
- doesn't provide a consistent end user experience (what if they *do*
install it in some virtual environments and not others?)
- doesn't provide any assistance in maintaining a correspondence
between pip versions and projects, which at least some freelancers and
other contractors see as a valuable capability
- should work *in theory* on multiple platforms, but has not been
tested broadly on them the way the current approach has

Hence, "potentially interesting from a technological perspective, but
doesn't solve any of the significant problems currently affecting end
users, so not actually all that interesting from a practical
perspective" is the natural assessment. It's neat that distil is a
standalone script, and hence can be easily run in a venv from outside
it, just as venv in Py 3.4 can bootstrap pip without first activating
the environment by running "<venv python> -m ensurepip". That's a
property of "standalone script" vs "installable package", though, and
one that has trade-offs of its own (some of which are noted above)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From saschpe at gmx.de  Tue Feb  4 17:46:42 2014
From: saschpe at gmx.de (Sascha Peilicke)
Date: Tue, 04 Feb 2014 17:46:42 +0100
Subject: [Distutils] PEP-459 feedback from openSUSE packaging
Message-ID: <2376307.yVndvNaN4C@bort.suse.de>

On Wednesday 05 February 2014 00:02:08 Nick Coghlan wrote:

> On 4 February 2014 21:43, Sascha Peilicke <saschpe at gmx.de> wrote:

> > Hi guys,
> > 
> > a colleague of mine hinted me to Nick's plea for feedback on various PEPs
> > from other distros perspectives.

> 
> Thanks! Happy to see my linux.conf.au presentation bearing fruit 
> 

> > I will provide some general remarks first and latter comment at least
> > PEP-459. So for openSUSE (and SLES), we automate Python package
> > generation as much as possible. For that we parse the metadata as found
> > on PyPI and maul the source distribution (yeah, the tarball) for every
> > usable bit. The latter is necessary to properly install files such as
> > README, AUTHORS, LICENSE(.txt,...). We also use it to grep for '*test*'
> > files to decide if it'S worth generating a RPM %check section.
> > 
> > Some modules do add 'package_data'. That's usually helpful since it goes
> > straight into %python_sitelib. For the files mentioned above, we have
> > 'data_files'. Not only is it seldomly used, it's almost universally done
> > wrong. Simply because every distro differs on where to put these files to.

> 
> PEP 426/459 will hopefully help with some of these, but I suspect only
> up to a point - pypi.python.org is always going to maintain a lower
> barrier to entry than the Linux distros that try to hammer their
> packages into a more integrated whole.

I agree and there's much value in keeping that barrier as low as possible. 
Most of the times, setting "--install-data" to something meaningful (while 
issueing setup.py install) is sufficient.


> 

> > A different cause of woe are install_requires / requires vs.
> > setup_requires
> > vs. test_require. Some people use 'requires', which is mostly
> > documentation
> > and lots of people put _everything_ in install_requires. From a
> > distribution> 
> > viewpoint, you have different sets of requirements:
> >  - build-time
> >  
> >    + optionally doc-requires
> >    + optionally test-requires
> >  
> >  - run-time
> > 
> > So setup_requires / test_require can be used to generate semi-accurate
> > BuildRequires: $bla RPM spec tags. But as said, few people use them and
> > less do it correct. Maybe because 'setup_requires' doesn't specificy
> > build-time reqs but 'setuptools-invocation-time' reqs (which is sth.
> > different). Also, we simply use 'install_requires' as both 'Requires:'
> > (runtime) and
> > 'BuildRequires:' (build-time). But that's a cludge. For example, projects
> > include 'Sphinx' in install_requires. What they meant is "if you want to
> > build docs, use Sphinx". What they specified is "you always need it".
> > Thankfully, the advent of pep allows us to check requirements.txt and
> > test-
> > requirements.txt. The latter are usually build-time (for the RPM %check
> > section). I guess I have to dig into the other PEPs first to see if this
> > really changed before being able to comment on that any further.

> 
> Yes, this aspect of the current system is a bit of a mess. One of the
> things we're aiming for with the wheel format is to clarify that even
> in the existing metadata, "install_requires" should refer to things
> needed to create a wheel, while "requires" should refer to things
> needed to actuall run the software after unpacking the wheel.
> 
> The proposed PEP 426 dependency tags are probably best summarised in
> this section:
> http://www.python.org/dev/peps/pep-0426/#mapping-dependencies-to-development
> -and-distribution-activities

I meanwhile read that and it looks promising so far. I have a bit of a fear 
that people may be overwhelmed by the options they have. 


> 
> Where a documentation dependency like Sphinx ends up would depend on
> the project. If the project has no bundled documentation (e.g. online
> docs only), then Sphinx would just be a "dev_requires" dependency.
> However, if it *did* ship with generated documentation that needed to
> be installed on the target system (e.g. man pages), then Sphinx would
> instead be a "build_requires" dependency.
> 
> I expect many upstream projects will still need help from the distros
> to get this right, but the ultimate aim for metadata 2.0 is to make it
> easier for distro repackagers to submit such patches upstream and *get
> them accepted as non-controversial changes*.

The good thing is that you can send pull requests to them with "do it that 
way" in the future rather than having to 'sed' around in setup.py files for 
the next 10 years.


> 

> > In general, the other metadata is good enough (except 'license', see
> > below), 'name', 'version', 'upstream_url' and 'description' are used for
> > their respective RPM spec counterparts. 'long_description' is used for
> > '%description $PKG_NAME'. The tarball download URL is used as 'Source0:'.
> > All other metadata tags are ignored because we don't need them to build a
> > RPM.

> 
> From a name and version point of view, I'm not at all familiar with
> the openSUSE policies, so I'd be interested in knowing if the
> restrictions in http://www.python.org/dev/peps/pep-0426/#name would
> also meet openSUSE naming guidelines.

That certainly help. The openSUSE naming policy is:

        python-$PYPI_UPSTREAM_NAME or
        python3-$PYPI_UPSTREAM_NAME

so we have packages like "python-zope.interface" or "python-Shed_Skin". I 
don't remember a module which had special characters (that RPM might frown 
upon) which we had to replace.



> 
> The version numbering PEP (http://www.python.org/dev/peps/pep-0440/)
> is annoyingly complicated, but it unfortunately needs to be in order
> to tolerate the diversity of versioning schems already in use on PyPI

Versioning is an arcane art. Of course we love semantic versioning (especially 
because RPM builds a lot of logic around), but if we discover sth crazy, we 
use this:

Version: 0.0.0+$WHATEVERNONSENSE

If $WHATEVERNONSE doesn't even increase alphabetically (needed for rpm -U to 
actually work), we put a date timestamp in front, sth. like:

Version 0.0.0+20140201.$WHATEVERNONSENSE

or just

Version 20140201.$WHATEVERNONSENSE

As long as the upstream version increases meaningfully, we're already content 




> > The 'license' metadata tag is causing the most issues for us. A perceived
> > 50% just put "GPL" in there. Which GPL version? GPL-only or actually
> > LGPL?. We have a legal crawler that tries to match the version from the
> > source code but often it's becomes a manual task or needs a check with
> > upstream. This tag is probably the least interesting for an upstream
> > developer but the most important one for any distro that has a corporate
> > legal entity somewhere in behind (I should say, sue-able entitity :-). So
> > with regards to PEP-459 specifically, I have specific recommendations for
> > the license tag. Instead of> 
> >         "This field SHOULD contain fewer than 512 characters and MUST
> >         contain
> > 
> > fewer
> > 
> >         than 2048.
> >         
> >         This field SHOULD NOT contain any line breaks."
> > 
> > I would propose:
> >         "This filed SHOULD contain a standardized license identifier as
> >         
> >          published by spdx.org."
> > 
> > SPDX-sytle license identifiers are short (less than 20 chars) and can be
> > parsed automatically. They are meant to be unambiguous and cross-distro.
> > SPDX.org license tags are used extensively inside openSUSE and SLES and
> > (to my knowledge) for Fedora and Debian too. That would be the single
> > most interesting change I'd be interest in.

> 
> Oh, I hadn't seen SPDX before - very interesting. I'm wondering if it
> may be a better fit for the PyPI Trove classifiers though - then it
> wouldn't even need to wait for metadata 2.0, we could just add them to
> the list of supported classifiers
> (https://pypi.python.org/pypi?%3Aaction=list_classifiers) and projects
> could start listing them in their current metadata.
> 
> Something like:
> 
>     License :: SPDX :: <tag>

That is actually a brilliant idea to add it to trove classifiers. Thing is, 
sdpx provides both a long version and a short identifier. Since I consider 
trove classifiers to be human-readable, I'm thinking of using the long version 
for trove and the short one as a recommendation for the License: tag. There's 
a rationale behind. Some projects really have complex licensing issues 
(regardless what they claim). Just to give an example, the license for python-
psycopg2 we have is:

License: LGPL-3.0-with-openssl-exception and (LGPL-3.0-with-openssl-exception 
or ZPL-2.0)

BTW. this is not what upstream thinks they have, this is what our license 
crawler found out (and what was approved by our legal guys). part of spdx are 
the boolean operators you see to specify _either_ license X _or_ license Y 
_and_ licence Z. Happens more often than you can imagine. Therefore, the short 
spdx identifiers are much more consise rather than using trove classifiers. 
Currently, you simply can't express 'and' / 'or' with trove classifiers.

-- 
Mit freundlichen Gr??en,
Sascha Peilicke

From ncoghlan at gmail.com  Tue Feb  4 17:47:26 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Feb 2014 02:47:26 +1000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <20140204112526.52851853@anarchist.wooz.org>
References: <CACac1F8guknVxNCEcEYHYCh8MLaOzcX1Qd1viJY5AbhVphQUJg@mail.gmail.com>
 <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
 <CADiSq7eUex8Nh6ESzAnW_ajW-2UJ=FR6DYSi1aqsurmFrvonzQ@mail.gmail.com>
 <20140204112526.52851853@anarchist.wooz.org>
Message-ID: <CADiSq7eahTJksiDuimTazwi8-WwRhBrJMAUNO8vywGX7=3D+hw@mail.gmail.com>

On 5 February 2014 02:25, Barry Warsaw <barry at python.org> wrote:
> On Feb 05, 2014, at 01:05 AM, Nick Coghlan wrote:
>
>>That only works if the system site packages is configured to be
>>visible inside the virtual environment - it usually isn't these days.
>
> Really?  I do this all the time.  It prevents downloading gobs of stuff from
> PyPI that my system already provides in reasonably up-to-date versions.

So do I - there's a bunch of RPMs in the system packages that are a
pain to install from source (if they're on PyPI at all - assorted
Fedora specific utilities aren't). But virtualenv doesn't work that
way by default, and the web/PaaS focused devs that write a lot of the
upstream "best practices" guidelines aren't typically fans of distro
provided packages because they're too old (scientific users often have
a similar complaint).

Playing peacemaker between the "move fast and break things" and "if
it's less than 5 years old it's still just a fad" schools of thought
is one of the things that makes packaging so "interesting" ;)

Cheers,
Nick.

>
> -Barry
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Tue Feb  4 17:50:00 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 4 Feb 2014 16:50:00 +0000
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
 Twitter's multi-platform executable archive format for Python))
In-Reply-To: <CADiSq7eahTJksiDuimTazwi8-WwRhBrJMAUNO8vywGX7=3D+hw@mail.gmail.com>
References: <CACac1F8guknVxNCEcEYHYCh8MLaOzcX1Qd1viJY5AbhVphQUJg@mail.gmail.com>
 <1391524435.174.YahooMailBasic@web172405.mail.ir2.yahoo.com>
 <CADiSq7eUex8Nh6ESzAnW_ajW-2UJ=FR6DYSi1aqsurmFrvonzQ@mail.gmail.com>
 <20140204112526.52851853@anarchist.wooz.org>
 <CADiSq7eahTJksiDuimTazwi8-WwRhBrJMAUNO8vywGX7=3D+hw@mail.gmail.com>
Message-ID: <CACac1F8wLmJ68VA_2=1HbC3sctxO=uSaLV32vp+4iZqdWxU_NQ@mail.gmail.com>

On 4 February 2014 16:47, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Playing peacemaker between the "move fast and break things" and "if
> it's less than 5 years old it's still just a fad" schools of thought
> is one of the things that makes packaging so "interesting" ;)

:-)

+1 QOTW

Paul

From vinay_sajip at yahoo.co.uk  Tue Feb  4 17:54:02 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 4 Feb 2014 16:54:02 +0000 (GMT)
Subject: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX -
	Twitter's multi-platform executable archive format for Python))
In-Reply-To: <CADiSq7cS8uRUEKEESgT9g8vV_q-o30qUc17rSYRqeg8xxX=PsA@mail.gmail.com>
Message-ID: <1391532842.90222.YahooMailBasic@web172401.mail.ir2.yahoo.com>


On Tue, 4/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> No hearing? You haven't answered *any* of our technical
> objections, instead dismissing them all as irrelevant.
 
Because you advanced no specific technical objections,
instead invoking generalisations. If you prefer the status quo
because it's easier to explain (i.e. there's nothing to explain),
that's not a technical objection in my book.

However you want to look at it, that's fine by me. The example
of distil shows that the status quo is not the only way of doing
things, and I'll remain interested in any feedback from anyone
regarding specific failures of distil in terms of practical
workflows that aren't possible. "Doesn't need to be
installed in a venv" or "runs as a single file" isn't a failure from
my POV, so we'll have to agree to disagree.

But, as you say, let's drop it. Jonathan Swift's observation
applies.

Regards,

Vinay Sajip

From speilicke at suse.com  Tue Feb  4 17:26:07 2014
From: speilicke at suse.com (Sascha Peilicke)
Date: Tue, 04 Feb 2014 17:26:07 +0100
Subject: [Distutils] Fwd: Re:  PEP-459 feedback from openSUSE packaging
Message-ID: <5174657.CgBxRkEvvh@bort.suse.de>

Mhm, I got:

"distutils-sig at python.org":
SMTP error from remote server after RCPT command:
host: mail.python.org
4.3.2 Service currently unavailable

trying again with corporate mail...
-- 
With kind regards,
Sascha Peilicke
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer HRB 16746 (AG N?rnberg)
-------------- next part --------------
An embedded message was scrubbed...
From: Sascha Peilicke <saschpe at gmx.de>
Subject: Re: [Distutils] PEP-459 feedback from openSUSE packaging
Date: Tue, 04 Feb 2014 15:31:19 +0100
Size: 10019
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140204/7275db76/attachment-0001.mht>

From python at aryehleib.com  Tue Feb  4 21:52:39 2014
From: python at aryehleib.com (Aryeh Leib Taurog)
Date: Tue, 4 Feb 2014 22:52:39 +0200
Subject: [Distutils] Package different sets of files with sdist and bdist
Message-ID: <20140204205239.GA495@m92.aryehleib.com>

I'm working on a django app, which I'd like to be able to package with 
setuptools.  I use sass/compass, coffeescript, and hamlpy, among other 
tools to generate supporting static content for my app.

I think my build/deploy process should look like this:

 1. On my development machine:

    [dev] $ python setup.py sdist

    This would create a tarball of all my sources,
    mainly *.py, *.iced, *.haml, *.sass, and one *.pyx

 2. On a build machine:

    [build] $ compass compile mypkg/static       # compile sass to css
    [build] $ coffee --compile -o mypkg/js mypkg/coffee  # compile coffee
    [build] $ python setup.py bdist_wheel

    This would create a binary wheel with my c-extension and all my 
    static content compiled.

 3. Deploy:

    [www] $ pip install mypkg-1.0-cp27-none-linux_x86_64.whl

The problem is that there doesn't seem to be any easy way to include 
only the *.coffee and *.sass files in the sdist and only the *.js and 
*.css files in the bdist, short of deleting the files I want to 
exclude at each stage prior to running setup.py, which I'd rather not 
do.

I'm using setuptools with VCS support and include_package_data=True.

Is there any way to do this, or a more pythonic workflow to accomplish 
the same goal?

Thanks,
Aryeh Leib Taurog

From merwok at netwok.org  Wed Feb  5 18:03:39 2014
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Wed, 05 Feb 2014 12:03:39 -0500
Subject: [Distutils] PEP 427
In-Reply-To: <CAG8k2+7VcmFLbUv4had73AFfZoVmjT8cjo0Jz7HodgXhcyac6A@mail.gmail.com>
References: <CFE861E8-162F-4622-8B62-23A52DDB6C5D@sazhin.us>
 <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com>
 <CAH3X-tmNcvsaa5SKo5nvb6o=j9csuZBoOGVmhKHKRchhwDkQaQ@mail.gmail.com>
 <A34E835E-4344-4640-B923-4711CA2A1B28@coderanger.net>
 <D311236F-8D64-43AD-A892-3E7DEFB352CD@sazhin.us>
 <CEE00B36-F5F2-456C-A134-3A957692DFB4@stufft.io>
 <A6B46779-FF01-4AC6-AC54-65F6C671B467@sazhin.us>
 <D5CCF29A-70E1-471D-9207-49478B85190C@stufft.io>
 <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us>
 <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io>
 <CAH3X-t=HhgSxwOu5pzOVmPn2KoVvnoFkEU6+d-vfBAYKAB09Ow@mail.gmail.com>
 <D65B9316-7490-42E9-B0C1-2F9373759466@stufft.io>
 <CAH3X-tnMMMpcMY1Gq6mGBnW6Y4p7msaM810GFrGr6x0n5rKKMA@mail.gmail.com>
 <CAG8k2+7VcmFLbUv4had73AFfZoVmjT8cjo0Jz7HodgXhcyac6A@mail.gmail.com>
Message-ID: <52F26EEB.1080004@netwok.org>

Le 30/01/2014 12:44, Daniel Holth a ?crit :
> We do actually need a better way to turn libraries on and off
> individually, for example, if I install the wrong set of Python
> libraries on Ubuntu the plugin system for somesuch package crashes;

Really?  I thought the Debuntu Python maintainer changed site-packages 
to dist-packages to avoid system programs being influenced by local 
installs.

Regards

> having everything installed importable all the time is a problem.

From dholth at gmail.com  Wed Feb  5 18:39:07 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 5 Feb 2014 12:39:07 -0500
Subject: [Distutils] PEP 427
In-Reply-To: <52F26EEB.1080004@netwok.org>
References: <CFE861E8-162F-4622-8B62-23A52DDB6C5D@sazhin.us>
 <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com>
 <CAH3X-tmNcvsaa5SKo5nvb6o=j9csuZBoOGVmhKHKRchhwDkQaQ@mail.gmail.com>
 <A34E835E-4344-4640-B923-4711CA2A1B28@coderanger.net>
 <D311236F-8D64-43AD-A892-3E7DEFB352CD@sazhin.us>
 <CEE00B36-F5F2-456C-A134-3A957692DFB4@stufft.io>
 <A6B46779-FF01-4AC6-AC54-65F6C671B467@sazhin.us>
 <D5CCF29A-70E1-471D-9207-49478B85190C@stufft.io>
 <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us>
 <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io>
 <CAH3X-t=HhgSxwOu5pzOVmPn2KoVvnoFkEU6+d-vfBAYKAB09Ow@mail.gmail.com>
 <D65B9316-7490-42E9-B0C1-2F9373759466@stufft.io>
 <CAH3X-tnMMMpcMY1Gq6mGBnW6Y4p7msaM810GFrGr6x0n5rKKMA@mail.gmail.com>
 <CAG8k2+7VcmFLbUv4had73AFfZoVmjT8cjo0Jz7HodgXhcyac6A@mail.gmail.com>
 <52F26EEB.1080004@netwok.org>
Message-ID: <CAG8k2+6eW3z424Mzp63OjfzY4=ewji7G9KV5WRN9p6UHmcJDfg@mail.gmail.com>

No non system packages were involved in the crash.
On Feb 5, 2014 12:04 PM, "?ric Araujo" <merwok at netwok.org> wrote:

> Le 30/01/2014 12:44, Daniel Holth a ?crit :
>
>> We do actually need a better way to turn libraries on and off
>> individually, for example, if I install the wrong set of Python
>> libraries on Ubuntu the plugin system for somesuch package crashes;
>>
>
> Really?  I thought the Debuntu Python maintainer changed site-packages to
> dist-packages to avoid system programs being influenced by local installs.
>
> Regards
>
>  having everything installed importable all the time is a problem.
>>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140205/9ebaadce/attachment.html>

From eugene at sazhin.us  Wed Feb  5 23:20:32 2014
From: eugene at sazhin.us (Evgeny Sazhin)
Date: Wed, 5 Feb 2014 17:20:32 -0500
Subject: [Distutils] wheel building help needed
Message-ID: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>

Hi,

So, i'm trying to get around building the wheel using pip wheel and
I'm having some troubles with it. I would appreciate if you could
clarify some of the questions for me:

I have a structure like that:
./
wheel/
    __main__.py
    my/__init__.py
        cool/__init__.py
              package/prog.py
                        __init__.py
MANIFEST.in
setup.py
setup.cfg

Where setup.cfg has only
[wheel]
universal=1

Manifest has:

include wheel/__main__.py
recursive_include wheel *.py

I'm issuing "pip wheel ." and it does put a wheel with some generated
info in it into the ./wheelhouse, but there are issues:

1. No python files added, neither __main__.py nor prog.py make it to
the wheel, so includes are not working
2. Despite the fact that I have universal=1 there is no py2.py3 in the
name it says cp33-cp33m

Could anybody hint anything?

In addition to that I failed to find the rules for manifest and
setup.cfg files (acceptable sections, values, syntax) so i would
appreciate if anybody could share a link to the stuff.

Thanks a lot!
Eugene

From p.f.moore at gmail.com  Thu Feb  6 00:01:17 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 5 Feb 2014 23:01:17 +0000
Subject: [Distutils] wheel building help needed
In-Reply-To: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
Message-ID: <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>

On 5 February 2014 22:20, Evgeny Sazhin <eugene at sazhin.us> wrote:
> 1. No python files added, neither __main__.py nor prog.py make it to
> the wheel, so includes are not working

The manifest is for sdists, not wheels. So includes won't "work" in
the sense that you seem to be expecting. You need to specify your
package details in setup.py - you don't say if you did. I suspect
you're not getting a universal wheel because it looks like it isn't
pure Python - maybe because it doesn't have the right files in it, I
can't tell from what you're showing.

You still seem to be making wheels with the intention that they are
executable. I apologise if I'm reading too much into your directory
structure, but if that's what you are trying to do I can well imagine
that you'll get odd results as you're trying to do something the
system wasn't designed for.

Paul

From eugene at sazhin.us  Thu Feb  6 00:41:12 2014
From: eugene at sazhin.us (Evgeny Sazhin)
Date: Wed, 5 Feb 2014 18:41:12 -0500
Subject: [Distutils] wheel building help needed
In-Reply-To: <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
Message-ID: <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>

On Wed, Feb 5, 2014 at 6:01 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 5 February 2014 22:20, Evgeny Sazhin <eugene at sazhin.us> wrote:
>> 1. No python files added, neither __main__.py nor prog.py make it to
>> the wheel, so includes are not working
>
> The manifest is for sdists, not wheels. So includes won't "work" in
> the sense that you seem to be expecting. You need to specify your
> package details in setup.py - you don't say if you did. I suspect
> you're not getting a universal wheel because it looks like it isn't
> pure Python - maybe because it doesn't have the right files in it, I
> can't tell from what you're showing.

I got rid of the wheel folder as the machinery behind the wheel
building (or between keyboard and chair) is not smart enough to
understand how to work with packages that are one level down from the
root.

my setup.py is :

setup(
name="prog",
version="0.1",
#etc,
packages=find_packages()

)

I got the wheel properly built using python setup.py bdist_wheel (i
was able to do it before, but didn't realize the manifest stuff is for
that way of building, not for the pip wheel). So "pip wheel" usage
remains to be unclear...

>
> You still seem to be making wheels with the intention that they are
> executable. I apologise if I'm reading too much into your directory
> structure, but if that's what you are trying to do I can well imagine
> that you'll get odd results as you're trying to do something the
> system wasn't designed for.
>
> Paul

Yes, not only i'm making wheel executable, but i also will most
probably build around the fact that it is zip-readable and therefore I
will add them to the PYTHONPATH to run, just like java classpath and
jar (I know - stubborn me) - that's the only thing that makes any
sense! After all those arguments about C extensions and whatnot i
still think that enforcing the unzipping and deploying stuff is not
what packaging should provide as a solution.
Plus as you see I'm trying to build universal wheel - pure python.
Nobody so far could prove or at least explain why it is bad idea *for
pure python*, so I think this is the way to go.


Thanks,
Eugene

From dholth at gmail.com  Thu Feb  6 03:10:31 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 5 Feb 2014 21:10:31 -0500
Subject: [Distutils] wheel building help needed
In-Reply-To: <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
Message-ID: <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>

You are probably just missing the package_dir argument to setup(),
adequately documented at
http://docs.python.org/2/distutils/examples.html

Pass

      package_dir={'': 'src'},

to indicate that the root module starts in 'src'.

MANIFEST.in is needed. It is always a bit awkward to write.

On Wed, Feb 5, 2014 at 6:41 PM, Evgeny Sazhin <eugene at sazhin.us> wrote:
> On Wed, Feb 5, 2014 at 6:01 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 5 February 2014 22:20, Evgeny Sazhin <eugene at sazhin.us> wrote:
>>> 1. No python files added, neither __main__.py nor prog.py make it to
>>> the wheel, so includes are not working
>>
>> The manifest is for sdists, not wheels. So includes won't "work" in
>> the sense that you seem to be expecting. You need to specify your
>> package details in setup.py - you don't say if you did. I suspect
>> you're not getting a universal wheel because it looks like it isn't
>> pure Python - maybe because it doesn't have the right files in it, I
>> can't tell from what you're showing.
>
> I got rid of the wheel folder as the machinery behind the wheel
> building (or between keyboard and chair) is not smart enough to
> understand how to work with packages that are one level down from the
> root.
>
> my setup.py is :
>
> setup(
> name="prog",
> version="0.1",
> #etc,
> packages=find_packages()
>
> )
>
> I got the wheel properly built using python setup.py bdist_wheel (i
> was able to do it before, but didn't realize the manifest stuff is for
> that way of building, not for the pip wheel). So "pip wheel" usage
> remains to be unclear...
>
>>
>> You still seem to be making wheels with the intention that they are
>> executable. I apologise if I'm reading too much into your directory
>> structure, but if that's what you are trying to do I can well imagine
>> that you'll get odd results as you're trying to do something the
>> system wasn't designed for.
>>
>> Paul
>
> Yes, not only i'm making wheel executable, but i also will most
> probably build around the fact that it is zip-readable and therefore I
> will add them to the PYTHONPATH to run, just like java classpath and
> jar (I know - stubborn me) - that's the only thing that makes any
> sense! After all those arguments about C extensions and whatnot i
> still think that enforcing the unzipping and deploying stuff is not
> what packaging should provide as a solution.
> Plus as you see I'm trying to build universal wheel - pure python.
> Nobody so far could prove or at least explain why it is bad idea *for
> pure python*, so I think this is the way to go.
>
>
> Thanks,
> Eugene
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From eugene at sazhin.us  Thu Feb  6 06:37:34 2014
From: eugene at sazhin.us (Evgeny Sazhin)
Date: Thu, 6 Feb 2014 00:37:34 -0500
Subject: [Distutils] wheel building help needed
In-Reply-To: <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
 <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>
Message-ID: <BFF4D834-BD08-4738-836F-0A1B005FACD4@sazhin.us>


On Feb 5, 2014, at 9:10 PM, Daniel Holth <dholth at gmail.com> wrote:

> You are probably just missing the package_dir argument to setup(),
> adequately documented at
> http://docs.python.org/2/distutils/examples.html
> 
> Pass
> 
>      package_dir={'': 'src'},
> 
> to indicate that the root module starts in 'src'.
> 
> MANIFEST.in is needed. It is always a bit awkward to write.

Thank you Daniel! I will definitely take a look at that.

I have to say though that this is one of the problems i have with the whole python packaging:
It is scattered all over the place. I?m importing setuptools in the setup.py to build wheel while i obviously miss some information about distutils? 

I think it could be very beneficial to have it all moved under one umbrella project or wrapper or something like that. I as a user am interested in centralized thing that knows how to take care of everything i need in order to make a wheel.
Why not to have only setup.cfg (even better wheel.cfg)  that describes all necessary things including includes and package details and then have ?wheel" command that knows to create it properly basing on this config? One config file, one dependencies file (requirements.txt) - one command that does the work = what the doctor ordered! No?

Thanks again,
Eugene  
 




> 
> On Wed, Feb 5, 2014 at 6:41 PM, Evgeny Sazhin <eugene at sazhin.us> wrote:
>> On Wed, Feb 5, 2014 at 6:01 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>> On 5 February 2014 22:20, Evgeny Sazhin <eugene at sazhin.us> wrote:
>>>> 1. No python files added, neither __main__.py nor prog.py make it to
>>>> the wheel, so includes are not working
>>> 
>>> The manifest is for sdists, not wheels. So includes won't "work" in
>>> the sense that you seem to be expecting. You need to specify your
>>> package details in setup.py - you don't say if you did. I suspect
>>> you're not getting a universal wheel because it looks like it isn't
>>> pure Python - maybe because it doesn't have the right files in it, I
>>> can't tell from what you're showing.
>> 
>> I got rid of the wheel folder as the machinery behind the wheel
>> building (or between keyboard and chair) is not smart enough to
>> understand how to work with packages that are one level down from the
>> root.
>> 
>> my setup.py is :
>> 
>> setup(
>> name="prog",
>> version="0.1",
>> #etc,
>> packages=find_packages()
>> 
>> )
>> 
>> I got the wheel properly built using python setup.py bdist_wheel (i
>> was able to do it before, but didn't realize the manifest stuff is for
>> that way of building, not for the pip wheel). So "pip wheel" usage
>> remains to be unclear...
>> 
>>> 
>>> You still seem to be making wheels with the intention that they are
>>> executable. I apologise if I'm reading too much into your directory
>>> structure, but if that's what you are trying to do I can well imagine
>>> that you'll get odd results as you're trying to do something the
>>> system wasn't designed for.
>>> 
>>> Paul
>> 
>> Yes, not only i'm making wheel executable, but i also will most
>> probably build around the fact that it is zip-readable and therefore I
>> will add them to the PYTHONPATH to run, just like java classpath and
>> jar (I know - stubborn me) - that's the only thing that makes any
>> sense! After all those arguments about C extensions and whatnot i
>> still think that enforcing the unzipping and deploying stuff is not
>> what packaging should provide as a solution.
>> Plus as you see I'm trying to build universal wheel - pure python.
>> Nobody so far could prove or at least explain why it is bad idea *for
>> pure python*, so I think this is the way to go.
>> 
>> 
>> Thanks,
>> Eugene
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig


From p.f.moore at gmail.com  Thu Feb  6 09:06:06 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 6 Feb 2014 08:06:06 +0000
Subject: [Distutils] wheel building help needed
In-Reply-To: <BFF4D834-BD08-4738-836F-0A1B005FACD4@sazhin.us>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
 <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>
 <BFF4D834-BD08-4738-836F-0A1B005FACD4@sazhin.us>
Message-ID: <CACac1F8n4f4FkLfA2-xvUte5KAG3f34cDYA6it63L5dVJq8cZg@mail.gmail.com>

On 6 February 2014 05:37, Evgeny Sazhin <eugene at sazhin.us> wrote:
> I think it could be very beneficial to have it all moved under one umbrella project or wrapper or something like that. I as a user am interested in centralized thing that knows how to take care of everything i need in order to make a wheel.
> Why not to have only setup.cfg (even better wheel.cfg)  that describes all necessary things including includes and package details and then have "wheel" command that knows to create it properly basing on this config? One config file, one dependencies file (requirements.txt) - one command that does the work = what the doctor ordered! No?

This is one of the major things being worked on at the moment -
initially, putting the documentation together in one place
(http://python-packaging-user-guide.readthedocs.org/en/latest/) and
eventually having a more "unified" build process, but that's some way
off. Experience and feedback like yours is immensely useful in making
sure things are documented in a way that is accessible to newcomers
(most of us round here are a bit too close to the problem to know
what's needed :-)). So many thanks for taking the time to post your
experiences.

Paul

From ncoghlan at gmail.com  Thu Feb  6 10:54:03 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 6 Feb 2014 19:54:03 +1000
Subject: [Distutils] wheel building help needed
In-Reply-To: <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
Message-ID: <CADiSq7cKk=y6J5Z2PEPTOGWBQZNvnB_4VUzyDmKQAP67-VTP-Q@mail.gmail.com>

Evgeny, while you can probably make wheels do what you want, if you're
interested in single file executables, you're almost certainly better off
using one of the tools designed to make those easier to work with (like
Twitter's recently discussed PEX format, or PEP 441).

As you have discovered, cross platform single file executables are an area
where the currently published documentation is even less polished and
coherent than that for the rest of the packaging infrastructure :)

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140206/c0e054a8/attachment.html>

From marius at pov.lt  Thu Feb  6 14:56:33 2014
From: marius at pov.lt (Marius Gedminas)
Date: Thu, 6 Feb 2014 15:56:33 +0200
Subject: [Distutils] wheel building help needed
In-Reply-To: <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
 <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>
Message-ID: <20140206135633.GA2502@fridge.pov.lt>

On Wed, Feb 05, 2014 at 09:10:31PM -0500, Daniel Holth wrote:
> MANIFEST.in is needed. It is always a bit awkward to write.

https://pypi.python.org/pypi/check-manifest is here to help.

Marius Gedminas
-- 
Real programmers don't write in BASIC.  Actually, no programmers write in
BASIC after reaching puberty.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140206/49a46cd7/attachment.sig>

From dholth at gmail.com  Thu Feb  6 19:06:54 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 6 Feb 2014 13:06:54 -0500
Subject: [Distutils] wheel building help needed
In-Reply-To: <BFF4D834-BD08-4738-836F-0A1B005FACD4@sazhin.us>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
 <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>
 <BFF4D834-BD08-4738-836F-0A1B005FACD4@sazhin.us>
Message-ID: <CAG8k2+7YqtRAhkHJC8Jdy6QNnStCs31BXks=Jo50DtzgcDzwoA@mail.gmail.com>

On Thu, Feb 6, 2014 at 12:37 AM, Evgeny Sazhin <eugene at sazhin.us> wrote:
>
> On Feb 5, 2014, at 9:10 PM, Daniel Holth <dholth at gmail.com> wrote:
>
>> You are probably just missing the package_dir argument to setup(),
>> adequately documented at
>> http://docs.python.org/2/distutils/examples.html
>>
>> Pass
>>
>>      package_dir={'': 'src'},
>>
>> to indicate that the root module starts in 'src'.
>>
>> MANIFEST.in is needed. It is always a bit awkward to write.
>
> Thank you Daniel! I will definitely take a look at that.
>
> I have to say though that this is one of the problems i have with the whole python packaging:
> It is scattered all over the place. I'm importing setuptools in the setup.py to build wheel while i obviously miss some information about distutils?
>
> I think it could be very beneficial to have it all moved under one umbrella project or wrapper or something like that. I as a user am interested in centralized thing that knows how to take care of everything i need in order to make a wheel.
> Why not to have only setup.cfg (even better wheel.cfg)  that describes all necessary things including includes and package details and then have "wheel" command that knows to create it properly basing on this config? One config file, one dependencies file (requirements.txt) - one command that does the work = what the doctor ordered! No?
>
> Thanks again,
> Eugene

I would like to see more people experiment with distutils
replacements. Bento and distil do that to some extent. We would like
to make it very easy to switch out the setup.py implementation for
something else.

From p at google-groups-2013.dobrogost.net  Thu Feb  6 12:56:48 2014
From: p at google-groups-2013.dobrogost.net (Piotr Dobrogost)
Date: Thu, 6 Feb 2014 03:56:48 -0800 (PST)
Subject: [Distutils] Dependency links are being followed without
 --process-dependency-links specified
Message-ID: <2bf0a7ab-3e04-4158-8851-9f55520902a6@googlegroups.com>

Running `pip install --no-deps -e .` with the following setup.py works
---- setup.py -------
from setuptools import setup, find_packages

setup(
    name="x",
    version='1.0',
    packages=find_packages(exclude=['ez_setup']),
    setup_requires=[
        "PasteScript>=1.7",
    ],
)

but after adding (dead) link in dependency_links like this

----- setup.py -----
from setuptools import setup, find_packages

setup(
    name="x",
    version='1.0',
    packages=find_packages(exclude=['ez_setup']),
    setup_requires=[
        "PasteScript>=1.7",
    ],
    dependency_links=[
        "http://www.pylonshq.com/download/"
    ],
)

it does no longer work. Please note I do not specify 
--process-dependency-links option. I get this error:


(ve)pdobrogost at host:~/tmp/x$ pip install --no-deps -e .
Obtaining file:///home/users/pdobrogost/tmp/x
  Running setup.py (path:/home/users/pdobrogost/tmp/x/setup.py) egg_info 
for package from file:///home/users/pdobrogost/tmp/x
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/users/pdobrogost/tmp/x/setup.py", line 13, in <module>
        "http://www.pylonshq.com/download/"
      File "/opt/python/2.7.5/lib/python2.7/distutils/core.py", line 112, 
in setup
        _setup_distribution = dist = klass(attrs)
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/dist.py", 
line 239, in __init__
        self.fetch_build_eggs(attrs.pop('setup_requires'))
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/dist.py", 
line 263, in fetch_build_eggs
        parse_requirements(requires), installer=self.fetch_build_egg
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/pkg_resources.py", 
line 564, in resolve
        dist = best[req.key] = env.best_match(req, self, installer)
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/pkg_resources.py", 
line 802, in best_match
        return self.obtain(req, installer) # try and download/install
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/pkg_resources.py", 
line 814, in obtain
        return installer(requirement)
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/dist.py", 
line 313, in fetch_build_egg
        return cmd.easy_install(req)
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/command/easy_install.py", 
line 581, in easy_install
        self.local_index
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 586, in fetch_distribution
        self.prescan()
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 484, in prescan
        list(map(self.scan_url, self.to_scan))
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 768, in scan_url
        self.process_url(url, True)
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 305, in process_url
        f = self.open_url(url, "Download error on %s: %%s -- Some packages 
may not be found!" % url)
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 702, in open_url
        return open_with_auth(url, self.opener)
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 901, in _socket_timeout
        return func(*args, **kwargs)
      File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 1014, in open_with_auth
        fp = opener(request)
      File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 404, in open
        response = self._open(req, data)
      File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 422, in _open
        '_open', req)
      File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 382, in 
_call_chain
        result = func(*args)
      File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 1214, in 
http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 1187, in 
do_open
        r = h.getresponse(buffering=True)
      File "/opt/python/2.7.5/lib/python2.7/httplib.py", line 1045, in 
getresponse
        response.begin()
      File "/opt/python/2.7.5/lib/python2.7/httplib.py", line 409, in begin
        version, status, reason = self._read_status()
      File "/opt/python/2.7.5/lib/python2.7/httplib.py", line 365, in 
_read_status
        line = self.fp.readline(_MAXLINE + 1)
      File "/opt/python/2.7.5/lib/python2.7/socket.py", line 476, in 
readline
        data = self._sock.recv(self._rbufsize)
    socket.timeout: timed out
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/home/users/pdobrogost/tmp/x/setup.py", line 13, in <module>

    "http://www.pylonshq.com/download/"

  File "/opt/python/2.7.5/lib/python2.7/distutils/core.py", line 112, in 
setup

    _setup_distribution = dist = klass(attrs)

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/dist.py", 
line 239, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/dist.py", 
line 263, in fetch_build_eggs

    parse_requirements(requires), installer=self.fetch_build_egg

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/pkg_resources.py", 
line 564, in resolve

    dist = best[req.key] = env.best_match(req, self, installer)

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/pkg_resources.py", 
line 802, in best_match

    return self.obtain(req, installer) # try and download/install

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/pkg_resources.py", 
line 814, in obtain

    return installer(requirement)

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/dist.py", 
line 313, in fetch_build_egg

    return cmd.easy_install(req)

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/command/easy_install.py", 
line 581, in easy_install

    self.local_index

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 586, in fetch_distribution

    self.prescan()

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 484, in prescan

    list(map(self.scan_url, self.to_scan))

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 768, in scan_url

    self.process_url(url, True)

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 305, in process_url

    f = self.open_url(url, "Download error on %s: %%s -- Some packages may 
not be found!" % url)

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 702, in open_url

    return open_with_auth(url, self.opener)

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 901, in _socket_timeout

    return func(*args, **kwargs)

  File 
"/home/users/pdobrogost/tmp/ve/lib/python2.7/site-packages/setuptools/package_index.py", 
line 1014, in open_with_auth

    fp = opener(request)

  File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 404, in open

    response = self._open(req, data)

  File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 422, in _open

    '_open', req)

  File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 382, in 
_call_chain

    result = func(*args)

  File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 1214, in http_open

    return self.do_open(httplib.HTTPConnection, req)

  File "/opt/python/2.7.5/lib/python2.7/urllib2.py", line 1187, in do_open

    r = h.getresponse(buffering=True)

  File "/opt/python/2.7.5/lib/python2.7/httplib.py", line 1045, in 
getresponse

    response.begin()

  File "/opt/python/2.7.5/lib/python2.7/httplib.py", line 409, in begin

    version, status, reason = self._read_status()

  File "/opt/python/2.7.5/lib/python2.7/httplib.py", line 365, in 
_read_status

    line = self.fp.readline(_MAXLINE + 1)

  File "/opt/python/2.7.5/lib/python2.7/socket.py", line 476, in readline

    data = self._sock.recv(self._rbufsize)

socket.timeout: timed out

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in 
/home/users/pdobrogost/tmp/x
Storing debug log for failure in /home/users/pdobrogost/.pip/pip.log
(ve)pdobrogost at host:~/tmp/x$

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140206/4549ecd0/attachment-0001.html>

From eugene at sazhin.us  Thu Feb  6 23:00:24 2014
From: eugene at sazhin.us (Evgeny Sazhin)
Date: Thu, 6 Feb 2014 17:00:24 -0500
Subject: [Distutils] wheel building help needed
In-Reply-To: <CADiSq7cKk=y6J5Z2PEPTOGWBQZNvnB_4VUzyDmKQAP67-VTP-Q@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
 <CADiSq7cKk=y6J5Z2PEPTOGWBQZNvnB_4VUzyDmKQAP67-VTP-Q@mail.gmail.com>
Message-ID: <CAH3X-tnJTU=DZZ_QEV-g18f1HXhWd2=1kFX9v1gr6HptvP-1-A@mail.gmail.com>

On Thu, Feb 6, 2014 at 4:54 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Evgeny, while you can probably make wheels do what you want, if you're
> interested in single file executables, you're almost certainly better off
> using one of the tools designed to make those easier to work with (like
> Twitter's recently discussed PEX format, or PEP 441).
>
> As you have discovered, cross platform single file executables are an area
> where the currently published documentation is even less polished and
> coherent than that for the rest of the packaging infrastructure :)
>
> Cheers,
> Nick.


Nick,

Thank you for pointing that out. I took a brief look at the pyz stuff
and while it might be good generally - it doesn't help me much. There
are two problems i'm trying to solve : dependency management and
jar-like behavior (as libraries or executables). Pip + wheel combo
seems to be geared better for that task (may be as you guys are
pointing out, not intentionally, but still seems to be better
approach).

We will definitely take a look at PEX as from the very scattered info
i got so far they seem to be very good at dependency management - so
that might be an option.


Thanks,
Eugene

From wickman at gmail.com  Thu Feb  6 23:23:13 2014
From: wickman at gmail.com (Brian Wickman)
Date: Thu, 6 Feb 2014 14:23:13 -0800
Subject: [Distutils] wheel building help needed
In-Reply-To: <CAH3X-tnJTU=DZZ_QEV-g18f1HXhWd2=1kFX9v1gr6HptvP-1-A@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
 <CADiSq7cKk=y6J5Z2PEPTOGWBQZNvnB_4VUzyDmKQAP67-VTP-Q@mail.gmail.com>
 <CAH3X-tnJTU=DZZ_QEV-g18f1HXhWd2=1kFX9v1gr6HptvP-1-A@mail.gmail.com>
Message-ID: <CAAmsm4jw+_EsGZyrpscSviiy0AxdJ65X02vbUGSaqNmcRnezkw@mail.gmail.com>

Happy to answer any specific questions about PEX here or off-thread.  I'm
hacking on wheel support in my spare time -- hope to ship that sometime in
the next couple weeks.  The other big upcoming change is standardizing
docstrings throughout so that we can get a proper API published on
readthedocs.

~brian


On Thu, Feb 6, 2014 at 2:00 PM, Evgeny Sazhin <eugene at sazhin.us> wrote:

> On Thu, Feb 6, 2014 at 4:54 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Evgeny, while you can probably make wheels do what you want, if you're
> > interested in single file executables, you're almost certainly better off
> > using one of the tools designed to make those easier to work with (like
> > Twitter's recently discussed PEX format, or PEP 441).
> >
> > As you have discovered, cross platform single file executables are an
> area
> > where the currently published documentation is even less polished and
> > coherent than that for the rest of the packaging infrastructure :)
> >
> > Cheers,
> > Nick.
>
>
> Nick,
>
> Thank you for pointing that out. I took a brief look at the pyz stuff
> and while it might be good generally - it doesn't help me much. There
> are two problems i'm trying to solve : dependency management and
> jar-like behavior (as libraries or executables). Pip + wheel combo
> seems to be geared better for that task (may be as you guys are
> pointing out, not intentionally, but still seems to be better
> approach).
>
> We will definitely take a look at PEX as from the very scattered info
> i got so far they seem to be very good at dependency management - so
> that might be an option.
>
>
> Thanks,
> Eugene
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140206/1d26d62d/attachment.html>

From eugene at sazhin.us  Thu Feb  6 23:40:37 2014
From: eugene at sazhin.us (Evgeny Sazhin)
Date: Thu, 6 Feb 2014 17:40:37 -0500
Subject: [Distutils] wheel building help needed
In-Reply-To: <CAAmsm4jw+_EsGZyrpscSviiy0AxdJ65X02vbUGSaqNmcRnezkw@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
 <CADiSq7cKk=y6J5Z2PEPTOGWBQZNvnB_4VUzyDmKQAP67-VTP-Q@mail.gmail.com>
 <CAH3X-tnJTU=DZZ_QEV-g18f1HXhWd2=1kFX9v1gr6HptvP-1-A@mail.gmail.com>
 <CAAmsm4jw+_EsGZyrpscSviiy0AxdJ65X02vbUGSaqNmcRnezkw@mail.gmail.com>
Message-ID: <CAH3X-tmOpw7GET-pK-WzpLieQFFevr9vgkJZ-CqMbdnKDWGseg@mail.gmail.com>

Brian,

that's very kind of you - i will definitely ping you for questions,
but after my small vacation;)

Thanks,
Eugene

On Thu, Feb 6, 2014 at 5:23 PM, Brian Wickman <wickman at gmail.com> wrote:
> Happy to answer any specific questions about PEX here or off-thread.  I'm
> hacking on wheel support in my spare time -- hope to ship that sometime in
> the next couple weeks.  The other big upcoming change is standardizing
> docstrings throughout so that we can get a proper API published on
> readthedocs.
>
> ~brian
>
>
> On Thu, Feb 6, 2014 at 2:00 PM, Evgeny Sazhin <eugene at sazhin.us> wrote:
>>
>> On Thu, Feb 6, 2014 at 4:54 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> > Evgeny, while you can probably make wheels do what you want, if you're
>> > interested in single file executables, you're almost certainly better
>> > off
>> > using one of the tools designed to make those easier to work with (like
>> > Twitter's recently discussed PEX format, or PEP 441).
>> >
>> > As you have discovered, cross platform single file executables are an
>> > area
>> > where the currently published documentation is even less polished and
>> > coherent than that for the rest of the packaging infrastructure :)
>> >
>> > Cheers,
>> > Nick.
>>
>>
>> Nick,
>>
>> Thank you for pointing that out. I took a brief look at the pyz stuff
>> and while it might be good generally - it doesn't help me much. There
>> are two problems i'm trying to solve : dependency management and
>> jar-like behavior (as libraries or executables). Pip + wheel combo
>> seems to be geared better for that task (may be as you guys are
>> pointing out, not intentionally, but still seems to be better
>> approach).
>>
>> We will definitely take a look at PEX as from the very scattered info
>> i got so far they seem to be very good at dependency management - so
>> that might be an option.
>>
>>
>> Thanks,
>> Eugene
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>

From eugene at sazhin.us  Fri Feb  7 01:13:26 2014
From: eugene at sazhin.us (Evgeny Sazhin)
Date: Thu, 6 Feb 2014 19:13:26 -0500
Subject: [Distutils] wheel building help needed
In-Reply-To: <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>
References: <CAH3X-tm6GecOaXeag41CsZHv7GfdF3hVq8CntaNKD85M=5n6=Q@mail.gmail.com>
 <CACac1F9pwqNC+jSLVPVYSrObdfgun+Whty6Yd2GkV7zQUsO3Yg@mail.gmail.com>
 <CAH3X-tkgZUmzghEw8goKcn7prxQx4vP=_gbQbXzXFTL+=zJYCg@mail.gmail.com>
 <CAG8k2+4mKf=bE07JGXgJRHg8rMzJUVxbTirksZUe4904GVBW4g@mail.gmail.com>
Message-ID: <CAH3X-tnz+vBGZwmd1+TUEaqv+jY32n_oG+k=Taqvid3CfvHrNQ@mail.gmail.com>

On Wed, Feb 5, 2014 at 9:10 PM, Daniel Holth <dholth at gmail.com> wrote:
> You are probably just missing the package_dir argument to setup(),
> adequately documented at
> http://docs.python.org/2/distutils/examples.html
>
> Pass
>
>       package_dir={'': 'src'},
>
> to indicate that the root module starts in 'src'.
>
> MANIFEST.in is needed. It is always a bit awkward to write.


I tried this and it did not work. It keeps missing the python files.
The only way it worked and created the expected wheel ( at least the
python files are there from the package and the universal=1 resulted
in correct tag) was when the package dir was directly in the root.

I also noticed another problem that in manifest

include . *.py

doesn't work, neither does the

include *

So my __main__.py now located in the root doesn't get inside the wheel.

So you said PEX is worth checking, huh?...;)

Thanks,
Eugene


>
> On Wed, Feb 5, 2014 at 6:41 PM, Evgeny Sazhin <eugene at sazhin.us> wrote:
>> On Wed, Feb 5, 2014 at 6:01 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>> On 5 February 2014 22:20, Evgeny Sazhin <eugene at sazhin.us> wrote:
>>>> 1. No python files added, neither __main__.py nor prog.py make it to
>>>> the wheel, so includes are not working
>>>
>>> The manifest is for sdists, not wheels. So includes won't "work" in
>>> the sense that you seem to be expecting. You need to specify your
>>> package details in setup.py - you don't say if you did. I suspect
>>> you're not getting a universal wheel because it looks like it isn't
>>> pure Python - maybe because it doesn't have the right files in it, I
>>> can't tell from what you're showing.
>>
>> I got rid of the wheel folder as the machinery behind the wheel
>> building (or between keyboard and chair) is not smart enough to
>> understand how to work with packages that are one level down from the
>> root.
>>
>> my setup.py is :
>>
>> setup(
>> name="prog",
>> version="0.1",
>> #etc,
>> packages=find_packages()
>>
>> )
>>
>> I got the wheel properly built using python setup.py bdist_wheel (i
>> was able to do it before, but didn't realize the manifest stuff is for
>> that way of building, not for the pip wheel). So "pip wheel" usage
>> remains to be unclear...
>>
>>>
>>> You still seem to be making wheels with the intention that they are
>>> executable. I apologise if I'm reading too much into your directory
>>> structure, but if that's what you are trying to do I can well imagine
>>> that you'll get odd results as you're trying to do something the
>>> system wasn't designed for.
>>>
>>> Paul
>>
>> Yes, not only i'm making wheel executable, but i also will most
>> probably build around the fact that it is zip-readable and therefore I
>> will add them to the PYTHONPATH to run, just like java classpath and
>> jar (I know - stubborn me) - that's the only thing that makes any
>> sense! After all those arguments about C extensions and whatnot i
>> still think that enforcing the unzipping and deploying stuff is not
>> what packaging should provide as a solution.
>> Plus as you see I'm trying to build universal wheel - pure python.
>> Nobody so far could prove or at least explain why it is bad idea *for
>> pure python*, so I think this is the way to go.
>>
>>
>> Thanks,
>> Eugene
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig

From debjyoti95 at gmail.com  Fri Feb  7 17:59:58 2014
From: debjyoti95 at gmail.com (Debjyoti Bhaumik)
Date: Fri, 7 Feb 2014 22:29:58 +0530
Subject: [Distutils] Pypi Openid login problem
Message-ID: <CAF5wJyEMv52joBG-eA9jHL_PfFvjPU8S3wWTxMir6XWVkmDE1A@mail.gmail.com>

Hi!
I am trying to login(register) to pypi with my google account. But I am
getting back this error:

OpenID response has been tampered with:NotAuthenticated('Replay attack
detected', 9)


I am using chrome on Linux.

I tried to login from different pages, both
this:https://pypi.python.org/pypi?:action=openid

But the results are same.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140207/7dfab428/attachment.html>

From ewdurbin at gmail.com  Sat Feb  8 23:15:21 2014
From: ewdurbin at gmail.com (Ernest W. Durbin III)
Date: Sat, 8 Feb 2014 17:15:21 -0500
Subject: [Distutils] PyPI Rate Limiting
Message-ID: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>

Since the launch of the new infrastructure for PyPI two weeks ago, I?ve been monitoring overall performance and reliability of PyPI for browsers, uploads, installers, and mirrors.

Overall I am very happy, but have noticed an ongoing issue with latency spikes and 5xx errors. I believe these issues are not new, but we don?t have any of the logs or monitoring that came along with the new infrastructure.

The cause of these issues is very apparently mirroring clients hitting PyPI with floods of requests at common cron intervals. Additionally, new mirrors coming online and performing their initial sync can easily cause extended periods of increased latency and errors for all users, especially if the number of workers configured to perform the sync is turned up.

At 2014-02-07 at about 00:00 UTC PyPI was effectively DoS?d for 45 minutes while a major research lab performing a sync via bandersnatch. It appears their worker count may have been configured as high as 50.

The design of PEP 381 mirroring clients requires calls to the PyPI XMLRPC to obtain changelogs and package serial numbers. As such, when clients are configured for high parallelism our backends can be quickly overwhelmed.

In order to maintain quality of service for all clients, we will begin rate limiting requests to the following routes:

  - /pypi
  - /mirrors
  - /id
  - /oauth
  - /security

The initial rates will be limited to 5 req/s per IP with bursts of 10 requests allowed. Client requests up to the burst limit will be delayed to maintain a 5 req/s maximum. Any requests past the 10 request burst will receive an HTTP 429 response code per RFC 6585.

Tuning these parameters will be painless, so if issues arise with mirroring clients we will be very responsive to necessary modifications.

Note that the routes used by installation clients (`/packages` and `/simple`) will remain unaffected as they are generally served from the CDN, and do not have as high of an overhead in our backend processes.

This rate-limiting is to be considered an interim solution, as I plan to begin a discussion on some updates to mirroring infrastructure guidelines.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140208/f289b30d/attachment.sig>

From robertc at robertcollins.net  Sun Feb  9 03:25:20 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Sun, 9 Feb 2014 15:25:20 +1300
Subject: [Distutils] PyPI Rate Limiting
In-Reply-To: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
References: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
Message-ID: <CAJ3HoZ2FPyi6zQn=uxS8WUGFep3YQR4DcObUVQjjs754xf-EYg@mail.gmail.com>

On 9 February 2014 11:15, Ernest W. Durbin III <ewdurbin at gmail.com> wrote:
> Since the launch of the new infrastructure for PyPI two weeks ago, I've been monitoring overall performance and reliability of PyPI for browsers, uploads, installers, and mirrors.
> The initial rates will be limited to 5 req/s per IP with bursts of 10 requests allowed. Client requests up to the burst limit will be delayed to maintain a 5 req/s maximum. Any requests past the 10 request burst will receive an HTTP 429 response code per RFC 6585.

5/s sounds really low - if the RPC's take less than 200ms to answer
(and I sure hope they do), a single threaded mirroring client (with
low latency to PyPI's servers // pipelined requests) can easily it.
Most folk I know writing API servers aim for response times in the
single to low 10's of ms digits... What is the 95% percentile for PyPI
to answer these problematic APIs ?

Can our infrastructure restrict concurrency etc (e.g. if we have
haproxy it can trivially limit by concurrency rather than rate)? That
would be IMO a better metric for overload.

> Tuning these parameters will be painless, so if issues arise with mirroring clients we will be very responsive to necessary modifications.
>
> Note that the routes used by installation clients (`/packages` and `/simple`) will remain unaffected as they are generally served from the CDN, and do not have as high of an overhead in our backend processes.
>
> This rate-limiting is to be considered an interim solution, as I plan to begin a discussion on some updates to mirroring infrastructure guidelines.

Ok, cool.

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From noah at coderanger.net  Sun Feb  9 07:28:51 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sat, 8 Feb 2014 22:28:51 -0800
Subject: [Distutils] PyPI Rate Limiting
In-Reply-To: <CAJ3HoZ2FPyi6zQn=uxS8WUGFep3YQR4DcObUVQjjs754xf-EYg@mail.gmail.com>
References: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
 <CAJ3HoZ2FPyi6zQn=uxS8WUGFep3YQR4DcObUVQjjs754xf-EYg@mail.gmail.com>
Message-ID: <D5DD4183-6F3A-4747-9624-B417BE8B6D61@coderanger.net>


On Feb 8, 2014, at 6:25 PM, Robert Collins <robertc at robertcollins.net> wrote:

> On 9 February 2014 11:15, Ernest W. Durbin III <ewdurbin at gmail.com> wrote:
>> Since the launch of the new infrastructure for PyPI two weeks ago, I've been monitoring overall performance and reliability of PyPI for browsers, uploads, installers, and mirrors.
>> The initial rates will be limited to 5 req/s per IP with bursts of 10 requests allowed. Client requests up to the burst limit will be delayed to maintain a 5 req/s maximum. Any requests past the 10 request burst will receive an HTTP 429 response code per RFC 6585.
> 
> 5/s sounds really low - if the RPC's take less than 200ms to answer
> (and I sure hope they do), a single threaded mirroring client (with
> low latency to PyPI's servers // pipelined requests) can easily it.
> Most folk I know writing API servers aim for response times in the
> single to low 10's of ms digits... What is the 95% percentile for PyPI
> to answer these problematic APIs ?
> 

If you are making lots of sequential requests, you should be putting a sleep in there. "as fast as possible" isn't a design goal, it is good service for all clients.

--Noah


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140208/854622b8/attachment.sig>

From robertc at robertcollins.net  Sun Feb  9 10:13:42 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Sun, 9 Feb 2014 22:13:42 +1300
Subject: [Distutils] PyPI Rate Limiting
In-Reply-To: <D5DD4183-6F3A-4747-9624-B417BE8B6D61@coderanger.net>
References: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
 <CAJ3HoZ2FPyi6zQn=uxS8WUGFep3YQR4DcObUVQjjs754xf-EYg@mail.gmail.com>
 <D5DD4183-6F3A-4747-9624-B417BE8B6D61@coderanger.net>
Message-ID: <CAJ3HoZ1J4jGsiwmhitJ-eTzc4erDH=WnQTJuCEzUThd2fuK9vA@mail.gmail.com>

On 9 February 2014 19:28, Noah Kantrowitz <noah at coderanger.net> wrote:
>
> On Feb 8, 2014, at 6:25 PM, Robert Collins <robertc at robertcollins.net> wrote:
>

>> 5/s sounds really low - if the RPC's take less than 200ms to answer
>> (and I sure hope they do), a single threaded mirroring client (with
>> low latency to PyPI's servers // pipelined requests) can easily it.
>> Most folk I know writing API servers aim for response times in the
>> single to low 10's of ms digits... What is the 95% percentile for PyPI
>> to answer these problematic APIs ?
>>
>
> If you are making lots of sequential requests, you should be putting a sleep in there. "as fast as possible" isn't a design goal, it is good service for all clients.

As fast as possible (on the server side) and good service for all
clients are very tightly correlated (and some would say there is a
causative relationship in fact).

On the client side, I totally support limiting concurrency, but I've
yet to see a convincing explanation for rate limiting already
serialised requests that doesn't boil down to 'assume the server is
badly written'. Note - I'm not assuming - or implying - that about
PyPI.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From donald at stufft.io  Sun Feb  9 17:50:50 2014
From: donald at stufft.io (Donald Stufft)
Date: Sun, 9 Feb 2014 11:50:50 -0500
Subject: [Distutils] PyPI Rate Limiting
In-Reply-To: <CAJ3HoZ1J4jGsiwmhitJ-eTzc4erDH=WnQTJuCEzUThd2fuK9vA@mail.gmail.com>
References: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
 <CAJ3HoZ2FPyi6zQn=uxS8WUGFep3YQR4DcObUVQjjs754xf-EYg@mail.gmail.com>
 <D5DD4183-6F3A-4747-9624-B417BE8B6D61@coderanger.net>
 <CAJ3HoZ1J4jGsiwmhitJ-eTzc4erDH=WnQTJuCEzUThd2fuK9vA@mail.gmail.com>
Message-ID: <F31C4536-7346-4A6B-B0C7-8229970180F3@stufft.io>


On Feb 9, 2014, at 4:13 AM, Robert Collins <robertc at robertcollins.net> wrote:

> On 9 February 2014 19:28, Noah Kantrowitz <noah at coderanger.net> wrote:
>> 
>> On Feb 8, 2014, at 6:25 PM, Robert Collins <robertc at robertcollins.net> wrote:
>> 
> 
>>> 5/s sounds really low - if the RPC's take less than 200ms to answer
>>> (and I sure hope they do), a single threaded mirroring client (with
>>> low latency to PyPI's servers // pipelined requests) can easily it.
>>> Most folk I know writing API servers aim for response times in the
>>> single to low 10's of ms digits... What is the 95% percentile for PyPI
>>> to answer these problematic APIs ?
>>> 
>> 
>> If you are making lots of sequential requests, you should be putting a sleep in there. "as fast as possible" isn't a design goal, it is good service for all clients.
> 
> As fast as possible (on the server side) and good service for all
> clients are very tightly correlated (and some would say there is a
> causative relationship in fact).
> 
> On the client side, I totally support limiting concurrency, but I've
> yet to see a convincing explanation for rate limiting already
> serialised requests that doesn't boil down to 'assume the server is
> badly written'. Note - I'm not assuming - or implying - that about
> PyPI.
> 
> -Rob
> 
> -- 
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

PyPI is a terrible code base that *is* pretty bad. There is an effort underway to make this better, but right now assuming that is a pretty safe bet :) It?s a really old code case (12 years or so) and has never had anyone on it full time so over the years it has accumulated lots of cruft and it comes from a time when web development practices were not very good.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140209/6840d599/attachment.sig>

From alexjeffburke at gmail.com  Sun Feb  9 15:26:48 2014
From: alexjeffburke at gmail.com (Alex Burke)
Date: Sun, 9 Feb 2014 15:26:48 +0100
Subject: [Distutils] guidance for contributing to packaging docs
Message-ID: <CABT47BLu1CoFCqc8f+LTQ4Eg+-6Zqo+hcMzsEuhDWuxsn_KJ_g@mail.gmail.com>

Hi all,

I've been a regular reader of the packaging mailing list for a little while
now and, though I don't yet feel up to working on code, wanted to see if I
could help with documentation as both an exercise in my understanding as
well as being able to help others (I struggled a little mid to late last
year trying to figure out best practices for packaging Python code).

My question might seem a little broad, but where would it be most useful to
focus documentation contributions? For example I know the python packaging
guide is being fleshed out, but it's seemed somewhat in a state of flux and
keeping up with reorganisations has been a little tricky.

Any pointers or specific suggestions would be really useful. Perhaps it
would it be best to wait until Python 3.4 ships with ensurepip given how
much advicethat might affect?

Incidentally, thought I'd mention it is a very welcome surprise to those I
have spoken to about Python that it will soon have an integrated and
blessed package managing solution that ships with default builds. It seems
to be a common point of confusion or even slight or against what is
otherwise considered a fantastic language. Kudos to everyone involved.

Thanks, Alex J Burke.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140209/7fd98610/attachment.html>

From noah at coderanger.net  Sun Feb  9 21:16:30 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sun, 9 Feb 2014 12:16:30 -0800
Subject: [Distutils] PyPI Rate Limiting
In-Reply-To: <CAJ3HoZ1J4jGsiwmhitJ-eTzc4erDH=WnQTJuCEzUThd2fuK9vA@mail.gmail.com>
References: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
 <CAJ3HoZ2FPyi6zQn=uxS8WUGFep3YQR4DcObUVQjjs754xf-EYg@mail.gmail.com>
 <D5DD4183-6F3A-4747-9624-B417BE8B6D61@coderanger.net>
 <CAJ3HoZ1J4jGsiwmhitJ-eTzc4erDH=WnQTJuCEzUThd2fuK9vA@mail.gmail.com>
Message-ID: <2C9CA06B-E6E6-421F-BCDD-A91F6F06C8CB@coderanger.net>


On Feb 9, 2014, at 1:13 AM, Robert Collins <robertc at robertcollins.net> wrote:

> On 9 February 2014 19:28, Noah Kantrowitz <noah at coderanger.net> wrote:
>> 
>> On Feb 8, 2014, at 6:25 PM, Robert Collins <robertc at robertcollins.net> wrote:
>> 
> 
>>> 5/s sounds really low - if the RPC's take less than 200ms to answer
>>> (and I sure hope they do), a single threaded mirroring client (with
>>> low latency to PyPI's servers // pipelined requests) can easily it.
>>> Most folk I know writing API servers aim for response times in the
>>> single to low 10's of ms digits... What is the 95% percentile for PyPI
>>> to answer these problematic APIs ?
>>> 
>> 
>> If you are making lots of sequential requests, you should be putting a sleep in there. "as fast as possible" isn't a design goal, it is good service for all clients.
> 
> As fast as possible (on the server side) and good service for all
> clients are very tightly correlated (and some would say there is a
> causative relationship in fact).
> 
> On the client side, I totally support limiting concurrency, but I've
> yet to see a convincing explanation for rate limiting already
> serialised requests that doesn't boil down to 'assume the server is
> badly written'. Note - I'm not assuming - or implying - that about
> PyPI.

I'm not sure what point you are trying to make. The server wouldn't artificially slow down requests, it (well, nginx) would just track requests and send 503s if limits are exceeded. Requests still complete as fast as possible, and we can ensure one client doesn't hog all the server resources.

--Noah


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140209/6de4bbd5/attachment.sig>

From qwcode at gmail.com  Sun Feb  9 22:15:32 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 9 Feb 2014 13:15:32 -0800
Subject: [Distutils] guidance for contributing to packaging docs
In-Reply-To: <CABT47BLu1CoFCqc8f+LTQ4Eg+-6Zqo+hcMzsEuhDWuxsn_KJ_g@mail.gmail.com>
References: <CABT47BLu1CoFCqc8f+LTQ4Eg+-6Zqo+hcMzsEuhDWuxsn_KJ_g@mail.gmail.com>
Message-ID: <CAPYWazp_AORN5yT7KefXi320gB32BavjkqhQDFYi2SPsCMG=tA@mail.gmail.com>

Yes, the structure for the packaging guide has changed a few times, but
hopefully, <knock on wood>, we've reached the tail end of that.  It's
currently marked up with numerous FIXME comments, specifically on the
"Advanced Topics" page:
https://python-packaging-user-guide.readthedocs.org/en/latest/additional.html

Beyond the guide, there's always the individual project docs:  pip,
virtualenv, setuptools, wheel, distlib
(btw, my plan is to refactor most of those to a consistent structure and
style;  I have issues open for the refactors in the user guide tracker; I'm
currently working on pip and setuptools)

As for the python.org docs, Nick Coghlan has already added notes to the top
of the old "Installing/Distributing Python Modules" guides pointing to the
new packaging user guide.  Eventually, both of the old guides need to
disappear in their current form, and be re-organized just as a distutils
reference that covers install schemes, commands, options etc... and that's
self-aware of it's place in the current ecosystem.

As for ensurepip/PEP453, I don't think that changes too much of the user
guide documentation, except for a few places on installation, and to make
"pyenv" more prominent (which is currently mentioned in footnotes only).
PEP453 is already mentioned in a couple places in the Guide.

--Marcus



On Sun, Feb 9, 2014 at 6:26 AM, Alex Burke <alexjeffburke at gmail.com> wrote:

> Hi all,
>
> I've been a regular reader of the packaging mailing list for a little
> while now and, though I don't yet feel up to working on code, wanted to see
> if I could help with documentation as both an exercise in my understanding
> as well as being able to help others (I struggled a little mid to late last
> year trying to figure out best practices for packaging Python code).
>
> My question might seem a little broad, but where would it be most useful
> to focus documentation contributions? For example I know the python
> packaging guide is being fleshed out, but it's seemed somewhat in a state
> of flux and keeping up with reorganisations has been a little tricky.
>
> Any pointers or specific suggestions would be really useful. Perhaps it
> would it be best to wait until Python 3.4 ships with ensurepip given how
> much advicethat might affect?
>
> Incidentally, thought I'd mention it is a very welcome surprise to those I
> have spoken to about Python that it will soon have an integrated and
> blessed package managing solution that ships with default builds. It seems
> to be a common point of confusion or even slight or against what is
> otherwise considered a fantastic language. Kudos to everyone involved.
>
> Thanks, Alex J Burke.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140209/b9ec8d31/attachment-0001.html>

From ncoghlan at gmail.com  Sun Feb  9 23:14:18 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 10 Feb 2014 08:14:18 +1000
Subject: [Distutils] guidance for contributing to packaging docs
In-Reply-To: <CAPYWazp_AORN5yT7KefXi320gB32BavjkqhQDFYi2SPsCMG=tA@mail.gmail.com>
References: <CABT47BLu1CoFCqc8f+LTQ4Eg+-6Zqo+hcMzsEuhDWuxsn_KJ_g@mail.gmail.com>
 <CAPYWazp_AORN5yT7KefXi320gB32BavjkqhQDFYi2SPsCMG=tA@mail.gmail.com>
Message-ID: <CADiSq7d1uLz444ghfcWaC6+fcN313mnNc82o=zRgZqwxxrzwmA@mail.gmail.com>

On 10 Feb 2014 07:16, "Marcus Smith" <qwcode at gmail.com> wrote:
>
> Yes, the structure for the packaging guide has changed a few times, but
hopefully, <knock on wood>, we've reached the tail end of that.  It's
currently marked up with numerous FIXME comments, specifically on the
"Advanced Topics" page:
https://python-packaging-user-guide.readthedocs.org/en/latest/additional.html
>
> Beyond the guide, there's always the individual project docs:  pip,
virtualenv, setuptools, wheel, distlib
> (btw, my plan is to refactor most of those to a consistent structure and
style;  I have issues open for the refactors in the user guide tracker; I'm
currently working on pip and setuptools)
>
> As for the python.org docs, Nick Coghlan has already added notes to the
top of the old "Installing/Distributing Python Modules" guides pointing to
the new packaging user guide.  Eventually, both of the old guides need to
disappear in their current form, and be re-organized just as a distutils
reference that covers install schemes, commands, options etc... and that's
self-aware of it's place in the current ecosystem.

Right, before 3.4 final goes out next month, I aim to at least have those
refactored as "For end users" and "For tool developers".

Longer term (especially with PEP 453 in place), I'm thinking we want to
minimise the standard library's role in the build and deployment process to
providing the tools with the info they need to do the right thing for the
specific installation of Python - we've been seeing OS vendors move that
way for a while (Visual Studio, XCode, RH Developer Toolset - all have
distinct lifecycles from the underlying OS).

>
> As for ensurepip/PEP453, I don't think that changes too much of the user
guide documentation, except for a few places on installation, and to make
"pyenv" more prominent (which is currently mentioned in footnotes only).
PEP453 is already mentioned in a couple places in the Guide.

I think we still need to mention "pip" vs "pip3" for POSIX system installs
somewhere, but aside from that, I should just need two targets to link to:
one that assumes pip is already installed (for 3.4+) and one that explains
how to bootstrap pip and virtualenv (for 3.3 and earlier).

Cheers,
Nick.

>
> --Marcus
>
>
>
> On Sun, Feb 9, 2014 at 6:26 AM, Alex Burke <alexjeffburke at gmail.com>
wrote:
>>
>> Hi all,
>>
>> I've been a regular reader of the packaging mailing list for a little
while now and, though I don't yet feel up to working on code, wanted to see
if I could help with documentation as both an exercise in my understanding
as well as being able to help others (I struggled a little mid to late last
year trying to figure out best practices for packaging Python code).
>>
>> My question might seem a little broad, but where would it be most useful
to focus documentation contributions? For example I know the python
packaging guide is being fleshed out, but it's seemed somewhat in a state
of flux and keeping up with reorganisations has been a little tricky.
>>
>> Any pointers or specific suggestions would be really useful. Perhaps it
would it be best to wait until Python 3.4 ships with ensurepip given how
much advicethat might affect?
>>
>> Incidentally, thought I'd mention it is a very welcome surprise to
those I have spoken to about Python that it will soon have an integrated
and blessed package managing solution that ships with default builds. It
seems to be a common point of confusion or even slight or against what is
otherwise considered a fantastic language. Kudos to everyone involved.
>>
>> Thanks, Alex J Burke.
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140210/b1c1efc9/attachment.html>

From chris.jerdonek at gmail.com  Mon Feb 10 10:48:44 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 10 Feb 2014 01:48:44 -0800
Subject: [Distutils] PyPI Rate Limiting
In-Reply-To: <2C9CA06B-E6E6-421F-BCDD-A91F6F06C8CB@coderanger.net>
References: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
 <CAJ3HoZ2FPyi6zQn=uxS8WUGFep3YQR4DcObUVQjjs754xf-EYg@mail.gmail.com>
 <D5DD4183-6F3A-4747-9624-B417BE8B6D61@coderanger.net>
 <CAJ3HoZ1J4jGsiwmhitJ-eTzc4erDH=WnQTJuCEzUThd2fuK9vA@mail.gmail.com>
 <2C9CA06B-E6E6-421F-BCDD-A91F6F06C8CB@coderanger.net>
Message-ID: <CAOTb1weHPOEsjk7Rhon4F0QXCyWUZRPqHDYbARGqJeU-0VcdCg@mail.gmail.com>

On Sun, Feb 9, 2014 at 12:16 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
>
> On Feb 9, 2014, at 1:13 AM, Robert Collins <robertc at robertcollins.net> wrote:
>
>> On 9 February 2014 19:28, Noah Kantrowitz <noah at coderanger.net> wrote:
>>>
>>> On Feb 8, 2014, at 6:25 PM, Robert Collins <robertc at robertcollins.net> wrote:
>>>
>>
>>>> 5/s sounds really low - if the RPC's take less than 200ms to answer
>>>> (and I sure hope they do), a single threaded mirroring client (with
>>>> low latency to PyPI's servers // pipelined requests) can easily it.
>>>> Most folk I know writing API servers aim for response times in the
>>>> single to low 10's of ms digits... What is the 95% percentile for PyPI
>>>> to answer these problematic APIs ?
>>>>
>>>
>>> If you are making lots of sequential requests, you should be putting a sleep in there. "as fast as possible" isn't a design goal, it is good service for all clients.
>>
>> As fast as possible (on the server side) and good service for all
>> clients are very tightly correlated (and some would say there is a
>> causative relationship in fact).
>>
>> On the client side, I totally support limiting concurrency, but I've
>> yet to see a convincing explanation for rate limiting already
>> serialised requests that doesn't boil down to 'assume the server is
>> badly written'. Note - I'm not assuming - or implying - that about
>> PyPI.
>
> I'm not sure what point you are trying to make. The server wouldn't artificially slow down requests, it (well, nginx) would just track requests and send 503s if limits are exceeded. Requests still complete as fast as possible, and we can ensure one client doesn't hog all the server resources.

I think he's saying that, given that the problems are being caused by
"clients configured for high parallelism," why not choose a
rate-limiting method that won't impact clients accessing it in a
single-threaded fashion.  It's a reasonable question.

Also, if the server isn't artificially slowing down requests, what
does, "Client requests up to the burst limit [of 10 requests] will be
delayed to maintain a 5 req/s maximum" mean?

--Chris

From noah at coderanger.net  Mon Feb 10 10:49:53 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Mon, 10 Feb 2014 01:49:53 -0800
Subject: [Distutils] PyPI Rate Limiting
In-Reply-To: <CAOTb1weHPOEsjk7Rhon4F0QXCyWUZRPqHDYbARGqJeU-0VcdCg@mail.gmail.com>
References: <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
 <CAJ3HoZ2FPyi6zQn=uxS8WUGFep3YQR4DcObUVQjjs754xf-EYg@mail.gmail.com>
 <D5DD4183-6F3A-4747-9624-B417BE8B6D61@coderanger.net>
 <CAJ3HoZ1J4jGsiwmhitJ-eTzc4erDH=WnQTJuCEzUThd2fuK9vA@mail.gmail.com>
 <2C9CA06B-E6E6-421F-BCDD-A91F6F06C8CB@coderanger.net>
 <CAOTb1weHPOEsjk7Rhon4F0QXCyWUZRPqHDYbARGqJeU-0VcdCg@mail.gmail.com>
Message-ID: <AD6F9807-9259-45CD-92AA-5B49E623A53A@coderanger.net>


On Feb 10, 2014, at 1:48 AM, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:

> On Sun, Feb 9, 2014 at 12:16 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
>> 
>> On Feb 9, 2014, at 1:13 AM, Robert Collins <robertc at robertcollins.net> wrote:
>> 
>>> On 9 February 2014 19:28, Noah Kantrowitz <noah at coderanger.net> wrote:
>>>> 
>>>> On Feb 8, 2014, at 6:25 PM, Robert Collins <robertc at robertcollins.net> wrote:
>>>> 
>>> 
>>>>> 5/s sounds really low - if the RPC's take less than 200ms to answer
>>>>> (and I sure hope they do), a single threaded mirroring client (with
>>>>> low latency to PyPI's servers // pipelined requests) can easily it.
>>>>> Most folk I know writing API servers aim for response times in the
>>>>> single to low 10's of ms digits... What is the 95% percentile for PyPI
>>>>> to answer these problematic APIs ?
>>>>> 
>>>> 
>>>> If you are making lots of sequential requests, you should be putting a sleep in there. "as fast as possible" isn't a design goal, it is good service for all clients.
>>> 
>>> As fast as possible (on the server side) and good service for all
>>> clients are very tightly correlated (and some would say there is a
>>> causative relationship in fact).
>>> 
>>> On the client side, I totally support limiting concurrency, but I've
>>> yet to see a convincing explanation for rate limiting already
>>> serialised requests that doesn't boil down to 'assume the server is
>>> badly written'. Note - I'm not assuming - or implying - that about
>>> PyPI.
>> 
>> I'm not sure what point you are trying to make. The server wouldn't artificially slow down requests, it (well, nginx) would just track requests and send 503s if limits are exceeded. Requests still complete as fast as possible, and we can ensure one client doesn't hog all the server resources.
> 
> I think he's saying that, given that the problems are being caused by
> "clients configured for high parallelism," why not choose a
> rate-limiting method that won't impact clients accessing it in a
> single-threaded fashion.  It's a reasonable question.
> 
> Also, if the server isn't artificially slowing down requests, what
> does, "Client requests up to the burst limit [of 10 requests] will be
> delayed to maintain a 5 req/s maximum" mean?

Any requests beyond the rate limits will get an HTTP 503 with an empty body.

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140210/dbbf5025/attachment.sig>

From marius at pov.lt  Mon Feb 10 12:43:06 2014
From: marius at pov.lt (Marius Gedminas)
Date: Mon, 10 Feb 2014 13:43:06 +0200
Subject: [Distutils] PyPI Rate Limiting
In-Reply-To: <AD6F9807-9259-45CD-92AA-5B49E623A53A@coderanger.net>
 <914FE14F-C5ED-4CEF-9CBD-946B631EE290@gmail.com>
Message-ID: <20140210114306.GA2450@fridge.pov.lt>

On Sat, Feb 08, 2014 at 05:15:21PM -0500, Ernest W. Durbin III wrote:
> The initial rates will be limited to 5 req/s per IP with bursts of 10
> requests allowed. Client requests up to the burst limit will be
> delayed to maintain a 5 req/s maximum. Any requests past the 10
> request burst will receive an HTTP 429 response code per RFC 6585.

On Mon, Feb 10, 2014 at 01:49:53AM -0800, Noah Kantrowitz wrote:
> On Feb 10, 2014, at 1:48 AM, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
> > Also, if the server isn't artificially slowing down requests, what
> > does, "Client requests up to the burst limit [of 10 requests] will be
> > delayed to maintain a 5 req/s maximum" mean?
> 
> Any requests beyond the rate limits will get an HTTP 503 with an empty body.

So is it 429 or 503?  And is there a delay or not?

Marius Gedminas
-- 
Linux is a fast moving project, with very fast evolving components. If you're
using an older distribution, older than 4 to 6 months (and anything with
"Enterprise" in the name is by definition old), please consider going to a
newer distribution.
                -- http://www.linuxpowertop.org/known.php
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140210/14f5fa00/attachment.sig>

From techtonik at gmail.com  Tue Feb 11 05:35:49 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 11 Feb 2014 07:35:49 +0300
Subject: [Distutils] .. *explicitly* mark the package as pre-release
In-Reply-To: <4640B83A-03C1-4DA9-8F81-42665C2799E6@stufft.io>
References: <CAPkN8xKZwpgVyyhtRkopmPn3npV9SkXEv+e30U-p=hKTKBBAGA@mail.gmail.com>
 <CAHrZfZD8nuKK6WawpVivNHkoJTXQZ+9ONnOwXvZC1sbmhR65nA@mail.gmail.com>
 <4640B83A-03C1-4DA9-8F81-42665C2799E6@stufft.io>
Message-ID: <CAPkN8xLPV1_gPCs4EvdtoO97EQQM_m-CeKaDnOdXqGr_L6XGkw@mail.gmail.com>

Opened the feature request https://github.com/pypa/pip/issues/1553

Why people don't try to make features optional? Posted rant to python-ideas.

On Sun, Dec 1, 2013 at 1:10 AM, Donald Stufft <donald at stufft.io> wrote:
> Hiding a release doesn?t hide it from the simple index, so installers will
> still find it.
>
> On Nov 30, 2013, at 4:59 PM, Richard Jones <richard at python.org> wrote:
>
> There is no way to mark a release as "pre-release" but you can hide it from
> view which would prevent an installer discovering any files related to the
> release. Log into PyPI and use the "releases" page for the project.
>
>
>
> On 30 November 2013 22:05, anatoly techtonik <techtonik at gmail.com> wrote:
>>
>> hi, please cc
>>
>> i want to release some versions to pypi, but hide them from installing
>> by explicitly marking as pre-release. i want to preserve versions
>> numbering as 0.1, 0.2 according to semver.org point 4. is that
>> possible?
>> --
>> anatoly t.
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>



-- 
anatoly t.

From techtonik at gmail.com  Tue Feb 11 16:45:33 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 11 Feb 2014 18:45:33 +0300
Subject: [Distutils] Metadata debate for ages
Message-ID: <CAPkN8xLRdSo2-kf0qdV84+qB-H7xHeGE4T1255AVgMYZk8Zo9w@mail.gmail.com>

On Tue, Feb 11, 2014 at 4:33 PM, Paul Moore <notifications at github.com>wrote:

> and I doubt that will happen, as the metadata debate has been ongoing for
> ages, and it's *very* late to be coming up with requirements with
> significant implications out of the blue like this
>
This is from thread with my rant about versioning, which set a dumb
question in my head. What is the role of dot zero part in metadata
versioning?

I mean if there is so much debates, how about choosing a minimal first
thing that can be released? Reach consensus, release. Increase version. Go
another round of debate. Release. Test. Increment version. Release.

It seems that people are trying to make it perfect loosing the real
conflicting points in a ton of information. I am looking at 50+ pages PEP
in http://www.python.org/dev/peps/pep-0426/ and the first thing I notice
that I don't understand why there are provisions like MUST and why PEP
named "Metadata" regulates what tools should or should not do and what
tools should exists at all.

To me, the "Metadata 2.0" PEP should specify only two things:
1. format(s) of the data, how it can be represented so that people can
process it
2. structure of the data - a set of fields, their values and cases when
they are applied

That's it. The things like "Automated tools, especially public index
servers, MAY impose additional length restrictions on metadata beyond those
enumerated in this PEP. Such limits SHOULD be imposed where necessary to
protect the integrity of a service, based on the available resources and
the service provider's judgment of reasonable metadata capacity
requirements." are out of scope of Metadata at all. This stuff belongs
elsewhere. If there will be a problem that metadata grows too large - it
should be recorded as an issue, and the next version of metadata should say
- fixed issues #..., #..., ... or just include a separate PEP for those who
don't know how to handle the load on their servers.

What do you think?
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140211/6f1cffd3/attachment.html>

From jaraco at jaraco.com  Wed Feb 12 05:09:39 2014
From: jaraco at jaraco.com (Jason R. Coombs)
Date: Wed, 12 Feb 2014 04:09:39 +0000
Subject: [Distutils] Setuptools 3.0b1 available for preview
Message-ID: <ab66596f76b347bd909640796848133e@BLUPR06MB434.namprd06.prod.outlook.com>

I'm pleased to announce that Setuptools 3.0b1 is now available for download from the project downloads page:

https://bitbucket.org/pypa/setuptools/downloads

This backward-incompatible release contains the changes detailed in the CHANGES.txt file:

https://bitbucket.org/pypa/setuptools/src/3.0b1/CHANGES.txt?at=default

Please give this beta release a try or review the changes for any impact this might have on your package. I anticipate most users will find these changes of no consequence, but if there are any issues that might affect your package that don't have an obvious solution, please file a ticket.

Regards,
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140212/f03a0a52/attachment.html>

From sdouche at gmail.com  Wed Feb 12 17:11:04 2014
From: sdouche at gmail.com (Sebastien Douche)
Date: Wed, 12 Feb 2014 17:11:04 +0100
Subject: [Distutils] Setuptools 3.0b1 available for preview
In-Reply-To: <ab66596f76b347bd909640796848133e@BLUPR06MB434.namprd06.prod.outlook.com>
References: <ab66596f76b347bd909640796848133e@BLUPR06MB434.namprd06.prod.outlook.com>
Message-ID: <CAAGHeXEumRO_qJcF9PqW9LjM5pahj7YHFkFzc1X5rigQjubuRA@mail.gmail.com>

On Wed, Feb 12, 2014 at 5:09 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:

Hi Jason

> This backward-incompatible release contains the changes detailed in the
> CHANGES.txt file:

Issue #148: When building (bdist_egg), setuptools no longer adds
  ``__init__.py`` files to namespace packages. Any packages that rely on this
  behavior will need to create ``__init__.py`` files and include the
  ``declare_namespace()``.

Thing like:

try:
    __import__('pkg_resources').declare_namespace(__name__)
except ImportError:
    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)
__init__.py (END)

are theses lines good?





-- 
Sebastien Douche <sdouche at gmail.com>
Twitter: @sdouche / G+: +sdouche

From tseaver at palladion.com  Wed Feb 12 20:26:37 2014
From: tseaver at palladion.com (Tres Seaver)
Date: Wed, 12 Feb 2014 14:26:37 -0500
Subject: [Distutils] Setuptools 3.0b1 available for preview
In-Reply-To: <ab66596f76b347bd909640796848133e@BLUPR06MB434.namprd06.prod.outlook.com>
References: <ab66596f76b347bd909640796848133e@BLUPR06MB434.namprd06.prod.outlook.com>
Message-ID: <ldght0$94q$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/11/2014 11:09 PM, Jason R. Coombs wrote:

> I'm pleased to announce that Setuptools 3.0b1 is now available for 
> download from the project downloads page:
> 
> https://bitbucket.org/pypa/setuptools/downloads
> 
> This backward-incompatible release contains the changes detailed in
> the CHANGES.txt file:
> 
> https://bitbucket.org/pypa/setuptools/src/3.0b1/CHANGES.txt?at=default
>
>  Please give this beta release a try or review the changes for any
> impact this might have on your package. I anticipate most users will
> find these changes of no consequence, but if there are any issues that
> might affect your package that don't have an obvious solution, please
> file a ticket.

I just tested SubstanceD using virtualenv's built with both Python 2.6.9
and Python 3.3.3.  All tests pass, exercising a stack which includes
ZODB, the zope.* packages, pyramid, etc.

I did see one at-exit failure (during the 'setup.py develop' under
2.6.9), but it was harmless AFAICT.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlL7yuwACgkQ+gerLs4ltQ7j8gCfd34nVzU3w/GCVfaxEAzx8jKm
gPsAn0gVS/YikXf64JkSr1L/G3n+aoXR
=s2z/
-----END PGP SIGNATURE-----


From donald at stufft.io  Wed Feb 12 21:00:23 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 12 Feb 2014 15:00:23 -0500
Subject: [Distutils] PyPI Status Update - PostgreSQL Tuning and CDN Shielding
Message-ID: <C70BC6AA-E63F-41E5-8E8D-BE3F4368DE20@stufft.io>

Just a quick update on some recent PyPI changes.

We?ve attempted to tune PostgreSQL. This has resulted in a particular problem
query going from a 23s execution time to a 5s execution time. This should
reduce or remove the 503 errors that occur when the mirrors slam PyPI at
midnight UTC looking for new packages. This has reduced the bulk of the problem
with the mirrors. Assuming stability remains the same as it is now we will not
be implementing rate limiting at this time as we would greatly prefer not to
have to do that.

We?ve also activated CDN Shielding for all non POST requests. This means that
whenever you contact a node in say the singapore POP if that node does not have
the response cached instead of directly contacting the backends in Rackspace's
IAD datacenter it will instead contact the Ashburn POP. This POP will serve
a response that it has cached, or contact the backend if it does not have a
cached response. This should further speed up PyPI as it's far more likely
that the shield POP (Ashburn) will have the response cached even if the edge
location does not. Further more the connectivity between the Fastly POPs are
more likely to be lower latency then the connectivity between any random
Fastly POP and the backends.

So essentially PostgreSQL should be faster and our caching CDN should be more
efficient.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140212/c00f7e40/attachment.sig>

From pje at telecommunity.com  Thu Feb 13 19:03:31 2014
From: pje at telecommunity.com (PJ Eby)
Date: Thu, 13 Feb 2014 13:03:31 -0500
Subject: [Distutils] Setuptools 3.0b1 available for preview
In-Reply-To: <CAAGHeXEumRO_qJcF9PqW9LjM5pahj7YHFkFzc1X5rigQjubuRA@mail.gmail.com>
References: <ab66596f76b347bd909640796848133e@BLUPR06MB434.namprd06.prod.outlook.com>
 <CAAGHeXEumRO_qJcF9PqW9LjM5pahj7YHFkFzc1X5rigQjubuRA@mail.gmail.com>
Message-ID: <CALeMXf74wQeqRFHu5KCPt29kiTS_KXm=zxTROJsuuz4CzgZzoA@mail.gmail.com>

On Wed, Feb 12, 2014 at 11:11 AM, Sebastien Douche <sdouche at gmail.com> wrote:
> On Wed, Feb 12, 2014 at 5:09 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:
>
> Hi Jason
>
>> This backward-incompatible release contains the changes detailed in the
>> CHANGES.txt file:
>
> Issue #148: When building (bdist_egg), setuptools no longer adds
>   ``__init__.py`` files to namespace packages. Any packages that rely on this
>   behavior will need to create ``__init__.py`` files and include the
>   ``declare_namespace()``.
>
> Thing like:
>
> try:
>     __import__('pkg_resources').declare_namespace(__name__)
> except ImportError:
>     from pkgutil import extend_path
>     __path__ = extend_path(__path__, __name__)
> __init__.py (END)
>
> are theses lines good?

Yes.  The main idea is just that the __init__.py must set up the
namespace package, and not rely on setuptools doing the declaration
implicitly/automatically.

From sdouche at gmail.com  Thu Feb 13 19:18:35 2014
From: sdouche at gmail.com (Sebastien Douche)
Date: Thu, 13 Feb 2014 19:18:35 +0100
Subject: [Distutils] Setuptools 3.0b1 available for preview
In-Reply-To: <CALeMXf74wQeqRFHu5KCPt29kiTS_KXm=zxTROJsuuz4CzgZzoA@mail.gmail.com>
References: <ab66596f76b347bd909640796848133e@BLUPR06MB434.namprd06.prod.outlook.com>
 <CAAGHeXEumRO_qJcF9PqW9LjM5pahj7YHFkFzc1X5rigQjubuRA@mail.gmail.com>
 <CALeMXf74wQeqRFHu5KCPt29kiTS_KXm=zxTROJsuuz4CzgZzoA@mail.gmail.com>
Message-ID: <CAAGHeXEVF76Me4kn28P8-O22aF+fpo1oTtC+huBbq9zWh8BFtw@mail.gmail.com>

On Thu, Feb 13, 2014 at 7:03 PM, PJ Eby <pje at telecommunity.com> wrote:
>> are theses lines good?
>
> Yes.  The main idea is just that the __init__.py must set up the
> namespace package, and not rely on setuptools doing the declaration
> implicitly/automatically.

Understood. We use these lines from the beginning. Thanks.


-- 
Sebastien Douche <sdouche at gmail.com>
Twitter: @sdouche / G+: +sdouche

From robertc at robertcollins.net  Sat Feb 15 06:56:35 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Sat, 15 Feb 2014 18:56:35 +1300
Subject: [Distutils] mirror update bandwidth?
Message-ID: <CAJ3HoZ3HmgQECfOh1Zt_noz-f=_XX1A_YOPUVfh1ZQQ5kvU_kA@mail.gmail.com>

Just curious - whats the typical monthly bandwidth taken up pulling
packages (say every 4 hours) from pypi with bandersnatch?

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From ewdurbin at gmail.com  Sat Feb 15 16:30:16 2014
From: ewdurbin at gmail.com (Ernest W. Durbin III)
Date: Sat, 15 Feb 2014 10:30:16 -0500
Subject: [Distutils] Removal of PyPI DNS mirroring infrastructure per PEP 449
Message-ID: <9267802D-A31A-4F6E-93D7-B9D99877C3A8@gmail.com>

This morning the infrastructure staff executed on the directives in PEP 449 regarding the removal of DNS entries for the PyPI mirroring infrastructure.
 
Reference: http://www.python.org/dev/peps/pep-0449/
 
The following DNS records were removed:
 
  - a.pypi.python.org
  - b.pypi.python.org
  - c.pypi.python.org
  - d.pypi.python.org
  - e.pypi.python.org
  - f.pypi.python.org
  - g.pypi.python.org
  - last.pypi.python.org
 
The copy on the PyPI website was updated to reflect these changes https://pypi.python.org/mirrors
 
Additionally the redirection infrastructure for these hostnames was removed from our HAproxy configuration at OSUOSL.
 
All requests for cheeseshop.python.org will continute to redirect to https://pypi.python.org with their appropriate HTTP query.
 
- Ernest W. Durbin III
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140215/006180eb/attachment.sig>

From eric at trueblade.com  Mon Feb 17 04:46:32 2014
From: eric at trueblade.com (Eric V. Smith)
Date: Sun, 16 Feb 2014 22:46:32 -0500
Subject: [Distutils] buildout bootstrap question
Message-ID: <53018618.7030105@trueblade.com>

In older versions of bootstrap.py, the --setup-source option would let
me select an alternate source of ez_setup.py. I just downloaded a new
version today from http://downloads.buildout.org/2/bootstrap.py, and I
notice this option is gone. bootstrap always will download ez_setup from
http://downloads.buildout.org/2/bootstrap.py (assuming setuptools isn't
installed, which in my case it is not).

I run buildout on machines that have access to my own internal http
servers, but do not have internet access. The requirement for internet
access breaks this setup.

Is there some reason --setup-source was removed?

Thanks.
Eric.

From jim at zope.com  Mon Feb 17 12:13:53 2014
From: jim at zope.com (Jim Fulton)
Date: Mon, 17 Feb 2014 06:13:53 -0500
Subject: [Distutils] buildout bootstrap question
In-Reply-To: <53018618.7030105@trueblade.com>
References: <53018618.7030105@trueblade.com>
Message-ID: <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>

On Sun, Feb 16, 2014 at 10:46 PM, Eric V. Smith <eric at trueblade.com> wrote:
> In older versions of bootstrap.py, the --setup-source option would let
> me select an alternate source of ez_setup.py. I just downloaded a new
> version today from http://downloads.buildout.org/2/bootstrap.py, and I
> notice this option is gone. bootstrap always will download ez_setup from
> http://downloads.buildout.org/2/bootstrap.py (assuming setuptools isn't
> installed, which in my case it is not).
>
> I run buildout on machines that have access to my own internal http
> servers, but do not have internet access. The requirement for internet
> access breaks this setup.
>
> Is there some reason --setup-source was removed?

I didn't know people needed it and was trying to reduce the complexity
of the bootstrap source.  We can add it back,

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From eric at trueblade.com  Mon Feb 17 12:37:35 2014
From: eric at trueblade.com (Eric V. Smith)
Date: Mon, 17 Feb 2014 06:37:35 -0500
Subject: [Distutils] buildout bootstrap question
In-Reply-To: <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
References: <53018618.7030105@trueblade.com>
 <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
Message-ID: <5301F47F.1070708@trueblade.com>

On 2/17/2014 6:13 AM, Jim Fulton wrote:
> On Sun, Feb 16, 2014 at 10:46 PM, Eric V. Smith <eric at trueblade.com> wrote:
>> In older versions of bootstrap.py, the --setup-source option would let
>> me select an alternate source of ez_setup.py. I just downloaded a new
>> version today from http://downloads.buildout.org/2/bootstrap.py, and I
>> notice this option is gone. bootstrap always will download ez_setup from
>> http://downloads.buildout.org/2/bootstrap.py (assuming setuptools isn't
>> installed, which in my case it is not).
>>
>> I run buildout on machines that have access to my own internal http
>> servers, but do not have internet access. The requirement for internet
>> access breaks this setup.
>>
>> Is there some reason --setup-source was removed?
> 
> I didn't know people needed it and was trying to reduce the complexity
> of the bootstrap source.  We can add it back,

That would be great, thanks.

I'm trying to move dozens of buldouts to buildout 2. In the meantime, I
think I'll just hard-code my url in bootstrap.py and then see how much
more progress I can make.

Eric.

From kiorky at cryptelium.net  Mon Feb 17 16:00:54 2014
From: kiorky at cryptelium.net (kiorky)
Date: Mon, 17 Feb 2014 16:00:54 +0100
Subject: [Distutils] buildout bootstrap question
In-Reply-To: <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
References: <53018618.7030105@trueblade.com>
 <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
Message-ID: <53022426.90102@cryptelium.net>


On 17/02/2014 12:13, Jim Fulton wrote:
> On Sun, Feb 16, 2014 at 10:46 PM, Eric V. Smith <eric at trueblade.com> wrote:
>> In older versions of bootstrap.py, the --setup-source option would let
>> me select an alternate source of ez_setup.py. I just downloaded a new
>> version today from http://downloads.buildout.org/2/bootstrap.py, and I
>> notice this option is gone. bootstrap always will download ez_setup from
>> http://downloads.buildout.org/2/bootstrap.py (assuming setuptools isn't
>> installed, which in my case it is not).
>>
>> I run buildout on machines that have access to my own internal http
>> servers, but do not have internet access. The requirement for internet
>> access breaks this setup.
>>
>> Is there some reason --setup-source was removed?
> I didn't know people needed it and was trying to reduce the complexity
> of the bootstrap source.  We can add it back,
>
> Jim
>
Yep, +1, please add it back for the sake of offline & restricted environments
support.

-- 
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF
Pensez ? l?environnement. N?imprimez ce courriel que si vous en avez vraiment
besoin.




-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140217/45a67a14/attachment.sig>

From eric at trueblade.com  Tue Feb 18 12:41:31 2014
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 18 Feb 2014 06:41:31 -0500
Subject: [Distutils] buildout bootstrap question
In-Reply-To: <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
References: <53018618.7030105@trueblade.com>
 <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
Message-ID: <530346EB.7050404@trueblade.com>

On 2/17/2014 6:13 AM, Jim Fulton wrote:
> On Sun, Feb 16, 2014 at 10:46 PM, Eric V. Smith <eric at trueblade.com> wrote:
>> In older versions of bootstrap.py, the --setup-source option would let
>> me select an alternate source of ez_setup.py. I just downloaded a new
>> version today from http://downloads.buildout.org/2/bootstrap.py, and I
>> notice this option is gone. bootstrap always will download ez_setup from
>> http://downloads.buildout.org/2/bootstrap.py (assuming setuptools isn't
>> installed, which in my case it is not).
>>
>> I run buildout on machines that have access to my own internal http
>> servers, but do not have internet access. The requirement for internet
>> access breaks this setup.
>>
>> Is there some reason --setup-source was removed?
> 
> I didn't know people needed it and was trying to reduce the complexity
> of the bootstrap source.  We can add it back,

While working up a patch for this, I realize that I'll also need to add
back --download-base to pass to ez_setup.use_setuptools.

Is the best way to do this to open an issue on
https://github.com/buildout/buildout, and create a fork with my proposed
changes? I'm not sure of the normal buildout workflow.

Eric.

From jim at zope.com  Tue Feb 18 13:23:52 2014
From: jim at zope.com (Jim Fulton)
Date: Tue, 18 Feb 2014 07:23:52 -0500
Subject: [Distutils] buildout bootstrap question
In-Reply-To: <530346EB.7050404@trueblade.com>
References: <53018618.7030105@trueblade.com>
 <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
 <530346EB.7050404@trueblade.com>
Message-ID: <CAPDm-FhhyDmphNV_XRNN6f7vBTvavUCJ+ENu1vX9+J-i_GjHfA@mail.gmail.com>

On Tue, Feb 18, 2014 at 6:41 AM, Eric V. Smith <eric at trueblade.com> wrote:
> On 2/17/2014 6:13 AM, Jim Fulton wrote:
>> On Sun, Feb 16, 2014 at 10:46 PM, Eric V. Smith <eric at trueblade.com> wrote:
>>> In older versions of bootstrap.py, the --setup-source option would let
>>> me select an alternate source of ez_setup.py. I just downloaded a new
>>> version today from http://downloads.buildout.org/2/bootstrap.py, and I
>>> notice this option is gone. bootstrap always will download ez_setup from
>>> http://downloads.buildout.org/2/bootstrap.py (assuming setuptools isn't
>>> installed, which in my case it is not).
>>>
>>> I run buildout on machines that have access to my own internal http
>>> servers, but do not have internet access. The requirement for internet
>>> access breaks this setup.
>>>
>>> Is there some reason --setup-source was removed?
>>
>> I didn't know people needed it and was trying to reduce the complexity
>> of the bootstrap source.  We can add it back,
>
> While working up a patch for this,

Thanks!

> I realize that I'll also need to add
> back --download-base to pass to ez_setup.use_setuptools.

Yup.  Feel free to propose something simpler.

> Is the best way to do this to open an issue on
> https://github.com/buildout/buildout, and create a fork with my proposed
> changes?

Yes.

> I'm not sure of the normal buildout workflow.

It's pretty standard open source workflow based on pull requests.

For a non-trivial change, you'd need to sign a Zope contributor
agreement, but I'd be happy to consider this trivial. :)

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From eric at trueblade.com  Tue Feb 18 13:35:09 2014
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 18 Feb 2014 07:35:09 -0500
Subject: [Distutils] buildout bootstrap question
In-Reply-To: <CAPDm-FhhyDmphNV_XRNN6f7vBTvavUCJ+ENu1vX9+J-i_GjHfA@mail.gmail.com>
References: <53018618.7030105@trueblade.com>
 <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
 <530346EB.7050404@trueblade.com>
 <CAPDm-FhhyDmphNV_XRNN6f7vBTvavUCJ+ENu1vX9+J-i_GjHfA@mail.gmail.com>
Message-ID: <5303537D.8000602@trueblade.com>

>> I realize that I'll also need to add
>> back --download-base to pass to ez_setup.use_setuptools.
> 
> Yup.  Feel free to propose something simpler.

I opened issue #165.

> For a non-trivial change, you'd need to sign a Zope contributor
> agreement, but I'd be happy to consider this trivial. :)

Either way. I'm happy to sign a contributor agreement.

Eric.


From matt at 3scan.com  Thu Feb 20 04:51:41 2014
From: matt at 3scan.com (Matt Goodman)
Date: Wed, 19 Feb 2014 19:51:41 -0800
Subject: [Distutils] Hello Greg/Anthony
Message-ID: <CABtF0Y+oCYO0GSDicTjR9cMX_mQw=jNRAf04CrzwbmKkoT8JdQ@mail.gmail.com>

I have a couple tweaks which makes distutils more functional for the mscv
family.  Namely exposing the compiler/link flags that are necessary for
linking against Python without dependence on the full msvc import being
successful.  There are a bunch of hack-arounds oribiting this problem, and
I would be willing to do some work to smooth it over.

Is there a github repo for this, or is it a part of python core.  In short,
is there any way I can contribute this code into distutils somehow?
--Matthew Goodman
===============
3Scan Computational Choreographer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140219/e4a6ec37/attachment.html>

From yacinechaouche at yahoo.com  Wed Feb 19 09:30:15 2014
From: yacinechaouche at yahoo.com (chaouche yacine)
Date: Wed, 19 Feb 2014 00:30:15 -0800 (PST)
Subject: [Distutils] How to link to a c library ?
Message-ID: <1392798615.16827.YahooMailNeo@web125504.mail.ne1.yahoo.com>



What I was trying to accomplish
-------------------------------

I tried to upgrade numpy
# pip install --upgrade numpy.../chaouche/build/numpy/numpy/numarray/_capi.c:198:undefined reference to `log10' collect2: ld returned 1 exit status error: Command "gcc -pthread -shared -Wl,--as-needed -Wl,--no-undefined -Wl,-z,relro -Wl,-O1 -Wl,--build-id -Wl,--enable-new-dtags build/temp.linux-i686-2.7/numpy/numarray/_capi.o -L/usr/lib -Lbuild/temp.linux-i686-2.7 -lpython2.7 -o build/lib.linux-i686-2.7/numpy/numarray/_capi.so" failed with exit status 1 ---------------------------------------- Rolling back uninstall of numpy Replacing /usr/lib/python2.7/site-packages/numpy-1.6.1-py2.7.egg-info
Command /usr/bin/python -c "import setuptools;__file__='/home/chaouche/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-lppaDW-record/install-record.txt failed with error code 1
Exception information:
Traceback (most recent call last):
...
InstallationError: Command /usr/bin/python -c "import setuptools;__file__='/home/chaouche/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-lppaDW-record/install-record.txt failed with error code 1



What the problem was
--------------------

I eventually figured out that they didn't link to the math library 
where all those symbols where defined (log10, sqrt, sincos etc.). I 
found that out by manually trying the gcc command with the -lm flag and 
it compiled fine.?


My question
------------
What files do I need to edit in order to add this -lm flag and install numpy correctly ?

From tseaver at palladion.com  Thu Feb 20 16:39:34 2014
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 20 Feb 2014 10:39:34 -0500
Subject: [Distutils] Hello Greg/Anthony
In-Reply-To: <CABtF0Y+oCYO0GSDicTjR9cMX_mQw=jNRAf04CrzwbmKkoT8JdQ@mail.gmail.com>
References: <CABtF0Y+oCYO0GSDicTjR9cMX_mQw=jNRAf04CrzwbmKkoT8JdQ@mail.gmail.com>
Message-ID: <530621B6.3020205@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/19/2014 10:51 PM, Matt Goodman wrote:
> I have a couple tweaks which makes distutils more functional for the
> mscv family.  Namely exposing the compiler/link flags that are
> necessary for linking against Python without dependence on the full
> msvc import being successful.  There are a bunch of hack-arounds
> oribiting this problem, and I would be willing to do some work to
> smooth it over.
> 
> Is there a github repo for this, or is it a part of python core.  In
> short, is there any way I can contribute this code into distutils
> somehow?

distutils is in the core as part of the standard library.  The canonical
repository is at:

  http://hg.python.org/cpython/

Submitters should follow the process outlined in the Python Developers'
Guide:

  http://docs.python.org/devguide/


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlMGIbAACgkQ+gerLs4ltQ7cuQCgnfoP8R+jLp5SHncVwbD3+5ac
is0An0YZkVXX017wUEiZ1pP3WpLhul+k
=M+Em
-----END PGP SIGNATURE-----


From donald at stufft.io  Fri Feb 21 02:38:45 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 20 Feb 2014 20:38:45 -0500
Subject: [Distutils] Pip 1.5.3 and virtualenv 1.11.3 released
Message-ID: <A06DB998-9C84-4BBE-B886-9254B03510AE@stufft.io>

I'm happy to announce the release of pip 1.5.3 and virtualenv 1.11.3. These
releases have the follow changelogs:


1.5.3 (2014-02-20)
------------------

* **DEPRECATION** ``pip install --build`` and ``pip install --no-clean`` are now
 deprecated.  See Issue #906 for discussion.
* Fixed #1112. Couldn't download directly from wheel paths/urls, and when wheel
 downloads did occur using requirement specifiers, dependencies weren't
 downloaded (PR #1527)
* Fixed #1320. ``pip wheel`` was not downloading wheels that already existed (PR
 #1524)
* Fixed #1111. ``pip install --download`` was failing using local
 ``--find-links`` (PR #1524)
* Workaround for Python bug http://bugs.python.org/issue20053 (PR #1544)
* Don't pass a unicode __file__ to setup.py on Python 2.x (PR #1583)
* Verify that the Wheel version is compatible with this pip (PR #1569)


1.11.3 (2014-02-20)
-------------------

* Updated setuptools to 2.2
* Updated pip to 1.5.3

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140220/6d730329/attachment.sig>

From donald at stufft.io  Fri Feb 21 02:36:43 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 20 Feb 2014 20:36:43 -0500
Subject: [Distutils] Pip 1.5.3 and virtualenv 1.11.3 released
Message-ID: <DA2C9815-AFE9-475D-B843-8DBC9AB73504@stufft.io>

I'm happy to announce the release of pip 1.5.3 and virtualenv 1.11.3. These
releases have the follow changelogs:


1.5.3 (2014-02-20)
------------------

* **DEPRECATION** ``pip install --build`` and ``pip install --no-clean`` are now
  deprecated.  See Issue #906 for discussion.
* Fixed #1112. Couldn't download directly from wheel paths/urls, and when wheel
  downloads did occur using requirement specifiers, dependencies weren't
  downloaded (PR #1527)
* Fixed #1320. ``pip wheel`` was not downloading wheels that already existed (PR
  #1524)
* Fixed #1111. ``pip install --download`` was failing using local
  ``--find-links`` (PR #1524)
* Workaround for Python bug http://bugs.python.org/issue20053 (PR #1544)
* Don't pass a unicode __file__ to setup.py on Python 2.x (PR #1583)
* Verify that the Wheel version is compatible with this pip (PR #1569)


1.11.3 (2014-02-20)
-------------------

* Updated setuptools to 2.2
* Updated pip to 1.5.3

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140220/464d5699/attachment.sig>

From cournape at gmail.com  Fri Feb 21 09:56:57 2014
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 21 Feb 2014 08:56:57 +0000
Subject: [Distutils] How to link to a c library ?
In-Reply-To: <1392798615.16827.YahooMailNeo@web125504.mail.ne1.yahoo.com>
References: <1392798615.16827.YahooMailNeo@web125504.mail.ne1.yahoo.com>
Message-ID: <CAGY4rcUT_PJMhqKsOCs2t7y1EYKqJ0v_dbs1j39ZF0MgT9qg0w@mail.gmail.com>

On Wed, Feb 19, 2014 at 8:30 AM, chaouche yacine
<yacinechaouche at yahoo.com>wrote:

>
>
> What I was trying to accomplish
> -------------------------------
>
> I tried to upgrade numpy
> # pip install --upgrade
> numpy.../chaouche/build/numpy/numpy/numarray/_capi.c:198:undefined
> reference to `log10' collect2: ld returned 1 exit status error: Command
> "gcc -pthread -shared -Wl,--as-needed -Wl,--no-undefined -Wl,-z,relro
> -Wl,-O1 -Wl,--build-id -Wl,--enable-new-dtags
> build/temp.linux-i686-2.7/numpy/numarray/_capi.o -L/usr/lib
> -Lbuild/temp.linux-i686-2.7 -lpython2.7 -o
> build/lib.linux-i686-2.7/numpy/numarray/_capi.so" failed with exit status 1
> ---------------------------------------- Rolling back uninstall of numpy
> Replacing /usr/lib/python2.7/site-packages/numpy-1.6.1-py2.7.egg-info
> Command /usr/bin/python -c "import
> setuptools;__file__='/home/chaouche/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n',
> '\n'), __file__, 'exec'))" install --single-version-externally-managed
> --record /tmp/pip-lppaDW-record/install-record.txt failed with error code 1
> Exception information:
> Traceback (most recent call last):
> ...
> InstallationError: Command /usr/bin/python -c "import
> setuptools;__file__='/home/chaouche/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n',
> '\n'), __file__, 'exec'))" install --single-version-externally-managed
> --record /tmp/pip-lppaDW-record/install-record.txt failed with error code 1
>
>
>
> What the problem was
> --------------------
>
> I eventually figured out that they didn't link to the math library
> where all those symbols where defined (log10, sqrt, sincos etc.). I
> found that out by manually trying the gcc command with the -lm flag and
> it compiled fine.
>
>
> My question
> ------------
> What files do I need to edit in order to add this -lm flag and install
> numpy correctly ?
>

numpy build links against the math library when needed, your problem most
likely comes from somewhere else.

You should contact the numpy mailing list, and post your full build log
there (not just the last lines).

David

> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/282b60cb/attachment-0001.html>

From donald at stufft.io  Fri Feb 21 13:29:18 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 21 Feb 2014 07:29:18 -0500
Subject: [Distutils] pip 1.5.4 and virtualenv 1.11.4
Message-ID: <ED29F4DE-74ED-49B8-A623-80ABEEEA59AE@stufft.io>

I've just released pip 1.5.4 and virtualenv 1.11.4 which corrects an issue
causing pip to always display a deprecation warning when invoked.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/c11f3cc6/attachment.sig>

From p at google-groups-2014.dobrogost.net  Fri Feb 21 09:55:27 2014
From: p at google-groups-2014.dobrogost.net (Piotr Dobrogost)
Date: Fri, 21 Feb 2014 09:55:27 +0100
Subject: [Distutils] Cross-platform way to get default directory for binary
 files like console scripts?
Message-ID: <CAA6tFMv8OQxJ6kapC0xCpEMR=7zSJOaVVHJTgU+EZGjSVtXeaA@mail.gmail.com>

Hi!

I posted (https://groups.google.com/forum/#!topic/comp.lang.python/s3vLkVy2xJk)
to comp.lang.python yesterday the following question:

-------
Is there cross-platform way to get default directory for binary files
(console scripts for instance) the same way one can use sys.executable
to get path to the Python's interpreter in cross-platform way?

Context:
There's Python script which runs various tools like pip using
subprocess and we would like to make sure we run tools that accompany
Python's interpreter used to run this script. Please note that the
script may be run from within virtualenv which had not been activated
- ./venv/bin/python our_script.py
-------

In replay Oscar Benjamin showed roundabout way to get this info

In [1]: from distutils.command.install import install
In [2]: from distutils.dist import Distribution
In [3]: c = install(Distribution())
In [4]: c.finalize_
c.finalize_options  c.finalize_other    c.finalize_unix
In [4]: c.finalize_options()
In [5]: c.insta
c.install_base       c.install_headers    c.install_lib
c.install_path_file  c.install_platlib    c.install_scripts
c.install_usersite
c.install_data       c.install_layout     c.install_libbase
c.install_platbase   c.install_purelib    c.install_userbase
In [5]: c.install_scripts
Out[5]: '/usr/local/bin'


Is it really that there's no API like sys.executable to get this info
and if so are there any plans to provide such API?


Best regards,
Piotr Dobrogost

From p.f.moore at gmail.com  Fri Feb 21 14:24:36 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 21 Feb 2014 13:24:36 +0000
Subject: [Distutils] Cross-platform way to get default directory for
 binary files like console scripts?
In-Reply-To: <CAA6tFMv8OQxJ6kapC0xCpEMR=7zSJOaVVHJTgU+EZGjSVtXeaA@mail.gmail.com>
References: <CAA6tFMv8OQxJ6kapC0xCpEMR=7zSJOaVVHJTgU+EZGjSVtXeaA@mail.gmail.com>
Message-ID: <CACac1F-AqdiDEqrTdGbaahS9-=rU9R5xWSvDEgM_xLjkcaRd+Q@mail.gmail.com>

On 21 February 2014 08:55, Piotr Dobrogost
<p at google-groups-2014.dobrogost.net> wrote:
> Hi!
>
> I posted (https://groups.google.com/forum/#!topic/comp.lang.python/s3vLkVy2xJk)
> to comp.lang.python yesterday the following question:
>
> -------
> Is there cross-platform way to get default directory for binary files
> (console scripts for instance) the same way one can use sys.executable
> to get path to the Python's interpreter in cross-platform way?

sysconfig.get_path("scripts") should work. If you're on a Python too
old to have sysconfig then sorry, I've no idea (other than "you should
upgrade" :-))

Paul

From vinay_sajip at yahoo.co.uk  Fri Feb 21 14:27:15 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Fri, 21 Feb 2014 13:27:15 +0000 (GMT)
Subject: [Distutils] Cross-platform way to get default directory for
	binary files like console scripts?
In-Reply-To: <CAA6tFMv8OQxJ6kapC0xCpEMR=7zSJOaVVHJTgU+EZGjSVtXeaA@mail.gmail.com>
Message-ID: <1392989235.54773.YahooMailBasic@web172403.mail.ir2.yahoo.com>


> Is there cross-platform way to get default directory for binary files
> (console scripts for instance)

Well, there's

$ /tmp/venv/bin/python
Python 3.3.0+ (3.3:c28b0b4e872b, Mar 25 2013, 17:51:34) 
[GCC 4.6.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig; print(sysconfig.get_path('scripts'))
/tmp/venv/bin
>>> 

$ python
Python 2.7.2+ (default, Jul 20 2012, 22:15:08) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig; print(sysconfig.get_path('scripts'))
/usr/local/bin

Regards,

Vinay Sajip

From oscar.j.benjamin at gmail.com  Fri Feb 21 18:04:38 2014
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Fri, 21 Feb 2014 17:04:38 +0000
Subject: [Distutils] Cross-platform way to get default directory for
 binary files like console scripts?
In-Reply-To: <CACac1F-AqdiDEqrTdGbaahS9-=rU9R5xWSvDEgM_xLjkcaRd+Q@mail.gmail.com>
References: <CAA6tFMv8OQxJ6kapC0xCpEMR=7zSJOaVVHJTgU+EZGjSVtXeaA@mail.gmail.com>
 <CACac1F-AqdiDEqrTdGbaahS9-=rU9R5xWSvDEgM_xLjkcaRd+Q@mail.gmail.com>
Message-ID: <CAHVvXxSajh6ugj3ffD0qpuudu--sFJ6w-nwpHtq+5rKfyNjb1w@mail.gmail.com>

On 21 February 2014 13:24, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> Is there cross-platform way to get default directory for binary files
>> (console scripts for instance) the same way one can use sys.executable
>> to get path to the Python's interpreter in cross-platform way?
>
> sysconfig.get_path("scripts") should work. If you're on a Python too
> old to have sysconfig then sorry, I've no idea (other than "you should
> upgrade" :-))

Ah, well that's better.

One question though: are these guaranteed to be consistent. I was
pointing at the actual code that distutils uses when installing where
as you're pointing at a module that independently lists an
over-lapping set of data:
http://hg.python.org/cpython/file/005d0678f93c/Lib/sysconfig.py#l21
http://hg.python.org/cpython/file/005d0678f93c/Lib/distutils/command/install.py#l28

For example sysconfig defines a scheme 'osx_framework_user' that
doesn't appear in distutils.command.install and uses slightly
different paths from posix_user. Does that mean that it is
inconsistent with what distutils would do?


Oscar

From p.f.moore at gmail.com  Fri Feb 21 19:45:19 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 21 Feb 2014 18:45:19 +0000
Subject: [Distutils] Cross-platform way to get default directory for
 binary files like console scripts?
In-Reply-To: <CAHVvXxSajh6ugj3ffD0qpuudu--sFJ6w-nwpHtq+5rKfyNjb1w@mail.gmail.com>
References: <CAA6tFMv8OQxJ6kapC0xCpEMR=7zSJOaVVHJTgU+EZGjSVtXeaA@mail.gmail.com>
 <CACac1F-AqdiDEqrTdGbaahS9-=rU9R5xWSvDEgM_xLjkcaRd+Q@mail.gmail.com>
 <CAHVvXxSajh6ugj3ffD0qpuudu--sFJ6w-nwpHtq+5rKfyNjb1w@mail.gmail.com>
Message-ID: <CACac1F-78s1kSdJQSwhRMj7xuTBkEwaYJF3vGXXDFVLhA6m+0w@mail.gmail.com>

On 21 February 2014 17:04, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
> On 21 February 2014 13:24, Paul Moore <p.f.moore at gmail.com> wrote:
>>>
>>> Is there cross-platform way to get default directory for binary files
>>> (console scripts for instance) the same way one can use sys.executable
>>> to get path to the Python's interpreter in cross-platform way?
>>
>> sysconfig.get_path("scripts") should work. If you're on a Python too
>> old to have sysconfig then sorry, I've no idea (other than "you should
>> upgrade" :-))
>
> Ah, well that's better.
>
> One question though: are these guaranteed to be consistent. I was
> pointing at the actual code that distutils uses when installing where
> as you're pointing at a module that independently lists an
> over-lapping set of data:
> http://hg.python.org/cpython/file/005d0678f93c/Lib/sysconfig.py#l21
> http://hg.python.org/cpython/file/005d0678f93c/Lib/distutils/command/install.py#l28
>
> For example sysconfig defines a scheme 'osx_framework_user' that
> doesn't appear in distutils.command.install and uses slightly
> different paths from posix_user. Does that mean that it is
> inconsistent with what distutils would do?

Well, it's difficult to tell. As you say, there are "schemes" involved
- the original question isn't actually well-defined as there are
"user" and "system" installs on each platform:

>>> sysconfig.get_path("scripts")
'C:\\Apps\\Python33\\Scripts'
>>> sysconfig.get_path("scripts", "nt_user")
'C:\\Users\\Gustav\\AppData\\Roaming\\Python\\Scripts'
>>> sysconfig.get_path("scripts", "nt")
'C:\\Apps\\Python33\\Scripts'

So the best you can really say is that sysconfig lets you get the
script path for a given scheme. But distutils predates sysconfig (and
setuptools hacks the internals of distutils in ways that potentially
add even more variability) so no, I can't guarantee that distutils
might not be consistent. If the two did give different results it's
probably a bug, though. But whether it's a bug taht would get *fixed*
is yet another problem (compatibility and all that).

For the use case described by the OP, using sysconfig is probably
sensible. But the tool should likely document that it won't find
executables installed with --user, and maybe provide a flag to say to
use the user scheme rather than the default scheme, so that the user
can control things. Depending on the complexity of the tool, and the
importance of being able to cope with unusual situations, having a
config file that allows the user to specify the full path on a
program-by-program basis might be necessary.

In summary - sysconfig is what you should use if you want the
"official" answer. But the question is not as easy as it looks.

Paul

From brett at yvrsfo.ca  Fri Feb 21 20:11:45 2014
From: brett at yvrsfo.ca (Brett Cannon)
Date: Fri, 21 Feb 2014 14:11:45 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
Message-ID: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>

So I'm trying to be a good Python project owner for
https://github.com/brettcannon/caniusepython3 so that means wanting to
produce a universal wheel. While reading up on exactly what is needed I
noticed there is `wheel keygen` which feeds `wheel sign`.

But what exactly is the keygen producing? I'm assuming it's a
private/public key but there is nothing about where those keys are stored,
if I should keep them when I change machines, etc. And if this is PKI then
I would assume I would want to get my public key signed by others in some
web-of-trust to make sure that the signing is more than just a content
hash. I do have a public/private GPG key from years ago when I tried to do
the right thing and got it signed at PyCon, but once again the wheel docs
don't say anything about GPG or reusing keys, etc. The wheel docs are so
non-committal it makes it feel like that whatever `gpg keygen` produces is
really not some performance shortcut and not really something to care about
perpetuating the output of.

So am I missing something or is `wheel keygen` just an optimization?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/6f368580/attachment.html>

From donald at stufft.io  Fri Feb 21 20:17:54 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 21 Feb 2014 14:17:54 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
In-Reply-To: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
References: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
Message-ID: <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>


On Feb 21, 2014, at 2:11 PM, Brett Cannon <brett at yvrsfo.ca> wrote:

> So I'm trying to be a good Python project owner for https://github.com/brettcannon/caniusepython3 so that means wanting to produce a universal wheel. While reading up on exactly what is needed I noticed there is `wheel keygen` which feeds `wheel sign`.
> 
> But what exactly is the keygen producing? I'm assuming it's a private/public key but there is nothing about where those keys are stored, if I should keep them when I change machines, etc. And if this is PKI then I would assume I would want to get my public key signed by others in some web-of-trust to make sure that the signing is more than just a content hash. I do have a public/private GPG key from years ago when I tried to do the right thing and got it signed at PyCon, but once again the wheel docs don't say anything about GPG or reusing keys, etc. The wheel docs are so non-committal it makes it feel like that whatever `gpg keygen` produces is really not some performance shortcut and not really something to care about perpetuating the output of.
> 
> So am I missing something or is `wheel keygen` just an optimization?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

In my opinion Wheel key signing is pointless. It has no trust model based with it and it?s Wheel specific. Right now there?s not a lot of benefit to signing but I would use the gpg signing that?s build into distutils. It?s generic and works across all file types.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/4f6c147f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/4f6c147f/attachment.sig>

From brett at yvrsfo.ca  Fri Feb 21 21:02:04 2014
From: brett at yvrsfo.ca (Brett Cannon)
Date: Fri, 21 Feb 2014 15:02:04 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
In-Reply-To: <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
References: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
 <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
Message-ID: <CAP1=2W6mnC0tX+xCfD5NPmXX-o+KC4JW31S_h3LonhcdY91ahA@mail.gmail.com>

Well, I'll at least use what twine supports. =)


On Fri, Feb 21, 2014 at 2:17 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Feb 21, 2014, at 2:11 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>
> So I'm trying to be a good Python project owner for
> https://github.com/brettcannon/caniusepython3 so that means wanting to
> produce a universal wheel. While reading up on exactly what is needed I
> noticed there is `wheel keygen` which feeds `wheel sign`.
>
> But what exactly is the keygen producing? I'm assuming it's a
> private/public key but there is nothing about where those keys are stored,
> if I should keep them when I change machines, etc. And if this is PKI then
> I would assume I would want to get my public key signed by others in some
> web-of-trust to make sure that the signing is more than just a content
> hash. I do have a public/private GPG key from years ago when I tried to do
> the right thing and got it signed at PyCon, but once again the wheel docs
> don't say anything about GPG or reusing keys, etc. The wheel docs are so
> non-committal it makes it feel like that whatever `gpg keygen` produces is
> really not some performance shortcut and not really something to care about
> perpetuating the output of.
>
> So am I missing something or is `wheel keygen` just an optimization?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> In my opinion Wheel key signing is pointless. It has no trust model based
> with it and it?s Wheel specific. Right now there?s not a lot of benefit to
> signing but I would use the gpg signing that?s build into distutils. It?s
> generic and works across all file types.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/ed595c6f/attachment.html>

From donald at stufft.io  Fri Feb 21 21:16:13 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 21 Feb 2014 15:16:13 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
In-Reply-To: <CAP1=2W6mnC0tX+xCfD5NPmXX-o+KC4JW31S_h3LonhcdY91ahA@mail.gmail.com>
References: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
 <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
 <CAP1=2W6mnC0tX+xCfD5NPmXX-o+KC4JW31S_h3LonhcdY91ahA@mail.gmail.com>
Message-ID: <BCCEC9EA-DBF8-4E8B-9CCD-AF5A090A72B7@stufft.io>

Twine just uses gpg like distutils upload does. It?ll even do the signing for you if you want.

twine upload -s dist/*

On Feb 21, 2014, at 3:02 PM, Brett Cannon <brett at yvrsfo.ca> wrote:

> Well, I'll at least use what twine supports. =)
> 
> 
> On Fri, Feb 21, 2014 at 2:17 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> On Feb 21, 2014, at 2:11 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
> 
>> So I'm trying to be a good Python project owner for https://github.com/brettcannon/caniusepython3 so that means wanting to produce a universal wheel. While reading up on exactly what is needed I noticed there is `wheel keygen` which feeds `wheel sign`.
>> 
>> But what exactly is the keygen producing? I'm assuming it's a private/public key but there is nothing about where those keys are stored, if I should keep them when I change machines, etc. And if this is PKI then I would assume I would want to get my public key signed by others in some web-of-trust to make sure that the signing is more than just a content hash. I do have a public/private GPG key from years ago when I tried to do the right thing and got it signed at PyCon, but once again the wheel docs don't say anything about GPG or reusing keys, etc. The wheel docs are so non-committal it makes it feel like that whatever `gpg keygen` produces is really not some performance shortcut and not really something to care about perpetuating the output of.
>> 
>> So am I missing something or is `wheel keygen` just an optimization?
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> 
> In my opinion Wheel key signing is pointless. It has no trust model based with it and it?s Wheel specific. Right now there?s not a lot of benefit to signing but I would use the gpg signing that?s build into distutils. It?s generic and works across all file types.
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/f84edc08/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/f84edc08/attachment-0001.sig>

From dholth at gmail.com  Fri Feb 21 21:27:35 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Feb 2014 15:27:35 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
In-Reply-To: <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
References: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
 <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
Message-ID: <CAG8k2+7N2zPyDAenQETJ+dO5MrZXme=qTesvcc_WFH=1jZgCcw@mail.gmail.com>

On Fri, Feb 21, 2014 at 2:17 PM, Donald Stufft <donald at stufft.io> wrote:
>
> On Feb 21, 2014, at 2:11 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>
> So I'm trying to be a good Python project owner for
> https://github.com/brettcannon/caniusepython3 so that means wanting to
> produce a universal wheel. While reading up on exactly what is needed I
> noticed there is `wheel keygen` which feeds `wheel sign`.
>
> But what exactly is the keygen producing? I'm assuming it's a private/public
> key but there is nothing about where those keys are stored, if I should keep
> them when I change machines, etc. And if this is PKI then I would assume I
> would want to get my public key signed by others in some web-of-trust to
> make sure that the signing is more than just a content hash. I do have a
> public/private GPG key from years ago when I tried to do the right thing and
> got it signed at PyCon, but once again the wheel docs don't say anything
> about GPG or reusing keys, etc. The wheel docs are so non-committal it makes
> it feel like that whatever `gpg keygen` produces is really not some
> performance shortcut and not really something to care about perpetuating the
> output of.
>
> So am I missing something or is `wheel keygen` just an optimization?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> In my opinion Wheel key signing is pointless. It has no trust model based
> with it and it's Wheel specific. Right now there's not a lot of benefit to
> signing but I would use the gpg signing that's build into distutils. It's
> generic and works across all file types.

'wheel keygen' creates an Ed25519 key that is stored using the Python
keyring library; the private key can end up in the system keychain on
Mac, the Gnome keychain, or a json file in ~/.config depending on
available bindings. The underlying idea was to give people a way to
request a package signed by a literal key in the same way that you can
request a download has a particular sha digest. But it wasn't
developed further.

Now we hope the 'tuf' project will be able to deliver useful package signing.

From brett at yvrsfo.ca  Fri Feb 21 21:32:00 2014
From: brett at yvrsfo.ca (Brett Cannon)
Date: Fri, 21 Feb 2014 15:32:00 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
In-Reply-To: <BCCEC9EA-DBF8-4E8B-9CCD-AF5A090A72B7@stufft.io>
References: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
 <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
 <CAP1=2W6mnC0tX+xCfD5NPmXX-o+KC4JW31S_h3LonhcdY91ahA@mail.gmail.com>
 <BCCEC9EA-DBF8-4E8B-9CCD-AF5A090A72B7@stufft.io>
Message-ID: <CAP1=2W4Y+ubXoTkgec+W9nyY-P2ogjbxtb7c2JhDb_F6bno0PA@mail.gmail.com>

Well, the docs gave the gpg command to use and made the good point that
doing so meant not typing your GPG passphrase into a strange app. Anyway,
https://pypi.python.org/pypi/caniusepython3 is now live and has both an
sdist and universal wheel which are both signed with my creaky GPG key.


On Fri, Feb 21, 2014 at 3:16 PM, Donald Stufft <donald at stufft.io> wrote:

> Twine just uses gpg like distutils upload does. It?ll even do the signing
> for you if you want.
>
> twine upload -s dist/*
>
> On Feb 21, 2014, at 3:02 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>
> Well, I'll at least use what twine supports. =)
>
>
> On Fri, Feb 21, 2014 at 2:17 PM, Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Feb 21, 2014, at 2:11 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>>
>> So I'm trying to be a good Python project owner for
>> https://github.com/brettcannon/caniusepython3 so that means wanting to
>> produce a universal wheel. While reading up on exactly what is needed I
>> noticed there is `wheel keygen` which feeds `wheel sign`.
>>
>> But what exactly is the keygen producing? I'm assuming it's a
>> private/public key but there is nothing about where those keys are stored,
>> if I should keep them when I change machines, etc. And if this is PKI then
>> I would assume I would want to get my public key signed by others in some
>> web-of-trust to make sure that the signing is more than just a content
>> hash. I do have a public/private GPG key from years ago when I tried to do
>> the right thing and got it signed at PyCon, but once again the wheel docs
>> don't say anything about GPG or reusing keys, etc. The wheel docs are so
>> non-committal it makes it feel like that whatever `gpg keygen` produces is
>> really not some performance shortcut and not really something to care about
>> perpetuating the output of.
>>
>> So am I missing something or is `wheel keygen` just an optimization?
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> In my opinion Wheel key signing is pointless. It has no trust model based
>> with it and it?s Wheel specific. Right now there?s not a lot of benefit to
>> signing but I would use the gpg signing that?s build into distutils. It?s
>> generic and works across all file types.
>>
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
>> DCFA
>>
>>
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/d58564a3/attachment.html>

From dholth at gmail.com  Fri Feb 21 21:33:18 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Feb 2014 15:33:18 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
In-Reply-To: <CAP1=2W4Y+ubXoTkgec+W9nyY-P2ogjbxtb7c2JhDb_F6bno0PA@mail.gmail.com>
References: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
 <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
 <CAP1=2W6mnC0tX+xCfD5NPmXX-o+KC4JW31S_h3LonhcdY91ahA@mail.gmail.com>
 <BCCEC9EA-DBF8-4E8B-9CCD-AF5A090A72B7@stufft.io>
 <CAP1=2W4Y+ubXoTkgec+W9nyY-P2ogjbxtb7c2JhDb_F6bno0PA@mail.gmail.com>
Message-ID: <CAG8k2+4O2CQpdORwNX1SGDbPB5bpt8WT01g10-8k65McPrvBFA@mail.gmail.com>

Woot

On Fri, Feb 21, 2014 at 3:32 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
> Well, the docs gave the gpg command to use and made the good point that
> doing so meant not typing your GPG passphrase into a strange app. Anyway,
> https://pypi.python.org/pypi/caniusepython3 is now live and has both an
> sdist and universal wheel which are both signed with my creaky GPG key.
>
>
> On Fri, Feb 21, 2014 at 3:16 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>> Twine just uses gpg like distutils upload does. It'll even do the signing
>> for you if you want.
>>
>> twine upload -s dist/*
>>
>> On Feb 21, 2014, at 3:02 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>>
>> Well, I'll at least use what twine supports. =)
>>
>>
>> On Fri, Feb 21, 2014 at 2:17 PM, Donald Stufft <donald at stufft.io> wrote:
>>>
>>>
>>> On Feb 21, 2014, at 2:11 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>>>
>>> So I'm trying to be a good Python project owner for
>>> https://github.com/brettcannon/caniusepython3 so that means wanting to
>>> produce a universal wheel. While reading up on exactly what is needed I
>>> noticed there is `wheel keygen` which feeds `wheel sign`.
>>>
>>> But what exactly is the keygen producing? I'm assuming it's a
>>> private/public key but there is nothing about where those keys are stored,
>>> if I should keep them when I change machines, etc. And if this is PKI then I
>>> would assume I would want to get my public key signed by others in some
>>> web-of-trust to make sure that the signing is more than just a content hash.
>>> I do have a public/private GPG key from years ago when I tried to do the
>>> right thing and got it signed at PyCon, but once again the wheel docs don't
>>> say anything about GPG or reusing keys, etc. The wheel docs are so
>>> non-committal it makes it feel like that whatever `gpg keygen` produces is
>>> really not some performance shortcut and not really something to care about
>>> perpetuating the output of.
>>>
>>> So am I missing something or is `wheel keygen` just an optimization?
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>> In my opinion Wheel key signing is pointless. It has no trust model based
>>> with it and it's Wheel specific. Right now there's not a lot of benefit to
>>> signing but I would use the gpg signing that's build into distutils. It's
>>> generic and works across all file types.
>>>
>>> -----------------
>>> Donald Stufft
>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
>>> DCFA
>>>
>>
>>
>>
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
>> DCFA
>>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From donald at stufft.io  Fri Feb 21 21:34:56 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 21 Feb 2014 15:34:56 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
In-Reply-To: <CAP1=2W4Y+ubXoTkgec+W9nyY-P2ogjbxtb7c2JhDb_F6bno0PA@mail.gmail.com>
References: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
 <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
 <CAP1=2W6mnC0tX+xCfD5NPmXX-o+KC4JW31S_h3LonhcdY91ahA@mail.gmail.com>
 <BCCEC9EA-DBF8-4E8B-9CCD-AF5A090A72B7@stufft.io>
 <CAP1=2W4Y+ubXoTkgec+W9nyY-P2ogjbxtb7c2JhDb_F6bno0PA@mail.gmail.com>
Message-ID: <06EA4F1B-E21D-4445-9805-AE28DD075D90@stufft.io>

Yea it does that too :) I was just being too lazy to type the docs out again :)

On Feb 21, 2014, at 3:32 PM, Brett Cannon <brett at yvrsfo.ca> wrote:

> Well, the docs gave the gpg command to use and made the good point that doing so meant not typing your GPG passphrase into a strange app. Anyway, https://pypi.python.org/pypi/caniusepython3 is now live and has both an sdist and universal wheel which are both signed with my creaky GPG key.
> 
> 
> On Fri, Feb 21, 2014 at 3:16 PM, Donald Stufft <donald at stufft.io> wrote:
> Twine just uses gpg like distutils upload does. It?ll even do the signing for you if you want.
> 
> twine upload -s dist/*
> 
> On Feb 21, 2014, at 3:02 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
> 
>> Well, I'll at least use what twine supports. =)
>> 
>> 
>> On Fri, Feb 21, 2014 at 2:17 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>> On Feb 21, 2014, at 2:11 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>> 
>>> So I'm trying to be a good Python project owner for https://github.com/brettcannon/caniusepython3 so that means wanting to produce a universal wheel. While reading up on exactly what is needed I noticed there is `wheel keygen` which feeds `wheel sign`.
>>> 
>>> But what exactly is the keygen producing? I'm assuming it's a private/public key but there is nothing about where those keys are stored, if I should keep them when I change machines, etc. And if this is PKI then I would assume I would want to get my public key signed by others in some web-of-trust to make sure that the signing is more than just a content hash. I do have a public/private GPG key from years ago when I tried to do the right thing and got it signed at PyCon, but once again the wheel docs don't say anything about GPG or reusing keys, etc. The wheel docs are so non-committal it makes it feel like that whatever `gpg keygen` produces is really not some performance shortcut and not really something to care about perpetuating the output of.
>>> 
>>> So am I missing something or is `wheel keygen` just an optimization?
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>> 
>> In my opinion Wheel key signing is pointless. It has no trust model based with it and it?s Wheel specific. Right now there?s not a lot of benefit to signing but I would use the gpg signing that?s build into distutils. It?s generic and works across all file types.
>> 
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
>> 
> 
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/d7cd5e31/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/d7cd5e31/attachment-0001.sig>

From brett at yvrsfo.ca  Fri Feb 21 21:40:18 2014
From: brett at yvrsfo.ca (Brett Cannon)
Date: Fri, 21 Feb 2014 15:40:18 -0500
Subject: [Distutils] what exactly does `wheel keygen` w/ `sign` do?
In-Reply-To: <06EA4F1B-E21D-4445-9805-AE28DD075D90@stufft.io>
References: <CAP1=2W4S=gk04Y7=48vZtYRPBkL3nx6gZMzFkyu09daR39XPYQ@mail.gmail.com>
 <55E8D80A-5E27-424F-B361-184C94FFC205@stufft.io>
 <CAP1=2W6mnC0tX+xCfD5NPmXX-o+KC4JW31S_h3LonhcdY91ahA@mail.gmail.com>
 <BCCEC9EA-DBF8-4E8B-9CCD-AF5A090A72B7@stufft.io>
 <CAP1=2W4Y+ubXoTkgec+W9nyY-P2ogjbxtb7c2JhDb_F6bno0PA@mail.gmail.com>
 <06EA4F1B-E21D-4445-9805-AE28DD075D90@stufft.io>
Message-ID: <CAP1=2W7NKUPPYC6E2-DF=M1B0bE6mdNTS3gSru8=JZhv=Zbrng@mail.gmail.com>

Hey, I'm just glad that `twine upload dist/*` with the .asc files in that
directory did the right thing. Something to mention if you ever decide to
update the docs.


On Fri, Feb 21, 2014 at 3:34 PM, Donald Stufft <donald at stufft.io> wrote:

> Yea it does that too :) I was just being too lazy to type the docs out
> again :)
>
> On Feb 21, 2014, at 3:32 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>
> Well, the docs gave the gpg command to use and made the good point that
> doing so meant not typing your GPG passphrase into a strange app. Anyway,
> https://pypi.python.org/pypi/caniusepython3 is now live and has both an
> sdist and universal wheel which are both signed with my creaky GPG key.
>
>
> On Fri, Feb 21, 2014 at 3:16 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> Twine just uses gpg like distutils upload does. It?ll even do the signing
>> for you if you want.
>>
>> twine upload -s dist/*
>>
>> On Feb 21, 2014, at 3:02 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>>
>> Well, I'll at least use what twine supports. =)
>>
>>
>> On Fri, Feb 21, 2014 at 2:17 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>>>
>>> On Feb 21, 2014, at 2:11 PM, Brett Cannon <brett at yvrsfo.ca> wrote:
>>>
>>> So I'm trying to be a good Python project owner for
>>> https://github.com/brettcannon/caniusepython3 so that means wanting to
>>> produce a universal wheel. While reading up on exactly what is needed I
>>> noticed there is `wheel keygen` which feeds `wheel sign`.
>>>
>>> But what exactly is the keygen producing? I'm assuming it's a
>>> private/public key but there is nothing about where those keys are stored,
>>> if I should keep them when I change machines, etc. And if this is PKI then
>>> I would assume I would want to get my public key signed by others in some
>>> web-of-trust to make sure that the signing is more than just a content
>>> hash. I do have a public/private GPG key from years ago when I tried to do
>>> the right thing and got it signed at PyCon, but once again the wheel docs
>>> don't say anything about GPG or reusing keys, etc. The wheel docs are so
>>> non-committal it makes it feel like that whatever `gpg keygen` produces is
>>> really not some performance shortcut and not really something to care about
>>> perpetuating the output of.
>>>
>>> So am I missing something or is `wheel keygen` just an optimization?
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>> In my opinion Wheel key signing is pointless. It has no trust model
>>> based with it and it?s Wheel specific. Right now there?s not a lot of
>>> benefit to signing but I would use the gpg signing that?s build into
>>> distutils. It?s generic and works across all file types.
>>>
>>> -----------------
>>> Donald Stufft
>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
>>> DCFA
>>>
>>>
>>
>>
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
>> DCFA
>>
>>
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140221/d6df21e1/attachment.html>

From ncoghlan at gmail.com  Sun Feb 23 10:49:10 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 23 Feb 2014 19:49:10 +1000
Subject: [Distutils] The PEP 426 defined metadata version will be metadata
	3.0
Message-ID: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>

Vinay brought to my attention that bdist_wheel currently publishes
pydist.json files that claim to contain content in the metadata 2.0
format. I was aware that Daniel had been experimenting with generating
pydist.json files, but I didn't realise that bdist_wheel did it by
default, nor that it used the real file name from the PEP rather than
a separate implementation specific file.

That's broken: it means that metadata format 2.0 can now never be well
defined, because we already have these experimental (and ultimately
non-conformant, since there have been backwards incompatible changes
to the draft spec) files floating around.

I will accordingly be updating the defined metadata version in PEP 426
to 3.0, and including an explicit admonition to *never* include
experimental metadata (whether in the base format or as part of an
experimental extension) in the main pydist.json file. Experimental
metadata should only ever appear in tool-specific files (which don't
need to guarantee any kind of interoperability with other tools).

In the meantime: please don't publish pydist.json files, use some
other filename if you want to experiment with JSON based metadata in
advance of the acceptance of PEP 426.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Feb 23 12:43:15 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 23 Feb 2014 21:43:15 +1000
Subject: [Distutils] Deleting legacy contents from core install guide?
Message-ID: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>

I finally started working on the docs.python.org updates for PEP 453
tonight, and I'm just not seeing much salvageable information in
http://docs.python.org/3/install/index.html.

While it was a big step forward back in 1998, it seems to me that it
has simply been left unmaintained for too long, and I think leaving
that content in there for 3.4 will do more harm than good. The content
won't be lost: not only will it continue to exist in source control,
but I won't be touching the 2.x docs at
http://docs.python.org/2/install/index.html.

With that content gone from the end user installation guide in 3.4+,
it can become just a very, very simple introduction to the core pip
commands, and then a reference out to the packaging user guide for
more info.

Thoughts?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Sun Feb 23 13:00:52 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 23 Feb 2014 12:00:52 +0000
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
Message-ID: <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>

On 23 February 2014 11:43, Nick Coghlan <ncoghlan at gmail.com> wrote:
> With that content gone from the end user installation guide in 3.4+,
> it can become just a very, very simple introduction to the core pip
> commands, and then a reference out to the packaging user guide for
> more info.
>
> Thoughts?

Are "setup.py install" and "setup.py build" being officially relegated
to undocumented status? That's a genuine question, I don't have a
particular reason to object if the answer is "yes".

But if they aren't, will there still be documentation anywhere that covers:

1. Running setup.py install will install your project.[1]
2. The locations of the distutils.cfg files.
3. How to use mingw on Windows, including setting the compiler in
distutils.cfg.[3]

[1] Ideally, that should also explain what this *doesn't* do in
comparison to pip. But that's a matter of enhancing the documentation
rather than removing legacy stuff.
[2] Given that mingw support has some long-standing bugs and issues,
it may be that it doesn't even work any more. But if that's true, we
should officially desupport it and mark all the relevant bugs as
closed/wontfix. If it *is* supported, though, we need to make sure
that somewhere we explain how to build with mingw. (At the moment, pip
says nothing on the matter, relying on the user setting his default
compiler appropriately, which means via the instructions in this
section, I guess). That may be something for the PUG, so moving this
content to the PUG might be reasonable.

I agree on the general principle of getting rid of stuff that's
outdated and we don't want to support, but "it's still in the 3.3 docs
and source control" isn't quite enough - we need to be clear that we
are effectively making stuff undocumented.

Paul

From ncoghlan at gmail.com  Sun Feb 23 13:21:13 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 23 Feb 2014 22:21:13 +1000
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
Message-ID: <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>

On 23 February 2014 22:00, Paul Moore <p.f.moore at gmail.com> wrote:
> On 23 February 2014 11:43, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> With that content gone from the end user installation guide in 3.4+,
>> it can become just a very, very simple introduction to the core pip
>> commands, and then a reference out to the packaging user guide for
>> more info.
>>
>> Thoughts?
>
> Are "setup.py install" and "setup.py build" being officially relegated
> to undocumented status? That's a genuine question, I don't have a
> particular reason to object if the answer is "yes".
>
> But if they aren't, will there still be documentation anywhere that covers:
>
> 1. Running setup.py install will install your project.[1]
> 2. The locations of the distutils.cfg files.
> 3. How to use mingw on Windows, including setting the compiler in
> distutils.cfg.[3]
>
> [1] Ideally, that should also explain what this *doesn't* do in
> comparison to pip. But that's a matter of enhancing the documentation
> rather than removing legacy stuff.
> [2] Given that mingw support has some long-standing bugs and issues,
> it may be that it doesn't even work any more. But if that's true, we
> should officially desupport it and mark all the relevant bugs as
> closed/wontfix. If it *is* supported, though, we need to make sure
> that somewhere we explain how to build with mingw. (At the moment, pip
> says nothing on the matter, relying on the user setting his default
> compiler appropriately, which means via the instructions in this
> section, I guess). That may be something for the PUG, so moving this
> content to the PUG might be reasonable.

You're right it still needs to be documented, but the problem I ran
into is that it no longer has any place in a document like the install
guide, which is primarily aimed at end users, including sysadmins that
may not care about Python in particular, but just want to know how to
install Python packages to a build root so they can create a platform
specific package.

However, after thinking a little more on the challenge, I believe the
lower level legacy content should fit right in as a new subsection in
the distutils docs at http://docs.python.org/3/distutils/index.html.

Cleaning *that* up can then be part of the longer term project of
sorting out the build side of the story, letting me focus on the
install side for the 3.4 docs.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p at google-groups-2014.dobrogost.net  Thu Feb 20 16:09:19 2014
From: p at google-groups-2014.dobrogost.net (Piotr Dobrogost)
Date: Thu, 20 Feb 2014 07:09:19 -0800 (PST)
Subject: [Distutils] Can't find "Handling the binary dependency management
 problem" thread using Google Groups interface
Message-ID: <21aeb722-69c1-45fd-af79-17a3c5fe0a2b@googlegroups.com>

Hi!

I found thread titled "Handling the binary dependency management problem" 
at 
https://mail.python.org/pipermail/distutils-sig/2013-December/023145.html 
and was trying to find it using Google Groups interface using this search 
https://groups.google.com/forum/#!searchin/distutils-sig/Handling$20the$20binary$20dependency$20management$20problem 
but with no luck.
Does someone know why I can't find this thread on Google Groups?


Best regards,
Piotr Dobrogost

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140220/c05d5d0f/attachment.html>

From techtonik at gmail.com  Sun Feb 23 08:46:39 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Sun, 23 Feb 2014 10:46:39 +0300
Subject: [Distutils] 503 errors from PyPI
In-Reply-To: <6C403E28-E543-4ABE-9B83-43446B44EAF1@stufft.io>
References: <CACac1F8UW8uF33ZWMcY8rYxUDFVOPCoskZZOnF-GKPSRzNHvKA@mail.gmail.com>
 <A2E35664-2428-4A28-B057-A1BE9D1D0C6C@stufft.io>
 <CACac1F_gSr9iLndL4_hEYf9GpQHM-eGH_EYjFVU5sAZBE2bm0w@mail.gmail.com>
 <F697C185-5E5D-4AD6-94C6-484E00C6DB12@stufft.io>
 <CACac1F_t1h5HvkB63rxyv10r-oQm1HmB_4+hCJAwbVc5w_e_FA@mail.gmail.com>
 <6C403E28-E543-4ABE-9B83-43446B44EAF1@stufft.io>
Message-ID: <CAPkN8x+rKjzLqDoQ3CLoMsXCvHXCGWktymXmF_rcpW1GtcykyQ@mail.gmail.com>

Getting this right now on all pypi pages

  Error 503 backend read error
  backend read error
  Guru Meditation:
  XID: 1718559534
  ________________________________
  Varnish cache server

http://status.python.org/ is silent.
Something is wrong with monitoring.

On Tue, Jun 18, 2013 at 3:40 PM, Donald Stufft <donald at stufft.io> wrote:
>
> On Jun 18, 2013, at 8:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
> On 18 June 2013 13:31, Donald Stufft <donald at stufft.io> wrote:
>>
>> Odd both of those work fine for me, investigating.
>
>
> It's working now - must have been an intermittent thing. Sorry for the
> noise.
> Paul
>
>
> Ok!
>
> For what it's worth. From my searching the "backend read error" just means
> the connection between the Fastly CDN server and PyPI ended erroneously. So
> it was probably just some network blips.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
anatoly t.

From qwcode at gmail.com  Sun Feb 23 22:48:35 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 23 Feb 2014 13:48:35 -0800
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
 <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
Message-ID: <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>

>
>
> However, after thinking a little more on the challenge, I believe the
> lower level legacy content should fit right in as a new subsection in
> the distutils docs at http://docs.python.org/3/distutils/index.html.
>

There's 3 things right now:

1. http://docs.python.org/3/distutils/index.html    ("Installing Python
Modules")
2. http://docs.python.org/3/install/index.html  ("Distributing Python
Modules")
3. http://docs.python.org/3/library/distutils.html  ("28.15. distutils")

I was thinking that #1 and #2 should disappear (or just provide a link to
the PUG).
I don't think you should bother to recreate any more guides in the
python.org docs.
The valuable "reference" content in #1 and #2 should be refactored into #3.

I'm willing to work on a patch for this, if you agree.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140223/ca5ba4e7/attachment-0001.html>

From qwcode at gmail.com  Sun Feb 23 22:50:07 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 23 Feb 2014 13:50:07 -0800
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
 <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
 <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
Message-ID: <CAPYWazocER1w9fyaF22rTL-DmUC5vo5YF00ffindX-B19TpeqQ@mail.gmail.com>

>
>
> 1. http://docs.python.org/3/distutils/index.html    ("Installing Python
> Modules")
> 2. http://docs.python.org/3/install/index.html  ("Distributing Python
> Modules")
>

woops, I reversed the links for these.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140223/b141cc9a/attachment.html>

From ncoghlan at gmail.com  Sun Feb 23 23:07:00 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 24 Feb 2014 08:07:00 +1000
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
 <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
 <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
Message-ID: <CADiSq7fvHg72X+SGF2UPtKy_eiHyEmfDL+qneYo1Vs+YhkYOJQ@mail.gmail.com>

On 24 Feb 2014 07:48, "Marcus Smith" <qwcode at gmail.com> wrote:
>>
>>
>> However, after thinking a little more on the challenge, I believe the
>> lower level legacy content should fit right in as a new subsection in
>> the distutils docs at http://docs.python.org/3/distutils/index.html.
>
>
> There's 3 things right now:
>
> 1. http://docs.python.org/3/distutils/index.html    ("Installing Python
Modules")
> 2. http://docs.python.org/3/install/index.html  ("Distributing Python
Modules")
> 3. http://docs.python.org/3/library/distutils.html  ("28.15. distutils")
>
> I was thinking that #1 and #2 should disappear (or just provide a link to
the PUG).
> I don't think you should bother to recreate any more guides in the
python.org docs.
> The valuable "reference" content in #1 and #2 should be refactored into
#3.

I'd like to keep 1 & 2, but in the form of a high level introduction to
their respective areas, including the difference between the standard
library and third party modules, and the role redistributors play in the
Python ecosystem. Aside from the fact I think that information is useful to
end users, and needs to be conveyed in the official docs to give it the
necessary authority,  my other rationale for that is that I would like to
keep the respective links in the top level overview page for the Python
docs, as well as avoid breaking any existing deep links to those pages. It
also ensures we have at least basic documentation in the offline help files.

> I'm willing to work on a patch for this, if you agree.

I do like the idea of moving most of the distutils docs into the distutils
module reference.

If you were willing to handle that task, that would be brilliant - I'd then
focus on writing *new* content for the installation and distribution
overview pages that introduces open source development principles, PyPI,
the Python Packaging Authority and the Python Packaging User Guide.

Cheers,
Nick.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140224/74b379fa/attachment.html>

From qwcode at gmail.com  Sun Feb 23 23:16:35 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 23 Feb 2014 14:16:35 -0800
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CADiSq7fvHg72X+SGF2UPtKy_eiHyEmfDL+qneYo1Vs+YhkYOJQ@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
 <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
 <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
 <CADiSq7fvHg72X+SGF2UPtKy_eiHyEmfDL+qneYo1Vs+YhkYOJQ@mail.gmail.com>
Message-ID: <CAPYWazok4KqzDRLS1W-jzFYRN_na9ZRnv-_QVwkrzKQGRhwM3A@mail.gmail.com>

>
>
> > I'm willing to work on a patch for this, if you agree.
>
> I do like the idea of moving most of the distutils docs into the distutils
> module reference.
>
> If you were willing to handle that task, that would be brilliant - I'd
> then focus on writing *new* content
>

ok, I'll see what I can come up with.

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140223/c4c5bd9d/attachment.html>

From chris.jerdonek at gmail.com  Sun Feb 23 23:32:19 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sun, 23 Feb 2014 14:32:19 -0800
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
 <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
 <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
Message-ID: <CAOTb1wdftWgfPHQ0jK2_OtPBOW3apT7d9Bt1tMOv9KA05P53Ng@mail.gmail.com>

On Sun, Feb 23, 2014 at 1:48 PM, Marcus Smith <qwcode at gmail.com> wrote:
>>
>> However, after thinking a little more on the challenge, I believe the
>> lower level legacy content should fit right in as a new subsection in
>> the distutils docs at http://docs.python.org/3/distutils/index.html.
>
>
> There's 3 things right now:
>
> 1. http://docs.python.org/3/distutils/index.html    ("Installing Python
> Modules")
> 2. http://docs.python.org/3/install/index.html  ("Distributing Python
> Modules")
> 3. http://docs.python.org/3/library/distutils.html  ("28.15. distutils")
>
> I was thinking that #1 and #2 should disappear (or just provide a link to
> the PUG).
> I don't think you should bother to recreate any more guides in the
> python.org docs.
> The valuable "reference" content in #1 and #2 should be refactored into #3.

To be explicit about one aspect of this process, I think content we
want documented going forward should only be removed once the
information has been added in its replacement form.  For example, for
stuff moving to PUG, I don't think we should remove a section from
"Distributing Python Modules" if the information we want there is not
yet in PUG.  A lot of PUG is still in a skeletal state.

By the way, I noticed that PUG's "Installation & Packaging Tutorial"
[1] is one section.  Shouldn't those be separate tutorials?  IIRC that
information was in separate sections before (one for installing and
one for packaging).

--Chris

[1] http://python-packaging-user-guide.readthedocs.org/en/latest/tutorial.html

From qwcode at gmail.com  Sun Feb 23 23:50:01 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 23 Feb 2014 14:50:01 -0800
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CAOTb1wdftWgfPHQ0jK2_OtPBOW3apT7d9Bt1tMOv9KA05P53Ng@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
 <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
 <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
 <CAOTb1wdftWgfPHQ0jK2_OtPBOW3apT7d9Bt1tMOv9KA05P53Ng@mail.gmail.com>
Message-ID: <CAPYWazr37ikAvq4WM72iPGwHmAnPk=aYm_1qOLQOUwuOj9c_VQ@mail.gmail.com>

>
>
> > I was thinking that #1 and #2 should disappear (or just provide a link to
> > the PUG).
> > I don't think you should bother to recreate any more guides in the
> > python.org docs.
> > The valuable "reference" content in #1 and #2 should be refactored into
> #3.
>
> To be explicit about one aspect of this process, I think content we
> want documented going forward should only be removed once the
> information has been added in its replacement form.  For example, for
> stuff moving to PUG, I don't think we should remove a section from
> "Distributing Python Modules" if the information we want there is not
> yet in PUG.
>

I agree, but again, my thinking is to refactor a lot of the content in #1
and #2, into #3, not necessarily move it to the PUG.
For example, details on the distutils install schemes and config files
would be in #3, not directly in the PUG.


> By the way, I noticed that PUG's "Installation & Packaging Tutorial"
> [1] is one section.  Shouldn't those be separate tutorials?  IIRC that
> information was in separate sections before (one for installing and
> one for packaging).
>

that's gone back and forth a bit.  at first it was 2 separate tutorials,
then 2 tutorials and quickstart, then just a quickstart, and I just
recently re-labeled the quickstart as a "tutorial" after a dicussion on
IRC, where it seemed odd to have a "quickstart" without a more detailed
tutorial later.   the tutorial is still developing.  maybe it could be
split into 2 pages later.   I'm more concerned with actually getting
content now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140223/44715316/attachment-0001.html>

From chris.jerdonek at gmail.com  Mon Feb 24 00:11:39 2014
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sun, 23 Feb 2014 15:11:39 -0800
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CAPYWazr37ikAvq4WM72iPGwHmAnPk=aYm_1qOLQOUwuOj9c_VQ@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
 <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
 <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
 <CAOTb1wdftWgfPHQ0jK2_OtPBOW3apT7d9Bt1tMOv9KA05P53Ng@mail.gmail.com>
 <CAPYWazr37ikAvq4WM72iPGwHmAnPk=aYm_1qOLQOUwuOj9c_VQ@mail.gmail.com>
Message-ID: <CAOTb1weBKGyehrTFgmhcySR3ixXgV878LkUsNaXbaUdXTpSMpA@mail.gmail.com>

On Sun, Feb 23, 2014 at 2:50 PM, Marcus Smith <qwcode at gmail.com> wrote:
>>
>> By the way, I noticed that PUG's "Installation & Packaging Tutorial"
>> [1] is one section.  Shouldn't those be separate tutorials?  IIRC that
>> information was in separate sections before (one for installing and
>> one for packaging).
>
>
> that's gone back and forth a bit.  at first it was 2 separate tutorials,
> then 2 tutorials and quickstart, then just a quickstart, and I just recently
> re-labeled the quickstart as a "tutorial" after a dicussion on IRC, where it
> seemed odd to have a "quickstart" without a more detailed tutorial later.
> the tutorial is still developing.  maybe it could be split into 2 pages
> later.   I'm more concerned with actually getting content now.

Yeah, that history still shows.  The first section called "Install the
Tools" is only about installing the tools needed for installation.
Later sections have you installing other "tools" like wheel and twine
when you get to the parts about packaging.  There is also no clear
separation between when the page switches from the installation parts
to the packaging parts (e.g. so users interested only in installing
can see more immediately what they need to be reading).

--Chris

From qwcode at gmail.com  Mon Feb 24 00:36:52 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 23 Feb 2014 15:36:52 -0800
Subject: [Distutils] Deleting legacy contents from core install guide?
In-Reply-To: <CAOTb1weBKGyehrTFgmhcySR3ixXgV878LkUsNaXbaUdXTpSMpA@mail.gmail.com>
References: <CADiSq7ek-yD1dsVwxtDmUkThczHHC9VzTvL+QjBKTggifSi4fg@mail.gmail.com>
 <CACac1F_6Wk1U8ZXXvr_-5LcO9rYa50-C8fPJcfjJpfcYH+ytyg@mail.gmail.com>
 <CADiSq7f8b9LXz9c6jOS9=o45P_bL3eDevsoNrry7yQV+iKot0A@mail.gmail.com>
 <CAPYWazpfJohs=RMknAcCBjx6uGt4=dFL2H-Zp=iSSrK1vgZdww@mail.gmail.com>
 <CAOTb1wdftWgfPHQ0jK2_OtPBOW3apT7d9Bt1tMOv9KA05P53Ng@mail.gmail.com>
 <CAPYWazr37ikAvq4WM72iPGwHmAnPk=aYm_1qOLQOUwuOj9c_VQ@mail.gmail.com>
 <CAOTb1weBKGyehrTFgmhcySR3ixXgV878LkUsNaXbaUdXTpSMpA@mail.gmail.com>
Message-ID: <CAPYWazqEEAbOnbYuW7fqmjwVZGXpRA1Q=Sg5rc5HTXgr7pja1w@mail.gmail.com>

>
>
> Yeah, that history still shows.  The first section called "Install the
> Tools" is only about installing the tools needed for installation.
>

Well, for packaging, minimally you'll need setuptools, and we recommend
using get-pip or pip to install setuptools.
And if you're going to work in "develop mode",  you'll often do that in a
virtual environment, and you'll need pip to get virtualenv (at least using
a "normal" installation of virtualenv)

But what we can do is add all the "tools" to this section (i.e. add twine
and wheel) to this section, and be more clear on what's needed for what.


>  There is also no clear
> separation between when the page switches from the installation parts
> to the packaging parts


I hear you, but otoh, if a section says "Create your own project", it's
about creating a project, not installation.
I guess I'd personally like to see it filled out more before deciding on
chopping it up *again*.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140223/49f31a61/attachment.html>

From maphew at gmail.com  Mon Feb 24 08:06:52 2014
From: maphew at gmail.com (Matt Wilkie)
Date: Sun, 23 Feb 2014 23:06:52 -0800
Subject: [Distutils] Can't find "Handling the binary dependency
 management problem" thread using Google Groups interface
In-Reply-To: <CANGB9XRnnfzC1hEOWZ=UvzRmy9z3R9E7mC0ch5rJA+7C4_RDkQ@mail.gmail.com>
References: <21aeb722-69c1-45fd-af79-17a3c5fe0a2b@googlegroups.com>
 <CANGB9XRnnfzC1hEOWZ=UvzRmy9z3R9E7mC0ch5rJA+7C4_RDkQ@mail.gmail.com>
Message-ID: <CANGB9XSGhUBp+p-8RMVu1--md4G1_SvNQZgu_91t32w5m6EO3g@mail.gmail.com>

...well now this is very curious. My reply to you bounced, because I'm not
subscribed to the google group version... but somehow I got your message in
the first place??

Manually CC'ing this one to distutils-sig at python.org, because I don't think
anyone there is aware of the problem.

-matt


On Sun, Feb 23, 2014 at 11:00 PM, Matt Wilkie <maphew at gmail.com> wrote:

> Hi Piotr,
>
> As best I can tell, it's because the Google Groups distutils-sig mailing
> list (distutils-sig at googlegroups.com) is a completely different list from
> this one (Distutils-SIG at python.org). As a matter of fact, looking at the
> compact view, it appears as if you have been the only active member since
> April of 2012!  http://i.imgur.com/z1E8Gxu.png
>
> Ahhh, here is why:
> https://groups.google.com/d/topic/distutils-sig/C2_VRYFO85s/discussion
>
> The google group distutils-sig was masquerading as a single member of the
> mailing list (and perhaps acted as a gateway?), that was automatically
> unsubscribed from the mailing list in April 2012 due to excessive bounces.
>
> Maybe someone here will remember who/how that mirror was setup and can get
> it reinstated, or removed from public view so it doesn't cause any further
> confusion.
>
> cheers,
>
> -matt
>
>
> On Thu, Feb 20, 2014 at 7:09 AM, Piotr Dobrogost <
> p at google-groups-2014.dobrogost.net> wrote:
>
>> Hi!
>>
>> I found thread titled "Handling the binary dependency management problem"
>> at
>> https://mail.python.org/pipermail/distutils-sig/2013-December/023145.htmland was trying to find it using Google Groups interface using this search
>> https://groups.google.com/forum/#!searchin/distutils-sig/Handling$20the$20binary$20dependency$20management$20problembut with no luck.
>> Does someone know why I can't find this thread on Google Groups?
>>
>>
>> Best regards,
>> Piotr Dobrogost
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140223/b912bfa6/attachment.html>

From p.f.moore at gmail.com  Mon Feb 24 08:22:09 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 24 Feb 2014 07:22:09 +0000
Subject: [Distutils] Can't find "Handling the binary dependency
 management problem" thread using Google Groups interface
In-Reply-To: <CANGB9XSGhUBp+p-8RMVu1--md4G1_SvNQZgu_91t32w5m6EO3g@mail.gmail.com>
References: <21aeb722-69c1-45fd-af79-17a3c5fe0a2b@googlegroups.com>
 <CANGB9XRnnfzC1hEOWZ=UvzRmy9z3R9E7mC0ch5rJA+7C4_RDkQ@mail.gmail.com>
 <CANGB9XSGhUBp+p-8RMVu1--md4G1_SvNQZgu_91t32w5m6EO3g@mail.gmail.com>
Message-ID: <CACac1F-yMgGzmAt-9h9wuUBRhDvgzhJ99jzZpqCFr9_ftKG=Fg@mail.gmail.com>

On 24 February 2014 07:06, Matt Wilkie <maphew at gmail.com> wrote:
> ...well now this is very curious. My reply to you bounced, because I'm not
> subscribed to the google group version... but somehow I got your message in
> the first place??

That happens to me too - I think Google Groups messes with the
reply-to header, breaking it. It also breaks threading and filtering,
in my experience.

> Manually CC'ing this one to distutils-sig at python.org, because I don't think
> anyone there is aware of the problem.

The Google Groups interfaces are known to be weird (I'm being polite
here when I want to say "broken"). I don't know about reading through
GG or missing messages, but certainly I'd recommend never posting by
that route.

Paul

From nad at acm.org  Mon Feb 24 09:09:23 2014
From: nad at acm.org (Ned Deily)
Date: Mon, 24 Feb 2014 00:09:23 -0800
Subject: [Distutils] Can't find "Handling the binary dependency
	management problem" thread using Google Groups interface
References: <21aeb722-69c1-45fd-af79-17a3c5fe0a2b@googlegroups.com>
 <CANGB9XRnnfzC1hEOWZ=UvzRmy9z3R9E7mC0ch5rJA+7C4_RDkQ@mail.gmail.com>
 <CANGB9XSGhUBp+p-8RMVu1--md4G1_SvNQZgu_91t32w5m6EO3g@mail.gmail.com>
 <CACac1F-yMgGzmAt-9h9wuUBRhDvgzhJ99jzZpqCFr9_ftKG=Fg@mail.gmail.com>
Message-ID: <nad-08DBE2.00092324022014@news.gmane.org>

In article 
<CACac1F-yMgGzmAt-9h9wuUBRhDvgzhJ99jzZpqCFr9_ftKG=Fg at mail.gmail.com>,
 Paul Moore <p.f.moore at gmail.com> wrote:
> The Google Groups interfaces are known to be weird (I'm being polite
> here when I want to say "broken"). I don't know about reading through
> GG or missing messages, but certainly I'd recommend never posting by
> that route.

Another option to using the mailing list or GG:

http://dir.gmane.org/gmane.comp.python.distutils.devel

-- 
 Ned Deily,
 nad at acm.org


From techtonik at gmail.com  Mon Feb 24 14:08:16 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 24 Feb 2014 16:08:16 +0300
Subject: [Distutils] Reopen the issue about --pre opt-out
Message-ID: <CAPkN8xL1viDFG7ZXwbu9rYDkYj+Kv-NFsBjF+qZMmkfjr02DbA@mail.gmail.com>

If anybody with authority to reopen finds
this useful, please do:
https://github.com/pypa/pip/issues/1557
-- 
anatoly t.

From robertc at robertcollins.net  Tue Feb 25 03:14:59 2014
From: robertc at robertcollins.net (Robert Collins)
Date: Tue, 25 Feb 2014 15:14:59 +1300
Subject: [Distutils] bandersnatch failing on
	https://pypi.python.org/simple/Mopidy-TuneIn/
Message-ID: <CAJ3HoZ3KrBpYTCk5yy3cfWO1zODnWyiB81ozi4pVc7AthFNpyA@mail.gmail.com>

This url is redirecting to itself, causing bandersnatch to fail, and
thus its never updating its serial, so its pulling gradually larger
and larger datasets.
https://pypi.python.org/simple/Mopidy-TuneIn
Reusing existing connection to pypi.python.org:443.
HTTP request sent, awaiting response...
  HTTP/1.1 301 Moved Permanently
  Date: Tue, 25 Feb 2014 02:14:48 GMT
  Server: Apache
  Location: /simple/Mopidy-TuneIn/
  Cache-Control: max-age=86400, public
  Strict-Transport-Security: max-age=31536000
  Content-Length: 0
  Accept-Ranges: bytes
  Age: 8204
  Keep-Alive: timeout=10, max=31
  Connection: Keep-Alive


-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From donald at stufft.io  Tue Feb 25 05:22:17 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 24 Feb 2014 23:22:17 -0500
Subject: [Distutils] bandersnatch failing on
	https://pypi.python.org/simple/Mopidy-TuneIn/
In-Reply-To: <CAJ3HoZ3KrBpYTCk5yy3cfWO1zODnWyiB81ozi4pVc7AthFNpyA@mail.gmail.com>
References: <CAJ3HoZ3KrBpYTCk5yy3cfWO1zODnWyiB81ozi4pVc7AthFNpyA@mail.gmail.com>
Message-ID: <60482F3D-C8A5-429D-ADA1-DA69BA95104F@stufft.io>

Should be fixed now.

On Feb 24, 2014, at 9:14 PM, Robert Collins <robertc at robertcollins.net> wrote:

> This url is redirecting to itself, causing bandersnatch to fail, and
> thus its never updating its serial, so its pulling gradually larger
> and larger datasets.
> https://pypi.python.org/simple/Mopidy-TuneIn
> Reusing existing connection to pypi.python.org:443.
> HTTP request sent, awaiting response...
>  HTTP/1.1 301 Moved Permanently
>  Date: Tue, 25 Feb 2014 02:14:48 GMT
>  Server: Apache
>  Location: /simple/Mopidy-TuneIn/
>  Cache-Control: max-age=86400, public
>  Strict-Transport-Security: max-age=31536000
>  Content-Length: 0
>  Accept-Ranges: bytes
>  Age: 8204
>  Keep-Alive: timeout=10, max=31
>  Connection: Keep-Alive
> 
> 
> -Rob
> 
> -- 
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140224/431aff4d/attachment.sig>

From eric at trueblade.com  Tue Feb 25 14:49:15 2014
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 25 Feb 2014 08:49:15 -0500
Subject: [Distutils] buildout bootstrap question
In-Reply-To: <5303537D.8000602@trueblade.com>
References: <53018618.7030105@trueblade.com>
 <CAPDm-FgDjsLfgW6kc8Q8EkE-gVrkF+Ub3-2+fg7FeOcQvt2uzw@mail.gmail.com>
 <530346EB.7050404@trueblade.com>
 <CAPDm-FhhyDmphNV_XRNN6f7vBTvavUCJ+ENu1vX9+J-i_GjHfA@mail.gmail.com>
 <5303537D.8000602@trueblade.com>
Message-ID: <530C9F5B.2070409@trueblade.com>

On 02/18/2014 07:35 AM, Eric V. Smith wrote:
>>> I realize that I'll also need to add
>>> back --download-base to pass to ez_setup.use_setuptools.
>>
>> Yup.  Feel free to propose something simpler.
> 
> I opened issue #165.

I added a patch to issue 165 that I believe is complete, except for
documentation and tests. I've been using it pretty extensively for a
week or so, and it does just what I need.

Before creating a pull request, I'd like to add some documentation. I
can't find any tests of bootstrap.py, so I'm not sure what to do there.
But before I spend more time on this, could someone (Jim?) pass judgment
on the code patch? I don't want to spend much more time on this if it
doesn't have a hope of getting accepted.

I addressed a few issues in the checkin message at
https://github.com/buildout/buildout/commit/e4f5809962fb3851f48dd6558a55dda0bd299563
. I'd like to make sure my decisions outlined there are acceptable, too.

Thanks.
Eric.


From donald at stufft.io  Wed Feb 26 19:00:47 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 26 Feb 2014 13:00:47 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
Message-ID: <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>


On Feb 23, 2014, at 4:49 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Vinay brought to my attention that bdist_wheel currently publishes
> pydist.json files that claim to contain content in the metadata 2.0
> format. I was aware that Daniel had been experimenting with generating
> pydist.json files, but I didn't realise that bdist_wheel did it by
> default, nor that it used the real file name from the PEP rather than
> a separate implementation specific file.
> 
> That's broken: it means that metadata format 2.0 can now never be well
> defined, because we already have these experimental (and ultimately
> non-conformant, since there have been backwards incompatible changes
> to the draft spec) files floating around.
> 
> I will accordingly be updating the defined metadata version in PEP 426
> to 3.0, and including an explicit admonition to *never* include
> experimental metadata (whether in the base format or as part of an
> experimental extension) in the main pydist.json file. Experimental
> metadata should only ever appear in tool-specific files (which don't
> need to guarantee any kind of interoperability with other tools).
> 
> In the meantime: please don't publish pydist.json files, use some
> other filename if you want to experiment with JSON based metadata in
> advance of the acceptance of PEP 426.
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

This shouldn?t matter. Tools shouldn?t be trusting a pydist.json just because
one exists and the Wheel format should have a version bump before you
can start trusting them anyways.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/7d5e12c1/attachment.sig>

From ncoghlan at gmail.com  Wed Feb 26 22:48:55 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Feb 2014 07:48:55 +1000
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
Message-ID: <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>

On 27 Feb 2014 04:00, "Donald Stufft" <donald at stufft.io> wrote:
> >
> > I will accordingly be updating the defined metadata version in PEP 426
> > to 3.0, and including an explicit admonition to *never* include
> > experimental metadata (whether in the base format or as part of an
> > experimental extension) in the main pydist.json file. Experimental
> > metadata should only ever appear in tool-specific files (which don't
> > need to guarantee any kind of interoperability with other tools).
> >
> > In the meantime: please don't publish pydist.json files, use some
> > other filename if you want to experiment with JSON based metadata in
> > advance of the acceptance of PEP 426.
> >
> > Regards,
> > Nick.
>
> This shouldn't matter. Tools shouldn't be trusting a pydist.json just
because
> one exists and the Wheel format should have a version bump before you
> can start trusting them anyways.

It's post-install and *post PEP 426 acceptance* that concerns me. Once PEP
426 is accepted, we'll need a reliable way to distinguish pydist.json files
in the standard format from these experimental prototypes.

Now, we could add some extra validity rules (such as "must come from a 1.1+
wheel file or a 2.0+ sdist"), but that seems fragile to me, since those
markers likely won't be available once the package is installed.

By contrast, skipping straight to 3.0 will make it easy for tools to
distinguish between files that are intended to be PEP compliant, and those
that may contain data in formats based on an earlier draft.

That said, I guess another alternative is to handle it through parsing
error fallbacks, and include a recommendation in PEP 426 that if
pydist.json is missing or fails to parse correctly, they should fall back
to checking for setuptools style metadata.

If Vinay is happy with that last option for handling the current
bdist_wheel misbehaviour that would mean we could leave the metadata
version alone, and just add the guidelines regarding publishing draft
metadata and handling malformed or missing metadata.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/7bc23fcf/attachment.html>

From donald at stufft.io  Wed Feb 26 22:51:58 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 26 Feb 2014 16:51:58 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
Message-ID: <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>


On Feb 26, 2014, at 4:48 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 27 Feb 2014 04:00, "Donald Stufft" <donald at stufft.io> wrote:
> > >
> > > I will accordingly be updating the defined metadata version in PEP 426
> > > to 3.0, and including an explicit admonition to *never* include
> > > experimental metadata (whether in the base format or as part of an
> > > experimental extension) in the main pydist.json file. Experimental
> > > metadata should only ever appear in tool-specific files (which don't
> > > need to guarantee any kind of interoperability with other tools).
> > >
> > > In the meantime: please don't publish pydist.json files, use some
> > > other filename if you want to experiment with JSON based metadata in
> > > advance of the acceptance of PEP 426.
> > >
> > > Regards,
> > > Nick.
> >
> > This shouldn?t matter. Tools shouldn?t be trusting a pydist.json just because
> > one exists and the Wheel format should have a version bump before you
> > can start trusting them anyways.
> 
> It's post-install and *post PEP 426 acceptance* that concerns me. Once PEP 426 is accepted, we'll need a reliable way to distinguish pydist.json files in the standard format from these experimental prototypes.
> 
> Now, we could add some extra validity rules (such as "must come from a 1.1+ wheel file or a 2.0+ sdist"), but that seems fragile to me, since those markers likely won't be available once the package is installed.
> 
> By contrast, skipping straight to 3.0 will make it easy for tools to distinguish between files that are intended to be PEP compliant, and those that may contain data in formats based on an earlier draft.
> 
> That said, I guess another alternative is to handle it through parsing error fallbacks, and include a recommendation in PEP 426 that if pydist.json is missing or fails to parse correctly, they should fall back to checking for setuptools style metadata.
> 
> If Vinay is happy with that last option for handling the current bdist_wheel misbehaviour that would mean we could leave the metadata version alone, and just add the guidelines regarding publishing draft metadata and handling malformed or missing metadata.
> 
> Cheers,
> Nick.

Well I don?t really care if we do 3.0 or 2.0 it?s just a number. I just mean that you shouldn?t parse a pydist.json inside of a Wheel unless you know it?s inside of a Wheel with Wheel-Version: Whatever-We-Formally-Add-Pydistjson-To-Wheel in.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/3bf77750/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/3bf77750/attachment.sig>

From p.f.moore at gmail.com  Wed Feb 26 23:03:21 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 26 Feb 2014 22:03:21 +0000
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
Message-ID: <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>

On 26 February 2014 21:51, Donald Stufft <donald at stufft.io> wrote:
> Well I don't really care if we do 3.0 or 2.0 it's just a number. I just mean
> that you shouldn't parse a pydist.json inside of a Wheel unless you know
> it's inside of a Wheel with Wheel-Version:
> Whatever-We-Formally-Add-Pydistjson-To-Wheel in.

pydist.json files appear in dist-info directories currently, as well.
Probably from installs of wheels (after all, installing a wheel
essentially just dumps everything into site-packages, including the
pydist.json). And there's no marker that I know of for the version of
a dist-info directory. (Actually, the WHEEL file is also in there, so
you could ignore pydist.json if there's also a WHEEL file saying
version 1.0, but that's more in the realm of heuristics than actual
standards-based rules...)

Paul

From donald at stufft.io  Wed Feb 26 23:07:12 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 26 Feb 2014 17:07:12 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
Message-ID: <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>


On Feb 26, 2014, at 5:03 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 26 February 2014 21:51, Donald Stufft <donald at stufft.io> wrote:
>> Well I don't really care if we do 3.0 or 2.0 it's just a number. I just mean
>> that you shouldn't parse a pydist.json inside of a Wheel unless you know
>> it's inside of a Wheel with Wheel-Version:
>> Whatever-We-Formally-Add-Pydistjson-To-Wheel in.
> 
> pydist.json files appear in dist-info directories currently, as well.
> Probably from installs of wheels (after all, installing a wheel
> essentially just dumps everything into site-packages, including the
> pydist.json). And there's no marker that I know of for the version of
> a dist-info directory. (Actually, the WHEEL file is also in there, so
> you could ignore pydist.json if there's also a WHEEL file saying
> version 1.0, but that's more in the realm of heuristics than actual
> standards-based rules...)
> 
> Paul

We should probably start versioning dist-info directories.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/f9eb8fc1/attachment.sig>

From donald at stufft.io  Wed Feb 26 23:38:57 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 26 Feb 2014 17:38:57 -0500
Subject: [Distutils] The PEP 426 defined metadata version will
	be	metadata 3.0
In-Reply-To: <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
Message-ID: <0F4495CE-B88B-4D97-9942-614AD00C724E@stufft.io>


On Feb 26, 2014, at 5:37 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> > If Vinay is happy with that last option for handling the current bdist_wheel misbehaviour
> > that would mean we could leave the metadata version alone
> 
> I've changed distlib to ignore pydist.json for wheels with version 1.0, where METADATA is used instead. Note that bdist_wheel also puts a metadata version of 2.0 in the METADATA file.
> 
> IMO versioning the pydist.json is equivalent to versioning the .dist-info directory - it certainly doesn't seem right to put a metadata version in the directory name itself, as it would effectively be noise almost all of the time.
> 
> I think it may be premature to promote wheels (as pythonwheels.com is doing with vigour) - nothing wrong with it in principle, of course, it's just the timing which seems wrong. Before such promotion to the community at large, It would seem advisable to get the PyPA house in order by (a) finalising the PEPs which are in limbo because of the focus on getting 3.4 out, (b) ensuring that the 1.1 spec for wheel covers any shortcomings which have been identified in 1.0, as well as more extensive testing for the implementations, and (c) revisiting accepted PEPs such as 425 to ensure that things are tightened up where needed (for example, there are some problems relating to tags, one of which IIRC Brian Wickman brought up a little while ago - about whether tags should follow the PyPy version numbering rather than the Python language version numbering).
> 
> Regards,
> 
> Vinay Sajip

It makes perfect sense to version the dist-info directory. You don?t know how to interpret the files inside that directory without it. You have to rely on heuristics and guessing.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/d9d1616f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/d9d1616f/attachment.sig>

From vinay_sajip at yahoo.co.uk  Wed Feb 26 23:37:35 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 26 Feb 2014 22:37:35 +0000 (GMT)
Subject: [Distutils] The PEP 426 defined metadata version will
	be	metadata 3.0
In-Reply-To: <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
Message-ID: <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>

>?If Vinay is happy with that last option for handling the current bdist_wheel misbehaviour
>?that would mean we could leave the metadata version alone

I've changed distlib to ignore pydist.json for wheels with version 1.0, where METADATA is used instead. Note that bdist_wheel also puts a metadata version of 2.0 in the METADATA file.

IMO versioning the pydist.json is equivalent to versioning the .dist-info directory - it certainly doesn't seem right to put a metadata version in the directory name itself, as it would effectively be noise almost all of the time.

I think it may be premature to promote wheels (as pythonwheels.com is doing with vigour) - nothing wrong with it in principle, of course, it's just the timing which seems wrong. Before such promotion to the community at large, It would seem advisable to get the PyPA house in order by (a) finalising the PEPs which are in limbo because of the focus on getting 3.4 out, (b) ensuring that the 1.1 spec for wheel covers any shortcomings which have been identified in 1.0, as well as more extensive testing for the implementations, and (c) revisiting accepted PEPs such as 425 to ensure that things are tightened up where needed (for example, there are some problems relating to tags, one of which IIRC Brian Wickman brought up a little while ago - about whether tags should follow the PyPy version numbering rather than the Python language version numbering).

Regards,


Vinay Sajip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/51ffa69c/attachment-0001.html>

From p.f.moore at gmail.com  Wed Feb 26 23:41:23 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 26 Feb 2014 22:41:23 +0000
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
Message-ID: <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>

On 26 February 2014 22:07, Donald Stufft <donald at stufft.io> wrote:
> We should probably start versioning dist-info directories.

Probably. I'm not actually sure there's a formal spec for what's in a
dist-info directory TBH. There's definitely a "plus anything else
people put there" clause in practice, but I don't know if there's a
list of the actual mandated files anywhere. PEP 426 says you put
pydist.json in there, but that's the only file I know of that's
documented.

Paul

From dholth at gmail.com  Wed Feb 26 23:52:51 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 26 Feb 2014 17:52:51 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
Message-ID: <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>

There is also the 'generator' field in pydist.json : "generator":
"bdist_wheel (0.22.0)". At the time I decided to do pydist.json I
thought I was tracking a spec that would be done any day now, but it
didn't work out that way. Do I recall an earlier complaint that I got
the filename wrong?

It's a bit silly that tools require draft-compliant pydist.json currently.

Calling it version 3 is a fine solution, and the spec has changed
enough from key/value 2.0 to json 2.0, and grown a year older, that
it's also appropriate to bump the version again. A follow on to the
unfinalized 1.3 started in 2005.

From ncoghlan at gmail.com  Thu Feb 27 00:06:36 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Feb 2014 09:06:36 +1000
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
Message-ID: <CADiSq7c8t3r1BSnsfReRZtC6=wiXYNbmvw4PVK9KKXs+OWLJSQ@mail.gmail.com>

On 27 Feb 2014 08:41, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 26 February 2014 22:07, Donald Stufft <donald at stufft.io> wrote:
> > We should probably start versioning dist-info directories.
>
> Probably. I'm not actually sure there's a formal spec for what's in a
> dist-info directory TBH. There's definitely a "plus anything else
> people put there" clause in practice, but I don't know if there's a
> list of the actual mandated files anywhere. PEP 426 says you put
> pydist.json in there, but that's the only file I know of that's
> documented.

http://www.python.org/dev/peps/pep-0376/

The installation database is one of the things listed at the top of PEP 426
as requiring an update to document the inclusion of pydist.json.

Cheers,
Nick.

>
> Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/2351e4f0/attachment.html>

From vinay_sajip at yahoo.co.uk  Thu Feb 27 00:17:46 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 26 Feb 2014 23:17:46 +0000 (GMT)
Subject: [Distutils] The PEP 426 defined metadata version will
	be	metadata 3.0
In-Reply-To: <0F4495CE-B88B-4D97-9942-614AD00C724E@stufft.io>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
 <0F4495CE-B88B-4D97-9942-614AD00C724E@stufft.io>
Message-ID: <1393456666.26535.YahooMailNeo@web172405.mail.ir2.yahoo.com>

> It makes perfect sense to version the dist-info directory.
> You don?t know how to interpret the files inside that directory without it.
> You have to rely on heuristics and guessing.



Not if you specify up front how it will work, which is doable.


It's not clear to me if you mean putting the metadata version in the directory name itself - if so, that's a -1 from me. It doesn't actually make things better as far as the PEP 376 database logic goes, given that a site-packages could have any number of dist-info dirs which could have been installed with different metadata versions.

If you mean putting some marker inside the directory, then pydist.json is as good a place as any, because as packaging evolves, the chances are that pydist.json will need to contain metadata version information telling how to process it, and there's little point in providing the information in two places. All of the legacy stuff is in separate files purely for speed of processing or through historical accident, but those are implementation details which should be under the hood as far as possible.

Regards,

Vinay Sajip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/a2a90309/attachment.html>

From donald at stufft.io  Thu Feb 27 00:22:15 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 26 Feb 2014 18:22:15 -0500
Subject: [Distutils] The PEP 426 defined metadata version will
	be	metadata 3.0
In-Reply-To: <1393456666.26535.YahooMailNeo@web172405.mail.ir2.yahoo.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
 <0F4495CE-B88B-4D97-9942-614AD00C724E@stufft.io>
 <1393456666.26535.YahooMailNeo@web172405.mail.ir2.yahoo.com>
Message-ID: <C3E3E281-62FF-44E6-9F7D-4CD2412728BC@stufft.io>


On Feb 26, 2014, at 6:17 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> > It makes perfect sense to version the dist-info directory.
> > You don?t know how to interpret the files inside that directory without it.
> > You have to rely on heuristics and guessing.
> 
> Not if you specify up front how it will work, which is doable.
> 
> It's not clear to me if you mean putting the metadata version in the directory name itself - if so, that's a -1 from me. It doesn't actually make things better as far as the PEP 376 database logic goes, given that a site-packages could have any number of dist-info dirs which could have been installed with different metadata versions.

No I don?t mean the directory name.

> 
> If you mean putting some marker inside the directory, then pydist.json is as good a place as any, because as packaging evolves, the chances are that pydist.json will need to contain metadata version information telling how to process it, and there's little point in providing the information in two places. All of the legacy stuff is in separate files purely for speed of processing or through historical accident, but those are implementation details which should be under the hood as far as possible.

There is always going to be multiple files, it?s kind of silly to tie the definition of the dist-info directory to the pydist.json when that?s perhaps not the file you care about how to interpret. How do you interpret the RECORD file? The INSTALLER file? 

The versioned definition of the dist-info directory would say, ?RECORD file is an XYZ file? ?pydist.json is an ABC file?. It?s the only sane way to handle the case where you can have an unbounded number of unknown files in the directory.

> 
> Regards,
> 
> Vinay Sajip
> 
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/c71227f7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/c71227f7/attachment-0001.sig>

From ncoghlan at gmail.com  Thu Feb 27 00:25:47 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Feb 2014 09:25:47 +1000
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
Message-ID: <CADiSq7egjPaPuNTEk8tuhaX9knbz74NoYT=VvriD+YUeGCLQSA@mail.gmail.com>

On 27 Feb 2014 08:37, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:
>
> > If Vinay is happy with that last option for handling the current
bdist_wheel misbehaviour
> > that would mean we could leave the metadata version alone
>
> I've changed distlib to ignore pydist.json for wheels with version 1.0,
where METADATA is used instead. Note that bdist_wheel also puts a metadata
version of 2.0 in the METADATA file.
>
> IMO versioning the pydist.json is equivalent to versioning the .dist-info
directory - it certainly doesn't seem right to put a metadata version in
the directory name itself, as it would effectively be noise almost all of
the time.
>
> I think it may be premature to promote wheels (as pythonwheels.com is
doing with vigour) - nothing wrong with it in principle, of course, it's
just the timing which seems wrong. Before such promotion to the community
at large, It would seem advisable to get the PyPA house in order by (a)
finalising the PEPs which are in limbo because of the focus on getting 3.4
out,

Wheels work almost as well with setuptools metadata as they will with the
initial release of the new metadata spec (because I will be removing the
metadata hook extension for the time being - I'm still not convinced that
mechanism is a good idea, and I don't think it should delay the other
benefits, like providing a format for PyPI to publish dependency metadata
directly).

> (b) ensuring that the 1.1 spec for wheel covers any shortcomings which
have been identified in 1.0, as well as more extensive testing for the
implementations, and

The wheel spec needs revision to cover more use cases. That's not a reason
to avoid encouraging it's use for the wide range of cases that it already
handles.

However, people do know that I'm not planning to write all these spec
updates myself, right? I've claimed 426/440/459, but if you wait for me to
write the others as well, we're going to be waiting a long time.

I know Donald already has a very early draft of wheel 2.0 in the works, but
I'm not aware of anyone that has even started thinking seriously about an
sdist 2.0 spec, or an update to the installation database spec - if the
latter could include an SQLite based caching mechanism, that would be
great, since it lays the foundation for allowing metaimporters to provide
installation database entries. And PEP 425 does need an update to cover at
least Linux distros (likely based on /etc/os-release).

Would it help if I created tasks for all the things that still need to be
done in the metadata formats repo, and assigned the ones I'm actually
working on to myself? That way people could see clearly where we already
know work is needed, but nobody has volunteered to do it yet.

> (c) revisiting accepted PEPs such as 425 to ensure that things are
tightened up where needed (for example, there are some problems relating to
tags, one of which IIRC Brian Wickman brought up a little while ago - about
whether tags should follow the PyPy version numbering rather than the
Python language version numbering).

PEP 425 explicitly covers that. Report bugs against tools that are
non-compliant because they expect distutils to automatically do the right
thing (when we know it doesn't, especially on Windows and in 2.x).

Cheers,
Nick.

>
> Regards,
>
> Vinay Sajip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/83d856ad/attachment.html>

From donald at stufft.io  Thu Feb 27 00:33:56 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 26 Feb 2014 18:33:56 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CADiSq7egjPaPuNTEk8tuhaX9knbz74NoYT=VvriD+YUeGCLQSA@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
 <CADiSq7egjPaPuNTEk8tuhaX9knbz74NoYT=VvriD+YUeGCLQSA@mail.gmail.com>
Message-ID: <EF24A65C-C836-4846-B1C7-B657857C8806@stufft.io>


On Feb 26, 2014, at 6:25 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 27 Feb 2014 08:37, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:
> >
> > > If Vinay is happy with that last option for handling the current bdist_wheel misbehaviour
> > > that would mean we could leave the metadata version alone
> >
> > I've changed distlib to ignore pydist.json for wheels with version 1.0, where METADATA is used instead. Note that bdist_wheel also puts a metadata version of 2.0 in the METADATA file.
> >
> > IMO versioning the pydist.json is equivalent to versioning the .dist-info directory - it certainly doesn't seem right to put a metadata version in the directory name itself, as it would effectively be noise almost all of the time.
> >
> > I think it may be premature to promote wheels (as pythonwheels.com is doing with vigour) - nothing wrong with it in principle, of course, it's just the timing which seems wrong. Before such promotion to the community at large, It would seem advisable to get the PyPA house in order by (a) finalising the PEPs which are in limbo because of the focus on getting 3.4 out,
> 
> Wheels work almost as well with setuptools metadata as they will with the initial release of the new metadata spec (because I will be removing the metadata hook extension for the time being - I'm still not convinced that mechanism is a good idea, and I don't think it should delay the other benefits, like providing a format for PyPI to publish dependency metadata directly).
> 
> > (b) ensuring that the 1.1 spec for wheel covers any shortcomings which have been identified in 1.0, as well as more extensive testing for the implementations, and
> 
> The wheel spec needs revision to cover more use cases. That's not a reason to avoid encouraging it's use for the wide range of cases that it already handles.
> 
> However, people do know that I'm not planning to write all these spec updates myself, right? I've claimed 426/440/459, but if you wait for me to write the others as well, we're going to be waiting a long time.
> 

I still need to review the changes to those :(
> I know Donald already has a very early draft of wheel 2.0 in the works, but I'm not aware of anyone that has even started thinking seriously about an sdist 2.0 spec, or an update to the installation database spec - if the latter could include an SQLite based caching mechanism, that would be great, since it lays the foundation for allowing metaimporters to provide installation database entries. And PEP 425 does need an update to cover at least Linux distros (likely based on /etc/os-release).
> 
> 

I have a Wheel 2.0 as well as a new one (or maybe just extending 425) that splits the filename convention out of Wheel.

Could possibly get a dist-info 2.0 in there but I also have two PyPI PEPS and a Python PEP on my TODO list too :/
> Would it help if I created tasks for all the things that still need to be done in the metadata formats repo, and assigned the ones I'm actually working on to myself? That way people could see clearly where we already know work is needed, but nobody has volunteered to do it yet.
> 
> > (c) revisiting accepted PEPs such as 425 to ensure that things are tightened up where needed (for example, there are some problems relating to tags, one of which IIRC Brian Wickman brought up a little while ago - about whether tags should follow the PyPy version numbering rather than the Python language version numbering).
> 
> PEP 425 explicitly covers that. Report bugs against tools that are non-compliant because they expect distutils to automatically do the right thing (when we know it doesn't, especially on Windows and in 2.x).
> 
> Cheers,
> Nick.
> 
> >
> > Regards,
> >
> > Vinay Sajip
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/bf725183/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/bf725183/attachment.sig>

From vinay_sajip at yahoo.co.uk  Thu Feb 27 01:16:52 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 27 Feb 2014 00:16:52 +0000 (GMT)
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CADiSq7egjPaPuNTEk8tuhaX9knbz74NoYT=VvriD+YUeGCLQSA@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>	<03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>	<CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>	<1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
 <CADiSq7egjPaPuNTEk8tuhaX9knbz74NoYT=VvriD+YUeGCLQSA@mail.gmail.com>
Message-ID: <1393460212.4332.YahooMailNeo@web172404.mail.ir2.yahoo.com>



> PEP 425 explicitly covers that.

It says "The version is?py_version_nodot. CPython gets away with no dot, but if one is needed the underscore?_?is used instead.?PyPy should probably use its own versions here?pp18,?pp19".?

The "probably" leaves some room for doubt as to what exactly is meant: both wheel and distlib use py_version_nodot, and the last sentence I quoted looks provisional rather than definitive.

Regards,

Vinay Sajip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/efb1e835/attachment-0001.html>

From ncoghlan at gmail.com  Thu Feb 27 01:28:12 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Feb 2014 10:28:12 +1000
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <1393460212.4332.YahooMailNeo@web172404.mail.ir2.yahoo.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
 <CADiSq7egjPaPuNTEk8tuhaX9knbz74NoYT=VvriD+YUeGCLQSA@mail.gmail.com>
 <1393460212.4332.YahooMailNeo@web172404.mail.ir2.yahoo.com>
Message-ID: <CADiSq7fofsY3cuGs2+f=DfRd4KpLDc_VX3cykc0AXBqA2EZ7Aw@mail.gmail.com>

On 27 Feb 2014 10:16, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:
>
>
> > PEP 425 explicitly covers that.
>
> It says "The version is py_version_nodot. CPython gets away with no dot,
but if one is needed the underscore _ is used instead. PyPy should probably
use its own versions here pp18, pp19".
>
> The "probably" leaves some room for doubt as to what exactly is meant:
both wheel and distlib use py_version_nodot, and the last sentence I quoted
looks provisional rather than definitive.

That's because API/ABI versioning is technically up to the individual
implementations. The PEP only mandates particular behaviour for CPython and
the implementation independent tags.

If if you want a definitive answer on that, ask the PyPy devs to write one
up.

I would have preferred to mandate behaviour based on sys.implementation,
but that still wouldn't have covered 2.x.

Cheers,
Nick.

>
> Regards,
>
> Vinay Sajip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/f730ab72/attachment.html>

From qwcode at gmail.com  Thu Feb 27 01:38:17 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 26 Feb 2014 16:38:17 -0800
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
Message-ID: <CAPYWazr-fx9LLudLr-PNi5qgnw4NoFbR9Orop9i46a18w-To7Q@mail.gmail.com>

>
>
> Wheel-Version: Whatever-We-Formally-Add-Pydistjson-To-Wheel in.
>

PEP427/Wheel-1.0 has this line:    "5.
 {distribution}-{version}.dist-info/METADATA is Metadata version 1.1 or
greater format metadata."

although it says "version 1.1 or greater", I guess we're saying it
indirectly excludes metadata 2.0, because it's referring to the key/value
"METADATA" file which 2.0 won't have.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/2d9345a9/attachment.html>

From qwcode at gmail.com  Thu Feb 27 01:46:00 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 26 Feb 2014 16:46:00 -0800
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CADiSq7egjPaPuNTEk8tuhaX9knbz74NoYT=VvriD+YUeGCLQSA@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
 <CADiSq7egjPaPuNTEk8tuhaX9knbz74NoYT=VvriD+YUeGCLQSA@mail.gmail.com>
Message-ID: <CAPYWazpOhG+F3M3LL2TDdw4NZkpWrHm=VT+nbdH1ZD1c8RUFpQ@mail.gmail.com>

>
>
> However, people do know that I'm not planning to write all these spec
> updates myself, right? I've claimed 426/440/459, but if you wait for me to
> write the others as well, we're going to be waiting a long time.
>
> I know Donald already has a very early draft of wheel 2.0 in the works,
> but I'm not aware of anyone that has even started thinking seriously about
> an sdist 2.0 spec, or an update to the installation database spec - if the
> latter could include an SQLite based caching mechanism, that would be
> great, since it lays the foundation for allowing metaimporters to provide
> installation database entries. And PEP 425 does need an update to cover at
> least Linux distros (likely based on /etc/os-release).
>
> Would it help if I created tasks for all the things that still need to be
> done in the metadata formats repo, and assigned the ones I'm actually
> working on to myself? That way people could see clearly where we already
> know work is needed, but nobody has volunteered to do it yet.
>

that would be good.  If you did, I would link to the tasks from the PUG
future page.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/0ed6aa88/attachment.html>

From qwcode at gmail.com  Thu Feb 27 02:22:43 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 26 Feb 2014 17:22:43 -0800
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
Message-ID: <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>

>
>
>
> Calling it version 3 is a fine solution, and the spec has changed
> enough from key/value 2.0 to json 2.0, and grown a year older, that
> it's also appropriate to bump the version again.


appropriate to bump? "key/value 2.0" was never a thing.
I don't like switching to 3.0, unless we really have to.
It looks odd from the outside.
like we're switching gears again in some major way

btw, what is in wheel's "2.0" METADATA file that is 2.0? and what's using
it?

can't bdist_wheel at least stop setting metadata to 2.0 from here on?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/371653a1/attachment.html>

From dholth at gmail.com  Thu Feb 27 04:02:01 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 26 Feb 2014 22:02:01 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
Message-ID: <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>

On Wed, Feb 26, 2014 at 8:22 PM, Marcus Smith <qwcode at gmail.com> wrote:
>>
>>
>> Calling it version 3 is a fine solution, and the spec has changed
>> enough from key/value 2.0 to json 2.0, and grown a year older, that
>> it's also appropriate to bump the version again.
>
>
> appropriate to bump? "key/value 2.0" was never a thing.
> I don't like switching to 3.0, unless we really have to.
> It looks odd from the outside.
> like we're switching gears again in some major way
>
> btw, what is in wheel's "2.0" METADATA file that is 2.0? and what's using
> it?
>
> can't bdist_wheel at least stop setting metadata to 2.0 from here on?

METADATA is the key/value file that setuptools uses when it finds a
.dist-info directory. It resembles pre-JSON PEP 426. It can be
classified as setuptools metadata now.

The next bdist_wheel will remove pydist.json.

From qwcode at gmail.com  Thu Feb 27 05:26:04 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Wed, 26 Feb 2014 20:26:04 -0800
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
Message-ID: <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>

>
>
> > btw, what is in wheel's "2.0" METADATA file that is 2.0? and what's using
> > it?
> >
> > can't bdist_wheel at least stop setting metadata to 2.0 from here on?
>
> METADATA is the key/value file that setuptools uses when it finds a
> .dist-info directory. It resembles pre-JSON PEP 426. It can be
> classified as setuptools metadata now.
>

roger, METADATA is a PEP376/dist-info file that
`pkg_resources.DistInfoDistribution` understands.
but my question was what are you adding, if anything, that warranted it
having "Metadata-Version: 2.0"?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140226/03e1dba2/attachment-0001.html>

From vinay_sajip at yahoo.co.uk  Thu Feb 27 09:16:00 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 27 Feb 2014 08:16:00 +0000 (GMT)
Subject: [Distutils] The PEP 426 defined metadata version will
	be	metadata 3.0
In-Reply-To: <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
Message-ID: <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>

> but my question was what are you adding, if anything,
> that warranted it having "Metadata-Version: 2.0"?


I believe that the fields?'Private-Version', 'Obsoleted-By', 'Setup-Requires-Dist',
'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an
early version of PEP 426, before the move to JSON.

Regards,

Vinay Sajip




_______________________________________________
Distutils-SIG maillist? -? Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/fdfab0ea/attachment.html>

From vinay_sajip at yahoo.co.uk  Thu Feb 27 09:30:17 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 27 Feb 2014 08:30:17 +0000 (GMT)
Subject: [Distutils] The PEP 426 defined metadata version will
	be	metadata 3.0
In-Reply-To: <C3E3E281-62FF-44E6-9F7D-4CD2412728BC@stufft.io>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <1393454255.87623.YahooMailNeo@web172402.mail.ir2.yahoo.com>
 <0F4495CE-B88B-4D97-9942-614AD00C724E@stufft.io>
 <1393456666.26535.YahooMailNeo@web172405.mail.ir2.yahoo.com>
 <C3E3E281-62FF-44E6-9F7D-4CD2412728BC@stufft.io>
Message-ID: <1393489817.91615.YahooMailNeo@web172402.mail.ir2.yahoo.com>

> There is always going to be multiple files, it?s kind of silly to tie the definition of
> the dist-info directory to the pydist.json when that?s perhaps not the file you
> care about how to interpret. How do you interpret the RECORD file?
> The INSTALLER file??



> The versioned definition of the dist-info directory would say, ?RECORD file is an
> XYZ file???pydist.json is an ABC file?. It?s the only sane way to handle the case
> where you can have?an unbounded number of unknown files in the directory.

Those files are either implementation specific (in which case, not our responsibility)
or they are standardised, in which case there will be a PEP (376 for the files apart
from pydist.json). If it is felt that the formats in PEP 376 should be future-proofed,
then perhaps they should be coalesced into a single JSON file with its own
metadata version and schema definition (/ducks ;-)

The PEP 376 files are INSTALLER, RECORD, REQUESTED and RESOURCES
(which AFAIK isn't used). I don't know of any reason why they couldn't be merged
into a single file.

Any implementation specific caches of parts of those files (e.g. exports, for speed
of scanning) would be up to individual implementations to maintain (and perhaps belong
in implementation-named subdirectories of dist-info, to avoid conflicts with other
implementations).

Regards,

Vinay Sajip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/f010491f/attachment.html>

From ncoghlan at gmail.com  Thu Feb 27 13:10:09 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Feb 2014 22:10:09 +1000
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
Message-ID: <CADiSq7eQCg7qaw5QCxAhhHv5-bwms+yJ1=Xh1O_4r5VL3JzN8g@mail.gmail.com>

On 27 February 2014 11:22, Marcus Smith <qwcode at gmail.com> wrote:
>> Calling it version 3 is a fine solution, and the spec has changed
>> enough from key/value 2.0 to json 2.0, and grown a year older, that
>> it's also appropriate to bump the version again.
>
> appropriate to bump? "key/value 2.0" was never a thing.
> I don't like switching to 3.0, unless we really have to.
> It looks odd from the outside.
> like we're switching gears again in some major way

Yeah, after chatting with Donald a bit, I think we can deal with it
without bumping the metadata version again (I think we could sell that
if we really had to, but I've been persuaded that isn't necessary at
this point).

1. Before reading a pydist.json file from a wheel file, check the
wheel version to be sure it's one that includes a valid pydist.json
file as part of the spec.
2/ Before reading it from a dist-info directory, check the layout
version there as well

While PEP 376 doesn't include a way to version dist-info directories,
we can add one in the same update that adds pydist.json to the
expected files list - the simplest way would be something like a
"LAYOUT" file with an incrementing number as its only contents (so PEP
376 would be considered layout 1, it's successor that adds pydist.json
layout 2, etc)

Coming up with an efficient sqlite backed install-time metadata
caching scheme or new metapath and path hook friendly metadata APIs
would be postponed for the time being, since those are both a long way
down our priority list at this point.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Feb 27 13:16:56 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Feb 2014 22:16:56 +1000
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
Message-ID: <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>

On 27 February 2014 18:16, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> but my question was what are you adding, if anything,
>> that warranted it having "Metadata-Version: 2.0"?
>
> I believe that the fields 'Private-Version', 'Obsoleted-By',
> 'Setup-Requires-Dist',
> 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an
> early version of PEP 426, before the move to JSON.

For the record,
http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last
version prior to the switch to JSON, and the summary of differences is
at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263

At that time, I think wheel still had a dependency on PEP 426 -
changing wheels to work with the setuptools metadata was a relatively
late change after we realised it made sense to decouple the two
activities.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Thu Feb 27 15:13:36 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 27 Feb 2014 09:13:36 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
Message-ID: <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>

I don't think it matters which number we use in the METADATA key/value
metadata... it's not even checked by anything.

On Thu, Feb 27, 2014 at 7:16 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27 February 2014 18:16, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>>> but my question was what are you adding, if anything,
>>> that warranted it having "Metadata-Version: 2.0"?
>>
>> I believe that the fields 'Private-Version', 'Obsoleted-By',
>> 'Setup-Requires-Dist',
>> 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an
>> early version of PEP 426, before the move to JSON.
>
> For the record,
> http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last
> version prior to the switch to JSON, and the summary of differences is
> at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263
>
> At that time, I think wheel still had a dependency on PEP 426 -
> changing wheels to work with the setuptools metadata was a relatively
> late change after we realised it made sense to decouple the two
> activities.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Thu Feb 27 15:14:41 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 27 Feb 2014 09:14:41 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
 <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>
Message-ID: <5D13088C-8315-428C-A16A-E7A349D4736D@stufft.io>


On Feb 27, 2014, at 9:13 AM, Daniel Holth <dholth at gmail.com> wrote:

> I don't think it matters which number we use in the METADATA key/value
> metadata... it's not even checked by anything.
> 
> On Thu, Feb 27, 2014 at 7:16 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 27 February 2014 18:16, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>>>> but my question was what are you adding, if anything,
>>>> that warranted it having "Metadata-Version: 2.0"?
>>> 
>>> I believe that the fields 'Private-Version', 'Obsoleted-By',
>>> 'Setup-Requires-Dist',
>>> 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an
>>> early version of PEP 426, before the move to JSON.
>> 
>> For the record,
>> http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last
>> version prior to the switch to JSON, and the summary of differences is
>> at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263
>> 
>> At that time, I think wheel still had a dependency on PEP 426 -
>> changing wheels to work with the setuptools metadata was a relatively
>> late change after we realised it made sense to decouple the two
>> activities.
>> 
>> Cheers,
>> Nick.
>> 
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Compliance to standards matter. It?s how you get reasonable interoperability.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/a7fb9af1/attachment.sig>

From dholth at gmail.com  Thu Feb 27 15:16:01 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 27 Feb 2014 09:16:01 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <5D13088C-8315-428C-A16A-E7A349D4736D@stufft.io>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
 <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>
 <5D13088C-8315-428C-A16A-E7A349D4736D@stufft.io>
Message-ID: <CAG8k2+6+AfMy2SWFsx2BCZ=YaDa4jYh42aC0Qnjx2mAKDpPtzA@mail.gmail.com>

Yes, but the METADATA version of the metadata is not a standard.

On Thu, Feb 27, 2014 at 9:14 AM, Donald Stufft <donald at stufft.io> wrote:
>
> On Feb 27, 2014, at 9:13 AM, Daniel Holth <dholth at gmail.com> wrote:
>
>> I don't think it matters which number we use in the METADATA key/value
>> metadata... it's not even checked by anything.
>>
>> On Thu, Feb 27, 2014 at 7:16 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> On 27 February 2014 18:16, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>>>>> but my question was what are you adding, if anything,
>>>>> that warranted it having "Metadata-Version: 2.0"?
>>>>
>>>> I believe that the fields 'Private-Version', 'Obsoleted-By',
>>>> 'Setup-Requires-Dist',
>>>> 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an
>>>> early version of PEP 426, before the move to JSON.
>>>
>>> For the record,
>>> http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last
>>> version prior to the switch to JSON, and the summary of differences is
>>> at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263
>>>
>>> At that time, I think wheel still had a dependency on PEP 426 -
>>> changing wheels to work with the setuptools metadata was a relatively
>>> late change after we realised it made sense to decouple the two
>>> activities.
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
> Compliance to standards matter. It's how you get reasonable interoperability.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>

From donald at stufft.io  Thu Feb 27 15:17:56 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 27 Feb 2014 09:17:56 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAG8k2+6+AfMy2SWFsx2BCZ=YaDa4jYh42aC0Qnjx2mAKDpPtzA@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
 <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>
 <5D13088C-8315-428C-A16A-E7A349D4736D@stufft.io>
 <CAG8k2+6+AfMy2SWFsx2BCZ=YaDa4jYh42aC0Qn
 jx2mAKDpPtzA@mail.gmail.com>
Message-ID: <C2A40700-3C14-43D5-B4EA-5DAE1F8C5871@stufft.io>


On Feb 27, 2014, at 9:16 AM, Daniel Holth <dholth at gmail.com> wrote:

> Yes, but the METADATA version of the metadata is not a standard.
> 
> On Thu, Feb 27, 2014 at 9:14 AM, Donald Stufft <donald at stufft.io> wrote:
>> 
>> On Feb 27, 2014, at 9:13 AM, Daniel Holth <dholth at gmail.com> wrote:
>> 
>>> I don't think it matters which number we use in the METADATA key/value
>>> metadata... it's not even checked by anything.
>>> 
>>> On Thu, Feb 27, 2014 at 7:16 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>> On 27 February 2014 18:16, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>>>>>> but my question was what are you adding, if anything,
>>>>>> that warranted it having "Metadata-Version: 2.0"?
>>>>> 
>>>>> I believe that the fields 'Private-Version', 'Obsoleted-By',
>>>>> 'Setup-Requires-Dist',
>>>>> 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an
>>>>> early version of PEP 426, before the move to JSON.
>>>> 
>>>> For the record,
>>>> http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last
>>>> version prior to the switch to JSON, and the summary of differences is
>>>> at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263
>>>> 
>>>> At that time, I think wheel still had a dependency on PEP 426 -
>>>> changing wheels to work with the setuptools metadata was a relatively
>>>> late change after we realised it made sense to decouple the two
>>>> activities.
>>>> 
>>>> Cheers,
>>>> Nick.
>>>> 
>>>> --
>>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>> 
>> Compliance to standards matter. It's how you get reasonable interoperability.
>> 
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 

Uh, it?s defined in the PEP standards.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/372ca8fc/attachment.sig>

From dholth at gmail.com  Thu Feb 27 15:24:26 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 27 Feb 2014 09:24:26 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <C2A40700-3C14-43D5-B4EA-5DAE1F8C5871@stufft.io>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
 <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>
 <5D13088C-8315-428C-A16A-E7A349D4736D@stufft.io>
 <CAG8k2+6+AfMy2SWFsx2BCZ=YaDa4jYh42aC0Qnjx2mAKDpPtzA@mail.gmail.com>
 <C2A40700-3C14-43D5-B4EA-5DAE1F8C5871@stufft.io>
Message-ID: <CAG8k2+62_T7TrQHvYC8duB2SPy0D0rNCDT6gkthpa=OSKoGEVQ@mail.gmail.com>

No PEP describes how METADATA supports setuptools' extras.

On Thu, Feb 27, 2014 at 9:17 AM, Donald Stufft <donald at stufft.io> wrote:
>
> On Feb 27, 2014, at 9:16 AM, Daniel Holth <dholth at gmail.com> wrote:
>
>> Yes, but the METADATA version of the metadata is not a standard.
>>
>> On Thu, Feb 27, 2014 at 9:14 AM, Donald Stufft <donald at stufft.io> wrote:
>>>
>>> On Feb 27, 2014, at 9:13 AM, Daniel Holth <dholth at gmail.com> wrote:
>>>
>>>> I don't think it matters which number we use in the METADATA key/value
>>>> metadata... it's not even checked by anything.
>>>>
>>>> On Thu, Feb 27, 2014 at 7:16 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>>> On 27 February 2014 18:16, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>>>>>>> but my question was what are you adding, if anything,
>>>>>>> that warranted it having "Metadata-Version: 2.0"?
>>>>>>
>>>>>> I believe that the fields 'Private-Version', 'Obsoleted-By',
>>>>>> 'Setup-Requires-Dist',
>>>>>> 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an
>>>>>> early version of PEP 426, before the move to JSON.
>>>>>
>>>>> For the record,
>>>>> http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last
>>>>> version prior to the switch to JSON, and the summary of differences is
>>>>> at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263
>>>>>
>>>>> At that time, I think wheel still had a dependency on PEP 426 -
>>>>> changing wheels to work with the setuptools metadata was a relatively
>>>>> late change after we realised it made sense to decouple the two
>>>>> activities.
>>>>>
>>>>> Cheers,
>>>>> Nick.
>>>>>
>>>>> --
>>>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>>> _______________________________________________
>>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>> Compliance to standards matter. It's how you get reasonable interoperability.
>>>
>>> -----------------
>>> Donald Stufft
>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>>
>
> Uh, it's defined in the PEP standards.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>

From dholth at gmail.com  Thu Feb 27 19:59:57 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 27 Feb 2014 13:59:57 -0500
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAG8k2+62_T7TrQHvYC8duB2SPy0D0rNCDT6gkthpa=OSKoGEVQ@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
 <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>
 <5D13088C-8315-428C-A16A-E7A349D4736D@stufft.io>
 <CAG8k2+6+AfMy2SWFsx2BCZ=YaDa4jYh42aC0Qnjx2mAKDpPtzA@mail.gmail.com>
 <C2A40700-3C14-43D5-B4EA-5DAE1F8C5871@stufft.io>
 <CAG8k2+62_T7TrQHvYC8duB2SPy0D0rNCDT6gkthpa=OSKoGEVQ@mail.gmail.com>
Message-ID: <CAG8k2+7Z4=5ukQwHvaa7nXxRY_qjeEJ8t=GVNuDuh3151-CH=w@mail.gmail.com>

Unfortunately "only follow the PEP-ratified standards" means "stop
doing useful packaging" until we have some additional standards.

Vinay had mentioned the pypy ABI tagging. I'd hoped implementations
would be interested in figuring this out, but I'm not sure the PyPy
group is or should be very excited about even having an ABI or a
preferred one from the many that their compilation process surely can
generate... for them, the best case might even be to call a library
with cffi, using no Python ABI at all. (the py2.py3-none-x86_64 style
of PEP 425 tag). So I still don't know the right answer for that part
of PEP 425.

From qwcode at gmail.com  Thu Feb 27 22:20:27 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Thu, 27 Feb 2014 13:20:27 -0800
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
Message-ID: <CAPYWazpsTK_fidyo65g7B3RmZRZOv+yt78c3VxeNo7CC-767zw@mail.gmail.com>

On Thu, Feb 27, 2014 at 12:16 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote:

> > but my question was what are you adding, if anything,
> > that warranted it having "Metadata-Version: 2.0"?
>
> I believe that the fields 'Private-Version', 'Obsoleted-By',
> 'Setup-Requires-Dist',
> 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in
> an
> early version of PEP 426, before the move to JSON.
>


from a quick scan of the wheel code, I only see "Provides-Extra" as
something additional to 1.2
Setuptools additionally acquired logic for "Provides-Extra" in
"DistInfoDistribution"
afaik, "Provides-Extra" is not literally in PEP426, so it has no PEP home?
but it's implemented?
it sounds like we need some version to capture this change?, or just
consider it a mess, and leave at that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/1c46fbad/attachment.html>

From qwcode at gmail.com  Fri Feb 28 06:41:22 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Thu, 27 Feb 2014 21:41:22 -0800
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAG8k2+62_T7TrQHvYC8duB2SPy0D0rNCDT6gkthpa=OSKoGEVQ@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
 <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>
 <5D13088C-8315-428C-A16A-E7A349D4736D@stufft.io>
 <CAG8k2+6+AfMy2SWFsx2BCZ=YaDa4jYh42aC0Qnjx2mAKDpPtzA@mail.gmail.com>
 <C2A40700-3C14-43D5-B4EA-5DAE1F8C5871@stufft.io>
 <CAG8k2+62_T7TrQHvYC8duB2SPy0D0rNCDT6gkthpa=OSKoGEVQ@mail.gmail.com>
Message-ID: <CAPYWazpkqG89jxu8=fnOrwfk9_VW+KNuPwsOUWjgrOAUy7BnVw@mail.gmail.com>

On Thu, Feb 27, 2014 at 6:24 AM, Daniel Holth <dholth at gmail.com> wrote:

> No PEP describes how METADATA supports setuptools' extras.
>

i.e. "Provides-Extras".   maybe we should at least add a footnote to
PEP-345, or *something*, so this is not a mystery down the road where this
came from, and how it ended being in "production" use for wheels
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/1a405fa1/attachment-0001.html>

From nilakhya.chatterjee at globallogic.com  Wed Feb 26 21:38:55 2014
From: nilakhya.chatterjee at globallogic.com (Nilakhya Chatterjee)
Date: Thu, 27 Feb 2014 02:08:55 +0530
Subject: [Distutils] Fwd: Error getting pyodbc from pypi shared some log
In-Reply-To: <CAPPfVQ2isRxaW8_C7Bu8aom9PbdY=7CE-q0MHFOLb-Dm2h_dFQ@mail.gmail.com>
References: <CAPPfVQ2isRxaW8_C7Bu8aom9PbdY=7CE-q0MHFOLb-Dm2h_dFQ@mail.gmail.com>
Message-ID: <CAPPfVQ2-UJRCwJ8F_JrMKe1gSNub-T9Ki75DSVQVn5aNL3EWkQ@mail.gmail.com>

Please look at https://pypi.python.org/pypi/pyodbc the download url of the
said package got missing.

---------- Forwarded message ----------
From: Nilakhya Chatterjee <nilakhya.chatterjee at globallogic.com>
Date: Wed, Feb 26, 2014 at 6:46 PM
Subject: Error getting pyodbc from pypi shared some log
To: michael at kleehammer.com



No distributions at all found for pyodbc>=3.0.7 (from
vertica-sqlalchemy-0.2)
Exception information:
Traceback (most recent call last):
  File
"/home/ubuntu/myenv/local/lib/python2.7/site-packages/pip/basecommand.py",
line 122, in main
    status = self.run(options, args)
  File
"/home/ubuntu/myenv/local/lib/python2.7/site-packages/pip/commands/install.py",
line 270, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle,
bundle=self.bundle)
  File "/home/ubuntu/myenv/local/lib/python2.7/site-packages/pip/req.py",
line 1157, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/home/ubuntu/myenv/local/lib/python2.7/site-packages/pip/index.py",
line 285, in find_requirement
    raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for pyodbc>=3.0.7 (from
vertica-sqlalchemy-0.2)

Let me know if you can quickly resolve this.

Tx !

-- 

Nilakhya | Consultant Engineering
GlobalLogic
P +x.xxx.xxx.xxxx  M +91.989.112.5770  S skype
www.globallogic.com
 <http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt



-- 

Nilakhya | Consultant Engineering
GlobalLogic
P +x.xxx.xxx.xxxx  M +91.989.112.5770  S skype
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140227/acbcc8b3/attachment.html>

From p at google-groups-2014.dobrogost.net  Thu Feb 27 12:49:19 2014
From: p at google-groups-2014.dobrogost.net (Piotr Dobrogost)
Date: Thu, 27 Feb 2014 12:49:19 +0100
Subject: [Distutils] Is there any sense to put setuptools as a requirement?
Message-ID: <CAA6tFMthwY2mN2PhUVOe+VMeN+6smy2GL8fh0=n6JjSzOk+T=A@mail.gmail.com>

Hi!

I've seen people putting 'setuptools' in 'install_requires' in
setup.py starting with import from setuptools like this:
from setuptools import setup, find_packages

Does it make any sense?
In what circumstances should 'setuptools' be listed in
'install_requires' or 'setup_requires'?


Regards,
Piotr Dobrogost