From ncoghlan at gmail.com  Tue Apr  1 00:41:12 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 1 Apr 2014 08:41:12 +1000
Subject: [Distutils] Pycon
In-Reply-To: <20140331132525.38a363ec@anarchist.wooz.org>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <20140331132525.38a363ec@anarchist.wooz.org>
Message-ID: <CADiSq7e3smt5dD7X7S_CUPq0yc1de8zsJUOj67CdMvU3HxE_PA@mail.gmail.com>

On 1 Apr 2014 03:26, "Barry Warsaw" <barry at python.org> wrote:
>
> On Mar 28, 2014, at 03:06 PM, Daniel Holth wrote:
>
> >Who is going to pycon? I will be there.
>
> I'll be there, for the duration (language summit through sprints).  It
would
> be great to hav an OpenSpace or BoF for discussing the intersection of
Python
> packaging issues and distros.

Oh, good idea. Toshio and a bunch of other Fedora folks will also be there
(unfortunately not Slavek this year - maybe 2015). Would be good to discuss
all the distro integration support in metadata 2.0 :)

Cheers,
Nick.

>
> -Barry
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140401/dc7c732c/attachment.html>

From a.badger at gmail.com  Tue Apr  1 02:04:47 2014
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Mon, 31 Mar 2014 17:04:47 -0700
Subject: [Distutils] Pycon
In-Reply-To: <CADiSq7e3smt5dD7X7S_CUPq0yc1de8zsJUOj67CdMvU3HxE_PA@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <20140331132525.38a363ec@anarchist.wooz.org>
 <CADiSq7e3smt5dD7X7S_CUPq0yc1de8zsJUOj67CdMvU3HxE_PA@mail.gmail.com>
Message-ID: <20140401000447.GA12602@rockytop.lan>

On Tue, Apr 01, 2014 at 08:41:12AM +1000, Nick Coghlan wrote:
> 
> On 1 Apr 2014 03:26, "Barry Warsaw" <barry at python.org> wrote:
> >
> > On Mar 28, 2014, at 03:06 PM, Daniel Holth wrote:
> >
> > >Who is going to pycon? I will be there.
> >
> > I'll be there, for the duration (language summit through sprints).  It would
> > be great to hav an OpenSpace or BoF for discussing the intersection of Python
> > packaging issues and distros.
> 
> Oh, good idea. Toshio and a bunch of other Fedora folks will also be there
> (unfortunately not Slavek this year - maybe 2015). Would be good to discuss all
> the distro integration support in metadata 2.0 :)

Yep, I'll be there from Language Summit until the end.  I'm not sure that
any of the other Fedora folks who are coming this year are interested in
packaging, we're a bit web-developer heavy this year.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140331/b911fe17/attachment.sig>

From dholth at gmail.com  Tue Apr  1 04:18:45 2014
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 31 Mar 2014 22:18:45 -0400
Subject: [Distutils] wheel 0.23.0 released
Message-ID: <CAG8k2+5dChcek41nfeGsMzOLgnt6ug6tT-g1Htcunz9KREs=yg@mail.gmail.com>

I've released version 0.23.0 of wheel, after verifying that its test
pass against Python 2.6, 2.7, PyPy, Python 3.3 and Python 3.4. Most of
the improvements in this release are from contributors. Thank you!

The most surprising feature to anyone who relies on wheel's own
installer is that bdist_wheel now relies on post-install setuptools
scripts wrapper generation - which is automatically provided by recent
wheel-supporting versions of pip, or manually by the wheel
install-scripts [distribution [distribution ...]] command. This change
also removes the --skip-scripts parameter from bdist_wheel.

https://pypi.python.org/pypi/wheel/

From pmiscml at gmail.com  Tue Apr  1 21:25:09 2014
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Tue, 1 Apr 2014 22:25:09 +0300
Subject: [Distutils] Creating distribution for Python *module*, not package
Message-ID: <20140401222509.52d11704@x34f>

Hello,

I would like to create a PyPIable packaged disribution for a
single-file python module (say, foo.py). How to achieve that? There
doesn't seem to be "modules" param to setup() (as a mirror of "packages"
param).

Thanks,
 Paul                          mailto:pmiscml at gmail.com

From pmiscml at gmail.com  Tue Apr  1 21:46:54 2014
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Tue, 1 Apr 2014 22:46:54 +0300
Subject: [Distutils] Creating distribution for Python *module*,
	not package
In-Reply-To: <20140401222509.52d11704@x34f>
References: <20140401222509.52d11704@x34f>
Message-ID: <20140401224654.36d875f9@x34f>

Hello,

On Tue, 1 Apr 2014 22:25:09 +0300
Paul Sokolovsky <pmiscml at gmail.com> wrote:

> Hello,
> 
> I would like to create a PyPIable packaged disribution for a
> single-file python module (say, foo.py). How to achieve that? There
> doesn't seem to be "modules" param to setup() (as a mirror of
> "packages" param).

Ah, pure magic: google it, search stackoverflow - find nothing. Post a
question to mailing list, and next step find the answer - the param is
called "py_modules".

Let's analyze: I would expect a complete reference of setup() params on
https://pythonhosted.org/setuptools/ , and/or explicit and very visible
mentioning that setuptools is extension to distutils and its docs
should be read first.

Instead I get:

Confusing TOC, first entry is called "Roadmap", which doesn't
provide any roadmap, but instead status. Next one "Supporting both
Python 2 and Python 3 with Setuptools" - oh, sure that's first thing I
want to know about setuptools. Or maybe no, first thing I want to know
what the heck it is. So, the section with which manual should start,
"Building and Distributing Packages with Setuptools" is #4 in TOC, no
wonder it takes a while for people to get to it. Searching for
"py_modules" returns 0 hits.

And kind suggestion for packaging tutorials writers - please start from
the start, by describing how to package a *module*, there's no need to
force package bloat on everyone. And the latter is exactly what 3
google-top tutorials I read did.


Keep up the great work!

-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From mmericke at gmail.com  Tue Apr  1 22:16:51 2014
From: mmericke at gmail.com (Michael Merickel)
Date: Tue, 1 Apr 2014 15:16:51 -0500
Subject: [Distutils] Creating distribution for Python *module*,
	not package
In-Reply-To: <20140401224654.36d875f9@x34f>
References: <20140401222509.52d11704@x34f> <20140401224654.36d875f9@x34f>
Message-ID: <CAKdhhwEvL6AVZ-sJ22q=4AKLPpfuUyk7ZMebufMinVDwVo0gQA@mail.gmail.com>

Yeah, setuptools is a bw-compat wrapper around distutils, so it tends to
only document its new features. This approach to documentation makes it
very difficult to understand for newcomers that are not familiar with
distutils. Especially now that setuptools is the de facto standard that
everyone is learning how to use.

https://docs.python.org/2/distutils/examples.html#pure-python-distribution-by-module


On Tue, Apr 1, 2014 at 2:46 PM, Paul Sokolovsky <pmiscml at gmail.com> wrote:

> Hello,
>
> On Tue, 1 Apr 2014 22:25:09 +0300
> Paul Sokolovsky <pmiscml at gmail.com> wrote:
>
> > Hello,
> >
> > I would like to create a PyPIable packaged disribution for a
> > single-file python module (say, foo.py). How to achieve that? There
> > doesn't seem to be "modules" param to setup() (as a mirror of
> > "packages" param).
>
> Ah, pure magic: google it, search stackoverflow - find nothing. Post a
> question to mailing list, and next step find the answer - the param is
> called "py_modules".
>
> Let's analyze: I would expect a complete reference of setup() params on
> https://pythonhosted.org/setuptools/ , and/or explicit and very visible
> mentioning that setuptools is extension to distutils and its docs
> should be read first.
>
> Instead I get:
>
> Confusing TOC, first entry is called "Roadmap", which doesn't
> provide any roadmap, but instead status. Next one "Supporting both
> Python 2 and Python 3 with Setuptools" - oh, sure that's first thing I
> want to know about setuptools. Or maybe no, first thing I want to know
> what the heck it is. So, the section with which manual should start,
> "Building and Distributing Packages with Setuptools" is #4 in TOC, no
> wonder it takes a while for people to get to it. Searching for
> "py_modules" returns 0 hits.
>
> And kind suggestion for packaging tutorials writers - please start from
> the start, by describing how to package a *module*, there's no need to
> force package bloat on everyone. And the latter is exactly what 3
> google-top tutorials I read did.
>
>
> Keep up the great work!
>
> --
> Best regards,
>  Paul                          mailto:pmiscml at gmail.com
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140401/1e426bde/attachment.html>

From dimino at gmail.com  Tue Apr  1 22:07:48 2014
From: dimino at gmail.com (Charles Dimino)
Date: Tue, 1 Apr 2014 13:07:48 -0700 (PDT)
Subject: [Distutils] setuptools and git flow
Message-ID: <e46c6589-7f83-48f3-8d04-c6e8ee033d46@googlegroups.com>

Hey all,

I'm trying to use setuptools to manage my project, and this project uses 
git flow.

Part of git flow (and in general) is that you don't know when you're 
working on develop branch what your next version will be yet.  In a way, 
this means that work done on your develop branch doesn't have *any* version 
at all.  Only once you create a release branch, will you know what your 
version will be.

This is at odds with the way setuptools documentation talks about re: daily 
builds and snapshot releases.  There's no option to change your project's 
"official" version identified entirely - only to append to it.

So what ends up happening is a project has its release (1.0.0, for 
example), and eventually the released version identifier makes its way back 
into the develop branch.  Then, when a new nightly develop build gets 
created, it's marked as 1.0.0-NIGHTLY-01012014.  This is a problem, because 
it's *not* a 1.0.0 nightly build, it's a POST-1.0.0 nightly build.

Since the develop branch isn't supposed to have an actual version, what are 
some recommendations?  Should the version value in setup.py just be set to 
something like "DEVELOP" in the develop branch?  That'd always create 
"DEVELOP-NIGHTLY-01012014" tags, which is acceptable.

Thanks,
Charlie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140401/38737395/attachment.html>

From p.f.moore at gmail.com  Tue Apr  1 22:41:21 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 1 Apr 2014 21:41:21 +0100
Subject: [Distutils] Creating distribution for Python *module*,
	not package
In-Reply-To: <CAKdhhwEvL6AVZ-sJ22q=4AKLPpfuUyk7ZMebufMinVDwVo0gQA@mail.gmail.com>
References: <20140401222509.52d11704@x34f> <20140401224654.36d875f9@x34f>
 <CAKdhhwEvL6AVZ-sJ22q=4AKLPpfuUyk7ZMebufMinVDwVo0gQA@mail.gmail.com>
Message-ID: <CACac1F9BYY_2L5wKQtmvWdf16nFCXrjA50JA4OJ9i7vvRc2CXw@mail.gmail.com>

On 1 April 2014 21:16, Michael Merickel <mmericke at gmail.com> wrote:
> Yeah, setuptools is a bw-compat wrapper around distutils, so it tends to
> only document its new features. This approach to documentation makes it very
> difficult to understand for newcomers that are not familiar with distutils.
> Especially now that setuptools is the de facto standard that everyone is
> learning how to use.

Agreed. The packaging documentation grew piecemeal over time, staring
from a basis of the distutils documentation which was itself pretty
incomprehensible. We're working on a new iteration of the packaging
documentation at
https://python-packaging-user-guide.readthedocs.org/en/latest/ but to
be honest, it's very easy to take for granted a lot of basics, simply
because the people doing the writing are too close to the packaging
ecosystem. For example, I don't think anyone considered documenting a
single Python module as mentioned by the OP (it's usually the case
that your needs very soon grow to the point where a single module
isn't enough, but that's not a reason to not cover it - just a
possible reason why we never thought of it).

We'd be very grateful for any contributions to the packaging user
guide, either in the form of pull requests or even just issues on the
tracker pointing at things we should cover (or should cover better).

Paul

From ncoghlan at gmail.com  Tue Apr  1 23:24:47 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 2 Apr 2014 07:24:47 +1000
Subject: [Distutils] Creating distribution for Python *module*,
	not package
In-Reply-To: <CACac1F9BYY_2L5wKQtmvWdf16nFCXrjA50JA4OJ9i7vvRc2CXw@mail.gmail.com>
References: <20140401222509.52d11704@x34f> <20140401224654.36d875f9@x34f>
 <CAKdhhwEvL6AVZ-sJ22q=4AKLPpfuUyk7ZMebufMinVDwVo0gQA@mail.gmail.com>
 <CACac1F9BYY_2L5wKQtmvWdf16nFCXrjA50JA4OJ9i7vvRc2CXw@mail.gmail.com>
Message-ID: <CADiSq7ceiYiR6qLOVMpJjU1SxgXt1kb-+1+P4jLCSQu8t0VdfQ@mail.gmail.com>

On 2 Apr 2014 06:41, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 1 April 2014 21:16, Michael Merickel <mmericke at gmail.com> wrote:
> > Yeah, setuptools is a bw-compat wrapper around distutils, so it tends to
> > only document its new features. This approach to documentation makes it
very
> > difficult to understand for newcomers that are not familiar with
distutils.
> > Especially now that setuptools is the de facto standard that everyone is
> > learning how to use.
>
> Agreed. The packaging documentation grew piecemeal over time, staring
> from a basis of the distutils documentation which was itself pretty
> incomprehensible. We're working on a new iteration of the packaging
> documentation at
> https://python-packaging-user-guide.readthedocs.org/en/latest/ but to
> be honest, it's very easy to take for granted a lot of basics, simply
> because the people doing the writing are too close to the packaging
> ecosystem.

Marcus has also been doing a lot of work trying to coordinate consistency
updates to the stdlib, pip, virtualenv and setuptools docs to get them in
sync with each other and bring them into line with the new overall user
guide (Now available as packaging.python.org, btw).

The fact several aspects of the setuptools docs assume readers are already
experienced distutils users just happens to be one of the harder problems
to resolve in that space, as fixing it requires extracting the relevant
info from the 15+ year old distutils docs and combining it with the already
complicated setuptools docs :P

(It's also influenced by the fact that more of the PyPA's focus for the
past year has been on cleaning up the installation side of the story. As
that has now reached a better state with the release of Python 3.4 and its
bundled pip bootstrapping mechanism, I expect we'll see more movement on
the build side of things this year)

Cheers,
Nick.

>For example, I don't think anyone considered documenting a
> single Python module as mentioned by the OP (it's usually the case
> that your needs very soon grow to the point where a single module
> isn't enough, but that's not a reason to not cover it - just a
> possible reason why we never thought of it).
>
> We'd be very grateful for any contributions to the packaging user
> guide, either in the form of pull requests or even just issues on the
> tracker pointing at things we should cover (or should cover better).
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140402/ef475ceb/attachment.html>

From donald at stufft.io  Tue Apr  1 23:31:55 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 1 Apr 2014 17:31:55 -0400
Subject: [Distutils] Creating distribution for Python *module*,
	not package
In-Reply-To: <CADiSq7ceiYiR6qLOVMpJjU1SxgXt1kb-+1+P4jLCSQu8t0VdfQ@mail.gmail.com>
References: <20140401222509.52d11704@x34f> <20140401224654.36d875f9@x34f>
 <CAKdhhwEvL6AVZ-sJ22q=4AKLPpfuUyk7ZMebufMinVDwVo0gQA@mail.gmail.com>
 <CACac1F9BYY_2L5wKQtmvWdf16nFCXrjA50JA4OJ9i7vvRc2CXw@mail.gmail.com>
 <CADiSq7ceiYiR6qLOVMpJjU1SxgXt1kb-+1+P4jLCSQu8t0VdfQ@mail.gmail.com>
Message-ID: <AC5DB9DB-C52F-47E0-ADF9-80C7852B913D@stufft.io>


On Apr 1, 2014, at 5:24 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 2 Apr 2014 06:41, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >
> > On 1 April 2014 21:16, Michael Merickel <mmericke at gmail.com> wrote:
> > > Yeah, setuptools is a bw-compat wrapper around distutils, so it tends to
> > > only document its new features. This approach to documentation makes it very
> > > difficult to understand for newcomers that are not familiar with distutils.
> > > Especially now that setuptools is the de facto standard that everyone is
> > > learning how to use.
> >
> > Agreed. The packaging documentation grew piecemeal over time, staring
> > from a basis of the distutils documentation which was itself pretty
> > incomprehensible. We're working on a new iteration of the packaging
> > documentation at
> > https://python-packaging-user-guide.readthedocs.org/en/latest/ but to
> > be honest, it's very easy to take for granted a lot of basics, simply
> > because the people doing the writing are too close to the packaging
> > ecosystem.
> 
> Marcus has also been doing a lot of work trying to coordinate consistency updates to the stdlib, pip, virtualenv and setuptools docs to get them in sync with each other and bring them into line with the new overall user guide (Now available as packaging.python.org, btw).
> 
> The fact several aspects of the setuptools docs assume readers are already experienced distutils users just happens to be one of the harder problems to resolve in that space, as fixing it requires extracting the relevant info from the 15+ year old distutils docs and combining it with the already complicated setuptools docs :P
> 
> (It's also influenced by the fact that more of the PyPA's focus for the past year has been on cleaning up the installation side of the story. As that has now reached a better state with the release of Python 3.4 and its bundled pip bootstrapping mechanism, I expect we'll see more movement on the build side of things this year)
> 
> Cheers,
> Nick.
> 
> >For example, I don't think anyone considered documenting a
> > single Python module as mentioned by the OP (it's usually the case
> > that your needs very soon grow to the point where a single module
> > isn't enough, but that's not a reason to not cover it - just a
> > possible reason why we never thought of it).
> >
> > We'd be very grateful for any contributions to the packaging user
> > guide, either in the form of pull requests or even just issues on the
> > tracker pointing at things we should cover (or should cover better).
> >
> > Paul
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

I was idly considering a setup.py.packaging.python.org or setup.py.python.org earlier that
was something like http://package.json.nodejitsu.com/

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140401/34bcefe5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140401/34bcefe5/attachment.sig>

From merwok at netwok.org  Fri Apr  4 00:46:39 2014
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Thu, 03 Apr 2014 18:46:39 -0400
Subject: [Distutils] Pycon
In-Reply-To: <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
Message-ID: <533DE4CF.1090403@netwok.org>

Hello,

Le 28/03/2014 19:10, Nick Coghlan a ?crit :
> On 29 March 2014 05:06, Daniel Holth <dholth at gmail.com> wrote:
>> Who is going to pycon? I will be there.
> As will I. ?ric Araujo was suggesting we do a Packaging Mini-Conf
> again as an open space, which sounded like a good idea to me, I just
> wasn't inclined to organise it myself this year :)
> 
> Open space page is at https://us.pycon.org/2014/community/openspaces/,
> but I don't know if ?ric already submitted a request for a space.
> 
> Also not sure if it would be better to do it before or after Noah's
> talk (the advantage of before is we might give him additional ideas,
> after is that we can go watch his talk first as a "year in review"
> kind of thing to remind us of how far we've come since this time last
> year)

I still think this is a good idea, but haven?t found time to set things
up.  If someone wants to take the idea and run with it, please do so as
soon as convenient!

Cheers

From lists at asheesh.org  Mon Apr  7 05:01:38 2014
From: lists at asheesh.org (Asheesh Laroia)
Date: Sun, 6 Apr 2014 20:01:38 -0700
Subject: [Distutils] "pip install -e ." vs. "python setup.py develop"
Message-ID: <CAFyrOVR=LRoR4yRoTyGWzZO8JnWgZFcGD0gHWTNEi3XGkM2tMQ@mail.gmail.com>

Hi lovely distutils people,

I have a question, as I prepare for my "Python packaging simplified, for
end users, app developers, and open source contributors" talk. I'm sure
I'll have more; I'll end up probably making a few threads about them, since
they'll come to me at random times.


For years, I've been recommending:

$ python setup.py develop


as a standard way to make something hackable and available in a virtualenv.
I notice that "python setup.py develop --user" exists, which is great, as
it means that you don't even need to bother with the virtualenv.


Having said that, I also notice that:

$ pip install -e .

does the same thing.

Should I be recommending one over the other?

I'm going to learn toward "pip install -e ." even though I haven't been
using it much personally, as it makes the talk more consistent -- I would
then be able to say, "Always use pip for doing your installing." But I
thought I'd ask about this. It seems that "pip install -e ." is the same as
"python setup.py develop" except that pip runs setup.py with setuptools
available, which addresses a problem where if the maintainer of a package's
setup.py file doesn't "from setuptools import setup", then "python setup.py
develop" won't work, whereas "pip install -e ." will always work.

Unless I'm mistaken. So the question is -- can someone sanity-check the
above?

I'm hoping to pretend to be an outsider for the purpose of empathizing with
the audience, and yet be enough of an insider to ask people on this list if
what I'm saying is consistent with Modern PyPA Doctrine (which generally
I'm happy to promote).

-- Asheesh.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140406/09f9a905/attachment.html>

From lists at asheesh.org  Mon Apr  7 05:04:08 2014
From: lists at asheesh.org (Asheesh Laroia)
Date: Sun, 6 Apr 2014 20:04:08 -0700
Subject: [Distutils] setup.py should use if __name__ == '__main__', right?
Message-ID: <CAFyrOVRLR2EAgNas11V+ROKVdmQOU=HNdy-j7xgukrPe0+9pVA@mail.gmail.com>

Hi nice distutils/PyPA people,

I had a question that probably is worth showing up in the archives. Namely:

It seems to me like bizarre bad form for the setup.py file to execute what
amounts to a main() function at import time.

I presume this is just some kind of historical accident, and if the early
authors of the distutils docs were careful, they'd have recommended:

from distutils.core import setup

if __name__ == '__main__':
    setup()  # FIXME args


rather than just:

from distutils.core import setup

setup()  # FIXME args



Is that an accurate assessment? If so, that's great, because I plan to
remark on this bemusedly in my talk. If there is a reason it is the way it
is, however, then I will avoid making a joke that is wrong in an important
way.

-- Asheesh.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140406/fec4b3ab/attachment.html>

From noah at coderanger.net  Mon Apr  7 05:05:17 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sun, 6 Apr 2014 20:05:17 -0700
Subject: [Distutils] "pip install -e ." vs. "python setup.py develop"
In-Reply-To: <CAFyrOVR=LRoR4yRoTyGWzZO8JnWgZFcGD0gHWTNEi3XGkM2tMQ@mail.gmail.com>
References: <CAFyrOVR=LRoR4yRoTyGWzZO8JnWgZFcGD0gHWTNEi3XGkM2tMQ@mail.gmail.com>
Message-ID: <E6D92399-933E-41D6-8A7C-C9EC8776CF9D@coderanger.net>

You should recommend using pip for it, mostly because as you said that will work even with packages that don't use setuptools :-) It also is required when doing a develop install with extras, though that requires a slightly more verbose syntax due to a bug in pip.

--Noah

On Apr 6, 2014, at 8:01 PM, Asheesh Laroia <lists at asheesh.org> wrote:

> Hi lovely distutils people,
> 
> I have a question, as I prepare for my "Python packaging simplified, for end users, app developers, and open source contributors" talk. I'm sure I'll have more; I'll end up probably making a few threads about them, since they'll come to me at random times.
> 
> 
> For years, I've been recommending:
> 
> $ python setup.py develop
> 
> 
> as a standard way to make something hackable and available in a virtualenv. I notice that "python setup.py develop --user" exists, which is great, as it means that you don't even need to bother with the virtualenv.
> 
> 
> Having said that, I also notice that:
> 
> $ pip install -e .
> 
> does the same thing.
> 
> Should I be recommending one over the other?
> 
> I'm going to learn toward "pip install -e ." even though I haven't been using it much personally, as it makes the talk more consistent -- I would then be able to say, "Always use pip for doing your installing." But I thought I'd ask about this. It seems that "pip install -e ." is the same as "python setup.py develop" except that pip runs setup.py with setuptools available, which addresses a problem where if the maintainer of a package's setup.py file doesn't "from setuptools import setup", then "python setup.py develop" won't work, whereas "pip install -e ." will always work.
> 
> Unless I'm mistaken. So the question is -- can someone sanity-check the above?
> 
> I'm hoping to pretend to be an outsider for the purpose of empathizing with the audience, and yet be enough of an insider to ask people on this list if what I'm saying is consistent with Modern PyPA Doctrine (which generally I'm happy to promote).
> 
> -- Asheesh.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140406/8be95b2e/attachment.sig>

From noah at coderanger.net  Mon Apr  7 05:06:20 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sun, 6 Apr 2014 20:06:20 -0700
Subject: [Distutils] setup.py should use if __name__ == '__main__',
	right?
In-Reply-To: <CAFyrOVRLR2EAgNas11V+ROKVdmQOU=HNdy-j7xgukrPe0+9pVA@mail.gmail.com>
References: <CAFyrOVRLR2EAgNas11V+ROKVdmQOU=HNdy-j7xgukrPe0+9pVA@mail.gmail.com>
Message-ID: <B025218D-5560-4C6F-B78F-7E81064CAECC@coderanger.net>

setup.py is not intended to be importable, so it has no "import time". Pretty sure I've never seen this patten used in a setup.py, nor would I think it has much semantic utility.

--Noah

On Apr 6, 2014, at 8:04 PM, Asheesh Laroia <lists at asheesh.org> wrote:

> Hi nice distutils/PyPA people,
> 
> I had a question that probably is worth showing up in the archives. Namely:
> 
> It seems to me like bizarre bad form for the setup.py file to execute what amounts to a main() function at import time.
> 
> I presume this is just some kind of historical accident, and if the early authors of the distutils docs were careful, they'd have recommended:
> 
> from distutils.core import setup
> 
> if __name__ == '__main__':
>     setup()  # FIXME args
> 
> 
> rather than just:
> 
> from distutils.core import setup
> 
> setup()  # FIXME args
> 
> 
> 
> Is that an accurate assessment? If so, that's great, because I plan to remark on this bemusedly in my talk. If there is a reason it is the way it is, however, then I will avoid making a joke that is wrong in an important way.
> 
> -- Asheesh.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140406/44e53677/attachment.sig>

From lists at asheesh.org  Mon Apr  7 06:07:40 2014
From: lists at asheesh.org (Asheesh Laroia)
Date: Sun, 6 Apr 2014 21:07:40 -0700
Subject: [Distutils] install_requires vs. requires_dist, and setup.cfg
Message-ID: <CAFyrOVT6w+XNBaymR3P7OLm9tdDUgM_Fw6pxybq2ePEG_k-esw@mail.gmail.com>

>From what I understand in PEP 345, "Requires-Dist" is the name in PKG-INFO
files for a distribution's dependencies on other distributions.

>From what I also understand from personal experience, when using
setuptools, install_requires is the name of the keyword argument to setup()
that specifies a dependency on another distribution.

Sanity-check: This doesn't mean that one should call setup() with a
"required_dist" keyword argument, right?

Relatedly,
https://mail.python.org/pipermail/distutils-sig/2010-January/015211.htmlseems
to suggest that one day in our glorious future, when we can lay down
our setup.py files and switch to setup.cfg, then we can call it
requires_dist in that format.

As I understand it. the setup.cfg revolution never happened, so I should
still encourage new people packaging their own things to write
install_requires in their calls to setuptools.setup().


Also, I just created a trivial setup.py that contains an "install_requires"
and doing "python setup.py develop --user" does seem to have created a
PKG-INFO file, but it doesn't have a "Requires-Dist" line.

Details of my test case:

?  rofl  python
Python 2.7.6 (default, Mar 22 2014, 15:40:47)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> print setuptools.__version__
3.4.2

?  rofl  cat setup.py
from setuptools import setup

setup(install_requires=['mwparser'])


?  rofl  grep mwparser UNKNOWN.egg-info/*
UNKNOWN.egg-info/requires.txt:mwparser


I expected to also see it in a Requires-Dist property within
UNKNOWN.egg-info/PKG-INFO. Is that a mistaken expectation? If so, what if
anything writes a "Requires-Dist" line anywhere?


Thanks for your patience with me, and for your speedy responses so far! I'm
really hoping to provide as much high quality information to people at
PyCon as I can, and grateful that so many wise people are willing to have
these conversations with me.

-- Asheesh.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140406/833007c5/attachment-0001.html>

From lists at asheesh.org  Mon Apr  7 06:32:40 2014
From: lists at asheesh.org (Asheesh Laroia)
Date: Sun, 6 Apr 2014 21:32:40 -0700
Subject: [Distutils] If you have nothing,
	begin with with putting get-pip.py in ~/bin/pip ???
Message-ID: <CAFyrOVT_vyCqB2irUONqadYMSgrgFkX++1OwkGYzfAD9+P2z_g@mail.gmail.com>

Hi all,

I'm trying to answer the question: "On a modern GNU/Linux, if you have no
'pip' command, and you want to live in the modern Python Packaging world,
how do you start?"

Hypothetically, let's say you're on the latest Python 2.7.

Also, I am presuming that you either don't have root on the system, or you
don't want to run things with sudo.


One answer I've seen (and given...) is to download virtualenv.py by itself,
and hope that it can create a virtualenv in e.g. ~/venv/base, and then
always activate it.

You can see that advice given here:
http://dubroy.com/blog/so-you-want-to-install-a-python-package/#the-better-way


This advice is not contradicted here <
http://www.virtualenv.org/en/latest/virtualenv.html#installation> although
it seems like the advice basically is, "Make sure you have pip already, but
if you don't, then we have no particular advice for you."


One alternative is to bootstrap via "get-pip.py", which happens (as far as
I can tell) to operate properly as a ~/bin/pip. Then you can "pip install
--user", and more.

However, get-pip.py <http://pip.readthedocs.org/en/latest/installing.html>
seems to, if you run it with no arguments, want to write to /usr/bin/pip.

What I'm hoping exists is a variant of get-pip.py that I can suggest people
put in ~/bin/pip (and tell them to fix their path to include $HOME/bin ),
and then they can pip install --user to their heart's content. You could
imagine this being called "heres-pip.py", rather than "get-pip.py", as in
my imagination, running "heres-pip.py" would not be supposed to have
side-effects like installing things unless you ask it to. You would just
drop it into place.


Shell transcripts follow, to help you see what I'm seeing, in case I'm
perhaps doing something wrong:

?  /tmp  python get-pip.py
Downloading/unpacking pip from
https://pypi.python.org/packages/py2.py3/p/pip/pip-1.5.4-py2.py3-none-any.whl#md5=f20bc8f31e322375ce06b26d73ce7b4f
  Downloading pip-1.5.4-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.4.1
    Uninstalling pip:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/tmp/tmpL4ws00/pip.zip/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/tmp/tmpL4ws00/pip.zip/pip/commands/install.py", line 279, in run
    requirement_set.install(install_options, global_options,
root=options.root_path)
  File "/tmp/tmpL4ws00/pip.zip/pip/req.py", line 1376, in install
    requirement.uninstall(auto_confirm=True)
  File "/tmp/tmpL4ws00/pip.zip/pip/req.py", line 594, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/tmp/tmpL4ws00/pip.zip/pip/req.py", line 1781, in remove
    renames(path, new_path)
  File "/tmp/tmpL4ws00/pip.zip/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/usr/lib/python2.7/shutil.py", line 303, in move
    os.unlink(src)
OSError: [Errno 13] Permission denied: '/usr/bin/pip'

?  /tmp  python get-pip.py --user --upgrade mwparser
Requirement already up-to-date: pip in
/home/paulproteus/.local/lib/python2.7/site-packages
Downloading/unpacking mwparser
  Downloading mwparser-0.1.tar.gz
  Running setup.py (path:/tmp/pip_build_paulproteus/mwparser/setup.py)
egg_info for package mwparser

Installing collected packages: mwparser
  Running setup.py install for mwparser
    /home/paulproteus/.local/lib/python2.7/site-packages/mwparser.py:13:
SyntaxWarning: assertion is always true, perhaps remove parentheses?
      assert('<' not in tag and '>' not in tag, "sample tag is 'ol'")
    /home/paulproteus/.local/lib/python2.7/site-packages/mwparser.py:14:
SyntaxWarning: assertion is always true, perhaps remove parentheses?
      assert(len(start_marker) == 1, "This code assumes start_marker's
length is 1")

Successfully installed mwparser
Cleaning up...



So it seems that you can (ab?)use get-pip.py as a personal ~/bin/pip. You
just have to know not to type "install". I'm guessing that "get-pip.py" is
a "pip" binary with "install pip" pre-pended to sys.argv, so by passing
more things, I happen to be able to cause it to do what I want. Either way,
I'd like to know more about what is sensible to recommend as a
bootstrapping process.

I presume that Mac OS users will have a similar issue as random GNU/Linux
users who lack pip.

I will say, if get-pip.py wanted to do "pip install --user" rather than
"pip install", then I'd be happy recommending it as-is.

(Other ideas welcome. I'm also semi-sure this is some kind of FAQ, but I
don't see it in my reading of the distutils-sig list, so I thought I'd
still ask it.)

Cheers,

Asheesh.

P.S. I am *so* tempted to add a no-op distribution called "install" to PyPI
so that "pip install install" would look like it works.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140406/9350804d/attachment.html>

From qwcode at gmail.com  Mon Apr  7 06:59:35 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 6 Apr 2014 21:59:35 -0700
Subject: [Distutils] install_requires vs. requires_dist, and setup.cfg
In-Reply-To: <CAFyrOVT6w+XNBaymR3P7OLm9tdDUgM_Fw6pxybq2ePEG_k-esw@mail.gmail.com>
References: <CAFyrOVT6w+XNBaymR3P7OLm9tdDUgM_Fw6pxybq2ePEG_k-esw@mail.gmail.com>
Message-ID: <CAPYWazpT+m7OSbbLcCRETPNiUD_DtAfz8__Y4Z7zh8w1Kbv2Tg@mail.gmail.com>

> From what I understand in PEP 345, "Requires-Dist" is the name in PKG-INFO
> files for a distribution's dependencies on other distributions.
>

setuptools/distutils has never implemented "Requires-Dist" from PEP345 for
PKG-INFO.
Otoh, wheel has, combined with PEP376, so you'll see it in *.dist-info
METADATA files, when installing from wheels.

 install_requires is the name of the keyword argument to setup() that
> specifies a dependency on another distribution.
>

yes


> Sanity-check: This doesn't mean that one should call setup() with a
> "required_dist" keyword argument, right?
>

right

As I understand it. the setup.cfg revolution never happened,
>

right, see http://packaging.python.org/en/latest/future.html



> Also, I just created a trivial setup.py that contains an
> "install_requires" and doing "python setup.py develop --user" does seem to
> have created a PKG-INFO file, but it doesn't have a "Requires-Dist" line.
>

look for the "requires.txt" file.  the dependency is in there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140406/b2976c44/attachment.html>

From qwcode at gmail.com  Mon Apr  7 07:32:45 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 6 Apr 2014 22:32:45 -0700
Subject: [Distutils] If you have nothing,
 begin with with putting get-pip.py in ~/bin/pip ???
In-Reply-To: <CAFyrOVT_vyCqB2irUONqadYMSgrgFkX++1OwkGYzfAD9+P2z_g@mail.gmail.com>
References: <CAFyrOVT_vyCqB2irUONqadYMSgrgFkX++1OwkGYzfAD9+P2z_g@mail.gmail.com>
Message-ID: <CAPYWazpheY5KA3DkVneK8Fmdb+EBcFdt2KJ1mBSf1q_wD8Ht4Q@mail.gmail.com>

>
>
> This advice is not contradicted here <
> http://www.virtualenv.org/en/latest/virtualenv.html#installation>
> although it seems like the advice basically is,
>
"Make sure you have pip already, but if you don't, then we have no
> particular advice for you."
>

that section actually offers 3 scenarios, not just pip
- "To install globally with pip"
- "To install globally from source"
- "To use locally from source"

One alternative is to bootstrap via "get-pip.py"
>

if you haven't seen this, take a look:
http://packaging.python.org/en/latest/tutorial.html

So it seems that you can (ab?)use get-pip.py as a personal ~/bin/pip. You
> just have to know not to type "install".
>

get-pip has pip inside it, and it's calling  "pip install -U pip
[setuptools]  sys.argv[1:]"
(see here:
https://raw.githubusercontent.com/pypa/pip/develop/contrib/get-pip.py)

the sys.argv[1:] is intended for passing arguments,  but technically you
can pass other packages and it will work.
I don't think we should be recommending "get-pip.py" be used as
"got-pip.py" in a talk.
the "got-pip.py" idea has come up before, and it's something that should be
discussed in full before recommending.

also, btw, it's currently under discussion to have pip's default behavior
(and get-pip's) be --user installs.
https://github.com/pypa/pip/issues/1668
https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0
https://github.com/pypa/python-packaging-user-guide/issues/43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140406/c01f3532/attachment-0001.html>

From carl at oddbird.net  Mon Apr  7 20:36:51 2014
From: carl at oddbird.net (Carl Meyer)
Date: Mon, 07 Apr 2014 12:36:51 -0600
Subject: [Distutils] setup.py should use if __name__ == '__main__',
	right?
In-Reply-To: <CAFyrOVRLR2EAgNas11V+ROKVdmQOU=HNdy-j7xgukrPe0+9pVA@mail.gmail.com>
References: <CAFyrOVRLR2EAgNas11V+ROKVdmQOU=HNdy-j7xgukrPe0+9pVA@mail.gmail.com>
Message-ID: <5342F043.3010101@oddbird.net>

On 04/06/2014 09:04 PM, Asheesh Laroia wrote:
> Hi nice distutils/PyPA people,
> 
> I had a question that probably is worth showing up in the archives. Namely:
> 
> It seems to me like bizarre bad form for the setup.py file to execute
> what amounts to a main() function at import time.
> 
> I presume this is just some kind of historical accident, and if the
> early authors of the distutils docs were careful, they'd have recommended:
> 
> from distutils.core import setup
> 
> if __name__ == '__main__':
>     setup()  # FIXME args
> 
> 
> rather than just:
> 
> from distutils.core import setup
> 
> setup()  # FIXME args
> 
> 
> 
> Is that an accurate assessment? If so, that's great, because I plan to
> remark on this bemusedly in my talk. If there is a reason it is the way
> it is, however, then I will avoid making a joke that is wrong in an
> important way.

I don't think there's anything wrong with guarding the 'setup()' call
with an 'if __name__ == "__main__"' (I have seen it done once or twice,
and wouldn't object to someone advocating for it), but it isn't common
practice, mostly because as Noah says there isn't any use case for
importing setup.py, so in this particular instance it's un-useful
boilerplate.

Carl

From dholth at gmail.com  Mon Apr  7 21:09:05 2014
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 7 Apr 2014 15:09:05 -0400
Subject: [Distutils] What is the syntax for passing conditional
 non-extra dependencies in setuptools?
In-Reply-To: <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
 <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
Message-ID: <CAG8k2+6bQ478_PTQ01wGgEY6PYgNhnMOw_7PW4ihvqXqCawfXQ@mail.gmail.com>

I think setuptools' :-separated environment markers mechanism is
great, and shouldn't need to be ratified anywhere. I also think the
actual environment markers specification is stable, although I noticed
the IPython wheel for example is using the deprecated .-separated
rather than _-separated identifiers.

These kinds of conditionals are obviously an important feature for
wheel and replace a common if/else statement inside setup.py, but can
only be specified in wheels by placing a copy of the requirements in
setup.cfg. It would be better if bdist_wheel could also just translate
the setuptools requirements.txt to its own format.

I assume extras_require = { ":sys_platform='win32'" :
"wincertstore==0.2" } would be likely to specify a non-extra (the
extra named '' or None?) dependency?


On Thu, Mar 27, 2014 at 2:39 PM, PJ Eby <pje at telecommunity.com> wrote:
> On Wed, Mar 26, 2014 at 11:29 PM, Daniel Holth <dholth at gmail.com> wrote:
>>
>> How do I specify a conditional (marker-guarded) non-extra dependency
>> in setuptools? The syntax for a conditional extra dependency is
>> currently:
>>
>>     extras_require = {
>>         "ssl:sys_platform=='win32'": "wincertstore==0.2",
>>         "certs": "certifi==1.0.1",
>>     },
>
>
> I only implemented support via extras, and the feature wasn't officially
> supported (still isn't, I don't think) because the PEP specifying the syntax
> wasn't fully baked yet.  I figured that if *only* setuptools itself used it,
> then if the syntax changed only setuptools would break...  but fix itself at
> the same time.
>
> The same cannot be said for any other package, so use at your own risk.  Or
> better yet, don't use it.  ;-)
>
> (At least, not until it's a documented feature w/a PEP-approved syntax.)
>
>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>

From dholth at gmail.com  Mon Apr  7 22:03:01 2014
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 7 Apr 2014 16:03:01 -0400
Subject: [Distutils] What is the syntax for passing conditional
 non-extra dependencies in setuptools?
In-Reply-To: <CAG8k2+6bQ478_PTQ01wGgEY6PYgNhnMOw_7PW4ihvqXqCawfXQ@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
 <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
 <CAG8k2+6bQ478_PTQ01wGgEY6PYgNhnMOw_7PW4ihvqXqCawfXQ@mail.gmail.com>
Message-ID: <CAG8k2+5X35aBRxYKS=-HucEOfa4GsSBH7V_9n4V4-C-Et=TxMA@mail.gmail.com>

OK, it does in fact work that way; empty-string-before-colon specifies
a default requirement with a marker.

Parses out to the following _dep_map with 'None' representing 'not an
extra dependency':

{None: [Requirement.parse('coding'), Requirement.parse('stuff')],
 'quux': [Requirement.parse('more-stuff')]}

requirements.txt:

coding

[:sys_platform=='linux2']
stuff

[quux:sys_platform=='linux2']
more_stuff


On Mon, Apr 7, 2014 at 3:09 PM, Daniel Holth <dholth at gmail.com> wrote:
> I think setuptools' :-separated environment markers mechanism is
> great, and shouldn't need to be ratified anywhere. I also think the
> actual environment markers specification is stable, although I noticed
> the IPython wheel for example is using the deprecated .-separated
> rather than _-separated identifiers.
>
> These kinds of conditionals are obviously an important feature for
> wheel and replace a common if/else statement inside setup.py, but can
> only be specified in wheels by placing a copy of the requirements in
> setup.cfg. It would be better if bdist_wheel could also just translate
> the setuptools requirements.txt to its own format.
>
> I assume extras_require = { ":sys_platform='win32'" :
> "wincertstore==0.2" } would be likely to specify a non-extra (the
> extra named '' or None?) dependency?
>
>
> On Thu, Mar 27, 2014 at 2:39 PM, PJ Eby <pje at telecommunity.com> wrote:
>> On Wed, Mar 26, 2014 at 11:29 PM, Daniel Holth <dholth at gmail.com> wrote:
>>>
>>> How do I specify a conditional (marker-guarded) non-extra dependency
>>> in setuptools? The syntax for a conditional extra dependency is
>>> currently:
>>>
>>>     extras_require = {
>>>         "ssl:sys_platform=='win32'": "wincertstore==0.2",
>>>         "certs": "certifi==1.0.1",
>>>     },
>>
>>
>> I only implemented support via extras, and the feature wasn't officially
>> supported (still isn't, I don't think) because the PEP specifying the syntax
>> wasn't fully baked yet.  I figured that if *only* setuptools itself used it,
>> then if the syntax changed only setuptools would break...  but fix itself at
>> the same time.
>>
>> The same cannot be said for any other package, so use at your own risk.  Or
>> better yet, don't use it.  ;-)
>>
>> (At least, not until it's a documented feature w/a PEP-approved syntax.)
>>
>>
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>

From pje at telecommunity.com  Tue Apr  8 01:00:23 2014
From: pje at telecommunity.com (PJ Eby)
Date: Mon, 7 Apr 2014 19:00:23 -0400
Subject: [Distutils] What is the syntax for passing conditional
 non-extra dependencies in setuptools?
In-Reply-To: <CAG8k2+5X35aBRxYKS=-HucEOfa4GsSBH7V_9n4V4-C-Et=TxMA@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
 <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
 <CAG8k2+6bQ478_PTQ01wGgEY6PYgNhnMOw_7PW4ihvqXqCawfXQ@mail.gmail.com>
 <CAG8k2+5X35aBRxYKS=-HucEOfa4GsSBH7V_9n4V4-C-Et=TxMA@mail.gmail.com>
Message-ID: <CALeMXf4wtB7isLm8rgPP=o9d4Wseno_avp4VruvnCvjAVDGVWw@mail.gmail.com>

On Mon, Apr 7, 2014 at 4:03 PM, Daniel Holth <dholth at gmail.com> wrote:

> OK, it does in fact work that way; empty-string-before-colon specifies
> a default requirement with a marker.
>
> Parses out to the following _dep_map with 'None' representing 'not an
> extra dependency':
>
> {None: [Requirement.parse('coding'), Requirement.parse('stuff')],
>  'quux': [Requirement.parse('more-stuff')]}
>
> requirements.txt:
>
> coding
>
> [:sys_platform=='linux2']
> stuff
>
> [quux:sys_platform=='linux2']
> more_stuff
>
>
Huh.  Either I did that intentionally and forgot about it, or it just fell
out as a brilliant accidental side effect of the orthogonal design of the
parsing functions.  Either way, I feel like I did something smart there.
;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140407/f07dcb42/attachment.html>

From dholth at gmail.com  Tue Apr  8 05:15:11 2014
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 7 Apr 2014 23:15:11 -0400
Subject: [Distutils] Comments on PEP 426 and 459
Message-ID: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>

I read through the latest versions of PEP 426 and 459 "Metadata 2.0
and extensions". Here are my comments.

The PEP suggests "setup.py dist_info" is a thing. Only "setup.py
egg_info" works. It might make sense to refactor bdist_wheel to
include a dist_info command, but if done badly it would break
distributions that rely on the pluggable nature of .egg-info
generation (most every part of the egg-info directory is written by a
separate plugin).

Why do we care about "Source labels MUST be unique within each project
and MUST NOT match any defined version for the project."?

I still think meta distributions are unnecessary and hard to
understand. It's either a lot of trouble to try to discourage ==
dependencies while still allowing them sometimes, or an
overcomplicated way to try to define requirements.txt 2.0 by allowing
distributions that aren't exactly distributions.

This sounds like a pain:
" To avoid malicious hijacking of names, when interpreting metadata retrieved
 from a public index server, automated tools MUST NOT pay any attention to
 ``"provides"`` entries that do not correspond to a published distribution. "

I'm not convinced that console scripts should not continue to be a
specially named entry point / export rather than their own very
similar thing.

Overall both peps are quite good. We need to be able to put them to
use to advance our goal of providing a transparently
backwards-compatible Python packaging system.

My peps repository on bitbucket has an updated json schema.

From ncoghlan at gmail.com  Tue Apr  8 05:32:22 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 8 Apr 2014 13:32:22 +1000
Subject: [Distutils] install_requires vs. requires_dist, and setup.cfg
In-Reply-To: <CAPYWazpT+m7OSbbLcCRETPNiUD_DtAfz8__Y4Z7zh8w1Kbv2Tg@mail.gmail.com>
References: <CAFyrOVT6w+XNBaymR3P7OLm9tdDUgM_Fw6pxybq2ePEG_k-esw@mail.gmail.com>
 <CAPYWazpT+m7OSbbLcCRETPNiUD_DtAfz8__Y4Z7zh8w1Kbv2Tg@mail.gmail.com>
Message-ID: <CADiSq7cTEu-7rN+7wxxC0QbqOJ5aV0NQ02=zswozZNAk=1v58Q@mail.gmail.com>

On 7 Apr 2014 00:59, "Marcus Smith" <qwcode at gmail.com> wrote:
>
>
>> From what I understand in PEP 345, "Requires-Dist" is the name in
PKG-INFO files for a distribution's dependencies on other distributions.
>
>
> setuptools/distutils has never implemented "Requires-Dist" from PEP345
for PKG-INFO.
> Otoh, wheel has, combined with PEP376, so you'll see it in *.dist-info
METADATA files, when installing from wheels.
>
>>  install_requires is the name of the keyword argument to setup() that
specifies a dependency on another distribution.
>
>
> yes
>
>>
>> Sanity-check: This doesn't mean that one should call setup() with a
"required_dist" keyword argument, right?
>
>
> right
>
>> As I understand it. the setup.cfg revolution never happened,
>
>
> right, see http://packaging.python.org/en/latest/future.html
>
>
>>
>> Also, I just created a trivial setup.py that contains an
"install_requires" and doing "python setup.py develop --user" does seem to
have created a PKG-INFO file, but it doesn't have a "Requires-Dist" line.
>
>
> look for the "requires.txt" file.  the dependency is in there.

And for some additional context: the currently active metadata standards
really consist of metadata 1.1 + the setuptools defined metadata files for
dependencies and entry points. The 1.x metadata PEPs really don't stand on
their own (not even 1.2).

We know this is confusing, but it has been working well enough to bump
metadata 2.0 down the priority list for the past several months. With any
luck, we'll be able to thrash out the remaining issues here at PyCon and
start to give people a clearer alternative (once the PEPs have been
accepted and setuptools/pip/etc updated accordingly).

Cheers,
Nick.

>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140408/64f159bd/attachment-0001.html>

From ncoghlan at gmail.com  Tue Apr  8 05:40:35 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 8 Apr 2014 13:40:35 +1000
Subject: [Distutils] What is the syntax for passing conditional
 non-extra dependencies in setuptools?
In-Reply-To: <CAG8k2+6bQ478_PTQ01wGgEY6PYgNhnMOw_7PW4ihvqXqCawfXQ@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
 <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
 <CAG8k2+6bQ478_PTQ01wGgEY6PYgNhnMOw_7PW4ihvqXqCawfXQ@mail.gmail.com>
Message-ID: <CADiSq7dmOiW-5hEYs+U4FY48qhCagd2W01CQT74ajHbMr4HR-g@mail.gmail.com>

On 7 Apr 2014 15:09, "Daniel Holth" <dholth at gmail.com> wrote:
>
> I think setuptools' :-separated environment markers mechanism is
> great, and shouldn't need to be ratified anywhere. I also think the
> actual environment markers specification is stable, although I noticed
> the IPython wheel for example is using the deprecated .-separated
> rather than _-separated identifiers.

I think the one open question was whether or not to tolerate those dots
from the previous iteration of the syntax where applicable.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140408/f35592ba/attachment.html>

From ncoghlan at gmail.com  Tue Apr  8 05:59:43 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 8 Apr 2014 13:59:43 +1000
Subject: [Distutils] Comments on PEP 426 and 459
In-Reply-To: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>
References: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>
Message-ID: <CADiSq7fExts3=d91=mz5tLGhCJ+eH769OAEdx0C09SXtvXq0aw@mail.gmail.com>

On 7 Apr 2014 23:15, "Daniel Holth" <dholth at gmail.com> wrote:
>
> I read through the latest versions of PEP 426 and 459 "Metadata 2.0
> and extensions". Here are my comments.
>
> The PEP suggests "setup.py dist_info" is a thing. Only "setup.py
> egg_info" works. It might make sense to refactor bdist_wheel to
> include a dist_info command, but if done badly it would break
> distributions that rely on the pluggable nature of .egg-info
> generation (most every part of the egg-info directory is written by a
> separate plugin).
>
> Why do we care about "Source labels MUST be unique within each project
> and MUST NOT match any defined version for the project."?

Hmm, I think that stems from an earlier versions that allowed requiring a
source label. I'll double check.

> I still think meta distributions are unnecessary and hard to
> understand. It's either a lot of trouble to try to discourage ==
> dependencies while still allowing them sometimes, or an
> overcomplicated way to try to define requirements.txt 2.0 by allowing
> distributions that aren't exactly distributions.

PyObjC at least needs them, and things like OpenStack releases would need
it (if published through PyPI).

Meta distribution is an essential part of encouraging decoupled component
design, without increasing the complexity of consuming an overall "release".

>
> This sounds like a pain:
> " To avoid malicious hijacking of names, when interpreting metadata
retrieved
>  from a public index server, automated tools MUST NOT pay any attention to
>  ``"provides"`` entries that do not correspond to a published
distribution. "

Yup. Necessary security feature for virtual provides in an untrusted
environment, though.

>
> I'm not convinced that console scripts should not continue to be a
> specially named entry point / export rather than their own very
> similar thing.

This is mostly a usability issue, but also relates to the stricter
requirements for installers.

>From a usability perspective, I don't believe users should have to learn
about the arbitrary entry point support just to request install time script
generation. "scripts are just a specific kind of plugin" is a case of
confusing one possible implementation model with the semantic model uses
actually care about (where scripts and plugins are different things).

>From an installer perspective, the key difference is that the *installer
itself* is one of the intended consumers of the metadata, so if the
installer doesn't know how to handle script metadata, it should fail,
rather than silently doing the wrong thing. By contrast, most other entry
points are only relevant at *runtime* - the installer doesn't need to know
anything about them.

>
> Overall both peps are quite good. We need to be able to put them to
> use to advance our goal of providing a transparently
> backwards-compatible Python packaging system.

Yeah, agreed - we're to the point now where finalising these PEPs is pretty
high on the packaging agenda :)

> My peps repository on bitbucket has an updated json schema.

Sweet, thanks.

Cheers,
Nick.

> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140408/ff28c75e/attachment.html>

From reinout at vanrees.org  Tue Apr  8 09:41:16 2014
From: reinout at vanrees.org (Reinout van Rees)
Date: Tue, 08 Apr 2014 09:41:16 +0200
Subject: [Distutils] mr.developer dev eggs not showing up in sys.path
Message-ID: <li096s$qo2$1@ger.gmane.org>

Hi,

A colleague came up with a problem yesterday (and last week) that only 
he seemed to have. He however found a stackoverflow question with the 
same problem: http://stackoverflow.com/q/21241451/27401

Basically: develop eggs from mr.developer don't show up in the sys.path 
of the bin/python (zc.recipe.egg script) and bin/django (djangorecipe 
part). Despite:

- The eggs being mentioned as dependency in the project's setup.py. I 
triple-verified this; bog standard setup we use in all our projects.

- One project uses buildout 1.4.4, the other 2.x.

- Tried the latest mr.developer and one from half a year ago or so.

- Yep, install_requires mentions them. I verified it with 
egg-info/requires.txt: they're there.

- Yep, develop-egg links are present.

- Upgraded setuptools, downgraded setuptools (all pretty recent 
versions, though): no change.

- It all runs fine on my computer...

The only solution is to add the missing eggs to the "eggs=" list of the 
relevant parts. Even though that should really be unnecessary as they're 
mentioned in the proper setup.py.


=> Anyone seen this behaviour before? Any idea of where to look? I'll 
try reproducing it locally, but it can't hurt to ask :-)


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From dholth at gmail.com  Tue Apr  8 14:44:35 2014
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 8 Apr 2014 08:44:35 -0400
Subject: [Distutils] Comments on PEP 426 and 459
In-Reply-To: <CADiSq7fExts3=d91=mz5tLGhCJ+eH769OAEdx0C09SXtvXq0aw@mail.gmail.com>
References: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>
 <CADiSq7fExts3=d91=mz5tLGhCJ+eH769OAEdx0C09SXtvXq0aw@mail.gmail.com>
Message-ID: <CAG8k2+4+=jDsBnPoq6Nz2jDxhEWVMuV-cGAtPFkg45n6ZOrykA@mail.gmail.com>

On Mon, Apr 7, 2014 at 11:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 7 Apr 2014 23:15, "Daniel Holth" <dholth at gmail.com> wrote:
>>
>> I read through the latest versions of PEP 426 and 459 "Metadata 2.0
>> and extensions". Here are my comments.
>>
>> The PEP suggests "setup.py dist_info" is a thing. Only "setup.py
>> egg_info" works. It might make sense to refactor bdist_wheel to
>> include a dist_info command, but if done badly it would break
>> distributions that rely on the pluggable nature of .egg-info
>> generation (most every part of the egg-info directory is written by a
>> separate plugin).
>>
>> Why do we care about "Source labels MUST be unique within each project
>> and MUST NOT match any defined version for the project."?
>
> Hmm, I think that stems from an earlier versions that allowed requiring a
> source label. I'll double check.
>
>> I still think meta distributions are unnecessary and hard to
>> understand. It's either a lot of trouble to try to discourage ==
>> dependencies while still allowing them sometimes, or an
>> overcomplicated way to try to define requirements.txt 2.0 by allowing
>> distributions that aren't exactly distributions.
>
> PyObjC at least needs them, and things like OpenStack releases would need it
> (if published through PyPI).
>
> Meta distribution is an essential part of encouraging decoupled component
> design, without increasing the complexity of consuming an overall "release".

Is the entire justification for this feature not "we don't trust you
to put == in run_requires", while exposing the same now-frustated
users to the same problem once they rename run_ to meta_? Instead of
having a separate but nearly identical top-level key, the == in
run_requires and not providing any actual modules could cause a
meta-distribution, and unwanted == in poorly written upstream
dependencies could be handled by a "patch" directive in the install
tool (like in buildout).

From vinay_sajip at yahoo.co.uk  Tue Apr  8 18:12:38 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 8 Apr 2014 17:12:38 +0100 (BST)
Subject: [Distutils] Comments on PEP 426 and 459
In-Reply-To: <CAG8k2+4+=jDsBnPoq6Nz2jDxhEWVMuV-cGAtPFkg45n6ZOrykA@mail.gmail.com>
References: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>
 <CADiSq7fExts3=d91=mz5tLGhCJ+eH769OAEdx0C09SXtvXq0aw@mail.gmail.com>
 <CAG8k2+4+=jDsBnPoq6Nz2jDxhEWVMuV-cGAtPFkg45n6ZOrykA@mail.gmail.com>
Message-ID: <1396973558.27702.YahooMailNeo@web172404.mail.ir2.yahoo.com>





> Is the entire justification for this feature not "we
> don't trust you?to put == in run_requires",

I'm confused - where would the == which you speak of go? IIUC a run_requires looks like

"run_requires": [ { "requires": ["SoftCushions"], "extra": "warmup" }
]


where each dictionary in the list could have "requires", "extra" and "environment" keys. The "environment" value might have "==" as part of a marker expression. Are you talking about something else?

Regards,

Vinay Sajip

From dholth at gmail.com  Tue Apr  8 19:04:32 2014
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 8 Apr 2014 13:04:32 -0400
Subject: [Distutils] Comments on PEP 426 and 459
In-Reply-To: <1396973558.27702.YahooMailNeo@web172404.mail.ir2.yahoo.com>
References: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>
 <CADiSq7fExts3=d91=mz5tLGhCJ+eH769OAEdx0C09SXtvXq0aw@mail.gmail.com>
 <CAG8k2+4+=jDsBnPoq6Nz2jDxhEWVMuV-cGAtPFkg45n6ZOrykA@mail.gmail.com>
 <1396973558.27702.YahooMailNeo@web172404.mail.ir2.yahoo.com>
Message-ID: <CAG8k2+62=JKOQbV9_ELo3ujdvKktF2TTAv=p95UC=-F4Q4Rk5A@mail.gmail.com>

On Tue, Apr 8, 2014 at 12:12 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>
>
>
>
>> Is the entire justification for this feature not "we
>> don't trust you to put == in run_requires",
>
> I'm confused - where would the == which you speak of go? IIUC a run_requires looks like
>
> "run_requires": [ { "requires": ["SoftCushions"], "extra": "warmup" }
> ]
>
>
> where each dictionary in the list could have "requires", "extra" and "environment" keys. The "environment" value might have "==" as part of a marker expression. Are you talking about something else?
>
> Regards,
>
> Vinay Sajip

That would be the == in "requires": ["SoftCushions == 4"] which IIUC
in the current PEP would be allowed in meta_ but disallowed in
run_requirements.

Nick has said in the past that he thinks it's easier to teach the 4
specific concepts without having to [first] explain the general case.
Most of our disagreements have been about this issue: whether a
distinction is important enough to deserve special treatment in the
spec, or whether it can be expressed when needed as a consequence of
having a more general format.

From dholth at gmail.com  Tue Apr  8 19:34:10 2014
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 8 Apr 2014 13:34:10 -0400
Subject: [Distutils] What is the syntax for passing conditional
 non-extra dependencies in setuptools?
In-Reply-To: <CADiSq7dmOiW-5hEYs+U4FY48qhCagd2W01CQT74ajHbMr4HR-g@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
 <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
 <CAG8k2+6bQ478_PTQ01wGgEY6PYgNhnMOw_7PW4ihvqXqCawfXQ@mail.gmail.com>
 <CADiSq7dmOiW-5hEYs+U4FY48qhCagd2W01CQT74ajHbMr4HR-g@mail.gmail.com>
Message-ID: <CAG8k2+5Zu5e4Jz8zicmHdOdbq61tjL1Yf2G0zY8wTV8nAV=NbQ@mail.gmail.com>

On Mon, Apr 7, 2014 at 11:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 7 Apr 2014 15:09, "Daniel Holth" <dholth at gmail.com> wrote:
>>
>> I think setuptools' :-separated environment markers mechanism is
>> great, and shouldn't need to be ratified anywhere. I also think the
>> actual environment markers specification is stable, although I noticed
>> the IPython wheel for example is using the deprecated .-separated
>> rather than _-separated identifiers.
>
> I think the one open question was whether or not to tolerate those dots from
> the previous iteration of the syntax where applicable.

I'd lean towards saying we are stuck with them for now in the
.dist-info directories or requirements.txt, but setuptools could
choose to forbid you from passing new dots through setup.py. The only
problem the dots caused was forgetting whether you should use dots or
underscores, and now it's easy to remember that the answer is always
underscore.

From vinay_sajip at yahoo.co.uk  Tue Apr  8 19:56:09 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 8 Apr 2014 18:56:09 +0100 (BST)
Subject: [Distutils] Comments on PEP 426 and 459
In-Reply-To: <CAG8k2+62=JKOQbV9_ELo3ujdvKktF2TTAv=p95UC=-F4Q4Rk5A@mail.gmail.com>
References: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>	<CADiSq7fExts3=d91=mz5tLGhCJ+eH769OAEdx0C09SXtvXq0aw@mail.gmail.com>	<CAG8k2+4+=jDsBnPoq6Nz2jDxhEWVMuV-cGAtPFkg45n6ZOrykA@mail.gmail.com>	<1396973558.27702.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CAG8k2+62=JKOQbV9_ELo3ujdvKktF2TTAv=p95UC=-F4Q4Rk5A@mail.gmail.com>
Message-ID: <1396979769.47727.YahooMailNeo@web172401.mail.ir2.yahoo.com>

>?That would be the == in "requires": ["SoftCushions == 4"] which IIUC
>?in the current PEP would be allowed in meta_ but disallowed in
>?run_requirements.


Okay, I see it now. Thanks.

Regards,

Vinay Sajip

From ws at gocept.com  Wed Apr  9 08:00:46 2014
From: ws at gocept.com (Wolfgang Schnerring)
Date: Wed, 9 Apr 2014 08:00:46 +0200
Subject: [Distutils] mr.developer dev eggs not showing up in sys.path
References: <li096s$qo2$1@ger.gmane.org>
Message-ID: <20140409080046.461f10b4@widnar>

* Reinout van Rees <reinout at vanrees.org> [2014-04-08 09:41]:
> Basically: develop eggs from mr.developer don't show up in the sys.path 
> of the bin/python (zc.recipe.egg script) and bin/django (djangorecipe 
> part). Despite:
> 
> - Yep, install_requires mentions them. I verified it with 
> egg-info/requires.txt: they're there.

Out-of-date/wrong egg-info would have been my guess (which can be devilishly
hard to find if one is not thingking about it). I had a similar situation
after I copied one egg to extract bits from it into a separate egg (say, I had
egg "foo" which contained foo.bar and foo.baz, and I wanted to move foo.baz
into its own package), but still had the old egg-info lying around (that said
"foo.baz can be found in the foo egg").

> => Anyone seen this behaviour before? Any idea of where to look? I'll 
> try reproducing it locally, but it can't hurt to ask :-)

Other than the above, I'm sorry, nope.

Wolfgang


From ncoghlan at gmail.com  Wed Apr  9 12:41:07 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 9 Apr 2014 06:41:07 -0400
Subject: [Distutils] Comments on PEP 426 and 459
In-Reply-To: <CAG8k2+4+=jDsBnPoq6Nz2jDxhEWVMuV-cGAtPFkg45n6ZOrykA@mail.gmail.com>
References: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>
 <CADiSq7fExts3=d91=mz5tLGhCJ+eH769OAEdx0C09SXtvXq0aw@mail.gmail.com>
 <CAG8k2+4+=jDsBnPoq6Nz2jDxhEWVMuV-cGAtPFkg45n6ZOrykA@mail.gmail.com>
Message-ID: <CADiSq7caBeDcTwzE67iHd8DJEw0Eo9rBx+iHWzd5rr9AOxxqAg@mail.gmail.com>

On 8 April 2014 08:44, Daniel Holth <dholth at gmail.com> wrote:
> On Mon, Apr 7, 2014 at 11:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 7 Apr 2014 23:15, "Daniel Holth" <dholth at gmail.com> wrote:
>> Meta distribution is an essential part of encouraging decoupled component
>> design, without increasing the complexity of consuming an overall "release".
>
> Is the entire justification for this feature not "we don't trust you
> to put == in run_requires", while exposing the same now-frustated
> users to the same problem once they rename run_ to meta_? Instead of
> having a separate but nearly identical top-level key, the == in
> run_requires and not providing any actual modules could cause a
> meta-distribution, and unwanted == in poorly written upstream
> dependencies could be handled by a "patch" directive in the install
> tool (like in buildout).

There's a reason I refer to the new dependency model as "semantic
dependencies" - the five categories are based on what the dependencies
*mean*, moreso than when they're relevant. Those meanings then
translate to whether or not pinning is appropriate on a public index
server.

With dev vs build vs run/meta vs test, the distinctions are obvious,
because they're dependencies that are needed for *specific
activities*:

- run the software (run+meta)
- run the automated test suite (test+run+meta)
- build the software (build)
- work on the software (dev+build+test+run+meta)

With run vs meta, the distinction is a little more subtle, as it
relates to the *commitments being made to users*, even though they
both apply to running the software.

The normal case is to use "run" - that means you're operating in a
pure publisher role, and integration and responding in a timely manner
to security updates for dependencies is the responsibility of the
software consumer. This is what you're going to want in almost all
cases when publishing to a public index server like PyPI.

The "meta" case is different, as that represents a *pre-integrated*
bundle of software - you're operating in a hybrid publisher/integrator
role, and hence taking on the responsibility for monitoring for and
responding to security updates in the pinned dependencies yourself. On
a public index server, this is designed to cover cases where projects
are developed and released in a coordinated fashion, and are able to
be consumed independently (so they don't depend on each other), but
when you combine them together, they must use consistent versions. I
just checked, and it turns out I haven't published a version of PEP
459 with the "constraints" extension defined at all yet (see
https://bitbucket.org/pypa/pypi-metadata-formats/commits/8cffbf2bacb0da8fd3394216882626e66d97bcc9),
but there's a case to be made for allowing version compatibility
constraints as part of that extension (it is currently aimed at
killing off the "Supports Environments" field in the main metadata
spec, as well as allowing the scientific community to handle ABI
compatibility constraints for NumPy).

Meta-requirements are also a good fit when distributing *applications*
(rather than libraries or frameworks), and that's a use case more
likely to arise on private index servers (although it's not impossible
to do on a public index server - it would potentially make sense for
open source server applications like graphite or sentry, for example,
to make selective use of meta-requirements).

By explicitly splitting meta requirements from normal runtime
requirements, that allows us to change the affordances of the tools to
nudge less experienced users in the right direction (as run_requires
is more intuitive than meta_requires, so a lot of folks will simply
believe version pinning isn't allowed on PyPI), while still supporting
the more advanced use cases where version pinning actually makes sense
(via meta_requires). The split also makes subsequent dependency audits
easier, as it makes it clearer where the responsibility for responding
to security updates in dependencies lies (run requires = with the
software consumer, meta requires = with the software publisher).

I'll likely add a bit more on this topic to the rationale section for
the new semantic dependency system in PEP 426 (or perhaps just a
reference back to this post).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Wed Apr  9 14:14:15 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 9 Apr 2014 08:14:15 -0400
Subject: [Distutils] Comments on PEP 426 and 459
In-Reply-To: <CADiSq7caBeDcTwzE67iHd8DJEw0Eo9rBx+iHWzd5rr9AOxxqAg@mail.gmail.com>
References: <CAG8k2+4pyBm5wr_zJYdR6Fj5nKPqD0xTW8L1bFpWO3_2=aBXdQ@mail.gmail.com>
 <CADiSq7fExts3=d91=mz5tLGhCJ+eH769OAEdx0C09SXtvXq0aw@mail.gmail.com>
 <CAG8k2+4+=jDsBnPoq6Nz2jDxhEWVMuV-cGAtPFkg45n6ZOrykA@mail.gmail.com>
 <CADiSq7caBeDcTwzE67iHd8DJEw0Eo9rBx+iHWzd5rr9AOxxqAg@mail.gmail.com>
Message-ID: <CAG8k2+7_c_4FMGm1FKgpEXvrXiUXRFBsGgKq6kx6DqbNA5RxvA@mail.gmail.com>

On Wed, Apr 9, 2014 at 6:41 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 8 April 2014 08:44, Daniel Holth <dholth at gmail.com> wrote:
>> On Mon, Apr 7, 2014 at 11:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>> On 7 Apr 2014 23:15, "Daniel Holth" <dholth at gmail.com> wrote:
>>> Meta distribution is an essential part of encouraging decoupled component
>>> design, without increasing the complexity of consuming an overall "release".
>>
>> Is the entire justification for this feature not "we don't trust you
>> to put == in run_requires", while exposing the same now-frustated
>> users to the same problem once they rename run_ to meta_? Instead of
>> having a separate but nearly identical top-level key, the == in
>> run_requires and not providing any actual modules could cause a
>> meta-distribution, and unwanted == in poorly written upstream
>> dependencies could be handled by a "patch" directive in the install
>> tool (like in buildout).
>
> There's a reason I refer to the new dependency model as "semantic
> dependencies" - the five categories are based on what the dependencies
> *mean*, moreso than when they're relevant. Those meanings then
> translate to whether or not pinning is appropriate on a public index
> server.
>
> With dev vs build vs run/meta vs test, the distinctions are obvious,
> because they're dependencies that are needed for *specific
> activities*:
>
> - run the software (run+meta)
> - run the automated test suite (test+run+meta)
> - build the software (build)
> - work on the software (dev+build+test+run+meta)
>
> With run vs meta, the distinction is a little more subtle, as it
> relates to the *commitments being made to users*, even though they
> both apply to running the software.
>
> The normal case is to use "run" - that means you're operating in a
> pure publisher role, and integration and responding in a timely manner
> to security updates for dependencies is the responsibility of the
> software consumer. This is what you're going to want in almost all
> cases when publishing to a public index server like PyPI.
>
> The "meta" case is different, as that represents a *pre-integrated*
> bundle of software - you're operating in a hybrid publisher/integrator
> role, and hence taking on the responsibility for monitoring for and
> responding to security updates in the pinned dependencies yourself. On
> a public index server, this is designed to cover cases where projects
> are developed and released in a coordinated fashion, and are able to
> be consumed independently (so they don't depend on each other), but
> when you combine them together, they must use consistent versions. I
> just checked, and it turns out I haven't published a version of PEP
> 459 with the "constraints" extension defined at all yet (see
> https://bitbucket.org/pypa/pypi-metadata-formats/commits/8cffbf2bacb0da8fd3394216882626e66d97bcc9),
> but there's a case to be made for allowing version compatibility
> constraints as part of that extension (it is currently aimed at
> killing off the "Supports Environments" field in the main metadata
> spec, as well as allowing the scientific community to handle ABI
> compatibility constraints for NumPy).
>
> Meta-requirements are also a good fit when distributing *applications*
> (rather than libraries or frameworks), and that's a use case more
> likely to arise on private index servers (although it's not impossible
> to do on a public index server - it would potentially make sense for
> open source server applications like graphite or sentry, for example,
> to make selective use of meta-requirements).
>
> By explicitly splitting meta requirements from normal runtime
> requirements, that allows us to change the affordances of the tools to
> nudge less experienced users in the right direction (as run_requires
> is more intuitive than meta_requires, so a lot of folks will simply
> believe version pinning isn't allowed on PyPI), while still supporting
> the more advanced use cases where version pinning actually makes sense
> (via meta_requires). The split also makes subsequent dependency audits
> easier, as it makes it clearer where the responsibility for responding
> to security updates in dependencies lies (run requires = with the
> software consumer, meta requires = with the software publisher).
>
> I'll likely add a bit more on this topic to the rationale section for
> the new semantic dependency system in PEP 426 (or perhaps just a
> reference back to this post).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

I can live with it.

From charlesc-distutils-python.org at pyropus.ca  Wed Apr  9 21:51:48 2014
From: charlesc-distutils-python.org at pyropus.ca (Charles Cazabon)
Date: Wed, 9 Apr 2014 13:51:48 -0600
Subject: [Distutils] Failure to extract svn revision
Message-ID: <20140409195148.GA28144@pyropus.ca>

Greetings,

I'm having trouble getting setuptools to properly extract the svn revision and
write it into the <package>.egg-info/PKG-INFO file in a new environment.  It
has worked correctly for years in our previous environments, so I'm not sure
what I'm doing wrong at this point.

The environments where this works are virtualenvs based on Python 2.6 (and
previously to that, 2.5) on various versions of Ubuntu.  The new environment
is a Python 2.7.6 virtualenv on what will become Ubuntu 14.04 "Trusty Tahr"
when that's finalized.

The system-wide Python used to bootstrap the virtualenv has setuptools 3.3
installed.  I've also installed setuptools 3.4.3 in the virtualenv to see if
that helped, but it makes no difference.  Our package's setup.cfg file
contains the usual:

  [egg_info]
  tag_build = dev
  tag_svn_revision = true

Running `python setup.py develop` (or egg_info directly, or one of the other
commands that call egg_info) results in the <package>.egg-info/PKG-INFO
containing:

  Metadata-Version: 1.0
  Name: <packagename>
  Version: 5.0dev

(with all other values being UNKNOWN, which is fine).  In our other
environments, the version properly gets the svn revision number appended, like
this:

  Version: 5.0dev-r15364

The upcoming Ubuntu release features subversion 1.8.8 instead of the 1.6.x in
the last long-term-support release, so I thought that might be the source of
the problem -- but manually running the code from svn_utils.py shows it seems
to work:

  >>> from setuptools.svn_utils import *
  >>> working_copy = '/path/to/srcdir/subdir'
  >>> SvnInfo().load(working_copy).get_revision()
  15364

In case it's relevant, setup.py/setup.cfg and the package I'm dealing with are
in a subdirectory of the working copy, so the .svn/ dir and its files etc are
in the parent directory, not the directory setup.py runs in.  `svnversion`
correctly fetches the revision number regardless, of course.

I haven't found any docs/FAQs/etc that would seem to be related - all the
subversion-related issues seem to be from the earlier transition to 1.5 or
1.6, which is long past.

Am I doing something wrong here?  Can anyone suggest why the revision tagging
is failing to work in my new environment?

Thanks,

Charles
-- 
------------------------------------------------------------------
Charles Cazabon         <charlesc-distutils-python.org at pyropus.ca>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

From charlesc-distutils-python.org at pyropus.ca  Thu Apr 10 21:09:53 2014
From: charlesc-distutils-python.org at pyropus.ca (Charles Cazabon)
Date: Thu, 10 Apr 2014 13:09:53 -0600
Subject: [Distutils] Failure to extract svn revision
In-Reply-To: <20140409195148.GA28144@pyropus.ca>
References: <20140409195148.GA28144@pyropus.ca>
Message-ID: <20140410190953.GA8791@pyropus.ca>

Charles Cazabon <charlesc-distutils-python.org at pyropus.ca> wrote:
> 
> I'm having trouble getting setuptools to properly extract the svn revision and
> write it into the <package>.egg-info/PKG-INFO file in a new environment.

Spent some time tracking this down, and it appears to be a bug in setuptools.
In my environment (Python 2.7.6 virtualenv with setuptools 3.4.3) and a
local package which is in a *subdir* of the svn 1.8.x working copy checkout,
this test in egg_info.py:tags():

  if self.tag_svn_revision and (
      os.path.exists('.svn') or os.path.exists('PKG-INFO')
  :)

... fails.  Fixing that test fixes the problem for me.  I'll file an issue on
the project bugtracker.

Charles
-- 
------------------------------------------------------------------
Charles Cazabon         <charlesc-distutils-python.org at pyropus.ca>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

From merwok at netwok.org  Fri Apr 11 19:20:39 2014
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Fri, 11 Apr 2014 13:20:39 -0400
Subject: [Distutils] Distribution and packaging mini-summit at PyCon: today
	5 to 6
In-Reply-To: <533DE4CF.1090403@netwok.org>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org>
Message-ID: <53482467.4000207@netwok.org>

  Hi all,

  Jason is leaving tomorrow so Nick and I figured the best time to meet
and talk about the Next Packaging Things would be today from 5pm to 6pm,
before the PyCon Dinners.

  I?m going to run to the 5th floor now and book one of the rooms.

  Cheers

From dholth at gmail.com  Fri Apr 11 19:22:06 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 11 Apr 2014 13:22:06 -0400
Subject: [Distutils] Distribution and packaging mini-summit at PyCon:
 today 5 to 6
In-Reply-To: <53482467.4000207@netwok.org>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org> <53482467.4000207@netwok.org>
Message-ID: <CAG8k2+5Y47X7CbPoqf9Hn48dUEBp332mkdF0Wzz+7NjFQ_Dm8A@mail.gmail.com>

See you there
On Apr 11, 2014 1:21 PM, "?ric Araujo" <merwok at netwok.org> wrote:

>   Hi all,
>
>   Jason is leaving tomorrow so Nick and I figured the best time to meet
> and talk about the Next Packaging Things would be today from 5pm to 6pm,
> before the PyCon Dinners.
>
>   I?m going to run to the 5th floor now and book one of the rooms.
>
>   Cheers
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140411/48615f73/attachment.html>

From Steve.Dower at microsoft.com  Fri Apr 11 20:56:56 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Fri, 11 Apr 2014 18:56:56 +0000
Subject: [Distutils] Distribution and packaging mini-summit at PyCon:
 today 5 to 6
In-Reply-To: <CAG8k2+5Y47X7CbPoqf9Hn48dUEBp332mkdF0Wzz+7NjFQ_Dm8A@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org> <53482467.4000207@netwok.org>,
 <CAG8k2+5Y47X7CbPoqf9Hn48dUEBp332mkdF0Wzz+7NjFQ_Dm8A@mail.gmail.com>
Message-ID: <4fa5cc888dd147a281f3b79bc89a0cf7@BLUPR03MB389.namprd03.prod.outlook.com>

I'll be there too.

Top-posted from my Windows Phone
________________________________
From: Daniel Holth<mailto:dholth at gmail.com>
Sent: ?4/?11/?2014 13:22
To: ?ric Araujo<mailto:merwok at netwok.org>
Cc: DistUtils mailing list <mailto:distutils-sig at python.org>
Subject: Re: [Distutils] Distribution and packaging mini-summit at PyCon: today 5 to 6


See you there

On Apr 11, 2014 1:21 PM, "?ric Araujo" <merwok at netwok.org<mailto:merwok at netwok.org>> wrote:
  Hi all,

  Jason is leaving tomorrow so Nick and I figured the best time to meet
and talk about the Next Packaging Things would be today from 5pm to 6pm,
before the PyCon Dinners.

  I?m going to run to the 5th floor now and book one of the rooms.

  Cheers
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org<mailto:Distutils-SIG at python.org>
https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140411/3e2e97dc/attachment.html>

From Steve.Dower at microsoft.com  Fri Apr 11 21:09:23 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Fri, 11 Apr 2014 19:09:23 +0000
Subject: [Distutils] Distribution and packaging mini-summit at PyCon:
 today 5 to 6
In-Reply-To: <4fa5cc888dd147a281f3b79bc89a0cf7@BLUPR03MB389.namprd03.prod.outlook.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org> <53482467.4000207@netwok.org>,
 <CAG8k2+5Y47X7CbPoqf9Hn48dUEBp332mkdF0Wzz+7NjFQ_Dm8A@mail.gmail.com>,
 <4fa5cc888dd147a281f3b79bc89a0cf7@BLUPR03MB389.namprd03.prod.outlook.com>
Message-ID: <b63aa4760a234ee8a6168b2471da45f8@BLUPR03MB389.namprd03.prod.outlook.com>

I've also let the conda guys know, so we'll have someone from their team too (probably Travis).
________________________________
From: Steve Dower<mailto:Steve.Dower at microsoft.com>
Sent: ?4/?11/?2014 14:57
To: Daniel Holth<mailto:dholth at gmail.com>; ?ric Araujo<mailto:merwok at netwok.org>
Cc: DistUtils mailing list<mailto:distutils-sig at python.org>
Subject: Re: [Distutils] Distribution and packaging mini-summit at PyCon: today 5 to 6

I'll be there too.

Top-posted from my Windows Phone
________________________________
From: Daniel Holth<mailto:dholth at gmail.com>
Sent: ?4/?11/?2014 13:22
To: ?ric Araujo<mailto:merwok at netwok.org>
Cc: DistUtils mailing list <mailto:distutils-sig at python.org>
Subject: Re: [Distutils] Distribution and packaging mini-summit at PyCon: today 5 to 6


See you there

On Apr 11, 2014 1:21 PM, "?ric Araujo" <merwok at netwok.org<mailto:merwok at netwok.org>> wrote:
  Hi all,

  Jason is leaving tomorrow so Nick and I figured the best time to meet
and talk about the Next Packaging Things would be today from 5pm to 6pm,
before the PyCon Dinners.

  I?m going to run to the 5th floor now and book one of the rooms.

  Cheers
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org<mailto:Distutils-SIG at python.org>
https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140411/acaff88f/attachment-0001.html>

From tismer at stackless.com  Fri Apr 11 23:41:38 2014
From: tismer at stackless.com (Christian Tismer)
Date: Fri, 11 Apr 2014 23:41:38 +0200
Subject: [Distutils] How do I specify the target OS version ?
Message-ID: <53486192.9040603@stackless.com>

Hey guys,

I am trying to publish some binaries with Wheel for the first time.

Building PySide with

    python bidst_wheel

I would like how the OS version is determined, or even more, how this is
defined.

Without any argument, when building PySide, I get two different
versions:
PySide-1.3.0dev-cp34-cp34m-macosx_10_9_x86_64.whl
PySide-1.3.0dev-cp27-none-macosx_10_8_x86_64.whl

For some reason, one python is supported by Mavericks and one is not.
Also, I am confused by the different output from the two.

I would like to be able to control this and say where the stuff should
run. I also cannot figure out how mandatory these strings are.
Is a version enforced? Is it recommended?
What will PyPi say?

And how do I see which version PyPi use when I """pip install xxx" ?

How do I define that? Did not find that in the docs, but that is due to
my impaired vision.

Any help/advice would be appreciated.

Cheers -- Chris


-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/

From merwok at netwok.org  Sat Apr 12 00:19:49 2014
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Fri, 11 Apr 2014 18:19:49 -0400
Subject: [Distutils] Distribution and packaging mini-summit at PyCon:
 today 5 to 6
In-Reply-To: <53482467.4000207@netwok.org>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org> <53482467.4000207@netwok.org>
Message-ID: <53486A85.5090707@netwok.org>

  Hi,

  One hour was awfully short.  Noah did a short review of the past year
and a half, Richard talked about future plans for PyPI, Nick clarified
some thoughts, and we stopped.  I suggested each PEP or tool maintainer
writes a short email here to describe briefly current status and
short-term roadmap.  I?ll say something about distutils2 and distutils
tomorrow, now it?s time for the PyCon dinners!

  Cheers

From dholth at gmail.com  Sun Apr 13 02:06:21 2014
From: dholth at gmail.com (Daniel Holth)
Date: Sat, 12 Apr 2014 20:06:21 -0400
Subject: [Distutils] Distribution and packaging mini-summit at PyCon:
 today 5 to 6
In-Reply-To: <53486A85.5090707@netwok.org>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org> <53482467.4000207@netwok.org>
 <53486A85.5090707@netwok.org>
Message-ID: <CAG8k2+5Mk9+suBJ8PpxXvZbt2dfpFzyqUUWgfP-+uwAm34VuhQ@mail.gmail.com>

On Fri, Apr 11, 2014 at 6:19 PM, ?ric Araujo <merwok at netwok.org> wrote:
>   Hi,
>
>   One hour was awfully short.  Noah did a short review of the past year
> and a half, Richard talked about future plans for PyPI, Nick clarified
> some thoughts, and we stopped.  I suggested each PEP or tool maintainer
> writes a short email here to describe briefly current status and
> short-term roadmap.  I?ll say something about distutils2 and distutils
> tomorrow, now it?s time for the PyCon dinners!
>
>   Cheers

The tip revision of wheel contains an updated json metadata generator.
It validates against the updated json metadata schema I did recently
to match the current pep. (The schema used in the wheel tests is
likely the most correct, compared to copies that exist amongst the
peps).

Daniel Holth

From ncoghlan at gmail.com  Sun Apr 13 11:19:08 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 13 Apr 2014 05:19:08 -0400
Subject: [Distutils] Distribution and packaging mini-summit at PyCon:
 today 5 to 6
In-Reply-To: <CAG8k2+5Mk9+suBJ8PpxXvZbt2dfpFzyqUUWgfP-+uwAm34VuhQ@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org> <53482467.4000207@netwok.org>
 <53486A85.5090707@netwok.org>
 <CAG8k2+5Mk9+suBJ8PpxXvZbt2dfpFzyqUUWgfP-+uwAm34VuhQ@mail.gmail.com>
Message-ID: <CADiSq7f6KRUMCMhi9WuRzZFoTTtM5F=yBb7=7vNC7g=mUt_=QQ@mail.gmail.com>

On 12 Apr 2014 20:36, "Daniel Holth" <dholth at gmail.com> wrote:
>
> On Fri, Apr 11, 2014 at 6:19 PM, ?ric Araujo <merwok at netwok.org> wrote:
> >   Hi,
> >
> >   One hour was awfully short.  Noah did a short review of the past year
> > and a half, Richard talked about future plans for PyPI, Nick clarified
> > some thoughts, and we stopped.  I suggested each PEP or tool maintainer
> > writes a short email here to describe briefly current status and
> > short-term roadmap.  I'll say something about distutils2 and distutils
> > tomorrow, now it's time for the PyCon dinners!
> >
> >   Cheers
>
> The tip revision of wheel contains an updated json metadata generator.
> It validates against the updated json metadata schema I did recently
> to match the current pep. (The schema used in the wheel tests is
> likely the most correct, compared to copies that exist amongst the
> peps).

For metadata 2.0, I'll be getting new versions up as part of the sprints,
but after the stdlib PEPs for the SSL and ensurepip feature backport
proposals.

The main near term need there is to get PEP 440 accepted so we can finalise
the pip and setuptools syntax for environment marker based version specific
dependencies.

The content of PEP 440 should be final at this point, but it's missing an
updated reference regex for the version parsing and a rerun of the
compatibility checks against an up to date PyPI dump. It would be great if
someone else could handle that task while I'm looking at the 2.7
enhancement PEPs.

Cheers,
Nick.

>
> Daniel Holth
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140413/0156cb7c/attachment.html>

From donald at stufft.io  Sun Apr 13 17:04:16 2014
From: donald at stufft.io (Donald Stufft)
Date: Sun, 13 Apr 2014 11:04:16 -0400
Subject: [Distutils] Distribution and packaging mini-summit at PyCon:
	today 5 to 6
In-Reply-To: <CADiSq7f6KRUMCMhi9WuRzZFoTTtM5F=yBb7=7vNC7g=mUt_=QQ@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org> <53482467.4000207@netwok.org>
 <53486A85.5090707@netwok.org>
 <CAG8k2+5Mk9+suBJ8PpxXvZbt2dfpFzyqUUWgfP-+uwAm34VuhQ@mail.gmail.com>
 <CADiSq7f6KRUMCMhi9WuRzZFoTTtM5F=yBb7=7vNC7g=mUt_=QQ@mail.gmail.com>
Message-ID: <7A97C243-109E-4E6D-B074-47CB35C77C60@stufft.io>


On Apr 13, 2014, at 5:19 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 12 Apr 2014 20:36, "Daniel Holth" <dholth at gmail.com> wrote:
> >
> > On Fri, Apr 11, 2014 at 6:19 PM, ?ric Araujo <merwok at netwok.org> wrote:
> > >   Hi,
> > >
> > >   One hour was awfully short.  Noah did a short review of the past year
> > > and a half, Richard talked about future plans for PyPI, Nick clarified
> > > some thoughts, and we stopped.  I suggested each PEP or tool maintainer
> > > writes a short email here to describe briefly current status and
> > > short-term roadmap.  I?ll say something about distutils2 and distutils
> > > tomorrow, now it?s time for the PyCon dinners!
> > >
> > >   Cheers
> >
> > The tip revision of wheel contains an updated json metadata generator.
> > It validates against the updated json metadata schema I did recently
> > to match the current pep. (The schema used in the wheel tests is
> > likely the most correct, compared to copies that exist amongst the
> > peps).
> 
> For metadata 2.0, I'll be getting new versions up as part of the sprints, but after the stdlib PEPs for the SSL and ensurepip feature backport proposals.
> 
> The main near term need there is to get PEP 440 accepted so we can finalise the pip and setuptools syntax for environment marker based version specific dependencies.
> 
> The content of PEP 440 should be final at this point, but it's missing an updated reference regex for the version parsing and a rerun of the compatibility checks against an up to date PyPI dump. It would be great if someone else could handle that task while I'm looking at the 2.7 enhancement PEPs.
> 
> Cheers,
> Nick.
> 
> >
> > Daniel Holth
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Will you be posting the updated PEPs here again?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140413/8f7752c9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140413/8f7752c9/attachment.sig>

From chris at simplistix.co.uk  Fri Apr 11 19:29:45 2014
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 11 Apr 2014 18:29:45 +0100
Subject: [Distutils] "pip install -e ." vs. "python setup.py develop"
In-Reply-To: <E6D92399-933E-41D6-8A7C-C9EC8776CF9D@coderanger.net>
References: <CAFyrOVR=LRoR4yRoTyGWzZO8JnWgZFcGD0gHWTNEi3XGkM2tMQ@mail.gmail.com>
 <E6D92399-933E-41D6-8A7C-C9EC8776CF9D@coderanger.net>
Message-ID: <53482689.5090402@simplistix.co.uk>

On 07/04/2014 04:05, Noah Kantrowitz wrote:
> You should recommend using pip for it, mostly because as you said that will work even with packages that don't use setuptools :-) It also is required when doing a develop install with extras, though that requires a slightly more verbose syntax due to a bug in pip.

What's the syntax?

Chris

>
> --Noah
>
> On Apr 6, 2014, at 8:01 PM, Asheesh Laroia <lists at asheesh.org> wrote:
>
>> Hi lovely distutils people,
>>
>> I have a question, as I prepare for my "Python packaging simplified, for end users, app developers, and open source contributors" talk. I'm sure I'll have more; I'll end up probably making a few threads about them, since they'll come to me at random times.
>>
>>
>> For years, I've been recommending:
>>
>> $ python setup.py develop
>>
>>
>> as a standard way to make something hackable and available in a virtualenv. I notice that "python setup.py develop --user" exists, which is great, as it means that you don't even need to bother with the virtualenv.
>>
>>
>> Having said that, I also notice that:
>>
>> $ pip install -e .
>>
>> does the same thing.
>>
>> Should I be recommending one over the other?
>>
>> I'm going to learn toward "pip install -e ." even though I haven't been using it much personally, as it makes the talk more consistent -- I would then be able to say, "Always use pip for doing your installing." But I thought I'd ask about this. It seems that "pip install -e ." is the same as "python setup.py develop" except that pip runs setup.py with setuptools available, which addresses a problem where if the maintainer of a package's setup.py file doesn't "from setuptools import setup", then "python setup.py develop" won't work, whereas "pip install -e ." will always work.
>>
>> Unless I'm mistaken. So the question is -- can someone sanity-check the above?
>>
>> I'm hoping to pretend to be an outsider for the purpose of empathizing with the audience, and yet be enough of an insider to ask people on this list if what I'm saying is consistent with Modern PyPA Doctrine (which generally I'm happy to promote).
>>
>> -- Asheesh.
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From noah at coderanger.net  Sun Apr 13 22:08:42 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sun, 13 Apr 2014 16:08:42 -0400
Subject: [Distutils] "pip install -e ." vs. "python setup.py develop"
In-Reply-To: <53482689.5090402@simplistix.co.uk>
References: <CAFyrOVR=LRoR4yRoTyGWzZO8JnWgZFcGD0gHWTNEi3XGkM2tMQ@mail.gmail.com>
 <E6D92399-933E-41D6-8A7C-C9EC8776CF9D@coderanger.net>
 <53482689.5090402@simplistix.co.uk>
Message-ID: <EEB5AAFE-2567-4F24-9474-7F0568E35ABC@coderanger.net>


On Apr 11, 2014, at 1:29 PM, Chris Withers <chris at simplistix.co.uk> wrote:

> On 07/04/2014 04:05, Noah Kantrowitz wrote:
>> You should recommend using pip for it, mostly because as you said that will work even with packages that don't use setuptools :-) It also is required when doing a develop install with extras, though that requires a slightly more verbose syntax due to a bug in pip.
> 
> What's the syntax?

pip install -e file:///path/to/thing[one,two]

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140413/3a40c78d/attachment.sig>

From marius at pov.lt  Mon Apr 14 09:15:13 2014
From: marius at pov.lt (Marius Gedminas)
Date: Mon, 14 Apr 2014 10:15:13 +0300
Subject: [Distutils] "pip install -e ." vs. "python setup.py develop"
In-Reply-To: <EEB5AAFE-2567-4F24-9474-7F0568E35ABC@coderanger.net>
References: <CAFyrOVR=LRoR4yRoTyGWzZO8JnWgZFcGD0gHWTNEi3XGkM2tMQ@mail.gmail.com>
 <E6D92399-933E-41D6-8A7C-C9EC8776CF9D@coderanger.net>
 <53482689.5090402@simplistix.co.uk>
 <EEB5AAFE-2567-4F24-9474-7F0568E35ABC@coderanger.net>
Message-ID: <20140414071512.GA20567@fridge.pov.lt>

On Sun, Apr 13, 2014 at 04:08:42PM -0400, Noah Kantrowitz wrote:
> 
> On Apr 11, 2014, at 1:29 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> 
> > On 07/04/2014 04:05, Noah Kantrowitz wrote:
> >> You should recommend using pip for it, mostly because as you said
> >> that will work even with packages that don't use setuptools :-) It
> >> also is required when doing a develop install with extras, though
> >> that requires a slightly more verbose syntax due to a bug in pip.
> > 
> > What's the syntax?
> 
> pip install -e file:///path/to/thing[one,two]

What's the bug?

https://github.com/pypa/pip/pull/500 was merged two years ago and was
supposed to enable

  pip install -e .[one,two]

Marius Gedminas
-- 
What can I do with Python that I can't do with C#?  You can go home on time at
the end of the day.
        -- Daniel Klein
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140414/7874af01/attachment.sig>

From tismer at stackless.com  Mon Apr 14 19:54:38 2014
From: tismer at stackless.com (Christian Tismer)
Date: Mon, 14 Apr 2014 19:54:38 +0200
Subject: [Distutils] How do I specify the target OS version ?
In-Reply-To: <53486192.9040603@stackless.com>
References: <53486192.9040603@stackless.com>
Message-ID: <534C20DE.6020804@stackless.com>

Thanks,

I answered this myself meanwhile.

ciao - Chris


On 11/04/14 23:41, Christian Tismer wrote:
> Hey guys,
> 
> I am trying to publish some binaries with Wheel for the first time.
> 
> Building PySide with
> 
>     python bidst_wheel
> 
> I would like how the OS version is determined, or even more, how this is
> defined.
> 
> Without any argument, when building PySide, I get two different
> versions:
> PySide-1.3.0dev-cp34-cp34m-macosx_10_9_x86_64.whl
> PySide-1.3.0dev-cp27-none-macosx_10_8_x86_64.whl
> 
> For some reason, one python is supported by Mavericks and one is not.
> Also, I am confused by the different output from the two.
> 
> I would like to be able to control this and say where the stuff should
> run. I also cannot figure out how mandatory these strings are.
> Is a version enforced? Is it recommended?
> What will PyPi say?
> 
> And how do I see which version PyPi use when I """pip install xxx" ?
> 
> How do I define that? Did not find that in the docs, but that is due to
> my impaired vision.
> 
> Any help/advice would be appreciated.
> 
> Cheers -- Chris
> 
> 


-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/

From ncoghlan at gmail.com  Tue Apr 15 22:57:49 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 15 Apr 2014 16:57:49 -0400
Subject: [Distutils] Distribution and packaging mini-summit at PyCon:
 today 5 to 6
In-Reply-To: <7A97C243-109E-4E6D-B074-47CB35C77C60@stufft.io>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <533DE4CF.1090403@netwok.org> <53482467.4000207@netwok.org>
 <53486A85.5090707@netwok.org>
 <CAG8k2+5Mk9+suBJ8PpxXvZbt2dfpFzyqUUWgfP-+uwAm34VuhQ@mail.gmail.com>
 <CADiSq7f6KRUMCMhi9WuRzZFoTTtM5F=yBb7=7vNC7g=mUt_=QQ@mail.gmail.com>
 <7A97C243-109E-4E6D-B074-47CB35C77C60@stufft.io>
Message-ID: <CADiSq7fPUEcwhZW8o8u5o-ziGB4G87ZCyJ5im64zaR8LsEu9-A@mail.gmail.com>

On 13 April 2014 11:04, Donald Stufft <donald at stufft.io> wrote:
> Will you be posting the updated PEPs here again?

Unfortunately, between software carpentry training yesterday, and
losing my wallet today, I don't know when I'll get to actually working
on anything this week :P

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tinchester at gmail.com  Wed Apr 16 18:18:16 2014
From: tinchester at gmail.com (=?UTF-8?B?VGluIFR2cnRrb3ZpxIc=?=)
Date: Wed, 16 Apr 2014 18:18:16 +0200
Subject: [Distutils] Enterprise Python - Multicomponent Projects
Message-ID: <534EAD48.8060506@gmail.com>

Hello packaging community,

I'm investigating ways of setting up Python projects at my workplace. 
We're predominantly a Java shop, but we might be dipping our toes in 
Python waters (finally!) due to a fortuitous project and my multi-year 
insistence, so I'm contemplating how to set up our Python build system 
to minimize workflow differences for other developers (well, and myself).

I've actually written u? a lengthy description of Maven and why we use 
it but I'll spare you for now. :) To keep the story short, I'm 
interested in options for setting up a multi-module Python project. By 
'multi-module' I don't mean a single setuptools-based project with 
several .py files inside, but a way of triggering a complex build 
process with a single command that would build all sub-modules 
(essentially sub-projects) and produce a number of end artifacts - just 
like Maven. Imagine a repository containing 30 separate Django apps, 
packaged independently, 10 utility libraries, 10 Django projects 
combining those app, and 10 RPM building projects for packaging it all 
up for deployment.

As far as I know, just using setuptools isn't adequate for a workflow 
like this - setuptools deals with the build process (testing, packaging, 
etc) of a single project only. Solutions that come to mind are: a 
hierarchy of Makefiles, shell scripts, or maybe Twitter's Pants, which 
sort of looks like Maven for Python but would probably need 
contributions to do what we want, and looks predisposed to building PEX 
files which, while very interesting, I'm not looking to do right now. 
None of these solutions are really ideal, especially if I want to 
support development on Windows (which I absolutely want).

I've even thought about actually using Maven, but that's just a 
Pandora's box of problems waiting to happen.

I'd appreciate insight on this from anyone who's thought about (and 
maybe solved) problems like this. I'm also willing to engage and 
contribute to improving the situation, especially if there's low hanging 
fruit to be picked. How do other companies handle large Python 
repositories with a lot of subcomponents?

Kind regards,
Tin

From jim at zope.com  Wed Apr 16 18:57:30 2014
From: jim at zope.com (Jim Fulton)
Date: Wed, 16 Apr 2014 12:57:30 -0400
Subject: [Distutils] Enterprise Python - Multicomponent Projects
In-Reply-To: <534EAD48.8060506@gmail.com>
References: <534EAD48.8060506@gmail.com>
Message-ID: <CAPDm-Fj13O3g3rrpJC5cfvKcjw291Vp=2sVSbRpQj1BaqzYCjA@mail.gmail.com>

On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
> Hello packaging community,
>
> I'm investigating ways of setting up Python projects at my workplace. We're
> predominantly a Java shop, but we might be dipping our toes in Python waters
> (finally!) due to a fortuitous project and my multi-year insistence, so I'm
> contemplating how to set up our Python build system to minimize workflow
> differences for other developers (well, and myself).
>
> I've actually written u? a lengthy description of Maven and why we use it
> but I'll spare you for now. :) To keep the story short, I'm interested in
> options for setting up a multi-module Python project. By 'multi-module' I
> don't mean a single setuptools-based project with several .py files inside,
> but a way of triggering a complex build process with a single command that
> would build all sub-modules (essentially sub-projects) and produce a number
> of end artifacts - just like Maven. Imagine a repository containing 30
> separate Django apps, packaged independently, 10 utility libraries, 10
> Django projects combining those app, and 10 RPM building projects for
> packaging it all up for deployment.
>
> As far as I know, just using setuptools isn't adequate for a workflow like
> this - setuptools deals with the build process (testing, packaging, etc) of
> a single project only. Solutions that come to mind are: a hierarchy of
> Makefiles, shell scripts, or maybe Twitter's Pants, which sort of looks like
> Maven for Python but would probably need contributions to do what we want,
> and looks predisposed to building PEX files which, while very interesting,
> I'm not looking to do right now. None of these solutions are really ideal,
> especially if I want to support development on Windows (which I absolutely
> want).
>
> I've even thought about actually using Maven, but that's just a Pandora's
> box of problems waiting to happen.
>
> I'd appreciate insight on this from anyone who's thought about (and maybe
> solved) problems like this. I'm also willing to engage and contribute to
> improving the situation, especially if there's low hanging fruit to be
> picked. How do other companies handle large Python repositories with a lot
> of subcomponents?

Checkout Buildout, http://www.buildout.org

It addresses a lot of the same problems.  Of course, for Python, compiling is
(mostly) not all that important.  For us (Zope Corporation) buildout
is more about
assembly/deployment, both for development and production, that building.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From tinchester at gmail.com  Wed Apr 16 19:02:25 2014
From: tinchester at gmail.com (=?UTF-8?B?VGluIFR2cnRrb3ZpxIc=?=)
Date: Wed, 16 Apr 2014 19:02:25 +0200
Subject: [Distutils] Enterprise Python - Multicomponent Projects
In-Reply-To: <CAPDm-Fj13O3g3rrpJC5cfvKcjw291Vp=2sVSbRpQj1BaqzYCjA@mail.gmail.com>
References: <534EAD48.8060506@gmail.com>
 <CAPDm-Fj13O3g3rrpJC5cfvKcjw291Vp=2sVSbRpQj1BaqzYCjA@mail.gmail.com>
Message-ID: <534EB7A1.9000004@gmail.com>

I confess I haven't had a chance to familiarize myself with Buildout, 
and will do so soon.

Thanks for the tip, Jim.

On 16.04.2014 18:57, Jim Fulton wrote:
> On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
>> Hello packaging community,
>>
>> I'm investigating ways of setting up Python projects at my workplace. We're
>> predominantly a Java shop, but we might be dipping our toes in Python waters
>> (finally!) due to a fortuitous project and my multi-year insistence, so I'm
>> contemplating how to set up our Python build system to minimize workflow
>> differences for other developers (well, and myself).
>>
>> I've actually written u? a lengthy description of Maven and why we use it
>> but I'll spare you for now. :) To keep the story short, I'm interested in
>> options for setting up a multi-module Python project. By 'multi-module' I
>> don't mean a single setuptools-based project with several .py files inside,
>> but a way of triggering a complex build process with a single command that
>> would build all sub-modules (essentially sub-projects) and produce a number
>> of end artifacts - just like Maven. Imagine a repository containing 30
>> separate Django apps, packaged independently, 10 utility libraries, 10
>> Django projects combining those app, and 10 RPM building projects for
>> packaging it all up for deployment.
>>
>> As far as I know, just using setuptools isn't adequate for a workflow like
>> this - setuptools deals with the build process (testing, packaging, etc) of
>> a single project only. Solutions that come to mind are: a hierarchy of
>> Makefiles, shell scripts, or maybe Twitter's Pants, which sort of looks like
>> Maven for Python but would probably need contributions to do what we want,
>> and looks predisposed to building PEX files which, while very interesting,
>> I'm not looking to do right now. None of these solutions are really ideal,
>> especially if I want to support development on Windows (which I absolutely
>> want).
>>
>> I've even thought about actually using Maven, but that's just a Pandora's
>> box of problems waiting to happen.
>>
>> I'd appreciate insight on this from anyone who's thought about (and maybe
>> solved) problems like this. I'm also willing to engage and contribute to
>> improving the situation, especially if there's low hanging fruit to be
>> picked. How do other companies handle large Python repositories with a lot
>> of subcomponents?
> Checkout Buildout, http://www.buildout.org
>
> It addresses a lot of the same problems.  Of course, for Python, compiling is
> (mostly) not all that important.  For us (Zope Corporation) buildout
> is more about
> assembly/deployment, both for development and production, that building.
>
> Jim
>


From wickman at gmail.com  Wed Apr 16 19:32:31 2014
From: wickman at gmail.com (Brian Wickman)
Date: Wed, 16 Apr 2014 13:32:31 -0400
Subject: [Distutils] Enterprise Python - Multicomponent Projects
In-Reply-To: <534EAD48.8060506@gmail.com>
References: <534EAD48.8060506@gmail.com>
Message-ID: <CAAmsm4hK8yTYmDE7hos7h2Q4a+2TPCV1n8FHc9i5-Q0zdeJL=Q@mail.gmail.com>

Pants can definitely do what you want, but you're probably right in that it
requires significant up-front investment.  It's not strictly tied to PEX
files (it can recursively generate setup.py-based projects from a
transitive closure of BUILD dependencies) but the learning curve is
non-trivial and most attention is given to the Java/Scala backends.  Python
support will improve over time but it may be a year or more before this is
really easy to adopt.

~brian



On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtkovi? <tinchester at gmail.com>wrote:

> Hello packaging community,
>
> I'm investigating ways of setting up Python projects at my workplace.
> We're predominantly a Java shop, but we might be dipping our toes in Python
> waters (finally!) due to a fortuitous project and my multi-year insistence,
> so I'm contemplating how to set up our Python build system to minimize
> workflow differences for other developers (well, and myself).
>
> I've actually written u? a lengthy description of Maven and why we use it
> but I'll spare you for now. :) To keep the story short, I'm interested in
> options for setting up a multi-module Python project. By 'multi-module' I
> don't mean a single setuptools-based project with several .py files inside,
> but a way of triggering a complex build process with a single command that
> would build all sub-modules (essentially sub-projects) and produce a number
> of end artifacts - just like Maven. Imagine a repository containing 30
> separate Django apps, packaged independently, 10 utility libraries, 10
> Django projects combining those app, and 10 RPM building projects for
> packaging it all up for deployment.
>
> As far as I know, just using setuptools isn't adequate for a workflow like
> this - setuptools deals with the build process (testing, packaging, etc) of
> a single project only. Solutions that come to mind are: a hierarchy of
> Makefiles, shell scripts, or maybe Twitter's Pants, which sort of looks
> like Maven for Python but would probably need contributions to do what we
> want, and looks predisposed to building PEX files which, while very
> interesting, I'm not looking to do right now. None of these solutions are
> really ideal, especially if I want to support development on Windows (which
> I absolutely want).
>
> I've even thought about actually using Maven, but that's just a Pandora's
> box of problems waiting to happen.
>
> I'd appreciate insight on this from anyone who's thought about (and maybe
> solved) problems like this. I'm also willing to engage and contribute to
> improving the situation, especially if there's low hanging fruit to be
> picked. How do other companies handle large Python repositories with a lot
> of subcomponents?
>
> Kind regards,
> Tin
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140416/80095292/attachment-0001.html>

From tseaver at palladion.com  Wed Apr 16 19:41:11 2014
From: tseaver at palladion.com (Tres Seaver)
Date: Wed, 16 Apr 2014 13:41:11 -0400
Subject: [Distutils] Enterprise Python - Multicomponent Projects
In-Reply-To: <CAPDm-Fj13O3g3rrpJC5cfvKcjw291Vp=2sVSbRpQj1BaqzYCjA@mail.gmail.com>
References: <534EAD48.8060506@gmail.com>
 <CAPDm-Fj13O3g3rrpJC5cfvKcjw291Vp=2sVSbRpQj1BaqzYCjA@mail.gmail.com>
Message-ID: <limfbp$uj1$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/16/2014 12:57 PM, Jim Fulton wrote:
> On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtkovi? <tinchester at gmail.com>
> wrote:
>> Hello packaging community,
>> 
>> I'm investigating ways of setting up Python projects at my
>> workplace. We're predominantly a Java shop, but we might be dipping
>> our toes in Python waters (finally!) due to a fortuitous project and
>> my multi-year insistence, so I'm contemplating how to set up our
>> Python build system to minimize workflow differences for other
>> developers (well, and myself).
>> 
>> I've actually written u? a lengthy description of Maven and why we
>> use it but I'll spare you for now. :) To keep the story short, I'm
>> interested in options for setting up a multi-module Python project.
>> By 'multi-module' I don't mean a single setuptools-based project
>> with several .py files inside, but a way of triggering a complex
>> build process with a single command that would build all sub-modules
>> (essentially sub-projects) and produce a number of end artifacts -
>> just like Maven. Imagine a repository containing 30 separate Django
>> apps, packaged independently, 10 utility libraries, 10 Django
>> projects combining those app, and 10 RPM building projects for 
>> packaging it all up for deployment.
>> 
>> As far as I know, just using setuptools isn't adequate for a
>> workflow like this - setuptools deals with the build process
>> (testing, packaging, etc) of a single project only. Solutions that
>> come to mind are: a hierarchy of Makefiles, shell scripts, or maybe
>> Twitter's Pants, which sort of looks like Maven for Python but would
>> probably need contributions to do what we want, and looks
>> predisposed to building PEX files which, while very interesting, I'm
>> not looking to do right now. None of these solutions are really
>> ideal, especially if I want to support development on Windows (which
>> I absolutely want).
>> 
>> I've even thought about actually using Maven, but that's just a
>> Pandora's box of problems waiting to happen.
>> 
>> I'd appreciate insight on this from anyone who's thought about (and
>> maybe solved) problems like this. I'm also willing to engage and
>> contribute to improving the situation, especially if there's low
>> hanging fruit to be picked. How do other companies handle large
>> Python repositories with a lot of subcomponents?
> 
> Checkout Buildout, http://www.buildout.org
> 
> It addresses a lot of the same problems.  Of course, for Python,
> compiling is (mostly) not all that important.  For us (Zope
> Corporation) buildout is more about assembly/deployment, both for
> development and production, that building.

buildout also allow for reproducible builds of things that *do* require
compilation (e.g., via the 'zc.reciple.cmmi' recipe for
'configure-make-make install' software).


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlNOwLcACgkQ+gerLs4ltQ46aACgts47yE/ErKtqag0FyOEpfCZP
4pAAoI8d04UodGt3NQB0AzPPJbJITPb/
=vBpI
-----END PGP SIGNATURE-----


From tinchester at gmail.com  Wed Apr 16 22:27:13 2014
From: tinchester at gmail.com (=?UTF-8?B?VGluIFR2cnRrb3ZpxIc=?=)
Date: Wed, 16 Apr 2014 22:27:13 +0200
Subject: [Distutils] Enterprise Python - Multicomponent Projects
In-Reply-To: <CAAmsm4hK8yTYmDE7hos7h2Q4a+2TPCV1n8FHc9i5-Q0zdeJL=Q@mail.gmail.com>
References: <534EAD48.8060506@gmail.com>
 <CAAmsm4hK8yTYmDE7hos7h2Q4a+2TPCV1n8FHc9i5-Q0zdeJL=Q@mail.gmail.com>
Message-ID: <534EE7A1.1080806@gmail.com>

Hi Brian, thanks for the reply.

I'm definitely excited about Pants. I see there's a pants-users group 
now, but it's currently empty (as in zero posts - crickets and 
tumbleweeds :) ). Would that be a good place to ask questions when I get 
around to playing with Pants?

It also looks like Pants development is very active right now. Would you 
suggest waiting a while before diving into the docs and code? I don't 
really need Pants to be super easy to use (your suggested one year 
period), just to be able to understand it and maybe write a few modules 
I need. I guess I'm asking if the core is stable enough for semi-serious 
plugin development.

Cheers!

On 16.04.2014 19:32, Brian Wickman wrote:
> Pants can definitely do what you want, but you're probably right in 
> that it requires significant up-front investment.  It's not strictly 
> tied to PEX files (it can recursively generate setup.py-based projects 
> from a transitive closure of BUILD dependencies) but the learning 
> curve is non-trivial and most attention is given to the Java/Scala 
> backends.  Python support will improve over time but it may be a year 
> or more before this is really easy to adopt.
>
> ~brian
>
>
>
> On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtkovi? <tinchester at gmail.com 
> <mailto:tinchester at gmail.com>> wrote:
>
>     Hello packaging community,
>
>     I'm investigating ways of setting up Python projects at my
>     workplace. We're predominantly a Java shop, but we might be
>     dipping our toes in Python waters (finally!) due to a fortuitous
>     project and my multi-year insistence, so I'm contemplating how to
>     set up our Python build system to minimize workflow differences
>     for other developers (well, and myself).
>
>     I've actually written u? a lengthy description of Maven and why we
>     use it but I'll spare you for now. :) To keep the story short, I'm
>     interested in options for setting up a multi-module Python
>     project. By 'multi-module' I don't mean a single setuptools-based
>     project with several .py files inside, but a way of triggering a
>     complex build process with a single command that would build all
>     sub-modules (essentially sub-projects) and produce a number of end
>     artifacts - just like Maven. Imagine a repository containing 30
>     separate Django apps, packaged independently, 10 utility
>     libraries, 10 Django projects combining those app, and 10 RPM
>     building projects for packaging it all up for deployment.
>
>     As far as I know, just using setuptools isn't adequate for a
>     workflow like this - setuptools deals with the build process
>     (testing, packaging, etc) of a single project only. Solutions that
>     come to mind are: a hierarchy of Makefiles, shell scripts, or
>     maybe Twitter's Pants, which sort of looks like Maven for Python
>     but would probably need contributions to do what we want, and
>     looks predisposed to building PEX files which, while very
>     interesting, I'm not looking to do right now. None of these
>     solutions are really ideal, especially if I want to support
>     development on Windows (which I absolutely want).
>
>     I've even thought about actually using Maven, but that's just a
>     Pandora's box of problems waiting to happen.
>
>     I'd appreciate insight on this from anyone who's thought about
>     (and maybe solved) problems like this. I'm also willing to engage
>     and contribute to improving the situation, especially if there's
>     low hanging fruit to be picked. How do other companies handle
>     large Python repositories with a lot of subcomponents?
>
>     Kind regards,
>     Tin
>     _______________________________________________
>     Distutils-SIG maillist  - Distutils-SIG at python.org
>     <mailto:Distutils-SIG at python.org>
>     https://mail.python.org/mailman/listinfo/distutils-sig
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140416/7b864624/attachment.html>

From ncoghlan at gmail.com  Wed Apr 16 22:29:51 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 16 Apr 2014 16:29:51 -0400
Subject: [Distutils] Enterprise Python - Multicomponent Projects
In-Reply-To: <CAAmsm4hK8yTYmDE7hos7h2Q4a+2TPCV1n8FHc9i5-Q0zdeJL=Q@mail.gmail.com>
References: <534EAD48.8060506@gmail.com>
 <CAAmsm4hK8yTYmDE7hos7h2Q4a+2TPCV1n8FHc9i5-Q0zdeJL=Q@mail.gmail.com>
Message-ID: <CADiSq7eWXg6f=BqHePp8ED33aUs7s-1K_-CrpOAzeV=xvciNTw@mail.gmail.com>

On 16 April 2014 13:32, Brian Wickman <wickman at gmail.com> wrote:
> Pants can definitely do what you want, but you're probably right in that it
> requires significant up-front investment.  It's not strictly tied to PEX
> files (it can recursively generate setup.py-based projects from a transitive
> closure of BUILD dependencies) but the learning curve is non-trivial and
> most attention is given to the Java/Scala backends.  Python support will
> improve over time but it may be a year or more before this is really easy to
> adopt.

It also depends heavily on whether cross-platform support is needed.
Fedora uses koji for this kind of thing, for example, as that's a
general purpose RPM build system, which handles Python along with
everything else.

The conda/binstar.org approach created by Continuum Analytics handles
the same issue from a cross-platform perspective.

Cheers,
Nick.

P.S. For folks that may sometimes wonder at the high level of
complexity in the metadata 2.0 specs, this kind of use case has a lot
to do with it. I want metadata 2.0 to work not only for the "direct
consumption of upstream packages" model favoured by
Software-as-a-Service developers,  but also to integrate cleanly with
downstream redistributors (whether Linux distros, redistributors like
Enthought, ActiveState and Continuum Analytics or custom in-house
deployment mechanisms).

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wickman at gmail.com  Thu Apr 17 01:36:26 2014
From: wickman at gmail.com (Brian Wickman)
Date: Wed, 16 Apr 2014 19:36:26 -0400
Subject: [Distutils] Enterprise Python - Multicomponent Projects
In-Reply-To: <534EE7A1.1080806@gmail.com>
References: <534EAD48.8060506@gmail.com>
 <CAAmsm4hK8yTYmDE7hos7h2Q4a+2TPCV1n8FHc9i5-Q0zdeJL=Q@mail.gmail.com>
 <534EE7A1.1080806@gmail.com>
Message-ID: <CAAmsm4jr1z6SkedsPKrBnCa61eYWPi3Ss2hZebP1SLtg3bDtLw@mail.gmail.com>

pants-users is brand new, hence no activity.  most activity is on
pants-devel (and the sister list, pants-reviews.)

the pants execution engine was refactored a couple years ago (the "goal"
framework) and the java/scala pipelines were ported to that.  the python
backend is currently legacy and needs to be ported, but due to codebase
fracturing has been delayed.  you should likely wait until that is done
before you write any plugins, as they would need to be rewritten after the
port is finished.  if you are interested in helping with the port,
pants-devel is the right place to go.


On Wed, Apr 16, 2014 at 4:27 PM, Tin Tvrtkovi? <tinchester at gmail.com> wrote:

>  Hi Brian, thanks for the reply.
>
> I'm definitely excited about Pants. I see there's a pants-users group now,
> but it's currently empty (as in zero posts - crickets and tumbleweeds :) ).
> Would that be a good place to ask questions when I get around to playing
> with Pants?
>
> It also looks like Pants development is very active right now. Would you
> suggest waiting a while before diving into the docs and code? I don't
> really need Pants to be super easy to use (your suggested one year period),
> just to be able to understand it and maybe write a few modules I need. I
> guess I'm asking if the core is stable enough for semi-serious plugin
> development.
>
> Cheers!
>
>
> On 16.04.2014 19:32, Brian Wickman wrote:
>
> Pants can definitely do what you want, but you're probably right in that
> it requires significant up-front investment.  It's not strictly tied to PEX
> files (it can recursively generate setup.py-based projects from a
> transitive closure of BUILD dependencies) but the learning curve is
> non-trivial and most attention is given to the Java/Scala backends.  Python
> support will improve over time but it may be a year or more before this is
> really easy to adopt.
>
>  ~brian
>
>
>
> On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtkovi? <tinchester at gmail.com>wrote:
>
>> Hello packaging community,
>>
>> I'm investigating ways of setting up Python projects at my workplace.
>> We're predominantly a Java shop, but we might be dipping our toes in Python
>> waters (finally!) due to a fortuitous project and my multi-year insistence,
>> so I'm contemplating how to set up our Python build system to minimize
>> workflow differences for other developers (well, and myself).
>>
>> I've actually written u? a lengthy description of Maven and why we use it
>> but I'll spare you for now. :) To keep the story short, I'm interested in
>> options for setting up a multi-module Python project. By 'multi-module' I
>> don't mean a single setuptools-based project with several .py files inside,
>> but a way of triggering a complex build process with a single command that
>> would build all sub-modules (essentially sub-projects) and produce a number
>> of end artifacts - just like Maven. Imagine a repository containing 30
>> separate Django apps, packaged independently, 10 utility libraries, 10
>> Django projects combining those app, and 10 RPM building projects for
>> packaging it all up for deployment.
>>
>> As far as I know, just using setuptools isn't adequate for a workflow
>> like this - setuptools deals with the build process (testing, packaging,
>> etc) of a single project only. Solutions that come to mind are: a hierarchy
>> of Makefiles, shell scripts, or maybe Twitter's Pants, which sort of looks
>> like Maven for Python but would probably need contributions to do what we
>> want, and looks predisposed to building PEX files which, while very
>> interesting, I'm not looking to do right now. None of these solutions are
>> really ideal, especially if I want to support development on Windows (which
>> I absolutely want).
>>
>> I've even thought about actually using Maven, but that's just a Pandora's
>> box of problems waiting to happen.
>>
>> I'd appreciate insight on this from anyone who's thought about (and maybe
>> solved) problems like this. I'm also willing to engage and contribute to
>> improving the situation, especially if there's low hanging fruit to be
>> picked. How do other companies handle large Python repositories with a lot
>> of subcomponents?
>>
>> Kind regards,
>> Tin
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140416/c9f91921/attachment-0001.html>

From jakkevanhees at gmail.com  Tue Apr 29 13:33:56 2014
From: jakkevanhees at gmail.com (Jan Van Hees)
Date: Tue, 29 Apr 2014 13:33:56 +0200
Subject: [Distutils] buildout prepends eggs to sys.path, performance issues
Message-ID: <433401cf639e$e84cbec0$b8e63c40$@gmail.com>

Hi list,

 

A bit of context: we?re running buildout 2.0.1, building zope and plone
applications, zope 2.13.10, combined with plone.recipe.zope2instance. This
question only impacts the performance of the startup of an instance, the the
performance of a running instance. But starting an instance is something
developpers do quite often on a days development.

 

While debugging some instance startup performance issues, I came across the
following.

The buildout Scripts prepends all the eggs to the system path, before the
python path.

 

In our setup this causes quite some delays, because imports from standard
python modules, also try to find that module in every eggs directory, before
it can find in in the default python location (because the eggs are
prepended). The eggs even go before the local folder, so even for importing
a local module, all eggs are poked first. If you are only working on local
disks, the starup performance difference is neglectable (speaking seconds),
but if the eggs are located on network disks, there is a performance
difference of about 30% in startup time (speaking minutes), with the path
prepended vs appended (where appended is the faster startup).

 

Some numbers from strace: with eggs appended to sys.path:

% time     seconds  usecs/call     calls        errors    syscall

------ ----------- ----------- --------- --------- ----------------

48.96    0.014658           0             142555    134820      open

 

And with eggs prepended to sys.path

 

% time     seconds  usecs/call     calls       errors    syscall

------ ----------- ----------- --------- --------- ----------------

58.82    0.035995           0              199897    192150    open

 

As you can see the amount of calls, and consequently the amount of errors,
is noticeable higher when prepending the eggs to the sys.path. On local
disks, like the above numbers, the difference is noticeable, but still
fairly small (timewise, not perdcentage wise), add the bit of extra delay of
a network disk, and the differences become really noticable.

 

As far as I?ve always understood, the default procedure working with paths
should be to append, unless you have a good reason. The good reason in this
case, that I see, could be that you want to prepend certain packages and
that way make sure you use your version instead of what?s present in
site-packages. Now my expectation would be that that is a fairly limited set
of packages that need to be prepended. If there are many, options like
virtualenv exist to avoid taking site-packages at all (that?s what we do
btw).

 

Am I missing a use case for the sys.path prepending, or has this never been
an issue before? Because if there is only the site-packages, or a like
issue, I?m happy to have a look into splitting the python path up in what
should be prepended (whole python path except site-packages?), and making
append the default (site-packages + eggs + 
), or looking how I could
provide a buildout syntax, where append would be default and one could
explicitly prepend some packages with buildout.cfg.

 

Another option I?ve investigated is to use the meta_path hook, providing my
own find_module and load_module, keeping a dictionary of module locations. I
don?t have similar timings about the numbers of syscalls like the ones above
for this scenario. I?m reconstructing that setup for the moment and see if
that makes a difference. Timing wise it?s a bit slower (but still
acceptable, instance starts in 30sec instead of 15/20), but I have no
measurements yet, that indicate something usefull.

 

 

Thanks for your feedback,

 

Jan

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140429/c23d10a4/attachment.html>

From jim at zope.com  Tue Apr 29 15:49:02 2014
From: jim at zope.com (Jim Fulton)
Date: Tue, 29 Apr 2014 09:49:02 -0400
Subject: [Distutils] buildout prepends eggs to sys.path,
	performance issues
In-Reply-To: <433401cf639e$e84cbec0$b8e63c40$@gmail.com>
References: <433401cf639e$e84cbec0$b8e63c40$@gmail.com>
Message-ID: <CAPDm-Fi8JLaLbdQqPRZgCMhzsfj9dJMOC09cJMqQcfNKatBhMQ@mail.gmail.com>

On Tue, Apr 29, 2014 at 7:33 AM, Jan Van Hees <jakkevanhees at gmail.com> wrote:
> Hi list,
>
>
>
> A bit of context: we?re running buildout 2.0.1, building zope and plone
> applications, zope 2.13.10, combined with plone.recipe.zope2instance. This
> question only impacts the performance of the startup of an instance, the the
> performance of a running instance. But starting an instance is something
> developpers do quite often on a days development.
>
>
>
> While debugging some instance startup performance issues, I came across the
> following.
>
> The buildout Scripts prepends all the eggs to the system path, before the
> python path.
>
>
>
> In our setup this causes quite some delays, because imports from standard
> python modules, also try to find that module in every eggs directory, before
> it can find in in the default python location (because the eggs are
> prepended).

...

You make a good performance argument.

> As far as I?ve always understood, the default procedure working with paths
> should be to append, unless you have a good reason. The good reason in this
> case, that I see, could be that you want to prepend certain packages and
> that way make sure you use your version instead of what?s present in
> site-packages.

Yes, exactly.

> Now my expectation would be that that is a fairly limited set
> of packages that need to be prepended.

But buildout can't know what they are.

> If there are many, options like
> virtualenv exist to avoid taking site-packages at all (that?s what we do
> btw).

Buildout doesn't require virtualenv.

I've always prepended appliocation paths for predicability.

I'm not interesated in changing this behavior of buildout, at
least not as a default.

I'll give this some thought, but some possible ideas:

- Provide an option to install the eggs for a part in a single
  directory.

  collective.recipe.omlette does this, although it doesn't seem to be
  used for run-time package lookup afaict from the docs.  You might
  try using this.

- Provide an option to append, rather than prepend eggs.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton