From setuptools at bugs.python.org  Thu Sep  1 01:45:47 2011
From: setuptools at bugs.python.org (Scott Cutler)
Date: Wed, 31 Aug 2011 23:45:47 +0000
Subject: [Distutils] [issue132] bad regular expression for build folder
Message-ID: <1314834347.56.0.486688211551.issue132@psf.upfronthosting.co.za>


New submission from Scott Cutler <cutler.scott at gmail.com>:

When using setup.py sdist, I found that my module folder "builders" was not being included. After using the debug flag I saw it used a regular expression to match the temporary build path and remove it. The regex is r'^build\.*' . It should be r'^build\\.*'

----------
messages: 635
nosy: wetfeet2000
priority: bug
status: unread
title: bad regular expression for build folder

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue132>
_______________________________________________

From lgp171188 at gmail.com  Thu Sep  1 07:29:17 2011
From: lgp171188 at gmail.com (Guruprasad)
Date: Thu, 1 Sep 2011 10:59:17 +0530
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <j3l9jn$rd0$1@dough.gmane.org>
References: <4E5CB250.3010709@gmail.com> <j3l9jn$rd0$1@dough.gmane.org>
Message-ID: <CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>

Hi,
On Wed, Aug 31, 2011 at 6:01 PM, Maurits van Rees
<m.van.rees at zestsoftware.nl> wrote:
> Op 30-08-11 11:50, L. Guruprasad schreef:

> You should have a line 'sources = sources' in the [buildout] section,
> otherwise the [sources] section is not used (unless that is the default, but
> explicit may be better than implicit here). ?When you run buildout do you
> actually get a 'src/<project name>' directory?

'wsb' is the name of the project. By default, the 'mr.developer'
extension checks out the source code under 'src' folder. Also by
default, it checks out the sources mentioned under the 'sources'
section. The 'sources' option in the 'buildout' section is to change
that default to something else.


>
> It looks to me like src/wsb is not actually used. ?You tell buildout that it
> can find a development package in src/wsb if it wants to, but I do not see a
> section that actually wants to use a package with that name.

Here is my buildout.cfg
<snip>
[buildout]
extensions = mr.developer
auto-checkout = wsb
develop = src/wsb
eggs = wsb
parts = django

[sources]
wsb = hg <repository URL>

[django]
recipe=djangorecipe
projectegg = wsb
extra-paths = src/
settings = settings

</snip>

> Or is 'wsb' the same as 'project name'?

Yes, wsb is the name of the project.

> Is 'projectegg' the correct spelling in djangorecipe? ?I see this in a
> buildout config I have here:
>
> [django]
> recipe = djangorecipe
> version = 1.2.4
> settings = production
> eggs =
> ? ?${buildout:eggs}
> project = projectname
> wsgi = true
>

djangorecipe automatically installs django and creates a default
django project by the name 'project'. The setting 'project' is used to
change the name of that default project that gets created. Here I
don't want a default project to be created and instead want to use an
existing develop egg, so the 'projectegg' setting can be used to
specify that to djangorecipe.

When I run buildout with this buildout.cfg, everything works fine and
djangorecipe sees the checked out develop egg that mr.developer
downloaded from my repository and sets the bin/django command
appropriately. But my problem here is that the dependencies mentioned
in the setup.py of the develop egg don't get installed automatically.

Thanks & Regards,
Guruprasad

From reinout at vanrees.org  Thu Sep  1 12:24:54 2011
From: reinout at vanrees.org (Reinout van Rees)
Date: Thu, 01 Sep 2011 12:24:54 +0200
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
References: <4E5CB250.3010709@gmail.com> <j3l9jn$rd0$1@dough.gmane.org>
	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
Message-ID: <j3nmhm$hke$1@dough.gmane.org>

On 01-09-11 07:29, Guruprasad wrote:
> Here is my buildout.cfg
> <snip>
> [buildout]
> extensions = mr.developer
> auto-checkout = wsb
> develop = src/wsb
> eggs = wsb
> parts = django
>
> [sources]
> wsb = hg<repository URL>

You should not need to have that "develop = src/wsb" line! When you 
enable a source with mr.developer, it gets installed as a development 
egg automatically.

So: you're doing things double, which *might* be the cause of the 
problems you see.


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From lgp171188 at gmail.com  Thu Sep  1 13:34:00 2011
From: lgp171188 at gmail.com (Guruprasad)
Date: Thu, 1 Sep 2011 17:04:00 +0530
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <j3nmhm$hke$1@dough.gmane.org>
References: <4E5CB250.3010709@gmail.com> <j3l9jn$rd0$1@dough.gmane.org>
	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
	<j3nmhm$hke$1@dough.gmane.org>
Message-ID: <CAFZPT6aXk1xBP5jzq=ozJes8MJTH=1DUPEHHz5BLUQKj4K+XRA@mail.gmail.com>

Hi,

On Thu, Sep 1, 2011 at 3:54 PM, Reinout van Rees <reinout at vanrees.org> wrote:
> On 01-09-11 07:29, Guruprasad wrote:
> You should not need to have that "develop = src/wsb" line! When you enable a
> source with mr.developer, it gets installed as a development egg
> automatically.
>
> So: you're doing things double, which *might* be the cause of the problems
> you see.

Even after removing the duplication, it doesn't work :-(

Thanks & Regards,
Guruprasad

From m.van.rees at zestsoftware.nl  Sat Sep  3 04:52:11 2011
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Sat, 03 Sep 2011 04:52:11 +0200
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <CAFZPT6aXk1xBP5jzq=ozJes8MJTH=1DUPEHHz5BLUQKj4K+XRA@mail.gmail.com>
References: <4E5CB250.3010709@gmail.com> <j3l9jn$rd0$1@dough.gmane.org>
	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
	<j3nmhm$hke$1@dough.gmane.org>
	<CAFZPT6aXk1xBP5jzq=ozJes8MJTH=1DUPEHHz5BLUQKj4K+XRA@mail.gmail.com>
Message-ID: <j3s4or$gul$1@dough.gmane.org>

Op 01-09-11 13:34, Guruprasad schreef:
> Hi,
>
> On Thu, Sep 1, 2011 at 3:54 PM, Reinout van Rees<reinout at vanrees.org>  wrote:
>> On 01-09-11 07:29, Guruprasad wrote:
>> You should not need to have that "develop = src/wsb" line! When you enable a
>> source with mr.developer, it gets installed as a development egg
>> automatically.
>>
>> So: you're doing things double, which *might* be the cause of the problems
>> you see.
>
> Even after removing the duplication, it doesn't work :-(

Your setup.py says:
from distutils.core import setup

I wonder if it would work if instead you say:

from setuptools import setup

And possibly add 'setuptools' in the install_requires in your setup.py.

Just guessing; I might be totally wrong.

-- 
Maurits van Rees
Web App Programmer at Zest Software: http://zestsoftware.nl
Personal website: http://maurits.vanrees.org/


From lgp171188 at gmail.com  Sat Sep  3 15:25:35 2011
From: lgp171188 at gmail.com (Guruprasad)
Date: Sat, 3 Sep 2011 18:55:35 +0530
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <j3s4or$gul$1@dough.gmane.org>
References: <4E5CB250.3010709@gmail.com> <j3l9jn$rd0$1@dough.gmane.org>
	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
	<j3nmhm$hke$1@dough.gmane.org>
	<CAFZPT6aXk1xBP5jzq=ozJes8MJTH=1DUPEHHz5BLUQKj4K+XRA@mail.gmail.com>
	<j3s4or$gul$1@dough.gmane.org>
Message-ID: <CAFZPT6aEBMx-8=biFnHq63gCMuPX0dT0An9fQ_2_pYnhb_QTtA@mail.gmail.com>

Hi,

On Sat, Sep 3, 2011 at 8:22 AM, Maurits van Rees
<m.van.rees at zestsoftware.nl> wrote:
> Op 01-09-11 13:34, Guruprasad schreef:
>> Even after removing the duplication, it doesn't work :-(
>
> Your setup.py says:
> from distutils.core import setup
>
> I wonder if it would work if instead you say:
>
> from setuptools import setup
>
> And possibly add 'setuptools' in the install_requires in your setup.py.
>
> Just guessing; I might be totally wrong.

Let me try that and get back. :-)

Thanks & Regards,
Guruprasad

From lgp171188 at gmail.com  Mon Sep  5 16:46:33 2011
From: lgp171188 at gmail.com (L. Guruprasad)
Date: Mon, 05 Sep 2011 20:16:33 +0530
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <j3s4or$gul$1@dough.gmane.org>
References: <4E5CB250.3010709@gmail.com>
	<j3l9jn$rd0$1@dough.gmane.org>	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>	<j3nmhm$hke$1@dough.gmane.org>	<CAFZPT6aXk1xBP5jzq=ozJes8MJTH=1DUPEHHz5BLUQKj4K+XRA@mail.gmail.com>
	<j3s4or$gul$1@dough.gmane.org>
Message-ID: <4E64E0C9.2010103@gmail.com>

On Saturday 03 September 2011 08:22 AM, Maurits van Rees wrote:
> Your setup.py says:
> from distutils.core import setup
>
> I wonder if it would work if instead you say:
>
> from setuptools import setup
>
> And possibly add 'setuptools' in the install_requires in your setup.py.
>
> Just guessing; I might be totally wrong.

With a lot of enthusiasm and hope I tried it, but the end result was the 
same disappointing one. :-(

Thanks & Regards,
Guruprasad

From brett at python.org  Tue Sep  6 07:02:22 2011
From: brett at python.org (Brett Cannon)
Date: Mon, 5 Sep 2011 22:02:22 -0700
Subject: [Distutils] Plans for a build command or --user support for
	pysetup3?
Message-ID: <CAP1=2W5th+t0OF4LyWL9Ev+5MtUqF4CjUh2GWK9t4TR9ncbsLw@mail.gmail.com>

I just tried creating a setup.cfg for coverage.py as
distribute_setup.py was hanging when run by Python default (worked
fine when I installed it directly). Two things that I missed was
wanting a build command (having trouble compiling coverage.py's
extension module) and --user suport for install.

Are either of those planned or in the works? If not, any specific reason why?

From merwok at netwok.org  Tue Sep  6 17:17:37 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Tue, 06 Sep 2011 17:17:37 +0200
Subject: [Distutils] Plans for a build command or --user support for
	pysetup3?
In-Reply-To: <CAP1=2W5th+t0OF4LyWL9Ev+5MtUqF4CjUh2GWK9t4TR9ncbsLw@mail.gmail.com>
References: <CAP1=2W5th+t0OF4LyWL9Ev+5MtUqF4CjUh2GWK9t4TR9ncbsLw@mail.gmail.com>
Message-ID: <4E663991.3020203@netwok.org>

Hi Brett,

> Two things that I missed was wanting a build command
A clarification: In distutils2, there is a difference between commands
(build, sdist, upload, i.e. similar to distutils commands) and actions
(metadata, list, search, run).  The pysetup install action is a
high-level interface to install a remote or local project.  If you?re
using it on a local setup.cfg-based project, then it will call the build
and install commands.

Now to answer your question: To call a command, use the run action:
   $ pysetup run build

> --user suport for install.
I?ve been surprised to discover that the install action lacks options to
control the target directory.  It?s probably because the action is not
complete yet.  IMO, we should support --user, --prefix, --home and all
other options supported by the install command itself.  I intended to
open a feature request for that; feel free to do it.

Regards

From merwok at netwok.org  Tue Sep  6 17:22:47 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Tue, 06 Sep 2011 17:22:47 +0200
Subject: [Distutils] Plans for a build command or --user support for
	pysetup3?
In-Reply-To: <4E663991.3020203@netwok.org>
References: <CAP1=2W5th+t0OF4LyWL9Ev+5MtUqF4CjUh2GWK9t4TR9ncbsLw@mail.gmail.com>
	<4E663991.3020203@netwok.org>
Message-ID: <4E663AC7.3060307@netwok.org>

Oh and BTW, I am working on huge patches
(http://bugs.python.org/issue12779) to update the packaging
documentation in the 3.3 docs.  If you find non-working things in the
current doc, it?s expected.

From m.van.rees at zestsoftware.nl  Wed Sep  7 18:15:50 2011
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 07 Sep 2011 18:15:50 +0200
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
References: <4E5CB250.3010709@gmail.com> <j3l9jn$rd0$1@dough.gmane.org>
	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
Message-ID: <j485bm$t80$1@dough.gmane.org>

Op 01-09-11 07:29, Guruprasad schreef:
> Hi,
> On Wed, Aug 31, 2011 at 6:01 PM, Maurits van Rees
> <m.van.rees at zestsoftware.nl>  wrote:
>> It looks to me like src/wsb is not actually used.  You tell buildout that it
>> can find a development package in src/wsb if it wants to, but I do not see a
>> section that actually wants to use a package with that name.
>
> Here is my buildout.cfg
> <snip>
> [buildout]
> extensions = mr.developer
> auto-checkout = wsb
> develop = src/wsb
> eggs = wsb
> parts = django
>
> [sources]
> wsb = hg<repository URL>
>
> [django]
> recipe=djangorecipe
> projectegg = wsb
> extra-paths = src/
> settings = settings

I reproduced your setup.  The problems is indeed that src/wsb is not 
actually used, or perhaps better said: the package there is not 
installed in the buildout.  This is because djangorecipe does not 
actually know that it needs to install the wsb package.  Maybe it 
should, as you do have the line 'projectegg = wsb'.

For me it works when I add 'eggs = ${buildout:eggs}' to the recipe, so 
like this:

[django]
recipe=djangorecipe
projectegg = wsb
eggs = ${buildout:eggs}
extra-paths = src/
settings = settings

(Or explicitly: eggs = wsb)

The difference in the resulting bin/django script is that these two 
extra lines are added to the path:

   '.../tmp/guru/src/wsb',
   '.../recaptcha_client-1.0.6-py2.6.egg',


I hope this helps,

-- 
Maurits van Rees
Web App Programmer at Zest Software: http://zestsoftware.nl
Personal website: http://maurits.vanrees.org/


From setuptools at bugs.python.org  Wed Sep  7 18:20:41 2011
From: setuptools at bugs.python.org (Sergei Stolyarov)
Date: Wed, 07 Sep 2011 16:20:41 +0000
Subject: [Distutils] [issue133] python setup.py upload_docs doesn't ask for
	login and	password
Message-ID: <1315412441.82.0.790823784349.issue133@psf.upfronthosting.co.za>


New submission from Sergei Stolyarov <sergei at regolit.com>:

python setup.py upload_docs doesn't ask for login and password instead it prints 

Upload failed (401): You must be identified to edit package information

Works only with valid ~/.pypirc (user and password)

----------
messages: 639
nosy: cancel
priority: bug
status: unread
title: python setup.py upload_docs doesn't ask for login and password

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue133>
_______________________________________________

From lgp171188 at gmail.com  Wed Sep  7 18:47:23 2011
From: lgp171188 at gmail.com (L. Guruprasad)
Date: Wed, 07 Sep 2011 22:17:23 +0530
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <j485bm$t80$1@dough.gmane.org>
References: <4E5CB250.3010709@gmail.com>
	<j3l9jn$rd0$1@dough.gmane.org>	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
	<j485bm$t80$1@dough.gmane.org>
Message-ID: <4E67A01B.5010609@gmail.com>

On Wednesday 07 September 2011 09:45 PM, Maurits van Rees wrote:
> Op 01-09-11 07:29, Guruprasad schreef:
> I reproduced your setup. The problems is indeed that src/wsb is not
> actually used, or perhaps better said: the package there is not
> installed in the buildout. This is because djangorecipe does not
> actually know that it needs to install the wsb package. Maybe it should,
> as you do have the line 'projectegg = wsb'.
>
> For me it works when I add 'eggs = ${buildout:eggs}' to the recipe, so
> like this:
>
> [django]
> recipe=djangorecipe
> projectegg = wsb
> eggs = ${buildout:eggs}
> extra-paths = src/
> settings = settings
>
> (Or explicitly: eggs = wsb)
>
> The difference in the resulting bin/django script is that these two
> extra lines are added to the path:
>
> '.../tmp/guru/src/wsb',
> '.../recaptcha_client-1.0.6-py2.6.egg',
>
>
> I hope this helps,

Bingo! It worked! Thanks! You made my day! :-) As you have pointed out, 
though I have a develop egg, djangorecipe needs to know that it has to 
use the egg and install it.

But now my question is if I have just a develop egg in my buildout.cfg 
with dependencies listed in setup.py and no other parts/recipes used, 
buildout will not install the dependencies right?

Thanks & Regards,
Guruprasad

From reinout at vanrees.org  Wed Sep  7 21:37:27 2011
From: reinout at vanrees.org (Reinout van Rees)
Date: Wed, 07 Sep 2011 21:37:27 +0200
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <4E67A01B.5010609@gmail.com>
References: <4E5CB250.3010709@gmail.com>
	<j3l9jn$rd0$1@dough.gmane.org>	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>
	<j485bm$t80$1@dough.gmane.org> <4E67A01B.5010609@gmail.com>
Message-ID: <j48h5n$o2g$1@dough.gmane.org>

On 07-09-11 18:47, L. Guruprasad wrote:
> But now my question is if I have just a develop egg in my buildout.cfg
> with dependencies listed in setup.py and no other parts/recipes used,
> buildout will not install the dependencies right?

In that case, nothing effectively uses the egg. Because buildout 
installs the develop egg, it probably grabs the dependencies just fine.

But if you don't actually use the egg, you won't see the results 
anywhere. Perhaps just a zc.recipe.egg part?


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From lgp171188 at gmail.com  Wed Sep  7 21:59:47 2011
From: lgp171188 at gmail.com (L. Guruprasad)
Date: Thu, 08 Sep 2011 01:29:47 +0530
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <j48h5n$o2g$1@dough.gmane.org>
References: <4E5CB250.3010709@gmail.com>	<j3l9jn$rd0$1@dough.gmane.org>	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>	<j485bm$t80$1@dough.gmane.org>
	<4E67A01B.5010609@gmail.com> <j48h5n$o2g$1@dough.gmane.org>
Message-ID: <4E67CD33.1010008@gmail.com>

On Thursday 08 September 2011 01:07 AM, Reinout van Rees wrote:
> On 07-09-11 18:47, L. Guruprasad wrote:
>> But now my question is if I have just a develop egg in my buildout.cfg
>> with dependencies listed in setup.py and no other parts/recipes used,
>> buildout will not install the dependencies right?
>
> In that case, nothing effectively uses the egg. Because buildout
> installs the develop egg, it probably grabs the dependencies just fine.
>
> But if you don't actually use the egg, you won't see the results
> anywhere. Perhaps just a zc.recipe.egg part?

Supposing I have a develop egg that uses 'recaptcha-client' as a dependency,

My buildout.cfg will look something like:
[buildout]
extensions = mr.developer
auto-checkout = wsb
develop = src/wsb
eggs = wsb
parts =

[sources]
wsb = hg <path to repo>

and my setup.py will have 'recaptcha-client' in 'install_requires' 
field. The develop egg will need the 'recaptcha-client', but buildout 
doesn't install it with the above configuration. Have I understood 
buildout incorrectly that I am trying something that is also incorrect?

Thanks & Regards,
Guruprasad

From reinout at vanrees.org  Thu Sep  8 10:18:10 2011
From: reinout at vanrees.org (Reinout van Rees)
Date: Thu, 08 Sep 2011 10:18:10 +0200
Subject: [Distutils] Installing the dependencies in the develop egg
	setup.py
In-Reply-To: <4E67CD33.1010008@gmail.com>
References: <4E5CB250.3010709@gmail.com>	<j3l9jn$rd0$1@dough.gmane.org>	<CAFZPT6au4v0eOVv4mQdousEr1U8wLre5vy=zk3epQPToM-rnFQ@mail.gmail.com>	<j485bm$t80$1@dough.gmane.org>
	<4E67A01B.5010609@gmail.com> <j48h5n$o2g$1@dough.gmane.org>
	<4E67CD33.1010008@gmail.com>
Message-ID: <j49to1$cda$1@dough.gmane.org>

On 07-09-11 21:59, L. Guruprasad wrote:
> and my setup.py will have 'recaptcha-client' in 'install_requires'
> field. The develop egg will need the 'recaptcha-client', but buildout
> doesn't install it with the above configuration. Have I understood
> buildout incorrectly that I am trying something that is also incorrect?

The one think you're missing: you're not telling buildout to actually 
install/do anything!

With your buildout.cfg, buildout *does* make the develop egg including 
recaptcha client available to any part that might need it. But in this 
buildout, you don't have any parts.

As an example, look in your bin/buildout. THere's a sys.path thingy in 
there with setuptools and buildout. You'll need something else in there 
with wsb and recaptcha and so in sys.path.


Can you add "console_scripts" to your parts list and then add this part 
and try again?

[console_scripts]
recipe = zc.recipe.egg
interpreter = python
eggs = ${buildout:eggs}

And then check the contents of bin/python.



Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From brett at python.org  Thu Sep  8 19:50:08 2011
From: brett at python.org (Brett Cannon)
Date: Thu, 8 Sep 2011 10:50:08 -0700
Subject: [Distutils] Plans for a build command or --user support for
	pysetup3?
In-Reply-To: <4E663991.3020203@netwok.org>
References: <CAP1=2W5th+t0OF4LyWL9Ev+5MtUqF4CjUh2GWK9t4TR9ncbsLw@mail.gmail.com>
	<4E663991.3020203@netwok.org>
Message-ID: <CAP1=2W4wZv+5uO-FYrszNEA83V9Swz6S0T1zxJsQO5grMhzAWg@mail.gmail.com>

http://bugs.python.org/issue12937 has the feature request.

On Tue, Sep 6, 2011 at 08:17, ?ric Araujo <merwok at netwok.org> wrote:
> Hi Brett,
>
>> Two things that I missed was wanting a build command
> A clarification: In distutils2, there is a difference between commands
> (build, sdist, upload, i.e. similar to distutils commands) and actions
> (metadata, list, search, run). ?The pysetup install action is a
> high-level interface to install a remote or local project. ?If you?re
> using it on a local setup.cfg-based project, then it will call the build
> and install commands.
>
> Now to answer your question: To call a command, use the run action:
> ? $ pysetup run build
>
>> --user suport for install.
> I?ve been surprised to discover that the install action lacks options to
> control the target directory. ?It?s probably because the action is not
> complete yet. ?IMO, we should support --user, --prefix, --home and all
> other options supported by the install command itself. ?I intended to
> open a feature request for that; feel free to do it.
>
> Regards
>

From quasipedia at gmail.com  Mon Sep 12 16:55:00 2011
From: quasipedia at gmail.com (Mac Ryan)
Date: Mon, 12 Sep 2011 16:55:00 +0200
Subject: [Distutils] Correct way to include (debian) program icons.
Message-ID: <20110912165500.0b66f752@jabbar>

Hello,

	I was wondering if there is a way in distutils to
include/indicate a program icon that will be used in the program
menus / desktop shortcuts.

	I'm actualy packaging for debian with `stdeb` and I was
expecting some stdeb-specific way to include the icon in the pacakge
but to the best of my knowledge there isn't.

	I'm presently including in my setup.py file:

data_files=[('/usr/share/pixmaps/', ['my-package-name.xpm'])],

	but this - while working - seems the wrong thing to do, as it
insert in the general python package something that is
platform-specific.

	Any ideas or primers on the subject?
	Thanks in advance,

/mac

From merwok at netwok.org  Mon Sep 12 17:20:06 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Mon, 12 Sep 2011 17:20:06 +0200
Subject: [Distutils] Correct way to include (debian) program icons.
In-Reply-To: <20110912165500.0b66f752@jabbar>
References: <20110912165500.0b66f752@jabbar>
Message-ID: <4E6E2326.7030701@netwok.org>

Hi Mac,

> 	I was wondering if there is a way in distutils to
> include/indicate a program icon that will be used in the program
> menus / desktop shortcuts.
Nothing standard, no.  You may find python-distutils-extra (that?s the
Debian/Ubuntu package name) interesting.

> 	but this - while working - seems the wrong thing to do, as it
> insert in the general python package something that is
> platform-specific.
That?s the state of things with distutils.  For distutils2/packaging
we?re working on a better system, but it?s not fully there yet.

Regards

From aspineux at gmail.com  Tue Sep 13 01:17:25 2011
From: aspineux at gmail.com (Alain Spineux)
Date: Tue, 13 Sep 2011 01:17:25 +0200
Subject: [Distutils] scripts are not converted by 2to3
Message-ID: <CACgTii_=_HosJY1H4Hrp=X0TUASH2hiJ0NvLJfAm+6sEn_hR7Q@mail.gmail.com>

I have this in my setup.py

setup(
    ...
      scripts=[ 'scripts/sendmail', 'scripts/infomail.py' ],
      use_2to3 = True
    ...

and none of sendmail (without .py) or infomail.py (with .py) are
converted when running

/opt/python-3.2.2/bin/python3 setup.py install

Everything else are working fine.
Did I miss something ?

Regards.

-- 
Alain Spineux ? ? ? ? ? ? ? ? ? | ?aspineux gmail com
Monitor your iT & Backups | ?http://www.magikmon.com
Free Backup front-end ? ? ? | http://www.magikmon.com/mksbackup
Your email 100% available | ?http://www.emailgency.com

From akabaila at pcug.org.au  Tue Sep 13 02:02:59 2011
From: akabaila at pcug.org.au (Algis Kabaila)
Date: Tue, 13 Sep 2011 10:02:59 +1000
Subject: [Distutils] Correct way to include (debian) program icons.
In-Reply-To: <4E6E2326.7030701@netwok.org>
References: <20110912165500.0b66f752@jabbar> <4E6E2326.7030701@netwok.org>
Message-ID: <201109131003.00751.akabaila@pcug.org.au>

On Tuesday 13 September 2011 01:20:06 ?ric Araujo wrote:
> Hi Mac,
> 
> > 	I was wondering if there is a way in distutils to
> > 
> > include/indicate a program icon that will be used in the program
> > menus / desktop shortcuts.
> 
> Nothing standard, no.  You may find python-distutils-extra (that?s the
> Debian/Ubuntu package name) interesting.
> 
> > 	but this - while working - seems the wrong thing to do, as it
> > 
> > insert in the general python package something that is
> > platform-specific.
> 
> That?s the state of things with distutils.  For distutils2/packaging
> we?re working on a better system, but it?s not fully there yet.
> 
> Regards

Is it possible to inlcuding icon with the Application in PyQt?  Clearly GUI is 
involved, if we talk about Aplication icons. I know the topic is out of scope 
here,  but I was just wondering...

OldAl.

From aspineux at gmail.com  Tue Sep 13 10:59:22 2011
From: aspineux at gmail.com (Alain Spineux)
Date: Tue, 13 Sep 2011 10:59:22 +0200
Subject: [Distutils] distribute_setup.py should be included automatically in
	sdist
Message-ID: <CACgTii-FkpUV-YdAUi_1rrQyYsgTyVNqPiSse8xh6eqSEi=ZBQ@mail.gmail.com>

Hi
When using this in my setup.py

from distribute_setup import use_setuptools
use_setuptools()

I think file distribute_setup.py should be included in source package
without adding it manually to the MANIFEST.IN.

Did I miss something ?

Regards.

-- 
Alain Spineux ? ? ? ? ? ? ? ? ? | ?aspineux gmail com
Monitor your iT & Backups | ?http://www.magikmon.com
Free Backup front-end ? ? ? | http://www.magikmon.com/mksbackup
Your email 100% available | ?http://www.emailgency.com

From regebro at gmail.com  Tue Sep 13 12:07:14 2011
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 13 Sep 2011 12:07:14 +0200
Subject: [Distutils] scripts are not converted by 2to3
In-Reply-To: <CACgTii_=_HosJY1H4Hrp=X0TUASH2hiJ0NvLJfAm+6sEn_hR7Q@mail.gmail.com>
References: <CACgTii_=_HosJY1H4Hrp=X0TUASH2hiJ0NvLJfAm+6sEn_hR7Q@mail.gmail.com>
Message-ID: <CAL0kPAWBeQOBs-MMVDaKLJYjY-Y6=wB35iaUqgkAuJqyV=J4Og@mail.gmail.com>

On Tue, Sep 13, 2011 at 01:17, Alain Spineux <aspineux at gmail.com> wrote:
> I have this in my setup.py
>
> setup(
> ? ?...
> ? ? ?scripts=[ 'scripts/sendmail', 'scripts/infomail.py' ],
> ? ? ?use_2to3 = True
> ? ?...
>
> and none of sendmail (without .py) or infomail.py (with .py) are
> converted when running
>
> /opt/python-3.2.2/bin/python3 setup.py install
>
> Everything else are working fine.
> Did I miss something ?

I don't think so, sounds like a bug.

//Lennart

From aspineux at gmail.com  Wed Sep 14 08:21:50 2011
From: aspineux at gmail.com (Alain Spineux)
Date: Wed, 14 Sep 2011 08:21:50 +0200
Subject: [Distutils] scripts are not converted by 2to3
In-Reply-To: <CAL0kPAWBeQOBs-MMVDaKLJYjY-Y6=wB35iaUqgkAuJqyV=J4Og@mail.gmail.com>
References: <CACgTii_=_HosJY1H4Hrp=X0TUASH2hiJ0NvLJfAm+6sEn_hR7Q@mail.gmail.com>
	<CAL0kPAWBeQOBs-MMVDaKLJYjY-Y6=wB35iaUqgkAuJqyV=J4Og@mail.gmail.com>
Message-ID: <CACgTii8bPk9ZGYqzi-DjAN2E93KdJHgeT62wOeJn0326FoEaDQ@mail.gmail.com>

On Tue, Sep 13, 2011 at 12:07 PM, Lennart Regebro <regebro at gmail.com> wrote:
> On Tue, Sep 13, 2011 at 01:17, Alain Spineux <aspineux at gmail.com> wrote:
>> I have this in my setup.py
>>
>> setup(
>> ? ?...
>> ? ? ?scripts=[ 'scripts/sendmail', 'scripts/infomail.py' ],
>> ? ? ?use_2to3 = True
>> ? ?...
>>
>> and none of sendmail (without .py) or infomail.py (with .py) are
>> converted when running
>>
>> /opt/python-3.2.2/bin/python3 setup.py install
>>
>> Everything else are working fine.
>> Did I miss something ?
>
> I don't think so, sounds like a bug.
>
> //Lennart
>

"test_suite" is not converted if not in packages list.
It looks like only .py file in "packages" are converted.

I have reported the bug anonymously.
It's not visible yet.


-- 
Alain Spineux ? ? ? ? ? ? ? ? ? | ?aspineux gmail com
Monitor your iT & Backups | ?http://www.magikmon.com
Free Backup front-end ? ? ? | http://www.magikmon.com/mksbackup
Your email 100% available | ?http://www.emailgency.com

From guy at rzn.co.il  Wed Sep 14 15:19:50 2011
From: guy at rzn.co.il (Guy Rozendorn)
Date: Wed, 14 Sep 2011 16:19:50 +0300
Subject: [Distutils] Becoming a distribute maintainer
Message-ID: <1E9930C0-67E5-46B6-AF95-E84F58D61E59@rzn.co.il>

Hi,

I would like to join the maintainers group of distribute.
Who do I need to talk about it?

Thanks,
Guy

From ozarow at gmail.com  Thu Sep 15 16:46:28 2011
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Thu, 15 Sep 2011 16:46:28 +0200
Subject: [Distutils] Correct way to include (debian) program icons.
In-Reply-To: <20110912165500.0b66f752@jabbar>
References: <20110912165500.0b66f752@jabbar>
Message-ID: <20110915144628.GX16705@piotro.eu>

[Mac Ryan, 2011-09-12]
> 	I was wondering if there is a way in distutils to
> include/indicate a program icon that will be used in the program
> menus / desktop shortcuts.

why do you want to use distutils for that? If it's Debian specific, use
Debian tools to install these files
(adding "dir/file.xpm /usr/share/pixmaps/" line to
debian/binary-package-name.install file should be enough)
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From quasipedia at gmail.com  Thu Sep 15 18:50:48 2011
From: quasipedia at gmail.com (Mac Ryan)
Date: Thu, 15 Sep 2011 18:50:48 +0200
Subject: [Distutils] Correct way to include (debian) program icons.
In-Reply-To: <20110915144628.GX16705@piotro.eu>
References: <20110912165500.0b66f752@jabbar> <20110915144628.GX16705@piotro.eu>
Message-ID: <20110915185048.1425b66e@jabbar>

On Thu, 15 Sep 2011 16:46:28 +0200
Piotr Ozarowski <ozarow at gmail.com> wrote:

> why do you want to use distutils for that? If it's Debian specific,
> use Debian tools to install these files

Hi Piotr,

	having an icon for a running program it's hardly
a debian-specific thing: more or less any OS have icons for
programs, so it seemed a candidate good enough to have been put in the
abstraction layer represented by setup.py.

	That said, I am very new to packaging, and so far the only way I
have been able to successfully build a package has been with:

	python setup.py --command-packages=stdeb.command bdist_deb

so I was just hoping there was a way to keep it this simple! :)

> (adding "dir/file.xpm /usr/share/pixmaps/" line to
> debian/binary-package-name.install file should be enough)

Thanks for this, sooner or later I will try again to have a crack at
packaging "from the scratch" but so far I got lost in the many
possible way one could do that... [I know - from the presentation you
gave somewhere and I found on the intertubes - that I should focus my
attention on dh_python2, but so far I haven't been able to find a
tutorial easy enough for me to follow. Recommendations on learning
material are most than welcome, of course!]

/mac

From barry at python.org  Thu Sep 15 20:15:52 2011
From: barry at python.org (Barry Warsaw)
Date: Thu, 15 Sep 2011 14:15:52 -0400
Subject: [Distutils] Correct way to include (debian) program icons.
In-Reply-To: <20110915185048.1425b66e@jabbar>
References: <20110912165500.0b66f752@jabbar> <20110915144628.GX16705@piotro.eu>
	<20110915185048.1425b66e@jabbar>
Message-ID: <20110915141552.05fecf31@resist.wooz.org>

On Sep 15, 2011, at 06:50 PM, Mac Ryan wrote:

>Thanks for this, sooner or later I will try again to have a crack at
>packaging "from the scratch" but so far I got lost in the many
>possible way one could do that... [I know - from the presentation you
>gave somewhere and I found on the intertubes - that I should focus my
>attention on dh_python2, but so far I haven't been able to find a
>tutorial easy enough for me to follow. Recommendations on learning
>material are most than welcome, of course!]

Packaging a simple, well-behaved Python package, with a well-written setup.py
*can* be pretty easy, although as I'm finding with my back-and-forth with
Piotr and Jakub on my flufl.* packages, not as easy as it *should* be.  Still,
I think it would be worthwhile to provide some concrete and modern
instructions for packaging Python applications from scratch.  I'm thinking
something more step-by-step instructional than the Debian Python Policy, and
less dependent on converting existing packages than TransitionToDHPython2.

Once the four of my packages pass muster and get sponsored into Debian, I'll
cobble something together for wiki.debian.org/Python.

Cheers,
-Barry

From setuptools at bugs.python.org  Thu Sep 15 21:12:13 2011
From: setuptools at bugs.python.org (Martin Qvist)
Date: Thu, 15 Sep 2011 19:12:13 +0000
Subject: [Distutils] [issue134] easy_install shebang should be /usr/bin/env
	python
Message-ID: <1316113932.99.0.985659121081.issue134@psf.upfronthosting.co.za>


New submission from Martin Qvist <q at maq.dk>:

easy_install uses #!/usr/bin/python but should use #!/usr/bin/env python. I wouldn't normally notice, but I have a system that uses Environment Modules (http://modules.sourceforge.net/), and there easy_install uses the wrong python version (the system one, not the one I load with modules).

----------
messages: 641
nosy: maq
priority: bug
status: unread
title: easy_install shebang should be /usr/bin/env python

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue134>
_______________________________________________

From ozarow at gmail.com  Fri Sep 16 17:16:36 2011
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Fri, 16 Sep 2011 17:16:36 +0200
Subject: [Distutils] Correct way to include (debian) program icons.
In-Reply-To: <20110915185048.1425b66e@jabbar>
References: <20110912165500.0b66f752@jabbar> <20110915144628.GX16705@piotro.eu>
	<20110915185048.1425b66e@jabbar>
Message-ID: <20110916151635.GZ16705@piotro.eu>

[Mac Ryan, 2011-09-15]
> 	having an icon for a running program it's hardly
> a debian-specific thing: more or less any OS have icons for
> programs, so it seemed a candidate good enough to have been put in the
> abstraction layer represented by setup.py.

/usr/share/pixmaps (or rather "menu" package that uses icons installed
there) is Debian specific. More general solution - Freedesktop.org's
desktop files is *nix specific (and used f.e. by GNOME or KDE in
Debian)... anyway packaging (AKA distutils2) will make it easier to
install such files.
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From reinout at vanrees.org  Sun Sep 18 01:27:25 2011
From: reinout at vanrees.org (Reinout van Rees)
Date: Sun, 18 Sep 2011 01:27:25 +0200
Subject: [Distutils] buildout and system packages: a simpler idea?
Message-ID: <j53acr$83s$1@dough.gmane.org>

Hi,

Buildout and system packages are apparently hard. Buildout 1.5.x tries 
hard to get it working, but I cannot get it to work (reliably), for 
instance.

Perhaps the problem is that we're trying too hard. Both buildout 1.5.x 
and the osc.recipe.sysegg recipe that I'm using on 1.4.x are trying to 
look up packages and trying to include them in the path.

What you normally want is just to use the system's PIL and mapnik and 
numpy. All stuff that you really don't want to build from source. So you 
assume it is installed with a clicky-clicky installer on windows and 
with apt-get on ubuntu.

Why not use this assumption? Add an "assumed-available" option to 
buildout that lists the eggs that are assumed to be available locally? 
No need to actually search them (which is the thing that breaks 1.5.x 
for me). Just make sure the regular python path is in place as the last 
item in sys.path.


Possible addition: allow a dotted path as a second parameter after the 
"assumed-available" eggs: buildout tries to import that path to check 
the availability of the egg that's assumed to be available.


So: if you need system eggs, you won't run buildout without system 
paths. Only thing left is to tell buildout that certain eggs are 
available locally so that it doesn't search for it. Just assume they're 
there and leave it at that.


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From dglick at gmail.com  Sun Sep 18 01:44:00 2011
From: dglick at gmail.com (David Glick)
Date: Sat, 17 Sep 2011 16:44:00 -0700
Subject: [Distutils] buildout and system packages: a simpler idea?
In-Reply-To: <j53acr$83s$1@dough.gmane.org>
References: <j53acr$83s$1@dough.gmane.org>
Message-ID: <4E7530C0.40508@gmail.com>

On 9/17/11 4:27 PM, Reinout van Rees wrote:
> Hi,
>
> Buildout and system packages are apparently hard. Buildout 1.5.x tries 
> hard to get it working, but I cannot get it to work (reliably), for 
> instance.
>
> Perhaps the problem is that we're trying too hard. Both buildout 1.5.x 
> and the osc.recipe.sysegg recipe that I'm using on 1.4.x are trying to 
> look up packages and trying to include them in the path.
>
> What you normally want is just to use the system's PIL and mapnik and 
> numpy. All stuff that you really don't want to build from source. So 
> you assume it is installed with a clicky-clicky installer on windows 
> and with apt-get on ubuntu.
>
> Why not use this assumption? Add an "assumed-available" option to 
> buildout that lists the eggs that are assumed to be available locally? 
> No need to actually search them (which is the thing that breaks 1.5.x 
> for me). Just make sure the regular python path is in place as the 
> last item in sys.path.
>
>
> Possible addition: allow a dotted path as a second parameter after the 
> "assumed-available" eggs: buildout tries to import that path to check 
> the availability of the egg that's assumed to be available.
>
>
> So: if you need system eggs, you won't run buildout without system 
> paths. Only thing left is to tell buildout that certain eggs are 
> available locally so that it doesn't search for it. Just assume 
> they're there and leave it at that.
collective.recipe.mockedeggs may be of use.

From reinout at vanrees.org  Sun Sep 18 01:52:11 2011
From: reinout at vanrees.org (Reinout van Rees)
Date: Sun, 18 Sep 2011 01:52:11 +0200
Subject: [Distutils] buildout and system packages: a simpler idea?
In-Reply-To: <4E7530C0.40508@gmail.com>
References: <j53acr$83s$1@dough.gmane.org> <4E7530C0.40508@gmail.com>
Message-ID: <j53br8$fb2$1@dough.gmane.org>

On 18-09-11 01:44, David Glick wrote:
> collective.recipe.mockedeggs may be of use.

I didn't know that one existed. Sounds like a good candidate.

Only thing that worries me: the version number. Is that mandatory? 
Perhaps it is a good idea, but I worry a bit about the possible results.


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From kirpit at gmail.com  Sun Sep 18 03:41:22 2011
From: kirpit at gmail.com (kirpit)
Date: Sun, 18 Sep 2011 04:41:22 +0300
Subject: [Distutils] finding a package namespace
Message-ID: <CAMzAXOi=5AxKXae6EzFaU=Y3XpSAbjLNwPWacvx64pabtEwFQQ@mail.gmail.com>

Hi Greg and Anthony.

I've been trying to deal with very simple question within python for weeks;
http://stackoverflow.com/questions/7184375/how-to-find-import-name-of-any-package-in-python

I modified the TokenMacGuy's code snippet a bit and put it on github;
https://gist.github.com/1176645

But still could not get that simple question working in a reliable way
for all the packages on pypi. It is failing for every second package.
That's why my little gadget pydoc.net is still not working and people
can't use it. I am getting hopeless, please help.

Roy E.

From quasipedia at gmail.com  Mon Sep 19 00:42:27 2011
From: quasipedia at gmail.com (Mac Ryan)
Date: Mon, 19 Sep 2011 00:42:27 +0200
Subject: [Distutils] Correct way to include (debian) program icons.
In-Reply-To: <20110915141552.05fecf31@resist.wooz.org>
References: <20110912165500.0b66f752@jabbar> <20110915144628.GX16705@piotro.eu>
	<20110915185048.1425b66e@jabbar>
	<20110915141552.05fecf31@resist.wooz.org>
Message-ID: <20110919004227.2b84fd7b@jabbar>

On Thu, 15 Sep 2011 14:15:52 -0400
Barry Warsaw <barry at python.org> wrote:

> I think it would be worthwhile to provide some concrete and modern
> instructions for packaging Python applications from scratch.  I'm
> thinking something more step-by-step instructional than the Debian
> Python Policy, and less dependent on converting existing packages
> than TransitionToDHPython2.
>
> Once the four of my packages pass muster
> and get sponsored into Debian, I'll cobble something together for
> wiki.debian.org/Python.

If you need somebody to "beta test" your tutorial (meaning: somebody
who does not know much about packaging and that will try to follow it
reporting parts that are unclear or question that do not find answer),
just drop me a line! :)

/mac

From balicki.aleksander at gmail.com  Sun Sep 18 23:06:13 2011
From: balicki.aleksander at gmail.com (Aleksander Balicki)
Date: Sun, 18 Sep 2011 23:06:13 +0200 (CEST)
Subject: [Distutils] easy_install is using umask during installation,
	it shouldn't
Message-ID: <alpine.LNX.2.00.1109182304270.3299@bialobrewy.alistra.ath.cx>

I have umask 077 on root, and always after installing a python package
with easy_install I get permission errors, because packages are being
installed to their directories, but users can't access them because of
the permissions generated by the root's umask.

The installed python packages should have 755/644 mods.

-- 
Aleksander "Alistra" Balicki

email:	balicki.aleksander at gmail.com
jabber:	wszystkie.inne.byly.zajete at gmail.com

From merwok at netwok.org  Mon Sep 19 15:46:15 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Mon, 19 Sep 2011 15:46:15 +0200
Subject: [Distutils] Given a distribution,
 find the Python modules (Was: finding a package namespace)
In-Reply-To: <CAMzAXOi=5AxKXae6EzFaU=Y3XpSAbjLNwPWacvx64pabtEwFQQ@mail.gmail.com>
References: <CAMzAXOi=5AxKXae6EzFaU=Y3XpSAbjLNwPWacvx64pabtEwFQQ@mail.gmail.com>
Message-ID: <4E7747A7.1020505@netwok.org>

Hi,

> I've been trying to deal with very simple question within python for weeks;
> http://stackoverflow.com/questions/7184375/how-to-find-import-name-of-any-package-in-python
What?s the use case?

> I modified the TokenMacGuy's code snippet a bit and put it on github;
You should be able to simplify that a bit, I?ll try to post something on
Stack Overflow later.

Here are some ideas:
- run setup.py install in a temp directory and detect modules
- run setuptools? egg_info command and introspect one of the files it
creates

Cheers

From benji at benjiyork.com  Mon Sep 19 19:24:22 2011
From: benji at benjiyork.com (Benji York)
Date: Mon, 19 Sep 2011 13:24:22 -0400
Subject: [Distutils] buildout and system packages: a simpler idea?
In-Reply-To: <j53acr$83s$1@dough.gmane.org>
References: <j53acr$83s$1@dough.gmane.org>
Message-ID: <CAC8tyQ5qB3OKxztYK_Tj5ha1yxtOGz8aFf48jcTo4eW5pqQp=w@mail.gmail.com>

On Sat, Sep 17, 2011 at 7:27 PM, Reinout van Rees <reinout at vanrees.org> wrote:
> Hi,
>
> Buildout and system packages are apparently hard. Buildout 1.5.x tries hard
> to get it working, but I cannot get it to work (reliably), for instance.
>
> Perhaps the problem is that we're trying too hard. Both buildout 1.5.x and
> the osc.recipe.sysegg recipe that I'm using on 1.4.x are trying to look up
> packages and trying to include them in the path.
>
> What you normally want is just to use the system's PIL and mapnik and numpy.
> All stuff that you really don't want to build from source. So you assume it
> is installed with a clicky-clicky installer on windows and with apt-get on
> ubuntu.

There's another thing you might want when using a system Python: to
exclude some packages.  I suspect that's what makes it hard.
-- 
Benji York

From reinout at vanrees.org  Mon Sep 19 22:39:36 2011
From: reinout at vanrees.org (Reinout van Rees)
Date: Mon, 19 Sep 2011 22:39:36 +0200
Subject: [Distutils] buildout and system packages: a simpler idea?
In-Reply-To: <CAC8tyQ5qB3OKxztYK_Tj5ha1yxtOGz8aFf48jcTo4eW5pqQp=w@mail.gmail.com>
References: <j53acr$83s$1@dough.gmane.org>
	<CAC8tyQ5qB3OKxztYK_Tj5ha1yxtOGz8aFf48jcTo4eW5pqQp=w@mail.gmail.com>
Message-ID: <j589a3$vrn$1@dough.gmane.org>

On 19-09-11 19:24, Benji York wrote:
>> What you normally want is just to use the system's PIL and mapnik and numpy.
>> >  All stuff that you really don't want to build from source. So you assume it
>> >  is installed with a clicky-clicky installer on windows and with apt-get on
>> >  ubuntu.
> There's another thing you might want when using a system Python: to
> exclude some packages.  I suspect that's what makes it hard.

My guess is that that's indeed a very hard task. But not that common. If 
you want to use something from your system python, it is more than 
reasonable to expect you to keep it somewhat clean and unpolluted by 
packages you don't want in most of your buildouts.


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From mithrandi at mithrandi.net  Wed Sep 21 16:20:08 2011
From: mithrandi at mithrandi.net (Tristan Seligmann)
Date: Wed, 21 Sep 2011 16:20:08 +0200
Subject: [Distutils] Twisted plugin system and Python packaging
Message-ID: <CAMcKhMQr77VEBJY1v8654T1j26saN8c+MSmXH6D6eGqtUuNyUA@mail.gmail.com>

This conversation originally started in a bug report against pip[1],
but I'm moving it here since it seems this might be a better venue;
you may wish to read the bug log for interest, but I'll summarize the
issue from scratch anyway.

Twisted has a plugin system (twisted.plugin[2]) which is used by
Twisted itself, as well as other projects (such as Axiom[3] and
Dosage[4]) to allow for pluggable extensibility. The implementation
looks for modules in a plugins package and then looks in those modules
for objects providing (in the zope.interface sense) the IPlugin
interface. For the sake of simplifying this explanation, I will only
refer to twisted.plugins in the rest of this mail, but assume that
everything applies similarly to plugin packages in other projects (eg.
axiom.plugins and dosage.plugins).

twisted/plugins/__init__.py sets __path__ as follows:

from twisted.plugin import pluginPackagePaths
__path__.extend(pluginPackagePaths(__name__))

The implementation of pluginPackagePaths[5] loops through the
directories on sys.path, and includes DIR/twisted/plugins/ for every
DIR on sys.path, *except* when DIR/twisted/plugins/__init__.py exists.
This means that if you drop a project dir into PYTHONPATH while
developing, the plugins from your project will be picked up, even
though they're not installed into the copy of Twisted you're using;
you can also have a project shipping plugins installed in
~/.local/lib/pythonX.Y/site-packages even though you're using a
site-wide install of Twisted in /usr/lib/pythonX.Y/site-packages.
However, if you have multiple versions of Twisted on sys.path (because
you have a newer version installed locally overriding the site-wide
one, for example), plugins from a different version of Twisted won't
be accidentally picked up.

Given the way __path__ is set, if you have a project that ships
Twisted plugins, it would be possible to install it to
.../site-packages/MyProject/myproject/... +
.../site-packages/MyProject/twisted/plugins/myproject_plugin.py, and
then include this in a .../site-packages/MyProject.pth. However, the
usual way to install your project is to install it to
.../site-packages/myproject/... +
.../site-packages/twisted/plugins/myproject_plugin.py; in other words,
directly into the existing Twisted installation. If you include
"twisted.plugins" in your setup.py, then this works fine with
distutils "setup.py install" as well as "pip install"; setuptools
"setup.py install" will install everything into an egg, which will
also work due to the way __path__ is set. However, since "twisted"
ends up in top_files.txt in the egg-info, "pip uninstall" will blow
away your whole Twisted install when uninstalling a project shipping
Twisted plugins that was installed with "pip install". Ironically, if
the project was installed with setuptools "setup.py install", there is
no problem, since pip just removes the egg that was installed, and
nothing else.

In the pip bug report, Carl Meyer suggests using the setuptools
"namespace packages" feature; this works around the immediate problem
(pip uninstall doesn't blow away your Twisted install anymore), but
causes somer othe problems of its own. First of all, code in
__init__.py is not functional for a namespace package, so the __path__
functionality described above ceases to function. Additionally,
twisted.plugins cannot be declared as a namespace package without also
declaring twisted as a namespace package, which thus also affects
twisted/__init__.py; the code in this module exports an __version__
attribute (identifying the version of Twisted) as well as importing
twisted.python.compat to ensure that various backwards compatibility
monkeypatches are installed.

So, how should Twisted and Twisted-related projects be packaged in
order to avoid these issues? Please bear in mind that the current
plugin system in Twisted was first introduced around March 2005
(replacing the even older plugin system in use at the time, I
believe), thus there are quite a number of users relying on this code;
any changes would need to be backwards-compatible to avoid causing
problems for all of the existing projects and users relying on the
functionality.

[1] https://github.com/pypa/pip/issues/355

[2] http://twistedmatrix.com/documents/current/core/howto/plugin.html

[3] https://launchpad.net/divmod-axiom

[4] https://launchpad.net/dosage

[5] http://twistedmatrix.com/trac/browser/trunk/twisted/plugin.py#L225
-- 
mithrandi, i Ainil en-Balandor, a faer Ambar

From pje at telecommunity.com  Sun Sep 25 05:01:44 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 24 Sep 2011 23:01:44 -0400
Subject: [Distutils] Twisted plugin system and Python packaging
In-Reply-To: <CAMcKhMQr77VEBJY1v8654T1j26saN8c+MSmXH6D6eGqtUuNyUA@mail.g
	mail.com>
References: <CAMcKhMQr77VEBJY1v8654T1j26saN8c+MSmXH6D6eGqtUuNyUA@mail.gmail.com>
Message-ID: <20110925030153.091993A40B2@sparrow.telecommunity.com>

At 04:20 PM 9/21/2011 +0200, Tristan Seligmann wrote:
>If you include "twisted.plugins" in your setup.py, then this works 
>fine with distutils "setup.py install" as well as "pip install"; 
>setuptools "setup.py install" will install everything into an egg, 
>which will also work due to the way __path__ is set. However, since 
>"twisted" ends up in top_files.txt in the egg-info, "pip uninstall" 
>will blow away your whole Twisted install when uninstalling a 
>project shipping Twisted plugins that was installed with "pip install".

This really sounds like a bug in pip; top_level.txt is not a 
replacement for a proper uninstall log.

>So, how should Twisted and Twisted-related projects be packaged in 
>order to avoid these issues? Please bear in mind that the current 
>plugin system in Twisted was first introduced around March 2005 
>(replacing the even older plugin system in use at the time, I 
>believe), thus there are quite a number of users relying on this 
>code; any changes would need to be backwards-compatible to avoid 
>causing problems for all of the existing projects and users relying 
>on the functionality.

I think you've answered your own question here: there *isn't* any way 
to package Twisted-related projects in a way that avoids the issue, 
due to the bug in pip.  It's not Twisted's fault that pip takes shortcuts here.


From glyph at twistedmatrix.com  Sun Sep 25 05:32:55 2011
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Sat, 24 Sep 2011 23:32:55 -0400
Subject: [Distutils] Twisted plugin system and Python packaging
In-Reply-To: <20110925030153.091993A40B2@sparrow.telecommunity.com>
References: <CAMcKhMQr77VEBJY1v8654T1j26saN8c+MSmXH6D6eGqtUuNyUA@mail.gmail.com>
	<20110925030153.091993A40B2@sparrow.telecommunity.com>
Message-ID: <59226611-0A96-4C98-865D-6E5D88954BFB@twistedmatrix.com>


On Sep 24, 2011, at 11:01 PM, P.J. Eby wrote:

> This really sounds like a bug in pip; top_level.txt is not a replacement for a proper uninstall log.

Thanks.  It appears that Pip's maintainers are (the teensiest bit grudgingly) coming around to this perspective as well: https://github.com/pypa/pip/issues/355#issuecomment-2182951

Hopefully we can get a solution that pretty much works everywhere, soon.

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20110924/069042dc/attachment.html>

From pje at telecommunity.com  Sun Sep 25 07:44:05 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 25 Sep 2011 01:44:05 -0400
Subject: [Distutils] Twisted plugin system and Python packaging
In-Reply-To: <59226611-0A96-4C98-865D-6E5D88954BFB@twistedmatrix.com>
References: <CAMcKhMQr77VEBJY1v8654T1j26saN8c+MSmXH6D6eGqtUuNyUA@mail.gmail.com>
	<20110925030153.091993A40B2@sparrow.telecommunity.com>
	<59226611-0A96-4C98-865D-6E5D88954BFB@twistedmatrix.com>
Message-ID: <20110925054409.573E83A40B2@sparrow.telecommunity.com>

At 11:32 PM 9/24/2011 -0400, Glyph Lefkowitz wrote:

>On Sep 24, 2011, at 11:01 PM, P.J. Eby wrote:
>
>>This really sounds like a bug in pip; top_level.txt is not a 
>>replacement for a proper uninstall log.
>
>Thanks.  It appears that Pip's maintainers are (the teensiest bit 
>grudgingly) coming around to this perspective as well: 
><https://github.com/pypa/pip/issues/355#issuecomment-2182951>https://github.com/pypa/pip/issues/355#issuecomment-2182951

You may want to note that Twisted isn't the only distribution that 
ships partial packages that aren't namespace packages.  The logilabs 
packages are (or at any rate were) distributed this way, with a 
top-level package in one distribution, and some modules (without an 
__init__) were shipped in other distributions.  (I could be wrong, 
but I think the mx.* stuff works this way, too.)

(These are all packages that are distutils-only, though, or have 
their own extensions, IIRC.)


From glyph at twistedmatrix.com  Mon Sep 26 02:52:11 2011
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Sun, 25 Sep 2011 20:52:11 -0400
Subject: [Distutils] Twisted plugin system and Python packaging
In-Reply-To: <20110925054409.573E83A40B2@sparrow.telecommunity.com>
References: <CAMcKhMQr77VEBJY1v8654T1j26saN8c+MSmXH6D6eGqtUuNyUA@mail.gmail.com>
	<20110925030153.091993A40B2@sparrow.telecommunity.com>
	<59226611-0A96-4C98-865D-6E5D88954BFB@twistedmatrix.com>
	<20110925054409.573E83A40B2@sparrow.telecommunity.com>
Message-ID: <59DD9F37-444B-423D-9316-DAE08A2BF3C4@twistedmatrix.com>


On Sep 25, 2011, at 1:44 AM, P.J. Eby wrote:

> You may want to note that Twisted isn't the only distribution that ships partial packages that aren't namespace packages.  The logilabs packages are (or at any rate were) distributed this way, with a top-level package in one distribution, and some modules (without an __init__) were shipped in other distributions.  (I could be wrong, but I think the mx.* stuff works this way, too.)

Thanks for that data point!  It is really good to know, I'll use it to bludgeon some people with next time this issue comes up (on this list, or on some Debian bug, or whatever) ;-).

> (These are all packages that are distutils-only, though, or have their own extensions, IIRC.)

You mean "extensions to distutils", or C extensions?  I think Twisted qualifies for both of those, and we definitely want to remain installable without setuptools, although we do use it for one or two things if it's available now.

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20110925/e5c1bf83/attachment.html>

From pje at telecommunity.com  Mon Sep 26 05:36:55 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 25 Sep 2011 23:36:55 -0400
Subject: [Distutils] Twisted plugin system and Python packaging
In-Reply-To: <59DD9F37-444B-423D-9316-DAE08A2BF3C4@twistedmatrix.com>
References: <CAMcKhMQr77VEBJY1v8654T1j26saN8c+MSmXH6D6eGqtUuNyUA@mail.gmail.com>
	<20110925030153.091993A40B2@sparrow.telecommunity.com>
	<59226611-0A96-4C98-865D-6E5D88954BFB@twistedmatrix.com>
	<20110925054409.573E83A40B2@sparrow.telecommunity.com>
	<59DD9F37-444B-423D-9316-DAE08A2BF3C4@twistedmatrix.com>
Message-ID: <20110926033710.9E7673A40AA@sparrow.telecommunity.com>

At 08:52 PM 9/25/2011 -0400, Glyph Lefkowitz wrote:
>On Sep 25, 2011, at 1:44 AM, P.J. Eby wrote:
>>(These are all packages that are distutils-only, though, or have 
>>their own extensions, IIRC.)
>
>You mean "extensions to distutils", or C extensions?

The former.  The mx.* stuff has a mx_setup or some such distutils 
extension.  Not sure about the ll.* stuff.


From ziade.tarek at gmail.com  Tue Sep 27 11:40:44 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 11:40:44 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
Message-ID: <CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>

I have sent that to the PSF list because there's a PSF project about PyPI infra.

But someone complained, saying that I was doing this discussion
"behind closed doors"

SInce this is not my goal, I am now spamming more lists...


---------- Forwarded message ----------
From: Tarek Ziad? <ziade.tarek at gmail.com>
Date: Tue, Sep 27, 2011 at 10:37 AM
Subject: The state of PyPI
To: PSF Members List <psf-members at python.org>
Cc: Richard Jones <r1chardj0n3s at gmail.com>, Steve Holden <steve at holdenweb.com>


Hey

This is just a mail that summarizes the current state of PyPI, the
existing features, and what can be done next to improve stuff.

I am sending this in the PSF members list because we had a project of
an infrastructure going on, and I want to make sure all involved
parties are in the same page.

1/ stability and high availability
2/ private mirrors
3/ private projects
4/ tutorial ?


= stability and high availability =

we went in two directions to improve PyPI :

1/ add the mirroring protocol
2/ make the PyPI server more reliable by pushing its storage in a
redundant cloud.

== mirroring ==

The mirroring protocol (PEP 381) is implemented on server-side, I've
worked with Martin on this, and we have mirrors now:

Look at http://pypi.python.org/mirrors

Also, there's a client that anyone can use to set up a mirror:
http://pypi.python.org/pypi/pep381client

The idea is that anyone in the community willing to maintain a mirror
can do so. We add the mirror in the CND, and make it available for
client tools to use. What's really missing right now is more
integration on client-side.

- Pip supports the mirroring protocol, and can fall back to a mirror,
but I am not 100% sure this is a default behavior. ?(please correct me
if it is now)
- Buildout knows how to use *another server* than the main PyPI, so
can manually switch to a mirror, but I don't think it's transparent.
It should.
- Distribute/Setuptools does not do anything for this, and should.
- everything is already implemented in packaging/distutils2

The effect of the mirrors is that PyPI being down should not impact
the community. This will be true once all tool are transparently using
the mirrors.

== better infra ==

I think the project is staled right now.


= private mirrors =

Having a private mirror makes a lot of sense, when companies need to
make sure their build systems are not relying on external services
like PyPI or a mirror. It's also a good way to dramatically reduce the
load for the community servers.

The idea is that a Jenkins server that builds hundreds of Python apps
every hour should not hammer PyPI.

We have everything needed these days to set up this kind of system,
with zc.buildout or pip good practices.

What we need is a good tutorial or a guide [*]

= private projects =

The part that we do not address in the community is private projects:
since we don't have any permissions/group/roles system in PyPI,
everything is public.

One way to solve this is to have a local repository for private
packages, that is looked by tools like pip or easy_install, with the
--find-links option.


What we need is a good tutorial or a guide [*]

= tutorial =

[*] If this helps, I am willing to work on a tutorial day for Pycon
US, that goes through all of this, to help people set up their dev.
environment the best way possible.

The material could then be published at python.org/pypi to help out.

I know Richard has some material already, so maybe this could be a
joint tutorial ?

HTH

Cheers
Tarek





--
Tarek Ziad? | http://ziade.org



-- 
Tarek Ziad? | http://ziade.org

From mal at egenix.com  Tue Sep 27 12:04:25 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 27 Sep 2011 12:04:25 +0200
Subject: [Distutils] [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
Message-ID: <4E819FA9.7020300@egenix.com>

Tarek Ziad? wrote:
> = stability and high availability =
> 
> we went in two directions to improve PyPI :
> 
> 1/ add the mirroring protocol
> 2/ make the PyPI server more reliable by pushing its storage in a
> redundant cloud.
> ...
> == better infra ==
> 
> I think the project is staled right now.

True, I don't have the needed cycles to make progress and neither
do the team members. We do have the Amazon infrastructure setup
(EC2, CloudFront, S3), and some progress has been made to sync
the S3 storage with the local copies of the packages.

The next steps would be to implement a trigger based sync scheme
for packages and the simple/ index, i.e. new uploads should trigger
an S3 copy process and update the simple/ index on S3 as well.

For more info, please have a look at the wiki page:

    http://wiki.python.org/moin/CloudPyPI

Here's the original proposal:

    http://wiki.python.org/moin/CloudPyPI/Proposal

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 27 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2011-10-04: PyCon DE 2011, Leipzig, Germany                 7 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From regebro at gmail.com  Tue Sep 27 12:07:15 2011
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 27 Sep 2011 12:07:15 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
Message-ID: <CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>

On Tue, Sep 27, 2011 at 11:40, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> 1/ stability and high availability

How are opinions on setting up country-specific PyPI mirrors? The lag
to the US is pretty severe in Poland, and I suspect my buildouts would
benefit from having a server in Poland. Now, of course, it could be
called x.pypi.python.org, but maybe we should have aliases such as
pl.pypi.python.org as well?

I have no strong opinion on the issue, what do others think?

//Lennart

From ziade.tarek at gmail.com  Tue Sep 27 12:10:40 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 12:10:40 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
Message-ID: <CAGSi+Q4FzCCWmbtKF_3ZDO3=mSJc+9DzBn+7Nx=ZzwFrqTqG3w@mail.gmail.com>

On Tue, Sep 27, 2011 at 12:07 PM, Lennart Regebro <regebro at gmail.com> wrote:
> On Tue, Sep 27, 2011 at 11:40, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> 1/ stability and high availability
>
> How are opinions on setting up country-specific PyPI mirrors? The lag
> to the US is pretty severe in Poland, and I suspect my buildouts would
> benefit from having a server in Poland. Now, of course, it could be
> called x.pypi.python.org, but maybe we should have aliases such as
> pl.pypi.python.org as well?
>
> I have no strong opinion on the issue, what do others think?

I think it's a good idea to have the closest mirror,.

One long-term goal I had was to add a client-side geloc code that
would prefer the closest mirror.

IOW, with your IP and the mirrors IP, pick the closest IP

>
> //Lennart
>



-- 
Tarek Ziad? | http://ziade.org

From jim at zope.com  Tue Sep 27 13:12:11 2011
From: jim at zope.com (Jim Fulton)
Date: Tue, 27 Sep 2011 07:12:11 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
Message-ID: <CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>

On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro <regebro at gmail.com> wrote:
> On Tue, Sep 27, 2011 at 11:40, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> 1/ stability and high availability
>
> How are opinions on setting up country-specific PyPI mirrors? The lag
> to the US is pretty severe in Poland, and I suspect my buildouts would
> benefit from having a server in Poland. Now, of course, it could be
> called x.pypi.python.org, but maybe we should have aliases such as
> pl.pypi.python.org as well?
>
> I have no strong opinion on the issue, what do others think?

Wouldn't CloudFront make this moot?

I'm not clear I follow the plans, but it appears that CloudFront would
provide greater availability with little or no changes to clients.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From ziade.tarek at gmail.com  Tue Sep 27 13:58:39 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 13:58:39 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
Message-ID: <CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>

2011/9/27 Jim Fulton <jim at zope.com>:
> On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro <regebro at gmail.com> wrote:
>> On Tue, Sep 27, 2011 at 11:40, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>>> 1/ stability and high availability
>>
>> How are opinions on setting up country-specific PyPI mirrors? The lag
>> to the US is pretty severe in Poland, and I suspect my buildouts would
>> benefit from having a server in Poland. Now, of course, it could be
>> called x.pypi.python.org, but maybe we should have aliases such as
>> pl.pypi.python.org as well?
>>
>> I have no strong opinion on the issue, what do others think?
>
> Wouldn't CloudFront make this moot?

If we state that PyPI completely depends on Amazon I guess yes.

But imho, it's saner for the long term to have a community-driven
protocol for mirroring so we don't rely on third-party vendors.

>
> I'm not clear I follow the plans, but it appears that CloudFront would
> provide greater availability with little or no changes to clients.
>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
>



-- 
Tarek Ziad? | http://ziade.org

From jim at zope.com  Tue Sep 27 14:27:02 2011
From: jim at zope.com (Jim Fulton)
Date: Tue, 27 Sep 2011 08:27:02 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
Message-ID: <CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>

On Tue, Sep 27, 2011 at 7:58 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> 2011/9/27 Jim Fulton <jim at zope.com>:
>> On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro <regebro at gmail.com> wrote:
>>> On Tue, Sep 27, 2011 at 11:40, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>>>> 1/ stability and high availability
>>>
>>> How are opinions on setting up country-specific PyPI mirrors? The lag
>>> to the US is pretty severe in Poland, and I suspect my buildouts would
>>> benefit from having a server in Poland. Now, of course, it could be
>>> called x.pypi.python.org, but maybe we should have aliases such as
>>> pl.pypi.python.org as well?
>>>
>>> I have no strong opinion on the issue, what do others think?
>>
>> Wouldn't CloudFront make this moot?
>
> If we state that PyPI completely depends on Amazon I guess yes.
>
> But imho, it's saner for the long term to have a community-driven
> protocol for mirroring so we don't rely on third-party vendors.

I understand where you're coming from but, ..
I think it's saner to rely on proven technology
than to invent our own protocol. NIH?

BTW, in looking at PEP 381 (yeah, I know, I'm a bad person
for waiting so long) I have lots of reservations about the protocol:

- It's potentially complex to implement efficiently, especially given that:

- We've had problems with mirrors getting out of date, meaning that,
potentially, clients should
  check multiple indexes,

- It either requires extra dns calls or relies to heavily on the last
mirror, which is probably likely
  to be the least reliable.

Life is short. We don't have to invent this ourselves.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From wichert at wiggy.net  Tue Sep 27 14:39:57 2011
From: wichert at wiggy.net (Wichert Akkerman)
Date: Tue, 27 Sep 2011 14:39:57 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
Message-ID: <4E81C41D.60703@wiggy.net>

On 09/27/2011 02:27 PM, Jim Fulton wrote:
> On Tue, Sep 27, 2011 at 7:58 AM, Tarek Ziad?<ziade.tarek at gmail.com>  wrote:
>> 2011/9/27 Jim Fulton<jim at zope.com>:
>>> On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro<regebro at gmail.com>  wrote:
>>>> On Tue, Sep 27, 2011 at 11:40, Tarek Ziad?<ziade.tarek at gmail.com>  wrote:
>>>>> 1/ stability and high availability
>>>> How are opinions on setting up country-specific PyPI mirrors? The lag
>>>> to the US is pretty severe in Poland, and I suspect my buildouts would
>>>> benefit from having a server in Poland. Now, of course, it could be
>>>> called x.pypi.python.org, but maybe we should have aliases such as
>>>> pl.pypi.python.org as well?
>>>>
>>>> I have no strong opinion on the issue, what do others think?
>>> Wouldn't CloudFront make this moot?
>> If we state that PyPI completely depends on Amazon I guess yes.
>>
>> But imho, it's saner for the long term to have a community-driven
>> protocol for mirroring so we don't rely on third-party vendors.
> I understand where you're coming from but, ..
> I think it's saner to rely on proven technology
> than to invent our own protocol. NIH?

This also feels like a problem that has already been solved in various 
ways by Debian, RedHat, CPAN and others.

Wichert.

From ziade.tarek at gmail.com  Tue Sep 27 14:40:26 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 14:40:26 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
Message-ID: <CAGSi+Q5Jqj5HQG2S=1Abc2Rrb_rpFaWgfNNZGfbp=Pd_L=ZFCg@mail.gmail.com>

On Tue, Sep 27, 2011 at 2:27 PM, Jim Fulton <jim at zope.com> wrote:
...
>
> I understand where you're coming from but, ..

Sorry, I don't understand what you imply here.


> I think it's saner to rely on proven technology
> than to invent our own protocol. NIH?

Ah sorry I misunderstood then. I thought CloudFront was a proprietary
platform, with its own protocol.

If you're saying that we can move away from CloudFront at any time and
have the same feature elsewhere, then it's perfect.

If you're saying that CloudFront is proven technology and that we
should not worry about relying on them, then I think we can do better
for the community to get locked-in for this, and continue to work on
an open protocol where everyone can participate by providing a spare
server.  But maybe that's just me ?

Most of the mirroring protocol was inspired by Perl's CPAN btw.

>
> BTW, in looking at PEP 381 (yeah, I know, I'm a bad person
> for waiting so long)

Yeah, started around 2 years ago, but comments are always welcome  :)

>  I have lots of reservations about the protocol:
>
> - It's potentially complex to implement efficiently, especially given that:
>
> - We've had problems with mirrors getting out of date, meaning that,
> potentially, clients should
> ?check multiple indexes,

Yeah, mirrors do get out of sync. There's a freshness time stamp.

But the use case is usually: PyPI is down, we fallback to a mirror. I
don't think it's more complicated than this.

>
> - It either requires extra dns calls or relies to heavily on the last
> mirror, which is probably likely
> ?to be the least reliable.

Once you have the list, I don't think you require extra call.

see http://hg.python.org/cpython/file/84280fac98b9/Lib/packaging/pypi/mirrors.py

>
> Life is short. We don't have to invent this ourselves.

Ah well, yeah -- Not sure what you are proposing right now.

If you imply that everything should be solved on server-side, and that
we should not have mirroring

>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
>



-- 
Tarek Ziad? | http://ziade.org

From ziade.tarek at gmail.com  Tue Sep 27 14:51:12 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 14:51:12 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <4E81C41D.60703@wiggy.net>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<4E81C41D.60703@wiggy.net>
Message-ID: <CAGSi+Q49-4uc9spgSUkEQSVJbv1+4VBjCUXcGE3jix2RNYCHng@mail.gmail.com>

On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman <wichert at wiggy.net> wrote:
..
>>
>> I understand where you're coming from but, ..
>> I think it's saner to rely on proven technology
>> than to invent our own protocol. NIH?
>
> This also feels like a problem that has already been solved in various ways
> by Debian, RedHat, CPAN and others.

Yes, and we've found a way similar to CPAN, with some Python specifics
(PyPI download statistics mainly)

Oh my, we're cycling again.

Nothing personal to you or Jim, but I have a sudden fatigue on
packaging because it seems like people are ignoring what's being done
to complain afterwards about us suffering of some kind of NIH :)

If you're seeing anything you don't like in PEP  381 (accepted a while
ago), go ahead and propose some improvements.

But please keep in mind that we've looked at other systems before we
wrote that PEP.

Cheers
Tarek

-- 
Tarek Ziad? | http://ziade.org

From wichert at wiggy.net  Tue Sep 27 15:00:57 2011
From: wichert at wiggy.net (Wichert Akkerman)
Date: Tue, 27 Sep 2011 15:00:57 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q49-4uc9spgSUkEQSVJbv1+4VBjCUXcGE3jix2RNYCHng@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>	<4E81C41D.60703@wiggy.net>
	<CAGSi+Q49-4uc9spgSUkEQSVJbv1+4VBjCUXcGE3jix2RNYCHng@mail.gmail.com>
Message-ID: <4E81C909.5060100@wiggy.net>

On 09/27/2011 02:51 PM, Tarek Ziad? wrote:
> On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman<wichert at wiggy.net>  wrote:
> ..
>>> I understand where you're coming from but, ..
>>> I think it's saner to rely on proven technology
>>> than to invent our own protocol. NIH?
>> This also feels like a problem that has already been solved in various ways
>> by Debian, RedHat, CPAN and others.
> Yes, and we've found a way similar to CPAN, with some Python specifics
> (PyPI download statistics mainly)
>
> Oh my, we're cycling again.
>
> Nothing personal to you or Jim, but I have a sudden fatigue on
> packaging because it seems like people are ignoring what's being done
> to complain afterwards about us suffering of some kind of NIH :)

It's just that my perspective is that of a simple user. And from my 
perspective nothing has changed in the last couple of years. Pypi still 
goes down occasionally, and when that happens many things start 
breaking. It may very well be that there are things planned or in 
progress, but until they are both usable and used by standard tools, 
which for me means buildout and setuptools, they are invisible.

Wichert.

From ziade.tarek at gmail.com  Tue Sep 27 15:07:53 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 15:07:53 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <4E81C909.5060100@wiggy.net>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<4E81C41D.60703@wiggy.net>
	<CAGSi+Q49-4uc9spgSUkEQSVJbv1+4VBjCUXcGE3jix2RNYCHng@mail.gmail.com>
	<4E81C909.5060100@wiggy.net>
Message-ID: <CAGSi+Q5S5M6r-R1jNQydL7WpvMY=UTBrCjbc5zWjKAxzO6RgDw@mail.gmail.com>

On Tue, Sep 27, 2011 at 3:00 PM, Wichert Akkerman <wichert at wiggy.net> wrote:
> On 09/27/2011 02:51 PM, Tarek Ziad? wrote:
>>
>> On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman<wichert at wiggy.net>
>> ?wrote:
>> ..
>>>>
>>>> I understand where you're coming from but, ..
>>>> I think it's saner to rely on proven technology
>>>> than to invent our own protocol. NIH?
>>>
>>> This also feels like a problem that has already been solved in various
>>> ways
>>> by Debian, RedHat, CPAN and others.
>>
>> Yes, and we've found a way similar to CPAN, with some Python specifics
>> (PyPI download statistics mainly)
>>
>> Oh my, we're cycling again.
>>
>> Nothing personal to you or Jim, but I have a sudden fatigue on
>> packaging because it seems like people are ignoring what's being done
>> to complain afterwards about us suffering of some kind of NIH :)
>
> It's just that my perspective is that of a simple user. And from my
> perspective nothing has changed in the last couple of years. Pypi still goes
> down occasionally, and when that happens many things start breaking. It may
> very well be that there are things planned or in progress, but until they
> are both usable and used by standard tools, which for me means buildout and
> setuptools, they are invisible.

Fair enough,

Pip has now the mirroring protocol implemented. I think they want to
make it a default option for the next major Pip release. IOW, you
should not suffer for downtimes using pip.

We'd need to add the same feature in easy_install and zc.buildout.
But since Pip did it, I think it's possible.

We have 5 mirrors run by the community
(http://pypi.python.org/mirrors) and I suspect porting pip's feature
to zc.buildout and easy_install would take less time than creating a
<Name your cloud service> app.


Cheers
Tarek

>
> Wichert.
>



-- 
Tarek Ziad? | http://ziade.org

From marrakis at gmail.com  Tue Sep 27 15:22:31 2011
From: marrakis at gmail.com (Mathieu Leduc-Hamel)
Date: Tue, 27 Sep 2011 09:22:31 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q5S5M6r-R1jNQydL7WpvMY=UTBrCjbc5zWjKAxzO6RgDw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<4E81C41D.60703@wiggy.net>
	<CAGSi+Q49-4uc9spgSUkEQSVJbv1+4VBjCUXcGE3jix2RNYCHng@mail.gmail.com>
	<4E81C909.5060100@wiggy.net>
	<CAGSi+Q5S5M6r-R1jNQydL7WpvMY=UTBrCjbc5zWjKAxzO6RgDw@mail.gmail.com>
Message-ID: <CABkuczXb--iriS4onhGVBoGafOi5h84jtaz8wZJCGn365c9+sA@mail.gmail.com>

Is there any plan also to work on an updated and better version of
Pypi, I mean, i understand that we need to work on the infrastructure
and make sure it will handle the load.

But i also think that the current codebase of Pypi would need be
updated. Is there any plan refactored in part of this work ? i Suspect
that this is something we'll have to do one day another if we want
continue to integrate things in it, to add feature...

On Tue, Sep 27, 2011 at 9:07 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Tue, Sep 27, 2011 at 3:00 PM, Wichert Akkerman <wichert at wiggy.net> wrote:
>> On 09/27/2011 02:51 PM, Tarek Ziad? wrote:
>>>
>>> On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman<wichert at wiggy.net>
>>> ?wrote:
>>> ..
>>>>>
>>>>> I understand where you're coming from but, ..
>>>>> I think it's saner to rely on proven technology
>>>>> than to invent our own protocol. NIH?
>>>>
>>>> This also feels like a problem that has already been solved in various
>>>> ways
>>>> by Debian, RedHat, CPAN and others.
>>>
>>> Yes, and we've found a way similar to CPAN, with some Python specifics
>>> (PyPI download statistics mainly)
>>>
>>> Oh my, we're cycling again.
>>>
>>> Nothing personal to you or Jim, but I have a sudden fatigue on
>>> packaging because it seems like people are ignoring what's being done
>>> to complain afterwards about us suffering of some kind of NIH :)
>>
>> It's just that my perspective is that of a simple user. And from my
>> perspective nothing has changed in the last couple of years. Pypi still goes
>> down occasionally, and when that happens many things start breaking. It may
>> very well be that there are things planned or in progress, but until they
>> are both usable and used by standard tools, which for me means buildout and
>> setuptools, they are invisible.
>
> Fair enough,
>
> Pip has now the mirroring protocol implemented. I think they want to
> make it a default option for the next major Pip release. IOW, you
> should not suffer for downtimes using pip.
>
> We'd need to add the same feature in easy_install and zc.buildout.
> But since Pip did it, I think it's possible.
>
> We have 5 mirrors run by the community
> (http://pypi.python.org/mirrors) and I suspect porting pip's feature
> to zc.buildout and easy_install would take less time than creating a
> <Name your cloud service> app.
>
>
> Cheers
> Tarek
>
>>
>> Wichert.
>>
>
>
>
> --
> Tarek Ziad? | http://ziade.org
> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Mathieu Leduc-Hamel

From aclark at aclark.net  Tue Sep 27 15:59:50 2011
From: aclark at aclark.net (Alex Clark)
Date: Tue, 27 Sep 2011 09:59:50 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q49-4uc9spgSUkEQSVJbv1+4VBjCUXcGE3jix2RNYCHng@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<4E81C41D.60703@wiggy.net>
	<CAGSi+Q49-4uc9spgSUkEQSVJbv1+4VBjCUXcGE3jix2RNYCHng@mail.gmail.com>
Message-ID: <j5skso$qa6$1@dough.gmane.org>

On 9/27/11 8:51 AM, Tarek Ziad? wrote:
> On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman<wichert at wiggy.net>  wrote:
> ..
>>>
>>> I understand where you're coming from but, ..
>>> I think it's saner to rely on proven technology
>>> than to invent our own protocol. NIH?
>>
>> This also feels like a problem that has already been solved in various ways
>> by Debian, RedHat, CPAN and others.
>
> Yes, and we've found a way similar to CPAN, with some Python specifics
> (PyPI download statistics mainly)
>
> Oh my, we're cycling again.
>
> Nothing personal to you or Jim, but I have a sudden fatigue on
> packaging because it seems like people are ignoring what's being done
> to complain afterwards about us suffering of some kind of NIH :)
>
> If you're seeing anything you don't like in PEP  381 (accepted a while
> ago), go ahead and propose some improvements.
>
> But please keep in mind that we've looked at other systems before we
> wrote that PEP.


+1 If someone (Tarek in this case) wants to take the lead on this then I 
don't care where the fix was invented ;-)

Obviously the less we make volunteers do, and the more we can outsource 
the better (I say this in Plone-land repeatedly) but sometimes 
inventing-something-here is the right thing to do. And it certainly 
would not preclude other (cloud) fixes AFAICT.



Alex






>
> Cheers
> Tarek
>


-- 
Alex Clark ? http://aclark.net


From aclark at aclark.net  Tue Sep 27 16:05:47 2011
From: aclark at aclark.net (Alex Clark)
Date: Tue, 27 Sep 2011 10:05:47 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q5S5M6r-R1jNQydL7WpvMY=UTBrCjbc5zWjKAxzO6RgDw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<4E81C41D.60703@wiggy.net>
	<CAGSi+Q49-4uc9spgSUkEQSVJbv1+4VBjCUXcGE3jix2RNYCHng@mail.gmail.com>
	<4E81C909.5060100@wiggy.net>
	<CAGSi+Q5S5M6r-R1jNQydL7WpvMY=UTBrCjbc5zWjKAxzO6RgDw@mail.gmail.com>
Message-ID: <j5sl7t$tbl$1@dough.gmane.org>

On 9/27/11 9:07 AM, Tarek Ziad? wrote:
> On Tue, Sep 27, 2011 at 3:00 PM, Wichert Akkerman<wichert at wiggy.net>  wrote:
>> On 09/27/2011 02:51 PM, Tarek Ziad? wrote:
>>>
>>> On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman<wichert at wiggy.net>
>>>   wrote:
>>> ..
>>>>>
>>>>> I understand where you're coming from but, ..
>>>>> I think it's saner to rely on proven technology
>>>>> than to invent our own protocol. NIH?
>>>>
>>>> This also feels like a problem that has already been solved in various
>>>> ways
>>>> by Debian, RedHat, CPAN and others.
>>>
>>> Yes, and we've found a way similar to CPAN, with some Python specifics
>>> (PyPI download statistics mainly)
>>>
>>> Oh my, we're cycling again.
>>>
>>> Nothing personal to you or Jim, but I have a sudden fatigue on
>>> packaging because it seems like people are ignoring what's being done
>>> to complain afterwards about us suffering of some kind of NIH :)
>>
>> It's just that my perspective is that of a simple user. And from my
>> perspective nothing has changed in the last couple of years. Pypi still goes
>> down occasionally, and when that happens many things start breaking. It may
>> very well be that there are things planned or in progress, but until they
>> are both usable and used by standard tools, which for me means buildout and
>> setuptools, they are invisible.
>
> Fair enough,
>
> Pip has now the mirroring protocol implemented. I think they want to
> make it a default option for the next major Pip release. IOW, you
> should not suffer for downtimes using pip.
>
> We'd need to add the same feature in easy_installand zc.buildout.
> But since Pip did it, I think it's possible.
>
> We have 5 mirrors run by the community
> (http://pypi.python.org/mirrors) and I suspect porting pip's feature
> to zc.buildout and easy_install would take less time than creating a
> <Name your cloud service>  app.


I'm not sure I fully understand why each-tool needs updating (vs. 
something that would provide HA but be invisible to the tools), but I 
suspect a lot of this is "legacy" related (i.e. PyPI was not originally 
designed to handle HA) and adding support to each tool, though 
potentially tedious-sounding (if not actually tedious) may be reasonable.



Alex





>
>
> Cheers
> Tarek
>
>>
>> Wichert.
>>
>
>
>


-- 
Alex Clark ? http://aclark.net


From jim at zope.com  Tue Sep 27 17:35:49 2011
From: jim at zope.com (Jim Fulton)
Date: Tue, 27 Sep 2011 11:35:49 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q5Jqj5HQG2S=1Abc2Rrb_rpFaWgfNNZGfbp=Pd_L=ZFCg@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<CAGSi+Q5Jqj5HQG2S=1Abc2Rrb_rpFaWgfNNZGfbp=Pd_L=ZFCg@mail.gmail.com>
Message-ID: <CAPDm-FhcsGTQutR4ov+83UWm14s+tfG8bpm2QnoskPk_FPWOKA@mail.gmail.com>

On Tue, Sep 27, 2011 at 8:40 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Tue, Sep 27, 2011 at 2:27 PM, Jim Fulton <jim at zope.com> wrote:
> ...
>>
>> I understand where you're coming from but, ..
>
> Sorry, I don't understand what you imply here.

I understand why you don't want to rely on a proprietary solution.

>> I think it's saner to rely on proven technology
>> than to invent our own protocol. NIH?
>
> Ah sorry I misunderstood then. I thought CloudFront was a proprietary
> platform, with its own protocol.

It's a reverse proxy.  You point it at s3 and at a web server and it caches.
Of course, it has aspects that are specific to it's implementation.

> If you're saying that we can move away from CloudFront at any time and
> have the same feature elsewhere, then it's perfect.

If we move to something else, *some* changes will be necessary,
but we can certainly move. I agree, it's perfect. ;)

> If you're saying that CloudFront is proven technology and that we
> should not worry about relying on them, then I think we can do better
> for the community to get locked-in for this, and continue to work on
> an open protocol where everyone can participate by providing a spare
> server. ?But maybe that's just me ?

It's nice to have a hobby. :)

But I don't want to have to update buildout *just* because of an itch
to have a custom protocol.

>
> Most of the mirroring protocol was inspired by Perl's CPAN btw.

<shrug>

> But the use case is usually: PyPI is down, we fallback to a mirror. I
> don't think it's more complicated than this.

I don't agree.  On multiple levels.  PYPI is often up but slow.
It's also in the wrong place.  A CDN should provide better performance,
reliability and locality.

A client has to:

- try pypi
- fallback to "last"
- If that's down, decide what other indexes to check

I don't see how having timestamps help unless you know
what the current timestamp is, unless you say that you'll reject
a mirror with a timestamp more than some period in the past.
It's not clear what this time delta should be and, in any case,
the client needs to first validate a mirror by checking it's timestamp.

I think this protocol is going to be hard to get right.

>>
>> - It either requires extra dns calls or relies to heavily on the last
>> mirror, which is probably likely
>> ?to be the least reliable.
>
> Once you have the list, I don't think you require extra call.
>
> see http://hg.python.org/cpython/file/84280fac98b9/Lib/packaging/pypi/mirrors.py

It has to make extra dns calls to resolve the other mirror names to ips.


>> Life is short. We don't have to invent this ourselves.
>
> Ah well, yeah -- Not sure what you are proposing right now.
>
> If you imply that everything should be solved on server-side, and that
> we should not have mirroring

I think we should pick a good CDN and use it.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From ziade.tarek at gmail.com  Tue Sep 27 19:25:11 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 19:25:11 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-FhcsGTQutR4ov+83UWm14s+tfG8bpm2QnoskPk_FPWOKA@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<CAGSi+Q5Jqj5HQG2S=1Abc2Rrb_rpFaWgfNNZGfbp=Pd_L=ZFCg@mail.gmail.com>
	<CAPDm-FhcsGTQutR4ov+83UWm14s+tfG8bpm2QnoskPk_FPWOKA@mail.gmail.com>
Message-ID: <CAGSi+Q5CE=9TJA_NKVfhQg4dMannJPpO7+Eg54DVahfGNk2O+w@mail.gmail.com>

On Tue, Sep 27, 2011 at 5:35 PM, Jim Fulton <jim at zope.com> wrote:

>>> I understand where you're coming from but, ..

>> Sorry, I don't understand what you imply here.

> I understand why you don't want to rely on a proprietary solution.

But it's true that I don't want to rely on a proprietary solution.
That's based on a good reason I think, mentioned at the end of this
mail.

...
>
>> If you're saying that CloudFront is proven technology and that we
>> should not worry about relying on them, then I think we can do better
>> for the community to get locked-in for this, and continue to work on
>> an open protocol where everyone can participate by providing a spare
>> server. ?But maybe that's just me ?
>
> It's nice to have a hobby. :)

I think you've missed what we, bunch of hobbyists, did in the past two years

+ 5 community mirrors are up and running, collecting download stats
that get merged
+ pip does work with the mirrors, and offer fallback options

It's too bad you were not there to tell us we were wasting our time
and how awesome CloudFront was ;)

But at this point, the shortest road to a better PyPI is to add the
mirroring support to other clients, pip showed the lead. And if
zc.buildout uses Distribute, it should get this feature at some point.

But having a CloudFront-based PyPI could also be interesting in
parallel, I am not saying it's not. But the project is stalled, and
has the defaults I've mentioned.

> But I don't want to have to update buildout *just* because of an itch
> to have a custom protocol.

I kind of wonder how hard it would be to have a standalone pypi
download client, ripped off from python 3.3's packaging, so you would
not have to worry about this.

And, well, you do not sound like you want to spend time in these
matters in any case, so if someone brings a patch I hope you will not
refuse it.

>> But the use case is usually: PyPI is down, we fallback to a mirror. I
>> don't think it's more complicated than this.
>
> I don't agree. ?On multiple levels. ?PYPI is often up but slow.

That's an orthogonal issue :  any server can be slow.

One better way to drastically speed up buildout is to  download /
build stuff in parallel imo.


> It's also in the wrong place. ?A CDN should provide better performance,
> reliability and locality.

Locality is indeed important, and picking up the nearest server is great.
Reliability is also solved by the mirrors.

>
> A client has to:
>
> - try pypi
> - fallback to "last"
> - If that's down, decide what other indexes to check
>
> I don't see how having timestamps help unless you know
> what the current timestamp is, unless you say that you'll reject
> a mirror with a timestamp more than some period in the past.

How hard it is to make those decisions ?

Do you really think getting the current timestamp is that hard ?

And the mirror timestamp,

  http://b.pypi.python.org/last-modified

In all you've said I fail to see how complicated it is, or long to do.

The ordering I see is:

normal behavior:
- if the cache is too old: get the list of mirrors  (-> the list of
mirrors and their timestamps get cached)
- pick the closest one
- use it

the server times out:
- try the "next closest"


> It's not clear what this time delta should be and, in any case,
> the client needs to first validate a mirror by checking it's timestamp.

This is the job of the client yes. An option that says, discard
mirrors that are > 1 day, or 5 hours etc.

Keeping a local cache that gets updated eventually is sufficient.

> I think this protocol is going to be hard to get right.

Maybe ? but if a v1 allows us to switch from server 1 being down to
server 2, it's already a success, no ?

servers that *we* the community, manage.



>>>
>>> - It either requires extra dns calls or relies to heavily on the last
>>> mirror, which is probably likely
>>> ?to be the least reliable.
>>
>> Once you have the list, I don't think you require extra call.
>>
>> see http://hg.python.org/cpython/file/84280fac98b9/Lib/packaging/pypi/mirrors.py
>
> It has to make extra dns calls to resolve the other mirror names to ips.

Yeah, once per session. but in any case, this is not a decision you're
making on every download. It's something you do when you start to
download stuff, and/or when a server times out.

You stick with a server once it's working

>
>
>>> Life is short. We don't have to invent this ourselves.
>>
>> Ah well, yeah -- Not sure what you are proposing right now.
>>
>> If you imply that everything should be solved on server-side, and that
>> we should not have mirroring
>
> I think we should pick a good CDN and use it.

I won't object, because this is orthogonal to the mirroring stuff, but
I am not going to scratch the mirroring efforts to move PyPI to a
single shop.

Every service on the planet, even Amazon, can be down.

oh, my:

- https://forums.aws.amazon.com/message.jspa?messageID=244986
- http://money.cnn.com/2011/04/22/technology/amazon_ec2_cloud_outage/index.htm.
- http://www.labnol.org/internet/amazon-s3-cloudfront-down/5667/
- https://forums.aws.amazon.com/message.jspa?messageID=134012
- https://forums.aws.amazon.com/message.jspa?messageID=177654

Do we really want Amazon to handle PyPI ?

I prefer a bunch of community mirrors. Heck, I have one at Mozilla,
and might make it public one day  :)

Or maybe the optimal solution is our own CND proxy so we don't deal
with this on client side.

<music in the background with trumpets, a flag with the Python logo
raises, slowly>

But in any case, I'd rather have a Pythoneer from our community behind
every mirror server, I can trust

</music>


Cheers
Tarek

-- 
Tarek Ziad? | http://ziade.org

From noah at coderanger.net  Tue Sep 27 19:24:02 2011
From: noah at coderanger.net (Noah Kantrowitz)
Date: Tue, 27 Sep 2011 10:24:02 -0700
Subject: [Distutils] [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
Message-ID: <DDF6FA6D-2EB1-4957-B871-590DAA750AFA@coderanger.net>

I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final.

On Sep 27, 2011, at 2:40 AM, Tarek Ziad? wrote:
> 
> == better infra ==
> 


The current plan is to have two primary load balancer VMs running Nginx acting as both balancer and SSL termination points. These will share a set of floating IPs using Heartbeat. Behind this will be the same Apache configuration currently in use (Apache serving static files and PyPI running as an FCGI script controlled by Apache) running on two VMs, both talking to the same master-slave Postgres 9 replication setup. Package files will initially be handled by a shared DRBD drive, however this may be obsoleted by the project to move file hosting to Cloudfront or another CDN.

A currently open question is how best to provide reliability and security for the SSH-based file upload system currently deployed on ximenez. Most likely we can setup the initial SSH endpoint on the load balancers to run a proxy to one of the main PyPI application servers, however failover would have to be semi-manual (possibly driven by Chef, meaning a chef-client run would have to happen before the tunnel would be updated, or anywhere up to 30 minutes). Given the relatively minimal public knowledge of this service, I think this is acceptable as a first-pass but a future solution involving HAProxy or another TCP load balancer to handle the SSH traffic might be appropriate. Similarly, automated database failover is not planned at this time due to the extra application-side complexity however the process will be well documented and able to be executed by all tier one on-call operations staff if the Postgres master goes offline for some reason.

This will only partially address the current reliability issues as many of the current problems are linked to Apache or mod_fcgid needing to be restarted. In that light I would like to see PyPI and the catalog-sig group investigate moving the codebase to work against mod_wsgi or gunicorn (no real preference between the two) to create a more reliable runtime environment.

--Noah
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20110927/f03dcb94/attachment.pgp>

From ziade.tarek at gmail.com  Tue Sep 27 19:33:58 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 19:33:58 +0200
Subject: [Distutils] [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <DDF6FA6D-2EB1-4957-B871-590DAA750AFA@coderanger.net>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<DDF6FA6D-2EB1-4957-B871-590DAA750AFA@coderanger.net>
Message-ID: <CAGSi+Q7YAfcoYRRsp=FUjPGnb1iazRv41JUO74kEHkm88qaaxw@mail.gmail.com>

On Tue, Sep 27, 2011 at 7:24 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
> I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final.

Thanks,

I've started this page with all the content :
http://wiki.python.org/moin/BetterPyPI





-- 
Tarek Ziad? | http://ziade.org

From noah at coderanger.net  Tue Sep 27 19:34:55 2011
From: noah at coderanger.net (Noah Kantrowitz)
Date: Tue, 27 Sep 2011 10:34:55 -0700
Subject: [Distutils] [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7YAfcoYRRsp=FUjPGnb1iazRv41JUO74kEHkm88qaaxw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<DDF6FA6D-2EB1-4957-B871-590DAA750AFA@coderanger.net>
	<CAGSi+Q7YAfcoYRRsp=FUjPGnb1iazRv41JUO74kEHkm88qaaxw@mail.gmail.com>
Message-ID: <5F826E99-A368-47E2-A26D-B691F8A05649@coderanger.net>


On Sep 27, 2011, at 10:33 AM, Tarek Ziad? wrote:

> On Tue, Sep 27, 2011 at 7:24 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
>> I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final.
> 
> Thanks,
> 
> I've started this page with all the content :
> http://wiki.python.org/moin/BetterPyPI

If people have feedback please also make sure to let me know so I we can discuss it.

--Noah
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20110927/18b10466/attachment.pgp>

From aclark at aclark.net  Tue Sep 27 19:59:40 2011
From: aclark at aclark.net (Alex Clark)
Date: Tue, 27 Sep 2011 13:59:40 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q5CE=9TJA_NKVfhQg4dMannJPpO7+Eg54DVahfGNk2O+w@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<CAGSi+Q5Jqj5HQG2S=1Abc2Rrb_rpFaWgfNNZGfbp=Pd_L=ZFCg@mail.gmail.com>
	<CAPDm-FhcsGTQutR4ov+83UWm14s+tfG8bpm2QnoskPk_FPWOKA@mail.gmail.com>
	<CAGSi+Q5CE=9TJA_NKVfhQg4dMannJPpO7+Eg54DVahfGNk2O+w@mail.gmail.com>
Message-ID: <j5t2ue$7cq$1@dough.gmane.org>

On 9/27/11 1:25 PM, Tarek Ziad? wrote:
> On Tue, Sep 27, 2011 at 5:35 PM, Jim Fulton<jim at zope.com>  wrote:
>
>>>> I understand where you're coming from but, ..
>
>>> Sorry, I don't understand what you imply here.
>
>> I understand why you don't want to rely on a proprietary solution.
>
> But it's true that I don't want to rely on a proprietary solution.
> That's based on a good reason I think, mentioned at the end of this
> mail.
>
> ...
>>
>>> If you're saying that CloudFront is proven technology and that we
>>> should not worry about relying on them, then I think we can do better
>>> for the community to get locked-in for this, and continue to work on
>>> an open protocol where everyone can participate by providing a spare
>>> server.  But maybe that's just me ?
>>
>> It's nice to have a hobby. :)
>
> I think you've missed what we, bunch of hobbyists, did in the past two years
>
> + 5 community mirrors are up and running, collecting download stats
> that get merged
> + pip does work with the mirrors, and offer fallback options
>
> It's too bad you were not there to tell us we were wasting our time
> and how awesome CloudFront was ;)
>
> But at this point, the shortest road to a better PyPI is to add the
> mirroring support to other clients, pip showed the lead. And if
> zc.buildout uses Distribute, it should get this feature at some point.
>
> But having a CloudFront-based PyPI could also be interesting in
> parallel, I am not saying it's not. But the project is stalled, and
> has the defaults I've mentioned.
>
>> But I don't want to have to update buildout *just* because of an itch
>> to have a custom protocol.
>
> I kind of wonder how hard it would be to have a standalone pypi
> download client, ripped off from python 3.3's packaging, so you would
> not have to worry about this.
>
> And, well, you do not sound like you want to spend time in these
> matters in any case, so if someone brings a patch I hope you will not
> refuse it.
>
>>> But the use case is usually: PyPI is down, we fallback to a mirror. I
>>> don't think it's more complicated than this.
>>
>> I don't agree.  On multiple levels.  PYPI is often up but slow.
>
> That's an orthogonal issue :  any server can be slow.
>
> One better way to drastically speed up buildout is to  download /
> build stuff in parallel imo.
>
>
>> It's also in the wrong place.  A CDN should provide better performance,
>> reliability and locality.
>
> Locality is indeed important, and picking up the nearest server is great.
> Reliability is also solved by the mirrors.
>
>>
>> A client has to:
>>
>> - try pypi
>> - fallback to "last"
>> - If that's down, decide what other indexes to check
>>
>> I don't see how having timestamps help unless you know
>> what the current timestamp is, unless you say that you'll reject
>> a mirror with a timestamp more than some period in the past.
>
> How hard it is to make those decisions ?
>
> Do you really think getting the current timestamp is that hard ?
>
> And the mirror timestamp,
>
>    http://b.pypi.python.org/last-modified
>
> In all you've said I fail to see how complicated it is, or long to do.
>
> The ordering I see is:
>
> normal behavior:
> - if the cache is too old: get the list of mirrors  (->  the list of
> mirrors and their timestamps get cached)
> - pick the closest one
> - use it
>
> the server times out:
> - try the "next closest"
>
>
>> It's not clear what this time delta should be and, in any case,
>> the client needs to first validate a mirror by checking it's timestamp.
>
> This is the job of the client yes. An option that says, discard
> mirrors that are>  1 day, or 5 hours etc.
>
> Keeping a local cache that gets updated eventually is sufficient.
>
>> I think this protocol is going to be hard to get right.
>
> Maybe ? but if a v1 allows us to switch from server 1 being down to
> server 2, it's already a success, no ?
>
> servers that *we* the community, manage.
>
>
>
>>>>
>>>> - It either requires extra dns calls or relies to heavily on the last
>>>> mirror, which is probably likely
>>>>   to be the least reliable.
>>>
>>> Once you have the list, I don't think you require extra call.
>>>
>>> see http://hg.python.org/cpython/file/84280fac98b9/Lib/packaging/pypi/mirrors.py
>>
>> It has to make extra dns calls to resolve the other mirror names to ips.
>
> Yeah, once per session. but in any case, this is not a decision you're
> making on every download. It's something you do when you start to
> download stuff, and/or when a server times out.
>
> You stick with a server once it's working
>
>>
>>
>>>> Life is short. We don't have to invent this ourselves.
>>>
>>> Ah well, yeah -- Not sure what you are proposing right now.
>>>
>>> If you imply that everything should be solved on server-side, and that
>>> we should not have mirroring
>>
>> I think we should pick a good CDN and use it.
>
> I won't object, because this is orthogonal to the mirroring stuff, but
> I am not going to scratch the mirroring efforts to move PyPI to a
> single shop.
>
> Every service on the planet, even Amazon, can be down.
>
> oh, my:
>
> - https://forums.aws.amazon.com/message.jspa?messageID=244986
> - http://money.cnn.com/2011/04/22/technology/amazon_ec2_cloud_outage/index.htm.
> - http://www.labnol.org/internet/amazon-s3-cloudfront-down/5667/
> - https://forums.aws.amazon.com/message.jspa?messageID=134012
> - https://forums.aws.amazon.com/message.jspa?messageID=177654
>
> Do we really want Amazon to handle PyPI ?
>
> I prefer a bunch of community mirrors. Heck, I have one at Mozilla,
> and might make it public one day  :)
>
> Or maybe the optimal solution is our own CND proxy so we don't deal
> with this on client side.
>
> <music in the background with trumpets, a flag with the Python logo
> raises, slowly>
>
> But in any case, I'd rather have a Pythoneer from our community behind
> every mirror server, I can trust
>
> </music>


Assuming Jim doesn't mind someone else adding support to zc.buildout for 
this, I wouldn't mind doing the work. I think stranger things have 
happened than we implement some feature not everyone agrees on, as long 
as it is not going to affect any other aspect of the software (which it 
wouldn't, IIUC).



Alex





>
>
> Cheers
> Tarek
>


-- 
Alex Clark ? http://aclark.net


From martin at v.loewis.de  Tue Sep 27 20:10:54 2011
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 27 Sep 2011 20:10:54 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
Message-ID: <4E8211AE.3040205@v.loewis.de>

Am 27.09.2011 13:12, schrieb Jim Fulton:
> On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro <regebro at gmail.com> wrote:
>> On Tue, Sep 27, 2011 at 11:40, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>>> 1/ stability and high availability
>>
>> How are opinions on setting up country-specific PyPI mirrors? The lag
>> to the US is pretty severe in Poland, and I suspect my buildouts would
>> benefit from having a server in Poland. Now, of course, it could be
>> called x.pypi.python.org, but maybe we should have aliases such as
>> pl.pypi.python.org as well?
>>
>> I have no strong opinion on the issue, what do others think?
> 
> Wouldn't CloudFront make this moot?

I personally don't believe the CloudFront project is feasible - IMO,
it just won't work. This is because there needs to be both dynamic
content (at least for uploads) and static content; CloudFront can
only mirror the static content. However, since this all forms a single
web site, integrating them will be either infeasible or pointless.

It would be possible to use CloudFront if the clients would be changed.
But if so, it would be best if the CloudFront copy would just be a
PEP 381 mirror.

Regards,
Martin

From martin at v.loewis.de  Tue Sep 27 20:15:07 2011
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 27 Sep 2011 20:15:07 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-FhcsGTQutR4ov+83UWm14s+tfG8bpm2QnoskPk_FPWOKA@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>	<CAGSi+Q5Jqj5HQG2S=1Abc2Rrb_rpFaWgfNNZGfbp=Pd_L=ZFCg@mail.gmail.com>
	<CAPDm-FhcsGTQutR4ov+83UWm14s+tfG8bpm2QnoskPk_FPWOKA@mail.gmail.com>
Message-ID: <4E8212AB.5080303@v.loewis.de>

> I understand why you don't want to rely on a proprietary solution.

It's not a proprietary solution. It uses standard technologies, such
as HTTP and XML-RPC. If you say that the specific URLs to access are
proprietary: true, but so would be a CloudFront mirror (which also
requires proprietary API to get the data into CloudFront).

> It's a reverse proxy.  You point it at s3 and at a web server and it caches.
> Of course, it has aspects that are specific to it's implementation.

That's not my understanding as to what the project to get PyPI onto
CloudFront is doing (IIUC).

Regards,
Martin

From jim at zope.com  Tue Sep 27 20:31:49 2011
From: jim at zope.com (Jim Fulton)
Date: Tue, 27 Sep 2011 14:31:49 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q5CE=9TJA_NKVfhQg4dMannJPpO7+Eg54DVahfGNk2O+w@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<CAGSi+Q5Jqj5HQG2S=1Abc2Rrb_rpFaWgfNNZGfbp=Pd_L=ZFCg@mail.gmail.com>
	<CAPDm-FhcsGTQutR4ov+83UWm14s+tfG8bpm2QnoskPk_FPWOKA@mail.gmail.com>
	<CAGSi+Q5CE=9TJA_NKVfhQg4dMannJPpO7+Eg54DVahfGNk2O+w@mail.gmail.com>
Message-ID: <CAPDm-FghqjuebfZU2MkrS_AqPDXQf1mCUat2F038OXo9S7Zukg@mail.gmail.com>

On Tue, Sep 27, 2011 at 1:25 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Tue, Sep 27, 2011 at 5:35 PM, Jim Fulton <jim at zope.com> wrote:

...

>> But I don't want to have to update buildout *just* because of an itch
>> to have a custom protocol.
>
> I kind of wonder how hard it would be to have a standalone pypi
> download client, ripped off from python 3.3's packaging, so you would
> not have to worry about this.

I doubt I'm going to be able to avoid worrying about it.

Still a reference client implementation would be useful.

> And, well, you do not sound like you want to spend time in these
> matters in any case,

I don't know what you mean.  Not sure I care. :)

> so if someone brings a patch I hope you will not
> refuse it.

No.  I'll eventually implement it if no one else does.

>>> But the use case is usually: PyPI is down, we fallback to a mirror. I
>>> don't think it's more complicated than this.
>>
>> I don't agree. ?On multiple levels. ?PYPI is often up but slow.
>
> That's an orthogonal issue : ?any server can be slow.

A service can be fast even if an individual server is slow.
Also, CDNs can make lots of horsepower available that
is shared among multiple customers.  I really doubt that
anything we build will be faster.

> One better way to drastically speed up buildout is to ?download /
> build stuff in parallel imo.

Thats true and something I'd like to do at some point. That's one of
the reasons I expect I'll have to worry about the protocol.

>
>> It's also in the wrong place. ?A CDN should provide better performance,
>> reliability and locality.
>
> Locality is indeed important, and picking up the nearest server is great.
> Reliability is also solved by the mirrors.

At the expense of increased complexity on the client.

>>
>> A client has to:
>>
>> - try pypi
>> - fallback to "last"
>> - If that's down, decide what other indexes to check
>>
>> I don't see how having timestamps help unless you know
>> what the current timestamp is, unless you say that you'll reject
>> a mirror with a timestamp more than some period in the past.
>
> How hard it is to make those decisions ?

It's not "hard" conceptually, but it's still a lot of
implementation complexity and a lot of extra network
requests.

> Do you really think getting the current timestamp is that hard ?
>
> And the mirror timestamp,
>
> ?http://b.pypi.python.org/last-modified
>
> In all you've said I fail to see how complicated it is, or long to do.

That's an extra HTTP request I need to make when I'm
considering use of a mirror.  If the first mirror I check seems to
be out of date, I may need to check all the mirrors.  It's an open
question what should be considered potentially out of date, a
timestamp older than an hour? a day?

> The ordering I see is:
>
> normal behavior:
> - if the cache is too old:

How old is too old?

> get the list of mirrors ?(-> the list of
> mirrors and their timestamps get cached)

They'll only get cached for the program invocation.
This means I have to potentially check lots of mirrors
every time someone runs buildout.  I can reduce latency
by doing this in parallel, but that's still a lot of requests.

> - pick the closest one

How do I decide what's closest? Did you mean closest?
or most up to date

> - use it
>
> the server times out:
> - try the "next closest"
>
>
>> It's not clear what this time delta should be and, in any case,
>> the client needs to first validate a mirror by checking it's timestamp.
>
> This is the job of the client yes. An option that says, discard
> mirrors that are > 1 day, or 5 hours etc.

"etc" is just waving hands.  Selecting the right value is hard, possibly
application dependent. Is this a configuration variable?  Now the
user has something to deal with.

> Keeping a local cache that gets updated eventually is sufficient.

In process, or on disk?  This just gets better and better. :)

>> I think this protocol is going to be hard to get right.
>
> Maybe ? but if a v1 allows us to switch from server 1 being down to
> server 2, it's already a success, no ?
>
> servers that *we* the community, manage.

I fail to see why this is inherently a good thing.  I don't like
"managing" things.  Less work is good.

...

> Do we really want Amazon to handle PyPI ?

Yes, or Rackspace, or Google, or AOL, or, whatever.  Just not us.

(I suspect some of these might even do it for free.)

> I prefer a bunch of community mirrors. Heck, I have one at Mozilla,
> and might make it public one day ?:)
>
> Or maybe the optimal solution is our own CND proxy so we don't deal
> with this on client side.
>
> <music in the background with trumpets, a flag with the Python logo
> raises, slowly>

Uh, yeah, sure.

FWIW, it hadn't occurred to me to use a CDN until a conversation a few
days ago. Doh.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From greg.ewing at canterbury.ac.nz  Tue Sep 27 22:59:49 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 28 Sep 2011 09:59:49 +1300
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
Message-ID: <4E823945.1070703@canterbury.ac.nz>

Jim Fulton wrote:

> Life is short. We don't have to invent this ourselves.

Are there really no existing open-source mirroring
systems out there?

This seems like a common enough thing that it must
have been solved already.

-- 
Greg

From barry at python.org  Tue Sep 27 23:05:19 2011
From: barry at python.org (Barry Warsaw)
Date: Tue, 27 Sep 2011 17:05:19 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <4E823945.1070703@canterbury.ac.nz>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>
	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>
	<4E823945.1070703@canterbury.ac.nz>
Message-ID: <20110927170519.09a2be0d@resist.wooz.org>

On Sep 28, 2011, at 09:59 AM, Greg Ewing wrote:

>Are there really no existing open-source mirroring
>systems out there?

rsync? :)

-Barry

From jim at zope.com  Tue Sep 27 23:23:17 2011
From: jim at zope.com (Jim Fulton)
Date: Tue, 27 Sep 2011 17:23:17 -0400
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <4E8211AE.3040205@v.loewis.de>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<4E8211AE.3040205@v.loewis.de>
Message-ID: <CAPDm-FinBu-+=im17fJ_MRXO7upkR1oFA3Mq9ELjmmUiZjB39g@mail.gmail.com>

On Tue, Sep 27, 2011 at 2:10 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Am 27.09.2011 13:12, schrieb Jim Fulton:
>> On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro <regebro at gmail.com> wrote:
>>> On Tue, Sep 27, 2011 at 11:40, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>>>> 1/ stability and high availability
>>>
>>> How are opinions on setting up country-specific PyPI mirrors? The lag
>>> to the US is pretty severe in Poland, and I suspect my buildouts would
>>> benefit from having a server in Poland. Now, of course, it could be
>>> called x.pypi.python.org, but maybe we should have aliases such as
>>> pl.pypi.python.org as well?
>>>
>>> I have no strong opinion on the issue, what do others think?
>>
>> Wouldn't CloudFront make this moot?
>
> I personally don't believe the CloudFront project is feasible - IMO,
> it just won't work. This is because there needs to be both dynamic
> content (at least for uploads) and static content; CloudFront can
> only mirror the static content.

That's the intent. To serve the static content.

> However, since this all forms a single
> web site, integrating them will be either infeasible or pointless.

There are 2 sites, "simple" and "pypi".  I would only host the simple
site in a CDN. I assume the mirrors are only mirroring the simple
site. pypi would, I assume, either stay where it is now, or be evolved
separately.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From martin at v.loewis.de  Tue Sep 27 23:25:03 2011
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 27 Sep 2011 23:25:03 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-FghqjuebfZU2MkrS_AqPDXQf1mCUat2F038OXo9S7Zukg@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>	<CAGSi+Q7gSWi8jr_5hzSME9DMRmiHHan=wBkSRnsd=hGXYyVtPA@mail.gmail.com>	<CAPDm-Fh_9DnHg9m=6oHv0R4bVXkPh2KT96s0UUDzZ1b9Vyzo8Q@mail.gmail.com>	<CAGSi+Q5Jqj5HQG2S=1Abc2Rrb_rpFaWgfNNZGfbp=Pd_L=ZFCg@mail.gmail.com>	<CAPDm-FhcsGTQutR4ov+83UWm14s+tfG8bpm2QnoskPk_FPWOKA@mail.gmail.com>	<CAGSi+Q5CE=9TJA_NKVfhQg4dMannJPpO7+Eg54DVahfGNk2O+w@mail.gmail.com>
	<CAPDm-FghqjuebfZU2MkrS_AqPDXQf1mCUat2F038OXo9S7Zukg@mail.gmail.com>
Message-ID: <4E823F2F.4010808@v.loewis.de>

> That's an extra HTTP request I need to make when I'm
> considering use of a mirror.  If the first mirror I check seems to
> be out of date, I may need to check all the mirrors.  It's an open
> question what should be considered potentially out of date, a
> timestamp older than an hour? a day?

15 minutes; if all mirrors are older than 15 minutes, it probably
means that the master is down, and you should then use the newest
mirror you can find.

> How old is too old?

More than 15 minutes.

>> get the list of mirrors  (-> the list of
>> mirrors and their timestamps get cached)
> 
> They'll only get cached for the program invocation.

No, you can also cache the fastest mirror across invocations,
and keep using it if it is younger than 15 minutes.

>> - pick the closest one
> 
> How do I decide what's closest? Did you mean closest?

He probably means the one that responds fastest.

> "etc" is just waving hands.  Selecting the right value is hard, possibly
> application dependent. Is this a configuration variable?  Now the
> user has something to deal with.

No, it will be 15 minutes. People might request being able to configure
it, but it won't really be necessary to do so.

Regards,
Martin

From martin at v.loewis.de  Tue Sep 27 23:27:19 2011
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 27 Sep 2011 23:27:19 +0200
Subject: [Distutils] Fwd: The state of PyPI
In-Reply-To: <CAPDm-FinBu-+=im17fJ_MRXO7upkR1oFA3Mq9ELjmmUiZjB39g@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<CAL0kPAUR5EmnHEcMkBZr_BB+ZezrLkxtecg4etr=24nrd5P1qw@mail.gmail.com>
	<CAPDm-FhGJH8UYErR0VxEq2oHuej_C9vM5xDQjND8czQN-7OZPQ@mail.gmail.com>
	<4E8211AE.3040205@v.loewis.de>
	<CAPDm-FinBu-+=im17fJ_MRXO7upkR1oFA3Mq9ELjmmUiZjB39g@mail.gmail.com>
Message-ID: <4E823FB7.3030500@v.loewis.de>

>> However, since this all forms a single
>> web site, integrating them will be either infeasible or pointless.
> 
> There are 2 sites, "simple" and "pypi".

No, it's one site: http://pypi.python.org/pypi, and
http://pypi.python.org/simple (plus there are other URLs, like
/packages).

Regards,
Martin

From chris at simplistix.co.uk  Wed Sep 28 21:09:47 2011
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 28 Sep 2011 20:09:47 +0100
Subject: [Distutils] buildout-versions 1.7 released!
Message-ID: <4E8370FB.1040203@simplistix.co.uk>

Hi All,

I'm pleased to announce a new release of buildout-versions.

This release allows you to specify the version of python that a buildout 
should be run with, for example:

[buildout]
extensions = buildout-versions
versions = versions
parts =

[versions]
python = 2.6
buildout-versions = 1.2
setuptools = 0.6c11
zc.buildout = 1.5.0

If you attempt to run the buildout with the wrong version of python, 
you'll get an error such as:

$ bin/buildout
While:
   Installing.
   Loading extensions.
Error: python version specified not found in sys.version: 2.6

The PyPI page is here:
http://pypi.python.org/pypi/buildout-versions

The documentation is here:
http://packages.python.org/buildout-versions/use.html

If you have any problems or suggestions, please let me know on this list 
or the Simplistix open source google group!

cheers,

Chris