[Catalog-sig] Rewrite PyPI for App Engine?

[Tarek Ziadé]

On Sat, Jun 19, 2010 at 1:58 AM, Ian Bicking <ianb at colorstudy.com> wrote:
..
>> Out of curiosity : have you ever worked with the current implementation ?
>>
>> I have hard time to understand why some people say it's hard to work with
>> it,
>> I don't think its a valid argument.
>
> I haven't looked at it in years, but I've poked around it some.  I found it
> difficult, yes.

Having worked with both code bases, it's much more simple that Pip, but suffers
from the same syndromes : some modules just grew too big, and there
are not enough tests  ;)

PyPI has for instance a huge webui.py file, which should be cut in pieces.

..
>> Even if I don't think its a priority in our concerns (community
>> mirrors come first), I wouldn't mind having the main PyPI UI in GAE.
>
> The priorities that motivate me are:
>
> 1. Make installation more reliable with respect to PyPI
> 2. Decrease overall maintenance burden
> 3. Decrease code liability
>
> Community mirrors only address 1 while App Engine addresses 2 and a rewrite
> addresses 3.

I agree with 2. but I don't understand 3. why AppEngine would decrease
code liability ?
a code can be liable in any environment.

> There's a significant number of failure conditions that a mirror network
> doesn't protect you from.  Connection refused, connection timed out, and 500
> errors are the only really obvious errors that will make a tool look to the
> next mirror.  Because of potential synchronization problems there's a lot of
> new problems a mirror network could introduce.

a mirror network is not the silver bullet, but I don't think the
number of failure conditions
is more significant than another solution. As a matter of fact,
potential synchronization problems should be addressed by the
mirroring protocol itself, if you think of any use case now, or if we
meet one later. but, the main use case from the client PoV : "the
sever is down" is fixed by falling back to another server.

>> Although, if PyPI was to be ported to GAE, couldn't we reuse the
>> existing code instead of rewriting from scratch ? we would just have
>> to rewrite the DB layer.
>
> I don't think it's worth reusing that code.

Why that ? As a contributor to the project, I know this will take some
time to be rewritten, even if the application is not big. Features are
still added in it. So rewriting something from scratch strikes me as a
bad idea.

Here are the priority I think we should take to solve the issues we
had with PyPI:

1. investigate deeper in why the PyPI server was down for some hours
2. make sure PyPI has more sysadmins in several timezones
3. make the existing mirrors, "official" mirrors, via PEP 381

1. 2. can be done right now.

Regards
Tarek