[Distutils] [Python-checkins] peps: PEP 470: use external indexes over link spidering

[Donald Stufft]

On May 15, 2014, at 1:13 PM, Stefan Krah <stefan-usenet at bytereef.org> wrote:

> Fixing the URLs:
> 
>> This is editorializing. In fact recently a former Debian project leader has
>> called PyPI "a bit flaky":
>> 
>>    https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html
> 
> https://mail.python.org/pipermail/distutils-sig/2014-May/024279.html

Ah yes. So that message actually requires a little bit of understanding. We have two
different “availability zones” on PyPI. The Web UI and the installer API. Right now
we’re seeing sporadic IO spikes on one of the glusterfs nodes which glusterfs isn’t
handling. However that’s having practically no affect on the installer /simple/ or
/packages/ API because we’ve engineered those to be extremely hard to lose.

First off, they are cached for up to 24 hours (With instant purging on changes) so
the likelihood is if the backend goes down they’ll still be served. We’ve also configured
a static mirror of PyPI that lives in a different datacenter, and if there are none of
our normal backends available to serve the request (because they’ve both gone down
and been marked as unhealthy) then it’ll seamlessly fallback to the static mirror and
serve that instead. Finally if somehow both the static mirror and the primary backend
are down, our cache will keep stale items for up to an extra day and serve them in
the case there is no healthy backends at all.

So to actually not be able to install something you need to have all of:

A) Both primary backends down
B) The static mirror down
C) The URL in question has not been accessed in the past 2 days.

We’re working on further enhancements that will make this even more robust.

HOWEVER, what the email in question is seeing is the Web UI. We degrade the
web UI early and we don’t have all of the failsafes for that. So the periodic IO spikes
which are effecting glusterfs which are taking the primary backends temporarily
out of rotation and falling back to the static mirror are causing the web UI to go
offline because the static mirror is read only. Again we’re working on more
enhancements to make it so we can even serve the web UI in a read only fashion
when the primary backends go down.

That being said, we’re investigating the cause of the intermittent IO spikes and we’ve
already replaced one node, but it’s still occurring so we’re looking at replacing the
whole cluster with PVHVM machines which get better IO performance.

> 
> 
>>> +* PyPI supports mirroring, both for private organizations and public mirrors.
>>> +  The legal terms of uploading to PyPI ensure that mirror operators, both
>>> +  public and private, have the right to distribute the software found on PyPI.
>> 
>> I don't think so:
>> 
>> https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html
> 
> https://mail.python.org/pipermail/python-legal-sig/2013-March/000003.html

As I said before, IANAL but i’m pretty sure that the two licenses running in parallel mean
that the uploader has to ensure that the two licenses are compatible before they upload
them. If they aren’t then they can’t legally upload the file. However since I am not a lawyer
I’ve reached out to VanL to get a clarification on that.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140515/c7a9d665/attachment.sig>