[Catalog-sig] pypi mirrors in bad shape

[Donald Stufft]

PyPI could use rsync for mirroring. You can setup anonymous rsync. 

It would require PyPI the software to write it's simple API files out
to disk and then expose that via RSync. Neither of those are
terribly hard and would improve the mirroring infrastructure IMO.


On Wednesday, January 16, 2013 at 7:33 PM, Aron Xu wrote:

> On Wed, Jan 16, 2013 at 5:43 PM, Jannis Leidel <jannis at leidel.info (mailto:jannis at leidel.info)> wrote:
> > 
> > On 15.01.2013, at 14:42, ken cochrane <kencochrane at gmail.com (mailto:kencochrane at gmail.com)> wrote:
> > 
> > > According to http://www.pypi-mirrors.org the pypi mirrors are in bad shape right now. We only have one mirror (f.pypi.python.org (http://f.pypi.python.org)) that isn't more then 8 hours out of date.
> > > 
> > > We have one server e.pypi.python.org (http://e.pypi.python.org) I can't even connect to anymore, and I'm not sure if is even online. Anyone know why?
> > > 
> > > c.pypi.python.org (http://c.pypi.python.org) and d.pypi.python.org (http://d.pypi.python.org) are both over 1 day old
> > > 
> > > g.pypi.python.org (http://g.pypi.python.org) is over 11 days old
> > > 
> > > b.pypi.python.org (http://b.pypi.python.org) is about 8 hours old at this point, I looked at the logs on this server and it looks like it is getting stuck trying to download packages for QSTK. Hopefully it resolves itself.
> > > 
> > > Restarting download of source/Q/QSTK/QSTK-0.2.1.tar.gz
> > > Restarting download of source/Q/QSTK/QSTK-0.2.0.tar.gz
> > > 
> > > 
> > > Even pypi.crate.io (http://pypi.crate.io) is over 4 hours out of date, and that one is usually pretty reliable.
> > > 
> > > I'm guessing there was something that was recently added to pypi, that broke the mirror replication, and each of these servers needs to be looked at to make sure they can get back in sync.
> > > 
> > > If I don't see any improvements in these mirrors by the end of the day, I'll start emailing the owners individually.
> > 
> > Sigh.
> > 
> > The mirror script was hanging on d, too. Since I've had to fix the mirror every other month now I wonder if the pep381client is up to the task. Does anyone have recommendations for fixing the situation? Is there an easy way to monitor the client and kill it if it runs longer than x?
> > 
> > I've now started a manual sync and hope to see d being up-to-date soonish.
> 
> Why not just use something based on rsync? I see no problem syncing
> Debian mirrors for years, which have lots of small files and update
> frequently. The script of Debian ftpsync cannot be used directly for
> pypi, but it's an example that worth considering:
> 
> http://ftp.debian.org/debian/project/ftpsync/ftpsync-current.tar.gz
> 
> 
> 
> -- 
> Regards,
> Aron Xu
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org (mailto:Catalog-SIG at python.org)
> http://mail.python.org/mailman/listinfo/catalog-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20130116/39871f04/attachment.html>