[Catalog-sig] PyPI overloaded(?)

["Martin v. Löwis"]

Jim Fulton schrieb:
>>> I should have noted in my note that my intent is not to complain.
>>> Mainly, I want to make sure that we we're in agreement about the goals
>>> and to make sure the PSF allocates the rsources needed to achieve them.
>> Can you volunteer yourself or somebody else to help in maintaining
>> PyPI?
> 
> Possibly.  Is it an issue of people?  I really don't know what the
> cause of the problems yesterday were.  Was it a software problem?
> Load?  I'd like to understand what the issues are first.

Primarily, it's people. Very few people understand the PyPI code well,
and nearly nobody has time research performance problems. My guess is
that the code could be improved to perform certain actions better.
For example, it's know that PyPI cannot really cope with web crawlers
too well. If too many of them hit him at a bad spot, it will go down.
It takes man-power to investigate these problems and propose solutions;
solutions based on speculation (like "let's mirror") are likely wrong.

I'm not sure whether there is any way to investigate performance
problems after the fact; if there is, it likely isn't possible in
this case. So one would have to define a performance analysis procedure
first, implement it, and wait for the performance to go down again
to understand what's commonly causing the problem. Of course, if
somebody would log in and analyse *while* the problem is happening,
there might be a much better chance at understanding the problem.
OTOH, it's really no fun to investigate something remotely on
an overloaded machine, so likely, whoever is in charge would just
make the problem go away instead of studying it.

Regards,
Martin