[Pandas-dev] pandas new infrastructure (OVH donation)

[Marc Garcia]

Hi all,

pandas has received a donation from OVHcloud <https://www.ovhcloud.com/> to
support the project infrastructure, with OVHcloud public cloud credits (an
initial amount of 10,000 EUR for a period of one year). OVH is open to
sponsor longer term and also other projects of the ecosystem (or NumFOCUS
as a whole), but we started with this to have feedback at a smaller scale
first.

The credits will be used initially for:
- Hosting of the pandas website
- Running the pandas benchmarks
- Speeding up the project CI

I detail next what I have in mind to set up for each. If anyone is
interested in getting involved, or has ideas, comments... please let me
know. I'll publish updates here as there is progress on this.


Website: I'm planning to experiment on splitting the website in two (it'll
be transparent for users). The website and the stable docs which receive
most of the traffic can probably be stored in Cloudflare pages. We're
already using Cloudflare as a CDN, so instead of using it as a cache, we
can publish the documents there. The rest of the docs (old versions and the
dev version) can be hosted in bucket storage of the OVHcloud. Response
times may be a bit slower, but our website is bigger than the Cloudflare
quota, and having old docs rarely accessed in a CDN seems unnecessary
anyway.

- Benchmarks: OVHcloud instances have guaranteed hardware, and we'll be
checking if this is enough for the results of the benchmarks to be
consistent over runs, or if there is too much variability and we need
dedicated hardware. If consistency is good enough that would be great,
since our benchmarks mostly use one core, and using dedicated hardware is
likely to be a decent waste of resources, since most servers will likely
have 16 cores or more. We'll discuss with OVH if dedicated hardware is
needed, as at the moment their public cloud doesn't offer it (there is an
alpha for providing dedicated instances, but we need to check with them).

- Faster CI: Our GitHub runners are small, and most builds take around one
hour or more to finish. We should be able to use bigger OVH instances for
our existing CI pretty easily, via their OpenStack API and CIrun.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/pandas-dev/attachments/20221105/29d0e1e0/attachment.html>