[Pandas-dev] pandas ASVs

[Marc Garcia]

Thanks Richard, really nice work on this front.

It took longer than expected, but I expect we should sign the new
partnership to get free hardware for pandas very soon now, probably early
next week. Only the final signatures are now missing.

With the new hosting, we will be moving our website as you say (the old
server is very expensive for what it is, and the new is free). But also we
can get dedicated hardware for benchmarks (and probably other things if
needed).

What I'd like to try is to set up a dedicated server as a github actions
worker, and set up a job in our CI with cirun to run benchmarks. Then test
two things:

- Timing: see if it could be feasible to run our benchmarks at every commit
like the rest of the CI. If this is possible, I think this would keep
things very simple, and catch all significant regressions easily before
they happen (also run for commits to main for the asv history)

- Stability: run the benchmarks for the same exact version of pandas many
times, and analyze the variance, understand well what's the noise in the
results, and see if we can take actions to reduce it to a minimum (fine
tunning OS settings...)

I don't know if a dedicated server as a githib actions worker with cirun
will be the best option, but I think it's worth trying the above, as it
should keep things simple and effective in my opinion. But we can surely
try other things and discuss other ideas, there may be better options.

On Sat, Aug 27, 2022, 02:16 Richard Shadrach <rhshadrach at gmail.com> wrote:

> I noticed a bit ago that the ASVs on the pandas webpage were no longer
> being updated.
>
> https://pandas.pydata.org/speed/pandas/
>
> Tom shared with me his setup and gave me permissions to
> https://github.com/asv-runner (thanks Tom!). I now have a machine
> dedicated to running the ASVs and updating the site here:
>
> https://asv-runner.github.io/asv-collection/pandas/
>
> I plan to be going through the regressions identified there and
> notifying PRs as necessary; manually at first but then looking into a more
> automated solution. I also plan to port Tom's work in asv-runner over to
> Docker.
>
> From Marc, it sounds like there might be some changes coming to the
> webserver, so I think it's best to wait until that dust settles before
> getting the ASVs on the pandas webpage to auto-update.
>
> Best,
> Richard
> _______________________________________________
> Pandas-dev mailing list
> Pandas-dev at python.org
> https://mail.python.org/mailman/listinfo/pandas-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/pandas-dev/attachments/20220827/951a1b0b/attachment.html>