From brett at python.org  Sun Jan  3 14:57:50 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 03 Jan 2016 19:57:50 +0000
Subject: [Speed] People interested in reworking the benchmark suite in 2016?
Message-ID: <CAP1=2W7m1BCpXJvr-Z6DKBkT5Kjda4xjaAU9iWKUQ4e9LefQdg@mail.gmail.com>

With the planned move to GitHub, there is an opportunity to try and rework
the set of benchmarks -- and anything else -- in 2016 by starting a new
benchmark repo from scratch. E.g., modern numeric benchmarks, long-running
benchmarks that warm up JITs, using pip with pegged bugfix versions so we
stop shipping library code with the benchmarks, etc.  We could also
standardize results output -- e.g. should we just make everything run under
codespeed? -- so that the benchmarks are easy to run locally for one-off
results as well as continuous benchmarking for trend details with a common
benchmark driver?

Would people be interested and motivated enough in getting representatives
from the various Python implementations together at PyCon and have a BoF to
discuss what we want from a proper, unified, baseline benchmark suite and
see if we can pull one together -- or at least start one -- in 2016?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20160103/602dab32/attachment.html>

From fijall at gmail.com  Sun Jan  3 16:45:58 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 3 Jan 2016 23:45:58 +0200
Subject: [Speed] People interested in reworking the benchmark suite in
 2016?
In-Reply-To: <CAP1=2W7m1BCpXJvr-Z6DKBkT5Kjda4xjaAU9iWKUQ4e9LefQdg@mail.gmail.com>
References: <CAP1=2W7m1BCpXJvr-Z6DKBkT5Kjda4xjaAU9iWKUQ4e9LefQdg@mail.gmail.com>
Message-ID: <CAK5idxQFP2F9BcfQ4APtV32Job1AuONmHhvkEEnoXLiD-m-1pQ@mail.gmail.com>

count me in (even if I don't make it to pycon)

On Sun, Jan 3, 2016 at 9:57 PM, Brett Cannon <brett at python.org> wrote:
> With the planned move to GitHub, there is an opportunity to try and rework
> the set of benchmarks -- and anything else -- in 2016 by starting a new
> benchmark repo from scratch. E.g., modern numeric benchmarks, long-running
> benchmarks that warm up JITs, using pip with pegged bugfix versions so we
> stop shipping library code with the benchmarks, etc.  We could also
> standardize results output -- e.g. should we just make everything run under
> codespeed? -- so that the benchmarks are easy to run locally for one-off
> results as well as continuous benchmarking for trend details with a common
> benchmark driver?
>
> Would people be interested and motivated enough in getting representatives
> from the various Python implementations together at PyCon and have a BoF to
> discuss what we want from a proper, unified, baseline benchmark suite and
> see if we can pull one together -- or at least start one -- in 2016?
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed
>

From solipsis at pitrou.net  Tue Jan 12 08:58:40 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 12 Jan 2016 14:58:40 +0100
Subject: [Speed] People interested in reworking the benchmark suite in
 2016?
References: <CAP1=2W7m1BCpXJvr-Z6DKBkT5Kjda4xjaAU9iWKUQ4e9LefQdg@mail.gmail.com>
Message-ID: <20160112145840.612676d0@fsol>

On Sun, 03 Jan 2016 19:57:50 +0000
Brett Cannon <brett at python.org> wrote:
> With the planned move to GitHub, there is an opportunity to try and rework
> the set of benchmarks -- and anything else -- in 2016 by starting a new
> benchmark repo from scratch. E.g., modern numeric benchmarks, long-running
> benchmarks that warm up JITs, using pip with pegged bugfix versions so we
> stop shipping library code with the benchmarks, etc.

That feature list should be discussed :-) For example I don't think
downloading dependencies with pip is desirable compared to the ease of
cloning a single repo (also what happens if the files become
unavailable?).

> We could also
> standardize results output -- e.g. should we just make everything run under
> codespeed? -- so that the benchmarks are easy to run locally for one-off
> results as well as continuous benchmarking for trend details with a common
> benchmark driver?

I don't know who uses codespeed apart from PyPy, but asv is a possible
benchmark runner:
https://asv.readthedocs.org/en/latest/index.html

> Would people be interested and motivated enough in getting representatives
> from the various Python implementations together at PyCon and have a BoF to
> discuss what we want from a proper, unified, baseline benchmark suite and
> see if we can pull one together -- or at least start one -- in 2016?

I probably won't be at PyCon.

Regards

Antoine.