[Speed] Do we want to stop vendoring source of third-party libraries with the benchmarks?

[Antoine Pitrou]

On Thu, 11 Feb 2016 18:35:29 +0000
Brett Cannon <brett at python.org> wrote:
> Maybe we should just have a requirements.txt file for Python 2 and another
> for Python 3 that are pegged to specific versions? We could even install
> things into a venv for isolation.

How does this impact interaction with the benchmarks suite?
E.g. does it increase the time of running a couple of benchmarks? Does
it make it easier or harder to benchmark a work-in-progress patch for
whatever interpreter?

> If we go this route then we could make
> the benchmark suite a package on PyPI and have people install the benchmark
> suite and then have instructions to run pip on the requirements files that
> we embed in the package.

I'm not fond of encouraging random users to run the benchmarks suite
without understanding what they're doing, and starting throwing around
pointless numbers and misconceptions about performance (which are then
very hard to fight since people tend to be irrationally captivated by
"performance numbers").  The benchmarks suite is mostly a tool for
developers of Python implementations, not the greater public.

Having the benchmarks suite only available through hg or git kind of
discourages those tendencies.

Regards

Antoine.