[Speed] [pypy-dev] Co-ordinating benchmarking

Thu Sep 1 08:45:00 CEST 2011

Okay, so all of a sudden there seem to be a *lot* of people looking at
this. This was a long thread quickly, and I only just got up to speed
with it. There are a lot of new names, and I don't know what existing
skills, interest and territories exist. Apologies for any faux pas.

I would like to organise the list of tasks a bit more clearly if that
is okay. I may be less familiar with the parts of this process than
others, so I just want to get it down clearly.

What I've done:
  -- Clone the Python repo into /home/speedracer/cpython/cpython
(updated to 2.7)
  -- Installed os packages to support a reasonable build.
  -- Built and installed python2.7 into /home/speedraver/cpython/27_bin

Presumably, people would like to be monitoring both PyPy and Cpython
as they progress over time. This means some kind of auto runner which
updates from the repo, re-runs the timings, and submits them to
codespeed. I am unclear on whether there is a "clear winner" for this
piece of technology.

List of Required Infrastructure on speed.python.org:
  -- A home for the cpython repo (check)
  -- An installed and running codespeed server (pending)
  -- A buildbot / automation for keeping up to date with the PyPy and
cpython repos (???)
  -- Automation to execute a process which must (???)
     1) Run the benchmarks
     2) Submit results to codespeed

I would suggest that the codespeed server be installed into
speedracer's home directory.

This must all be installable and configurable from Chef (which appears
to me like Fabric, i.e. an automation tool for deployment and
management of systems). This is not yet accomplished.

We also clearly need some kind of wiki documentation on what is going
on, so that contributors (or just newbies like me) can figure out
where things are at and what is going on. The bitbucket project is
great, but the task titles are currently rather brief if someone isn't
already totally up-to-speed on what is going on.

There appear to me to be two unresolved questions:
  1) What piece of technology should we use for a buildbot / build automation
  2) What piece of technology should we use for the benchmark runner?

I have no suggestions on (1), it's not a strong point for me.

As regards (2), I am ignorant to what others might already be using,
except to say the landscape seems unclear and fractured to me. My
work, benchmarker.py, is likely to be adaptable to our needs and I am
more than happy to support the package so it can be applied here. As I
understand it, the GSOC project was about the actual benchmarking
functions, not so much about automation and support for managing the
results of benchmarking. If an already-working alternative to
benchmarker.py exists and makes more sense to use, then that is fine
by me. I would still like to help out to learn more about
benchmarking.

The main issue with me as a contributor will be time. I have a full
plate as a result of Real Life, so I will sometimes go dark for a week
or so. However, I'm motivated and interested, and can put in a few
hours a week most weeks.

Do I have this right? Is that a reasonable description of the work
breakdown? Do we have clear names against tasks so that co-ordination
can be done through those people (rather than via the whole list)?

Regards,
-Tennessee

On Thu, Sep 1, 2011 at 6:28 AM, Noah Kantrowitz <noah at coderanger.net> wrote:
> Its all branches all the way down, so we can start work anywhere and push it to an "official" PSF bin later I think. I'm sure we will want to host a mirror of it on the python.org hg server too, just for discoverability.
>
> --Noah
>
> On Aug 31, 2011, at 1:12 PM, Miquel Torres wrote:
>
>> Oh, cool, so there will be an Opscode hosted account for the PSF,
>> right? Then the Chef repo should be for the PSF. Maybe in a current
>> account somewhere? What do you propose?
>>
>> Miquel
>>
>>
>> 2011/8/31 Noah Kantrowitz <noah at coderanger.net>:
>>> Opscode has already agreed to donate a Hosted account as long we keep it under ~20 clients :-) I can hand out the info for it to anyone that wants. As for setting up the Chef repo, just remember we are trying to not manage this system in isolation and that it will be part of a bigger PSF infrastructure management effort.
>>>
>>> --Noah
>>>
>>> On Aug 31, 2011, at 11:34 AM, Miquel Torres wrote:
>>>
>>>> Hi all,
>>>>
>>>> though I took up on the task of installing a Codespeed instance
>>>> myself, I didn't have time until now. This weekend I will definitely
>>>> have  a *lot* of time to work on this, so count on that task being
>>>> done by then.
>>>>
>>>> The bitbucket issue tracker is a start (though a organization account
>>>> would be better) and the splash page is great. So let's get started
>>>> organizing things.
>>>>
>>>> Regarding the deployment strategy, it turns out I use Chef at work, so
>>>> I am in full agreement with Noah here (yey!). Actually, I am the
>>>> author of LittleChef (which we can use as a tool to execute Chef on
>>>> the node).
>>>>
>>>> So, Configuration Management. I would propose that Noah starts the
>>>> repo with the Chef cookbooks (preferably a complete LittleChef
>>>> kitchen, but that is not a must :), and gets the main recipes (apache,
>>>> django) going, while I create a cookbook for Codespeed. What do you
>>>> think?
>>>>
>>>> The benchmark runner question is still open. We need to clarify that.
>>>> Use the pypy runner? Tennessee's work?
>>>>
>>>> Regarding repositories and issues, we could maybe have a "speed"
>>>> organization account (not sure on Bitbucket, you can do that in
>>>> Github), where we have a wiki, issues, and runner + config management
>>>> repo + other stuff.
>>>>
>>>> Cheers,
>>>> Miquel
>>>>
>>>> 2011/8/31 Jesse Noller <jnoller at gmail.com>:
>>>>> I've put up a splash page for the project this AM:
>>>>>
>>>>> http://speed.python.org/
>>>>>
>>>>> jesse
>>>>> _______________________________________________
>>>>> pypy-dev mailing list
>>>>> pypy-dev at python.org
>>>>> http://mail.python.org/mailman/listinfo/pypy-dev
>>>>>
>>>> _______________________________________________
>>>> Speed mailing list
>>>> Speed at python.org
>>>> http://mail.python.org/mailman/listinfo/speed
>>>
>>>
>
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> http://mail.python.org/mailman/listinfo/speed
>
>

-- 
--------------------------------------------------
Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"