[Python-Dev] Benchmark results across all major Python implementations

[Stewart, David C]

+Stefan (owner of the 0-day lab)




On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" <python-dev-bounces+david.c.stewart=intel.com at python.org on behalf of rdmurray at bitdance.com> wrote:

>On Mon, 16 Nov 2015 23:37:06 +0000, "Stewart, David C" <david.c.stewart at intel.com> wrote:
>> Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet
>> 
>> You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/
>> (I have this running constantly on a dedicated screen in my office).
>
>Just took a look at this.  Pretty cool.  The web page is a bit confusing,
>though.  It doesn't give any clue as to what is being measured by the
>numbers presented...it isn't obvious whether those downward sloping
>lines represent progress or regression.  Also, the intro talks about
>historical data, but other than the older dates[*] in the graph there's
>no access to it.  Do you have plans to provide access to the raw data?
>It also doesn't show all of the test shown in the example email in your
>blog post or the emails to python-checkins...do you plan to make those
>graphs available in the future as well?

The data on this website has been normalized so "up" is "good" so far as the slope of the line. The daily email has a lot more detail about the hardware and software configuration and the versions being compared. We run workloads multiple times and visually show the relative standard distribution on the graph.

No plans to show the raw data.

I think showing multiple workloads graphically sounds useful, we should look into that.

>
>Also, in the emails, what is the PGO column percentage relative to?

It's the performance boost on the current rev from just using PGO. Another way to think about it is, this is the performance that you leave on the table by *not* building Cpython with PGO. For example, from last night's run, we would see an 18.54% boost in django_v2 by building Python using PGO.

Note: PGO is not the default way to build Python because it is relatively slow to compile it that way. (I think it should be the default). 

Here are the instructions for using it (thanks to Peter Wang for the instructions):

hg clone https://hg.python.org/cpython cpython
cd cpython
hg update 2.7
./configure
make profile-opt



>
>I suppose that for this to have maximum effect someone would have to
>specifically be paying attention to performance and figuring out why
>every (real) regression happened.  I don't suppose we have anyone in the
>community currently who is taking on that role, though we certainly do
>have people who are *interested* in Python performance :)

We're trying to fill that role as much as we can. When there is a significant (and unexplained) regression that we see, I usually ask our engineers to bisect it to identify the offending patch and root-cause it.

>
>--David
>
>[*] Personally I'd find it easier to read those dates in MM-DD form,
>but I suppose that's a US quirk, since in the US when using slashes
>the month comes first...

You and me both. As you surmised, the site was developed by our friends in Europe. :-)

>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: https://mail.python.org/mailman/options/python-dev/david.c.stewart%40intel.com