From matti.picus at gmail.com  Tue Mar 20 11:31:59 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 20 Mar 2018 17:31:59 +0200
Subject: [Speed] steps to get pypy benchmarks running
In-Reply-To: <CADiSq7csT8PCBB3wAFYP8u9waJCzw0zTXymcYPcdwiFyseJy2A@mail.gmail.com>
References: <279813bf-e7e1-693b-418b-af5e04dca098@gmail.com>
 <9c02bbf1-a61a-4f1d-597f-d0f93a0c04b7@hotpy.org>
 <CADiSq7csT8PCBB3wAFYP8u9waJCzw0zTXymcYPcdwiFyseJy2A@mail.gmail.com>
Message-ID: <e6655227-cdb7-1293-7ac7-3ee0c3033a2f@gmail.com>

On 14/02/18 20:18, Nick Coghlan wrote:
> On 14 February 2018 at 07:52, Mark Shannon <mark at hotpy.org> wrote:
>> Hi,
>>
>> On 13/02/18 14:27, Matti Picus wrote:
>>> I have begun to dive into the performance/perf code. My goal is to get
>>> pypy benchmarks running on http://speed.python.org. Since PyPy has a JIT,
>>> the benchmark runs must have a warmup stage.
>> Why?
>> The other interpreters don't get an arbitrary chunk of time for free, so
>> neither should PyPy. Warmup in an inherent cost of dynamic optimisers. The
>> benefits should outweigh the costs, but the costs shouldn't be ignored.
> For speed.python.org purposes, that would likely be most usefully
> reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
> the former runs under the same conditions as CPython, while the latter
> is given the benefit of warming up the JIT first).
>
> Only reporting the former would miss the point of PyPy's main use case
> (i.e. long lived processes), while only reporting the latter would
> miss one of the main answers to "Why hasn't everyone already switched
> to PyPy for all their Python needs?" (i.e. when the app doesn't run
> long enough to pay back the increased start-up overhead).
>
> Cheers,
> Nick.
So would it be reasonable as a first step to get the PyPy runner(s) into 
operation by modifying the nightly runs to download from the latest 
nightly builds [1], [2]?
We can deal with reporting cold/warm statistics later. As people have 
said, they are really two orthogonal issues.

[1] 
http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 
for python 2.7
[2] 
http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2 
for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)

Matti

From matti.picus at gmail.com  Sun Mar 25 10:37:11 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 25 Mar 2018 17:37:11 +0300
Subject: [Speed] steps to get pypy benchmarks running
In-Reply-To: <e6655227-cdb7-1293-7ac7-3ee0c3033a2f@gmail.com>
References: <279813bf-e7e1-693b-418b-af5e04dca098@gmail.com>
 <9c02bbf1-a61a-4f1d-597f-d0f93a0c04b7@hotpy.org>
 <CADiSq7csT8PCBB3wAFYP8u9waJCzw0zTXymcYPcdwiFyseJy2A@mail.gmail.com>
 <e6655227-cdb7-1293-7ac7-3ee0c3033a2f@gmail.com>
Message-ID: <13b624c7-4b53-b337-737a-6fe4dbcd20ee@gmail.com>

On 20/03/18 17:31, Matti Picus wrote:
> On 14/02/18 20:18, Nick Coghlan wrote:
>> On 14 February 2018 at 07:52, Mark Shannon <mark at hotpy.org> wrote:
>>> Hi,
>>>
>>> On 13/02/18 14:27, Matti Picus wrote:
>>>> I have begun to dive into the performance/perf code. My goal is to get
>>>> pypy benchmarks running on http://speed.python.org. Since PyPy has 
>>>> a JIT,
>>>> the benchmark runs must have a warmup stage.
>>> Why?
>>> The other interpreters don't get an arbitrary chunk of time for 
>>> free, so
>>> neither should PyPy. Warmup in an inherent cost of dynamic 
>>> optimisers. The
>>> benefits should outweigh the costs, but the costs shouldn't be ignored.
>> For speed.python.org purposes, that would likely be most usefully
>> reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
>> the former runs under the same conditions as CPython, while the latter
>> is given the benefit of warming up the JIT first).
>>
>> Only reporting the former would miss the point of PyPy's main use case
>> (i.e. long lived processes), while only reporting the latter would
>> miss one of the main answers to "Why hasn't everyone already switched
>> to PyPy for all their Python needs?" (i.e. when the app doesn't run
>> long enough to pay back the increased start-up overhead).
>>
>> Cheers,
>> Nick.
> So would it be reasonable as a first step to get the PyPy runner(s) 
> into operation by modifying the nightly runs to download from the 
> latest nightly builds [1], [2]?
> We can deal with reporting cold/warm statistics later. As people have 
> said, they are really two orthogonal issues.
>
> [1] 
> http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 
> for python 2.7
> [2] 
> http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2 
> for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)
>
> Matti

No responses, maybe I asked the wrong question.
I would be willing to issue a pull request to get PyPy runners into 
operation on "the beast" so it can report results to speed.python.org.
Which repo holds the code that stages `performance` runs and reports to 
speed.pypy.org?
Matti

From brett at python.org  Mon Mar 26 14:03:45 2018
From: brett at python.org (Brett Cannon)
Date: Mon, 26 Mar 2018 18:03:45 +0000
Subject: [Speed] steps to get pypy benchmarks running
In-Reply-To: <13b624c7-4b53-b337-737a-6fe4dbcd20ee@gmail.com>
References: <279813bf-e7e1-693b-418b-af5e04dca098@gmail.com>
 <9c02bbf1-a61a-4f1d-597f-d0f93a0c04b7@hotpy.org>
 <CADiSq7csT8PCBB3wAFYP8u9waJCzw0zTXymcYPcdwiFyseJy2A@mail.gmail.com>
 <e6655227-cdb7-1293-7ac7-3ee0c3033a2f@gmail.com>
 <13b624c7-4b53-b337-737a-6fe4dbcd20ee@gmail.com>
Message-ID: <CAP1=2W6BdjFapSbAkmNJ=sF0fRUBG4H5wU2FG-AsQjO193UhDg@mail.gmail.com>

On Sun, 25 Mar 2018 at 07:37 Matti Picus <matti.picus at gmail.com> wrote:

> On 20/03/18 17:31, Matti Picus wrote:
> > On 14/02/18 20:18, Nick Coghlan wrote:
> >> On 14 February 2018 at 07:52, Mark Shannon <mark at hotpy.org> wrote:
> >>> Hi,
> >>>
> >>> On 13/02/18 14:27, Matti Picus wrote:
> >>>> I have begun to dive into the performance/perf code. My goal is to get
> >>>> pypy benchmarks running on http://speed.python.org. Since PyPy has
> >>>> a JIT,
> >>>> the benchmark runs must have a warmup stage.
> >>> Why?
> >>> The other interpreters don't get an arbitrary chunk of time for
> >>> free, so
> >>> neither should PyPy. Warmup in an inherent cost of dynamic
> >>> optimisers. The
> >>> benefits should outweigh the costs, but the costs shouldn't be ignored.
> >> For speed.python.org purposes, that would likely be most usefully
> >> reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
> >> the former runs under the same conditions as CPython, while the latter
> >> is given the benefit of warming up the JIT first).
> >>
> >> Only reporting the former would miss the point of PyPy's main use case
> >> (i.e. long lived processes), while only reporting the latter would
> >> miss one of the main answers to "Why hasn't everyone already switched
> >> to PyPy for all their Python needs?" (i.e. when the app doesn't run
> >> long enough to pay back the increased start-up overhead).
> >>
> >> Cheers,
> >> Nick.
> > So would it be reasonable as a first step to get the PyPy runner(s)
> > into operation by modifying the nightly runs to download from the
> > latest nightly builds [1], [2]?
> > We can deal with reporting cold/warm statistics later. As people have
> > said, they are really two orthogonal issues.
> >
> > [1]
> > http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2
> > for python 2.7
> > [2]
> > http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2
> > for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)
> >
> > Matti
>
> No responses, maybe I asked the wrong question.
>

I think the people who have traditionally maintained speed.python.org are
just not available to answer the question, not that it was the wrong
question.


> I would be willing to issue a pull request to get PyPy runners into
> operation on "the beast" so it can report results to speed.python.org.
> Which repo holds the code that stages `performance` runs and reports to
> speed.pypy.org?
>

Unfortunately I don't know.

-Brett


> Matti
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20180326/bee2d222/attachment.html>

From zachary.ware+pydev at gmail.com  Mon Mar 26 14:10:07 2018
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Mon, 26 Mar 2018 13:10:07 -0500
Subject: [Speed] steps to get pypy benchmarks running
In-Reply-To: <CAP1=2W6BdjFapSbAkmNJ=sF0fRUBG4H5wU2FG-AsQjO193UhDg@mail.gmail.com>
References: <279813bf-e7e1-693b-418b-af5e04dca098@gmail.com>
 <9c02bbf1-a61a-4f1d-597f-d0f93a0c04b7@hotpy.org>
 <CADiSq7csT8PCBB3wAFYP8u9waJCzw0zTXymcYPcdwiFyseJy2A@mail.gmail.com>
 <e6655227-cdb7-1293-7ac7-3ee0c3033a2f@gmail.com>
 <13b624c7-4b53-b337-737a-6fe4dbcd20ee@gmail.com>
 <CAP1=2W6BdjFapSbAkmNJ=sF0fRUBG4H5wU2FG-AsQjO193UhDg@mail.gmail.com>
Message-ID: <CAKJDb-MCJXkKhSyQgzU95wpjSfYtOtOw0joTaDi3hXeUtnuq8A@mail.gmail.com>

On Mon, Mar 26, 2018 at 1:03 PM, Brett Cannon <brett at python.org> wrote:
> On Sun, 25 Mar 2018 at 07:37 Matti Picus <matti.picus at gmail.com> wrote:
>> No responses, maybe I asked the wrong question.
>
> I think the people who have traditionally maintained speed.python.org are
> just not available to answer the question, not that it was the wrong
> question.

This.  I think at this point we might be better served to just get
Matti access to the relevant machines.

-- 
Zach

From brett at python.org  Mon Mar 26 14:15:17 2018
From: brett at python.org (Brett Cannon)
Date: Mon, 26 Mar 2018 18:15:17 +0000
Subject: [Speed] steps to get pypy benchmarks running
In-Reply-To: <CAKJDb-MCJXkKhSyQgzU95wpjSfYtOtOw0joTaDi3hXeUtnuq8A@mail.gmail.com>
References: <279813bf-e7e1-693b-418b-af5e04dca098@gmail.com>
 <9c02bbf1-a61a-4f1d-597f-d0f93a0c04b7@hotpy.org>
 <CADiSq7csT8PCBB3wAFYP8u9waJCzw0zTXymcYPcdwiFyseJy2A@mail.gmail.com>
 <e6655227-cdb7-1293-7ac7-3ee0c3033a2f@gmail.com>
 <13b624c7-4b53-b337-737a-6fe4dbcd20ee@gmail.com>
 <CAP1=2W6BdjFapSbAkmNJ=sF0fRUBG4H5wU2FG-AsQjO193UhDg@mail.gmail.com>
 <CAKJDb-MCJXkKhSyQgzU95wpjSfYtOtOw0joTaDi3hXeUtnuq8A@mail.gmail.com>
Message-ID: <CAP1=2W6_Dfk2o0WX3TSxLF-ysQuUF3VEMqzpWdRRg2S_Baht0A@mail.gmail.com>

On Mon, 26 Mar 2018 at 11:10 Zachary Ware <zachary.ware+pydev at gmail.com>
wrote:

> On Mon, Mar 26, 2018 at 1:03 PM, Brett Cannon <brett at python.org> wrote:
> > On Sun, 25 Mar 2018 at 07:37 Matti Picus <matti.picus at gmail.com> wrote:
> >> No responses, maybe I asked the wrong question.
> >
> > I think the people who have traditionally maintained speed.python.org
> are
> > just not available to answer the question, not that it was the wrong
> > question.
>
> This.  I think at this point we might be better served to just get
> Matti access to the relevant machines.
>

No objections from me.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20180326/6f467fb4/attachment.html>

From matti.picus at gmail.com  Tue Mar 27 03:59:47 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 27 Mar 2018 10:59:47 +0300
Subject: [Speed] steps to get pypy benchmarks running
In-Reply-To: <CAP1=2W6_Dfk2o0WX3TSxLF-ysQuUF3VEMqzpWdRRg2S_Baht0A@mail.gmail.com>
References: <279813bf-e7e1-693b-418b-af5e04dca098@gmail.com>
 <9c02bbf1-a61a-4f1d-597f-d0f93a0c04b7@hotpy.org>
 <CADiSq7csT8PCBB3wAFYP8u9waJCzw0zTXymcYPcdwiFyseJy2A@mail.gmail.com>
 <e6655227-cdb7-1293-7ac7-3ee0c3033a2f@gmail.com>
 <13b624c7-4b53-b337-737a-6fe4dbcd20ee@gmail.com>
 <CAP1=2W6BdjFapSbAkmNJ=sF0fRUBG4H5wU2FG-AsQjO193UhDg@mail.gmail.com>
 <CAKJDb-MCJXkKhSyQgzU95wpjSfYtOtOw0joTaDi3hXeUtnuq8A@mail.gmail.com>
 <CAP1=2W6_Dfk2o0WX3TSxLF-ysQuUF3VEMqzpWdRRg2S_Baht0A@mail.gmail.com>
Message-ID: <2841d05a-fa58-82d0-7296-3c4455af0951@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20180327/9c28aef4/attachment-0001.html>

From tobami at gmail.com  Fri Mar 30 12:08:08 2018
From: tobami at gmail.com (Miquel Torres)
Date: Fri, 30 Mar 2018 18:08:08 +0200
Subject: [Speed] steps to get pypy benchmarks running
In-Reply-To: <2841d05a-fa58-82d0-7296-3c4455af0951@gmail.com>
References: <279813bf-e7e1-693b-418b-af5e04dca098@gmail.com>
 <9c02bbf1-a61a-4f1d-597f-d0f93a0c04b7@hotpy.org>
 <CADiSq7csT8PCBB3wAFYP8u9waJCzw0zTXymcYPcdwiFyseJy2A@mail.gmail.com>
 <e6655227-cdb7-1293-7ac7-3ee0c3033a2f@gmail.com>
 <13b624c7-4b53-b337-737a-6fe4dbcd20ee@gmail.com>
 <CAP1=2W6BdjFapSbAkmNJ=sF0fRUBG4H5wU2FG-AsQjO193UhDg@mail.gmail.com>
 <CAKJDb-MCJXkKhSyQgzU95wpjSfYtOtOw0joTaDi3hXeUtnuq8A@mail.gmail.com>
 <CAP1=2W6_Dfk2o0WX3TSxLF-ysQuUF3VEMqzpWdRRg2S_Baht0A@mail.gmail.com>
 <2841d05a-fa58-82d0-7296-3c4455af0951@gmail.com>
Message-ID: <CAGf+9VyzgWfk4O0dv0Ln7Qxq-TOBbRKfbra2=sJpW17Vv1VN_w@mail.gmail.com>

Hi Matti,

if you need any help configuring the app on speed.python.org just ask and
I'll gladly help.

Cheers,
Miquel


2018-03-27 9:59 GMT+02:00 Matti Picus <matti.picus at gmail.com>:

> On 26/03/18 21:15, Brett Cannon wrote:
>
>
>
> On Mon, 26 Mar 2018 at 11:10 Zachary Ware <zachary.ware+pydev at gmail.com>
> wrote:
>
>> On Mon, Mar 26, 2018 at 1:03 PM, Brett Cannon <brett at python.org> wrote:
>> > On Sun, 25 Mar 2018 at 07:37 Matti Picus <matti.picus at gmail.com> wrote:
>> >> No responses, maybe I asked the wrong question.
>> >
>> > I think the people who have traditionally maintained speed.python.org
>> are
>> > just not available to answer the question, not that it was the wrong
>> > question.
>>
>> This.  I think at this point we might be better served to just get
>> Matti access to the relevant machines.
>>
>
> No objections from me.
>
>
> Thanks for the help, I now can access "the beast", and am exploring the
> best way to run the benchmarks and upload the results to speed.python.org.
>
> It seems there is a script to do just that - run_and_upload.py. The
> process is currently not automated. For now that should be enough to get
> some first impressions.
>
> Matti
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20180330/9f9bdeab/attachment.html>