From victor.stinner at gmail.com  Tue Jan  9 10:09:16 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jan 2018 16:09:16 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7 and
 PyPy2?
Message-ID: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>

Hi,

I regularly update requirements of the performance benchmark suite.
Django 2.0 is now proposed, but it's incompatible with Python 2.7 and
PyPy2.

* Is it ok to skip the django_template benchmark on Python 2.7 and PyPy2?
* Should we add a different benchmark?
* Or test different Django version depending on the Python version?

I'm fine with testing Django 1.11 on Python 2.7 and test Django >= 2.0
on Python 3, but maybe others have different opinion on that?

The main question is the expected behaviour when comparing Python2 and
Python3 with performance on the Django template benchmark.

Victor

From solipsis at pitrou.net  Tue Jan  9 10:17:57 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Jan 2018 16:17:57 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
Message-ID: <20180109161757.1fa74a52@fsol>

On Tue, 9 Jan 2018 16:09:16 +0100
Victor Stinner <victor.stinner at gmail.com>
wrote:
> Hi,
> 
> I regularly update requirements of the performance benchmark suite.
> Django 2.0 is now proposed, but it's incompatible with Python 2.7 and
> PyPy2.
> 
> * Is it ok to skip the django_template benchmark on Python 2.7 and PyPy2?
> * Should we add a different benchmark?

How do you plan to make numbers comparable if you change the Django
version for a given benchmark?  The only solution IMHO is to add a
different benchmark.

Perhaps you should also probably wait for a bugfix release or two :-)

Regards

Antoine.



From songofacandy at gmail.com  Tue Jan  9 10:42:29 2018
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 10 Jan 2018 00:42:29 +0900
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <20180109161757.1fa74a52@fsol>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
Message-ID: <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>

On Wed, Jan 10, 2018 at 12:17 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 9 Jan 2018 16:09:16 +0100
> Victor Stinner <victor.stinner at gmail.com>
> wrote:
>> Hi,
>>
>> I regularly update requirements of the performance benchmark suite.
>> Django 2.0 is now proposed, but it's incompatible with Python 2.7 and
>> PyPy2.
>>
>> * Is it ok to skip the django_template benchmark on Python 2.7 and PyPy2?
>> * Should we add a different benchmark?
>
> How do you plan to make numbers comparable if you change the Django
> version for a given benchmark?  The only solution IMHO is to add a
> different benchmark.
>

We already compare different libraries.  For example, pickle is very different
between Python 2.7 and 3.6.
Even though it's not good for comparing interpreter performance, it's good
for people comparing Python 2 and 3.

If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python 2.7,
it's nice carrot for people moving forward.


> Perhaps you should also probably wait for a bugfix release or two :-)

I agree.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed


-- 
INADA Naoki  <songofacandy at gmail.com>

From victor.stinner at gmail.com  Tue Jan  9 11:39:33 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jan 2018 17:39:33 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
Message-ID: <CAMpsgwbXDaSWeWhiYCPFbN6Ufr2YcAKkpVNkDsDC7kOraJddgw@mail.gmail.com>

2018-01-09 16:42 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> We already compare different libraries.  For example, pickle is very different
> between Python 2.7 and 3.6.
> Even though it's not good for comparing interpreter performance, it's good
> for people comparing Python 2 and 3.
>
> If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python 2.7,
> it's nice carrot for people moving forward.

I agree. Antoine: what do you think?

>> Perhaps you should also probably wait for a bugfix release or two :-)
>
> I agree.

Well, I wrote "2.0", but the latest release is "2.0.1", released a few days ago:
https://pypi.org/project/Django/

Victor

From victor.stinner at gmail.com  Tue Jan  9 11:42:34 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jan 2018 17:42:34 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <20180109161757.1fa74a52@fsol>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
Message-ID: <CAMpsgwZEngCm9sMiHPLdHaVFU09BsquaF6fyb-iBh_fRB44Mmg@mail.gmail.com>

2018-01-09 16:17 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> How do you plan to make numbers comparable if you change the Django
> version for a given benchmark?  The only solution IMHO is to add a
> different benchmark.

I mostly use performance to compare Python versions, like compare
Python 3.7 to Python 2.7. If the Django-1.11 benchmark becomes
different, I'm not able anymore to compare django_template between
Python 2 and Python 3. In this case, I would prefer simply to ignore
the benchmark on Python 2.

The benchmark has nothing interesting. It's just another templating
library, we already have other benchmarks testing strings and
templates.

Victor

From solipsis at pitrou.net  Tue Jan  9 11:46:57 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Jan 2018 17:46:57 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAMpsgwbXDaSWeWhiYCPFbN6Ufr2YcAKkpVNkDsDC7kOraJddgw@mail.gmail.com>
Message-ID: <20180109174657.0d566596@fsol>

On Tue, 9 Jan 2018 17:39:33 +0100
Victor Stinner <victor.stinner at gmail.com>
wrote:
> 2018-01-09 16:42 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> > We already compare different libraries.  For example, pickle is very different
> > between Python 2.7 and 3.6.
> > Even though it's not good for comparing interpreter performance, it's good
> > for people comparing Python 2 and 3.
> >
> > If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python 2.7,
> > it's nice carrot for people moving forward.  
> 
> I agree. Antoine: what do you think?

I disagree.  pickle is an integral part of Python, it's versioned
*with* Python.  Django is not and it's misleading to compare results
obtained using two differents of it.

Regards

Antoine.



From victor.stinner at gmail.com  Tue Jan  9 11:51:36 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jan 2018 17:51:36 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <20180109174657.0d566596@fsol>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAMpsgwbXDaSWeWhiYCPFbN6Ufr2YcAKkpVNkDsDC7kOraJddgw@mail.gmail.com>
 <20180109174657.0d566596@fsol>
Message-ID: <CAMpsgwaN-9V5Jf=O2CASpdY4X0WHqueTFa-xZ4zuZp2wQH7ALA@mail.gmail.com>

Ok. So would you be ok to simply drop this benchmark? Or does anyone need it?

http://pyperformance.readthedocs.io/benchmarks.html#django-template
---
Use the Django template system to build a 150x150-cell HTML table.

Use Context and Template classes of the django.template module.
---

Other template benchmarks:

* http://pyperformance.readthedocs.io/benchmarks.html#chameleon
* http://pyperformance.readthedocs.io/benchmarks.html#genshi
* http://pyperformance.readthedocs.io/benchmarks.html#mako

Victor

2018-01-09 17:46 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> On Tue, 9 Jan 2018 17:39:33 +0100
> Victor Stinner <victor.stinner at gmail.com>
> wrote:
>> 2018-01-09 16:42 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
>> > We already compare different libraries.  For example, pickle is very different
>> > between Python 2.7 and 3.6.
>> > Even though it's not good for comparing interpreter performance, it's good
>> > for people comparing Python 2 and 3.
>> >
>> > If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python 2.7,
>> > it's nice carrot for people moving forward.
>>
>> I agree. Antoine: what do you think?
>
> I disagree.  pickle is an integral part of Python, it's versioned
> *with* Python.  Django is not and it's misleading to compare results
> obtained using two differents of it.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed

From solipsis at pitrou.net  Tue Jan  9 11:47:24 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Jan 2018 17:47:24 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAMpsgwZEngCm9sMiHPLdHaVFU09BsquaF6fyb-iBh_fRB44Mmg@mail.gmail.com>
Message-ID: <20180109174724.1885e95b@fsol>

On Tue, 9 Jan 2018 17:42:34 +0100
Victor Stinner <victor.stinner at gmail.com>
wrote:
> 2018-01-09 16:17 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> > How do you plan to make numbers comparable if you change the Django
> > version for a given benchmark?  The only solution IMHO is to add a
> > different benchmark.  
> 
> I mostly use performance to compare Python versions, like compare
> Python 3.7 to Python 2.7. If the Django-1.11 benchmark becomes
> different, I'm not able anymore to compare django_template between
> Python 2 and Python 3.

Why would it become different?



From victor.stinner at gmail.com  Tue Jan  9 11:57:13 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jan 2018 17:57:13 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <20180109174724.1885e95b@fsol>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAMpsgwZEngCm9sMiHPLdHaVFU09BsquaF6fyb-iBh_fRB44Mmg@mail.gmail.com>
 <20180109174724.1885e95b@fsol>
Message-ID: <CAMpsgwYb0Uc6NBJqBkhw1mmS5yQEEiUSc_zbRVWoenxA-5aaZA@mail.gmail.com>

I would prefer to not have to install Django 1.11 *and* Django 2.0 in
the same virtual environment (I'm not sure that it's technically
possible). It means that the Django 1.11 benchmark would be specific
to Python 2, whereas the Django 2.0 benchmark would be specific to
Python 3. So with an hypothetical future performance version, it would
become impossible to compare django template benchmark between Python
2 and Python 3. I'm not sure that the Python 2 benchmark would be
useful.

Victor

2018-01-09 17:47 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> On Tue, 9 Jan 2018 17:42:34 +0100
> Victor Stinner <victor.stinner at gmail.com>
> wrote:
>> 2018-01-09 16:17 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
>> > How do you plan to make numbers comparable if you change the Django
>> > version for a given benchmark?  The only solution IMHO is to add a
>> > different benchmark.
>>
>> I mostly use performance to compare Python versions, like compare
>> Python 3.7 to Python 2.7. If the Django-1.11 benchmark becomes
>> different, I'm not able anymore to compare django_template between
>> Python 2 and Python 3.
>
> Why would it become different?
>
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed

From solipsis at pitrou.net  Tue Jan  9 12:01:19 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Jan 2018 18:01:19 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAMpsgwZEngCm9sMiHPLdHaVFU09BsquaF6fyb-iBh_fRB44Mmg@mail.gmail.com>
 <20180109174724.1885e95b@fsol>
 <CAMpsgwYb0Uc6NBJqBkhw1mmS5yQEEiUSc_zbRVWoenxA-5aaZA@mail.gmail.com>
Message-ID: <20180109180119.390a4af3@fsol>

On Tue, 9 Jan 2018 17:57:13 +0100
Victor Stinner <victor.stinner at gmail.com>
wrote:
> I would prefer to not have to install Django 1.11 *and* Django 2.0 in
> the same virtual environment (I'm not sure that it's technically
> possible). It means that the Django 1.11 benchmark would be specific
> to Python 2, whereas the Django 2.0 benchmark would be specific to
> Python 3.

I see, it's limitation due to how the benchmarking script works.  Then
I guess it's up to you :-)

I admit, I find the whole virtual environment thing annoying.  If I
even want to run *one* benchmark, it starts downloading and
installing *every* potentially useful third-party library.

Regards

Antoine.



From solipsis at pitrou.net  Tue Jan  9 11:55:49 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Jan 2018 17:55:49 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAMpsgwbXDaSWeWhiYCPFbN6Ufr2YcAKkpVNkDsDC7kOraJddgw@mail.gmail.com>
 <20180109174657.0d566596@fsol>
 <CAMpsgwaN-9V5Jf=O2CASpdY4X0WHqueTFa-xZ4zuZp2wQH7ALA@mail.gmail.com>
Message-ID: <20180109175549.68eeaab5@fsol>

On Tue, 9 Jan 2018 17:51:36 +0100
Victor Stinner <victor.stinner at gmail.com>
wrote:
> Ok. So would you be ok to simply drop this benchmark? Or does anyone need it?

I don't know, what is the point of dropping it?  It makes it less easy
to compare Python 2 and Python 3 performance, right?

Regards

Antoine.



From berker.peksag at gmail.com  Tue Jan  9 12:11:04 2018
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Tue, 9 Jan 2018 20:11:04 +0300
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <CAMpsgwaN-9V5Jf=O2CASpdY4X0WHqueTFa-xZ4zuZp2wQH7ALA@mail.gmail.com>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAMpsgwbXDaSWeWhiYCPFbN6Ufr2YcAKkpVNkDsDC7kOraJddgw@mail.gmail.com>
 <20180109174657.0d566596@fsol>
 <CAMpsgwaN-9V5Jf=O2CASpdY4X0WHqueTFa-xZ4zuZp2wQH7ALA@mail.gmail.com>
Message-ID: <CAF4280KoG6b37P14x0ox1BmSUFwrV_d7QkTgkC5_9hywFQk2DA@mail.gmail.com>

On Tue, Jan 9, 2018 at 7:51 PM, Victor Stinner <victor.stinner at gmail.com> wrote:
> Ok. So would you be ok to simply drop this benchmark? Or does anyone need it?
>
> http://pyperformance.readthedocs.io/benchmarks.html#django-template
> ---
> Use the Django template system to build a 150x150-cell HTML table.
>
> Use Context and Template classes of the django.template module.
> ---
>
> Other template benchmarks:
>
> * http://pyperformance.readthedocs.io/benchmarks.html#chameleon
> * http://pyperformance.readthedocs.io/benchmarks.html#genshi
> * http://pyperformance.readthedocs.io/benchmarks.html#mako

Django's template system is probably the most popular one out there.
Django 2.0 doesn't introduce a lot of new features or performance
improvements to the template engine and 1.11 is an LTS release. I'd
say keep Django 1.11 for now.

--Berker

From victor.stinner at gmail.com  Tue Jan  9 12:11:28 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jan 2018 18:11:28 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <20180109180119.390a4af3@fsol>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAMpsgwZEngCm9sMiHPLdHaVFU09BsquaF6fyb-iBh_fRB44Mmg@mail.gmail.com>
 <20180109174724.1885e95b@fsol>
 <CAMpsgwYb0Uc6NBJqBkhw1mmS5yQEEiUSc_zbRVWoenxA-5aaZA@mail.gmail.com>
 <20180109180119.390a4af3@fsol>
Message-ID: <CAMpsgwb9ugLo0zCeWfRSCVuTX_FMeNbpwBmGToHkFZw_KJV8MQ@mail.gmail.com>

2018-01-09 18:01 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> I admit, I find the whole virtual environment thing annoying.  If I
> even want to run *one* benchmark, it starts downloading and
> installing *every* potentially useful third-party library.

If Django is installed in your PYTHONPATH, you can run directly the benchmark:

cd performance/benchmarks
python3 bm_django_template.py

The pyperformance script / python3 -m performance is an helper to
store all results in a single file and to install dependencies in a
virtual environment.

Note: you can run multiple benchmarks and write all results into a
single JSON file using --append option instead of -o / --output.

Victor

From victor.stinner at gmail.com  Tue Jan  9 12:13:00 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Jan 2018 18:13:00 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <20180109175549.68eeaab5@fsol>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAMpsgwbXDaSWeWhiYCPFbN6Ufr2YcAKkpVNkDsDC7kOraJddgw@mail.gmail.com>
 <20180109174657.0d566596@fsol>
 <CAMpsgwaN-9V5Jf=O2CASpdY4X0WHqueTFa-xZ4zuZp2wQH7ALA@mail.gmail.com>
 <20180109175549.68eeaab5@fsol>
Message-ID: <CAMpsgwZ3zXonHq4Z=9q2vbhJzUMrPjg=CjLjcEtOGTjm83WDiA@mail.gmail.com>

2018-01-09 17:55 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> I don't know, what is the point of dropping it?

I'm trying to keep pyperformance up to date: always update the next
performance release to latest versions of requirements.

Use an older performance version if you want older requirements.

Victor

From solipsis at pitrou.net  Tue Jan  9 12:16:26 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Jan 2018 18:16:26 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAMpsgwZEngCm9sMiHPLdHaVFU09BsquaF6fyb-iBh_fRB44Mmg@mail.gmail.com>
 <20180109174724.1885e95b@fsol>
 <CAMpsgwYb0Uc6NBJqBkhw1mmS5yQEEiUSc_zbRVWoenxA-5aaZA@mail.gmail.com>
 <20180109180119.390a4af3@fsol>
 <CAMpsgwb9ugLo0zCeWfRSCVuTX_FMeNbpwBmGToHkFZw_KJV8MQ@mail.gmail.com>
Message-ID: <20180109181626.25d780ac@fsol>

On Tue, 9 Jan 2018 18:11:28 +0100
Victor Stinner <victor.stinner at gmail.com>
wrote:
> 2018-01-09 18:01 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> > I admit, I find the whole virtual environment thing annoying.  If I
> > even want to run *one* benchmark, it starts downloading and
> > installing *every* potentially useful third-party library.  
> 
> If Django is installed in your PYTHONPATH, you can run directly the benchmark:
> 
> cd performance/benchmarks
> python3 bm_django_template.py

Thanks, but that's not very functional.

Regards

Antoine.



From solipsis at pitrou.net  Tue Jan  9 12:18:25 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Jan 2018 18:18:25 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAMpsgwbXDaSWeWhiYCPFbN6Ufr2YcAKkpVNkDsDC7kOraJddgw@mail.gmail.com>
 <20180109174657.0d566596@fsol>
 <CAMpsgwaN-9V5Jf=O2CASpdY4X0WHqueTFa-xZ4zuZp2wQH7ALA@mail.gmail.com>
 <20180109175549.68eeaab5@fsol>
 <CAMpsgwZ3zXonHq4Z=9q2vbhJzUMrPjg=CjLjcEtOGTjm83WDiA@mail.gmail.com>
Message-ID: <20180109181825.3c9c1a89@fsol>

On Tue, 9 Jan 2018 18:13:00 +0100
Victor Stinner <victor.stinner at gmail.com>
wrote:
> 2018-01-09 17:55 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> > I don't know, what is the point of dropping it?  
> 
> I'm trying to keep pyperformance up to date: always update the next
> performance release to latest versions of requirements.

But up to date with what?  Just because your benchmark is running with
an older version of Django doesn't mean that the performance
indications it gives are not valid anymore.

Regards

Antoine.



From anto.cuni at gmail.com  Tue Jan  9 15:16:07 2018
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 9 Jan 2018 21:16:07 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <CAMpsgwaN-9V5Jf=O2CASpdY4X0WHqueTFa-xZ4zuZp2wQH7ALA@mail.gmail.com>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAMpsgwbXDaSWeWhiYCPFbN6Ufr2YcAKkpVNkDsDC7kOraJddgw@mail.gmail.com>
 <20180109174657.0d566596@fsol>
 <CAMpsgwaN-9V5Jf=O2CASpdY4X0WHqueTFa-xZ4zuZp2wQH7ALA@mail.gmail.com>
Message-ID: <CAHPg78xjWmxnh1aAmofxofF8ZWKeARszBkOo9gT2tBMrWMoyuQ@mail.gmail.com>

Why do you want to drop it?
It's a piece of Python code which does lots of string manipulation, so it's
a very good benchmark to compare different Python versions and/or different
Python implementations.

If the reason is "this is no longer going to represent the overall
performance of django because the newest version of django uses something
different", then I suggest to simply change the name of the benchmark.


On Tue, Jan 9, 2018 at 5:51 PM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Ok. So would you be ok to simply drop this benchmark? Or does anyone need
> it?
>
> http://pyperformance.readthedocs.io/benchmarks.html#django-template
> ---
> Use the Django template system to build a 150x150-cell HTML table.
>
> Use Context and Template classes of the django.template module.
> ---
>
> Other template benchmarks:
>
> * http://pyperformance.readthedocs.io/benchmarks.html#chameleon
> * http://pyperformance.readthedocs.io/benchmarks.html#genshi
> * http://pyperformance.readthedocs.io/benchmarks.html#mako
>
> Victor
>
> 2018-01-09 17:46 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> > On Tue, 9 Jan 2018 17:39:33 +0100
> > Victor Stinner <victor.stinner at gmail.com>
> > wrote:
> >> 2018-01-09 16:42 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> >> > We already compare different libraries.  For example, pickle is very
> different
> >> > between Python 2.7 and 3.6.
> >> > Even though it's not good for comparing interpreter performance, it's
> good
> >> > for people comparing Python 2 and 3.
> >> >
> >> > If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python
> 2.7,
> >> > it's nice carrot for people moving forward.
> >>
> >> I agree. Antoine: what do you think?
> >
> > I disagree.  pickle is an integral part of Python, it's versioned
> > *with* Python.  Django is not and it's misleading to compare results
> > obtained using two differents of it.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > _______________________________________________
> > Speed mailing list
> > Speed at python.org
> > https://mail.python.org/mailman/listinfo/speed
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20180109/a4d123ae/attachment.html>

From ncoghlan at gmail.com  Tue Jan  9 21:50:26 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 Jan 2018 12:50:26 +1000
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <CAMpsgwYb0Uc6NBJqBkhw1mmS5yQEEiUSc_zbRVWoenxA-5aaZA@mail.gmail.com>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAMpsgwZEngCm9sMiHPLdHaVFU09BsquaF6fyb-iBh_fRB44Mmg@mail.gmail.com>
 <20180109174724.1885e95b@fsol>
 <CAMpsgwYb0Uc6NBJqBkhw1mmS5yQEEiUSc_zbRVWoenxA-5aaZA@mail.gmail.com>
Message-ID: <CADiSq7cJT0XA7P=eZeiXrcUpT0Wszy+E0FYiTjaYXdVtk61dDQ@mail.gmail.com>

On 10 January 2018 at 02:57, Victor Stinner <victor.stinner at gmail.com> wrote:
> I would prefer to not have to install Django 1.11 *and* Django 2.0 in
> the same virtual environment (I'm not sure that it's technically
> possible). It means that the Django 1.11 benchmark would be specific
> to Python 2, whereas the Django 2.0 benchmark would be specific to
> Python 3. So with an hypothetical future performance version, it would
> become impossible to compare django template benchmark between Python
> 2 and Python 3. I'm not sure that the Python 2 benchmark would be
> useful.

The point of the benchmark is to test non-trivial string manipulation,
not to test the latest version of Django's template engine.

Just lock the dependency at Django 1.11, and explain why.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Jan 10 19:36:16 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 11 Jan 2018 10:36:16 +1000
Subject: [Speed] Impact of Meltdown/Spectre OS patches on benchmark results?
Message-ID: <CADiSq7enSyXjk=X8GUYuTTPp-Rk1sQ=uf140j6+31kXpUgqYXQ@mail.gmail.com>

Hi folks,

Reading https://medium.com/implodinggradients/meltdown-c24a9d5e254e
prompts me to ask: are speed.python.org benchmark results produced now
actually going to be comparable with those executed last year?

Or will the old results need to be backfilled again with the new
baseline OS performance?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From paul at paulgraydon.co.uk  Wed Jan 10 23:54:36 2018
From: paul at paulgraydon.co.uk (Paul Graydon)
Date: Wed, 10 Jan 2018 20:54:36 -0800
Subject: [Speed] Impact of Meltdown/Spectre OS patches on benchmark
 results?
In-Reply-To: <CADiSq7enSyXjk=X8GUYuTTPp-Rk1sQ=uf140j6+31kXpUgqYXQ@mail.gmail.com>
References: <CADiSq7enSyXjk=X8GUYuTTPp-Rk1sQ=uf140j6+31kXpUgqYXQ@mail.gmail.com>
Message-ID: <75CF2C44-3BF1-4C3B-B1FF-60324F6F3F89@paulgraydon.co.uk>

That's not a great comparison there.  They've gone from a 2+ year old, somewhat customised kernel, and then compared it to a bleeding edge vanilla upstream kernel.

It's worth pointing out that Ubuntu has currently opted to only put in Meltdown patches with the kernel they released yesterday.  Spectre stuff is still to come. (https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown <https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown>) so that benchmark probably won't reflect what anyone actually sees anyway.

Impact is going to be very variable based on what is being benchmarked.  I've been doing a whole bunch of benchmarking at work as part of some broad investigations of the impact for our customers, leveraging a few different resources.
I haven't grabbed and run the speed.python.org <http://speed.python.org/> benchmarks but I may be able to spin up a bare metal CentOS instance tomorrow and run through the paces with a pre/post Spectre and Meltdown patched kernel, if it's straightforward to do. (are there any documents people could point me to with instructions?)

I'd consider the patches here to largely invalidate any historical performance benchmarks for a comparison perspective.  Even on a CPU with PCID capabilities there is definitely a hit on certain types of operations.

We're also very much in the "patch to make it secure" phase, with an optimisation phase still to come, and it's not certain what Intel, AMD, and/or ARM are going to push out by way of updates that may mitigate the impact.  There's going to be a lot of focus on improving the performance situation over the next while, so the odds are reasonable that full historical benchmarks will need to be run repeatedly, I guess?

Paul

 
On January 11, 2018 12:36:16 AM UTC, Nick Coghlan <ncoghlan at gmail.com> wrote:
Hi folks,

Reading https://medium.com/implodinggradients/meltdown-c24a9d5e254e <https://medium.com/implodinggradients/meltdown-c24a9d5e254e>
prompts me to ask: are speed.python.org benchmark results produced now
actually going to be comparable with those executed last year?

Or will the old results need to be backfilled again with the new
baseline OS performance?

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20180110/c4026fd7/attachment.html>

From victor.stinner at gmail.com  Thu Jan 11 04:36:19 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 11 Jan 2018 10:36:19 +0100
Subject: [Speed] performance 0.6.1 released
Message-ID: <CAMpsgwakdfb-NWeKLmi7gJRAxDiEX55Bc197NAmMgQ2u6bC2-Q@mail.gmail.com>

Hi,

I'm happy to announce the release of performance 0.6.1:

http://pyperformance.readthedocs.io/

Changes:

* Fix inherit-environ: propagate to recursive invocations of ``performance``
  in ``compile`` and ``compile_all`` commands.
* Fix the ``--track-memory`` option thanks to the update to perf 1.5.
* Update requirements

  - certifi: 2017.4.17 => 2017.11.5
  - Chameleon: 3.1 => 3.2
  - Django: 1.11.3 => 1.11.9
  - docutils: 0.13.1 => 0.14
  - dulwich: 0.17.3 => 0.18.6
  - html5lib: 0.999999999 => 1.0.1
  - Mako: 1.0.6 => 1.0.7
  - mercurial: 4.2.2 => 4.4.2
  - mpmath: 0.19 => 1.0.0
  - perf: 1.4 => 1.5.1 (fix ``--track-memory`` option)
  - psutil: 5.2.2 => 5.4.3
  - pyaes: 1.6.0 => 1.6.1
  - six: 1.10.0 => 1.11.0
  - SQLAlchemy: 1.1.11 => 1.2.0
  - sympy: 1.0 => 1.1.1
  - tornado: 4.5.1 => 4.5.3

Note: Django wasn't upgraded to 2.0 to keep Python 2 compatibility.

Victor

From solipsis at pitrou.net  Thu Jan 11 07:12:40 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 11 Jan 2018 13:12:40 +0100
Subject: [Speed] Impact of Meltdown/Spectre OS patches on benchmark
 results?
References: <CADiSq7enSyXjk=X8GUYuTTPp-Rk1sQ=uf140j6+31kXpUgqYXQ@mail.gmail.com>
Message-ID: <20180111131240.0df9936d@fsol>

On Thu, 11 Jan 2018 10:36:16 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> Hi folks,
> 
> Reading https://medium.com/implodinggradients/meltdown-c24a9d5e254e
> prompts me to ask: are speed.python.org benchmark results produced now
> actually going to be comparable with those executed last year?
> 
> Or will the old results need to be backfilled again with the new
> baseline OS performance?

AFAIU the Meltdown patches will only significantly affect workloads that
do a lot of system calls. *Ideally* our benchmark suite doesn't, since
it's meant to benchmark Python, not the outside environment :-)

Spectre seems to be more uncertain, as it can affect any code
(userspace code as well as kernel code), and will apparently require
careful auditing of sensitive routines to avoid potential information
leaks.  This all depends which particular routines (e.g. something in
glibc) get patched.

In any case we can only tell with actual numbers :-)

Regards

Antoine.



From songofacandy at gmail.com  Tue Jan 16 06:37:08 2018
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 16 Jan 2018 20:37:08 +0900
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
Message-ID: <CAEfz+TxN6v1HiPU-caZ+q7ehybZ8dwhL7qObZvfJvCC8+GGG3g@mail.gmail.com>

> Even though it's not good for comparing interpreter performance, it's good
> for people comparing Python 2 and 3.
>
> If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python 2.7,
> it's nice carrot for people moving forward.
>

FYI, Django 2 is about 2x faster than 1.11 on django_template benchmark!
It's because Django 1.11 calls force_text() many times for Python 2
compatibility.

https://github.com/django/django/blob/419705bbe84e27c3d5be85f198a0352a6724927e/django/utils/encoding.py#L58
https://github.com/django/django/blob/419705bbe84e27c3d5be85f198a0352a6724927e/django/template/defaulttags.py#L225

Actually, dropping Python 2 support makes Django faster.  It's nice news!

Bests,

From stefan_ml at behnel.de  Fri Jan 19 05:28:10 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jan 2018 11:28:10 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <CAEfz+TxN6v1HiPU-caZ+q7ehybZ8dwhL7qObZvfJvCC8+GGG3g@mail.gmail.com>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAEfz+TxN6v1HiPU-caZ+q7ehybZ8dwhL7qObZvfJvCC8+GGG3g@mail.gmail.com>
Message-ID: <3f9d41fe-7f01-4338-2640-55b9dfb594a9@behnel.de>

INADA Naoki schrieb am 16.01.2018 um 12:37:
>> Even though it's not good for comparing interpreter performance, it's good
>> for people comparing Python 2 and 3.
>>
>> If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python 2.7,
>> it's nice carrot for people moving forward.
>>
> 
> FYI, Django 2 is about 2x faster than 1.11 on django_template benchmark!
> It's because Django 1.11 calls force_text() many times for Python 2
> compatibility.
> 
> https://github.com/django/django/blob/419705bbe84e27c3d5be85f198a0352a6724927e/django/utils/encoding.py#L58
> https://github.com/django/django/blob/419705bbe84e27c3d5be85f198a0352a6724927e/django/template/defaulttags.py#L225
> 
> Actually, dropping Python 2 support makes Django faster.  It's nice news!

That suggests adding Django 2 as a new Py3-only benchmark.

Stefan

From victor.stinner at gmail.com  Fri Jan 19 05:39:31 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 19 Jan 2018 11:39:31 +0100
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <3f9d41fe-7f01-4338-2640-55b9dfb594a9@behnel.de>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAEfz+TxN6v1HiPU-caZ+q7ehybZ8dwhL7qObZvfJvCC8+GGG3g@mail.gmail.com>
 <3f9d41fe-7f01-4338-2640-55b9dfb594a9@behnel.de>
Message-ID: <CAMpsgwbzn+O_EesOo1kfo5GfFQHOtpcxPo8Qv22Tt4Y5RS3TdA@mail.gmail.com>

2018-01-19 11:28 GMT+01:00 Stefan Behnel <stefan_ml at behnel.de>:
> That suggests adding Django 2 as a new Py3-only benchmark.

Again, the practical issue is to install Django 2 and Django 1.11 in
the same virtual environment. I'm not sure that it's doable.

I would prefer to not have to create a different virtualenv for
Python3-only dependencies.

I needed to release quickly a bugfix release, fix --track-memory,
feature asked by Xiang Zhang, so I released performance 0.6.1 which
only updated Django from 1.11.3 to 1.11.9.

Or we need to redesign how performance install dependencies, but
that's a larger project :-)

Victor

From ncoghlan at gmail.com  Sat Jan 27 08:39:10 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 27 Jan 2018 23:39:10 +1000
Subject: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7
 and PyPy2?
In-Reply-To: <CAMpsgwbzn+O_EesOo1kfo5GfFQHOtpcxPo8Qv22Tt4Y5RS3TdA@mail.gmail.com>
References: <CAMpsgwaEHyU1F8-at2F36g+gJ3mwg_sz2ZbpE+Z-S0B5sa480w@mail.gmail.com>
 <20180109161757.1fa74a52@fsol>
 <CAEfz+Tw6aRjqGJYPXZ4B=ODeYiNh+cF20Fzp=s7MJnX_U-ibMQ@mail.gmail.com>
 <CAEfz+TxN6v1HiPU-caZ+q7ehybZ8dwhL7qObZvfJvCC8+GGG3g@mail.gmail.com>
 <3f9d41fe-7f01-4338-2640-55b9dfb594a9@behnel.de>
 <CAMpsgwbzn+O_EesOo1kfo5GfFQHOtpcxPo8Qv22Tt4Y5RS3TdA@mail.gmail.com>
Message-ID: <CADiSq7dSrir9u2yC1scPQcuUtovz+e5Tq15Pi2vCv-HBQhXyvQ@mail.gmail.com>

On 19 January 2018 at 20:39, Victor Stinner <victor.stinner at gmail.com>
wrote:

> 2018-01-19 11:28 GMT+01:00 Stefan Behnel <stefan_ml at behnel.de>:
> > That suggests adding Django 2 as a new Py3-only benchmark.
>
> Again, the practical issue is to install Django 2 and Django 1.11 in
> the same virtual environment. I'm not sure that it's doable.
>
> I would prefer to not have to create a different virtualenv for
> Python3-only dependencies.
>
> I needed to release quickly a bugfix release, fix --track-memory,
> feature asked by Xiang Zhang, so I released performance 0.6.1 which
> only updated Django from 1.11.3 to 1.11.9.
>
> Or we need to redesign how performance install dependencies, but
> that's a larger project :-)
>

It may be worth looking at using pew to set up a separate virtual
environment for each benchmark, but then use `pew add` to share common
components (like perf itself) between them. That way you won't have
conflicting dependencies between benchmarks (since they'll be in separate
venvs), without having to duplicate *all* the common components.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20180127/662f4ada/attachment.html>

From matti.picus at gmail.com  Tue Jan 30 15:37:52 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 30 Jan 2018 22:37:52 +0200
Subject: [Speed] adding a pypy runner for speed.python.org
Message-ID: <3296f52d-dd6b-1e69-2c4a-b7d4e3216bf2@gmail.com>

I am cross-posting to both speed and pypy-dev to ask what needs to be 
done to get pypy2 and pypy3 benchmark runners onto speed.python.org
I am willing to be the contact person from the PyPy side, who do we need 
to talk to on the speed maintainers' side?
Instead of spamming the lists again, we could discuss this off-line, and 
report back with a plan, required resources, and a timeline.

Thanks,
Matti Picus

From victor.stinner at gmail.com  Wed Jan 31 08:47:01 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 Jan 2018 14:47:01 +0100
Subject: [Speed] adding a pypy runner for speed.python.org
In-Reply-To: <3296f52d-dd6b-1e69-2c4a-b7d4e3216bf2@gmail.com>
References: <3296f52d-dd6b-1e69-2c4a-b7d4e3216bf2@gmail.com>
Message-ID: <CAMpsgwZbCByt5g4jKLRquSnRn1-AR2xD38+j10ONFTRszP_4MQ@mail.gmail.com>

Hi,

I tried but failed to find someone in PyPy to adjust performance
benchmarks for PyPy. Currently, the JIT is not properly warmed up, and
the results can be dishonnest or not reliable.

My latest attempt to support is PyPy is:
http://vstinner.readthedocs.io/pypy_warmups.html

IMHO we need to develop a statistical methodology in perf to compute
when values become stable. That's hard to define and it was proven
that "performance stability" doesn't exist (see " Virtual Machine
Warmup Blows Hot and Cold" paper).

Fijal from PyPy would like to use hardcoded configuration for the
number of warmup values. I like the idea, but nobody implemented it
yet.

Victor

2018-01-30 21:37 GMT+01:00 Matti Picus <matti.picus at gmail.com>:
> I am cross-posting to both speed and pypy-dev to ask what needs to be done
> to get pypy2 and pypy3 benchmark runners onto speed.python.org
> I am willing to be the contact person from the PyPy side, who do we need to
> talk to on the speed maintainers' side?
> Instead of spamming the lists again, we could discuss this off-line, and
> report back with a plan, required resources, and a timeline.
>
> Thanks,
> Matti Picus
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed

From tobami at gmail.com  Wed Jan 31 09:14:13 2018
From: tobami at gmail.com (Miquel Torres)
Date: Wed, 31 Jan 2018 14:14:13 +0000
Subject: [Speed] adding a pypy runner for speed.python.org
In-Reply-To: <CAMpsgwZbCByt5g4jKLRquSnRn1-AR2xD38+j10ONFTRszP_4MQ@mail.gmail.com>
References: <3296f52d-dd6b-1e69-2c4a-b7d4e3216bf2@gmail.com>
 <CAMpsgwZbCByt5g4jKLRquSnRn1-AR2xD38+j10ONFTRszP_4MQ@mail.gmail.com>
Message-ID: <CAGf+9VzmS9zt7O12VnzKOWto_F==RN8fuWGjaUYRLBd1A=Ag0A@mail.gmail.com>

Hi Matti,

I'll gladly help setting up the speed site

El El mi?, 31 ene 2018 a las 13:47, Victor Stinner <victor.stinner at gmail.com>
escribi?:

> Hi,
>
> I tried but failed to find someone in PyPy to adjust performance
> benchmarks for PyPy. Currently, the JIT is not properly warmed up, and
> the results can be dishonnest or not reliable.
>
> My latest attempt to support is PyPy is:
> http://vstinner.readthedocs.io/pypy_warmups.html
>
> IMHO we need to develop a statistical methodology in perf to compute
> when values become stable. That's hard to define and it was proven
> that "performance stability" doesn't exist (see " Virtual Machine
> Warmup Blows Hot and Cold" paper).
>
> Fijal from PyPy would like to use hardcoded configuration for the
> number of warmup values. I like the idea, but nobody implemented it
> yet.
>
> Victor
>
> 2018-01-30 21:37 GMT+01:00 Matti Picus <matti.picus at gmail.com>:
> > I am cross-posting to both speed and pypy-dev to ask what needs to be
> done
> > to get pypy2 and pypy3 benchmark runners onto speed.python.org
> > I am willing to be the contact person from the PyPy side, who do we need
> to
> > talk to on the speed maintainers' side?
> > Instead of spamming the lists again, we could discuss this off-line, and
> > report back with a plan, required resources, and a timeline.
> >
> > Thanks,
> > Matti Picus
> > _______________________________________________
> > Speed mailing list
> > Speed at python.org
> > https://mail.python.org/mailman/listinfo/speed
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20180131/9fccd359/attachment.html>