From eva_maia at sapo.pt Tue Feb 2 15:06:45 2010 From: eva_maia at sapo.pt (eva_maia at sapo.pt) Date: Tue, 02 Feb 2010 14:06:45 +0000 Subject: [pypy-dev] Type inference of PyPy Message-ID: <20100202140645.92365nra6hdfzwcg@w18.mail.sapo.pt> Hi, I'm a Masters student in CS at University of Porto and the purpose of my thesis is the development of a Why module for Python. Why is a software verification platform (VCG). There is already such tools for C (frama-c) and Java (krakatoa). To that end, I need to infer the type of program variables. I'd like to use type inference of PyPy (for RPython). I know there is a statement (x.knowntype) that given the type of the arguments it gives the type of return variable, but I want to know if it is also possible to access the type of local variables. Thanks, Eva. From arigo at tunes.org Tue Feb 2 17:17:51 2010 From: arigo at tunes.org (Armin Rigo) Date: Tue, 2 Feb 2010 17:17:51 +0100 Subject: [pypy-dev] Type inference of PyPy In-Reply-To: <20100202140645.92365nra6hdfzwcg@w18.mail.sapo.pt> References: <20100202140645.92365nra6hdfzwcg@w18.mail.sapo.pt> Message-ID: <20100202161751.GA10351@code0.codespeak.net> Hi, On Tue, Feb 02, 2010 at 02:06:45PM +0000, eva_maia at sapo.pt wrote: > To that end, I need to infer the type of program variables. I'd like > to use type inference of PyPy (for RPython). Two warnings before you go any further (I guess you already know about it but it doesn't hurt repeating it): 1. You can't infer the type of most program variables in Python: http://codespeak.net/pypy/dist/pypy/doc/faq.html#can-pypy-compile-normal-python-programs 2. PyPy's type inference only works for RPython, not for Python programs. > I know there is a statement (x.knowntype) that given the type of the > arguments it gives the type of return variable, but I want to know if > it is also possible to access the type of local variables. In the RPythonAnnotator instance there is a dictionary called 'bindings' that maps local variables to the corresponding SomeXxx instances. What you don't get "out of the box" is the mapping from the real local variables in your Python source code -- only from Variable instances, which are produced by the flow object space. A bientot, Armin. From reynaudd at loria.fr Mon Feb 8 20:52:14 2010 From: reynaudd at loria.fr (Daniel Reynaud) Date: Mon, 8 Feb 2010 20:52:14 +0100 Subject: [pypy-dev] virtual machine translation tutorial Message-ID: <6dbefc261002081152m8fd8a88ofa7cbc2076b77136@mail.gmail.com> Hello pypy folk, For the past 2 days, I have struggled with virtual machine translation and JIT'ing. I relied heavily on the patience of everyone on #pypy, so I tried to aggregate the accumulated knowledge in a blog post. Here it is: http://indefinitestudies.org/2010/02/08/creating-a-toy-virtual-machine-with-pypy/ Cheers, dan From cfbolz at gmx.de Thu Feb 11 13:15:19 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 11 Feb 2010 13:15:19 +0100 Subject: [pypy-dev] =?iso-8859-1?q?cobra_server_at_the_Uni_D=FCsseldorf?= Message-ID: <4B73F4D7.1040508@gmx.de> Hi all, we need to do some server restructuring at the university and in the process need to use Cobra for other things (we will get some other machine back after the reshuffling). Sorry about the disturbance. Could people please check their home directories for things that need to be kept and move them elsewhere? We plan to shut Cobra down on the 19th of February, if somebody cannot make it till then, please send me a mail. Cheers, Carl Friedrich From ncbray at gmail.com Thu Feb 18 21:36:39 2010 From: ncbray at gmail.com (Nick Bray) Date: Thu, 18 Feb 2010 14:36:39 -0600 Subject: [pypy-dev] Modeling attributes in the annotator Message-ID: <5d66556e1002181236g2b3ec989kbeeee713430ab90a@mail.gmail.com> I'm doing a lit review on Python analysis techniques. Right now I'm trying to understand how PyPy's annotator models object attributes. Are the descriptions on the web page accurate? There are two corner cases that I do not understand how the annotator, as described, deals with. 1) o.f = 1 ??? = o.__dict__['f'] 2) Any attribute lookup involving a descriptor that is not a function object. I realize the annotator is only intended to work for RPython. Given the information I have, however, I do not see what restrictions in RPython would disallow these cases. ("They just are" is an acceptable answer, I just want to make sure I'm not being particularly dense and missing something.) From benjamin at python.org Thu Feb 18 22:12:58 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 18 Feb 2010 15:12:58 -0600 Subject: [pypy-dev] Modeling attributes in the annotator In-Reply-To: <5d66556e1002181236g2b3ec989kbeeee713430ab90a@mail.gmail.com> References: <5d66556e1002181236g2b3ec989kbeeee713430ab90a@mail.gmail.com> Message-ID: <1afaf6161002181312p427d10aco474f339216f65133@mail.gmail.com> 2010/2/18 Nick Bray : > I'm doing a lit review on Python analysis techniques. ?Right now I'm > trying to understand how PyPy's annotator models object attributes. > Are the descriptions on the web page accurate? ?There are two corner > cases that I do not understand how the annotator, as described, deals > with. > > 1) > o.f = 1 > ??? = o.__dict__['f'] > > 2) Any attribute lookup involving a descriptor that is not a function object. > > I realize the annotator is only intended to work for RPython. Given > the information I have, however, I do not see what restrictions in > RPython would disallow these cases. ?("They just are" is an acceptable > answer, I just want to make sure I'm not being particularly dense and > missing something.) They just are. __dict__ isn't RPython. -- Regards, Benjamin From fijall at gmail.com Thu Feb 18 22:58:20 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 18 Feb 2010 16:58:20 -0500 Subject: [pypy-dev] Modeling attributes in the annotator In-Reply-To: <1afaf6161002181312p427d10aco474f339216f65133@mail.gmail.com> References: <5d66556e1002181236g2b3ec989kbeeee713430ab90a@mail.gmail.com> <1afaf6161002181312p427d10aco474f339216f65133@mail.gmail.com> Message-ID: <693bc9ab1002181358r132078b6x4c02dc6b5216fc7@mail.gmail.com> On Thu, Feb 18, 2010 at 4:12 PM, Benjamin Peterson wrote: > 2010/2/18 Nick Bray : >> I'm doing a lit review on Python analysis techniques. ?Right now I'm >> trying to understand how PyPy's annotator models object attributes. >> Are the descriptions on the web page accurate? ?There are two corner >> cases that I do not understand how the annotator, as described, deals >> with. >> >> 1) >> o.f = 1 >> ??? = o.__dict__['f'] >> >> 2) Any attribute lookup involving a descriptor that is not a function object. >> >> I realize the annotator is only intended to work for RPython. Given >> the information I have, however, I do not see what restrictions in >> RPython would disallow these cases. ?("They just are" is an acceptable >> answer, I just want to make sure I'm not being particularly dense and >> missing something.) > > They just are. __dict__ isn't RPython. > > > > -- > Regards, > Benjamin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev There is a bit of documentation on RPython. http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python From arigo at tunes.org Mon Feb 22 16:05:12 2010 From: arigo at tunes.org (Armin Rigo) Date: Mon, 22 Feb 2010 16:05:12 +0100 Subject: [pypy-dev] Graphs moved: cobra->wyvern Message-ID: <20100222150511.GA27169@code0.codespeak.net> Hi all, The 'cobra' machine is going down, so I moved the graph plot servers on 'wyvern'. For a reminder, they are: without JIT: http://wyvern.cs.uni-duesseldorf.de:8098/index.html with JIT: http://codespeak.net/pypy/jitplots.html A bientot, Armin. From arigo at tunes.org Thu Feb 25 10:18:04 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 25 Feb 2010 10:18:04 +0100 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: References: <20091002114028.GK15455@trillke.net> Message-ID: <20100225091804.GA15891@code0.codespeak.net> Hi Miquel, May I point out a couple of comments about http://speed.pypy.org/ ? The first is that it looks great, indeed; thank you very much for doing such a site! :-) The comments are mostly about the graphics in /timeline: * they should also display, or allow to display, the speed of CPython for comparison; * they should have a longer maximum history than 100, to see more clearly the long-term evolution. All in all it looks great! A bientot, Armin. From santagada at gmail.com Thu Feb 25 14:39:12 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 25 Feb 2010 10:39:12 -0300 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: <20100225091804.GA15891@code0.codespeak.net> References: <20091002114028.GK15455@trillke.net> <20100225091804.GA15891@code0.codespeak.net> Message-ID: I also have a bunch of comments, all are my opinion and should not be taken as demands (or even as a good review). - When you first visit the site I think it would be better to be on the timeline like on http://buildbot.pypy.org/plotsummary.html that shows all benchmarks on the same page, but just the last 50 revisions or it becomes too hard to see the recent improvements. (being able to show more history is important too, maybe there is a way to show the trend and the last x revisions clearly). - Overview window comments: - rename the column "result" to "time" (or timing?); - create a new column called "previous time" with the last time of the previous revision. - rename "current change" to "improvement" and invert the values (instead of a green -2% you end up with a green 2%); - with the last two ones it will make understanding the improvement and trends a lot easier; - trend should have the number of revisions on the label (and the values reversed like the "improvement" column); - rename the column "times cpython" to something else (eg. "compared to cpython") and maybe have values like "2x slow down" and "2.5x speedup"; - the graph should say what it is graphing; - maybe invert the order of results. - Timeline window comments: - showing all benchmarks in the same page would be good - Pypy with jit should come first in the list of interpreters, and while you only test with hybrid gc there is no need to state that explicitly; - see all benchmarks together, so you don't have to hunt around (maybe have a detailed view); - describe all axis; - as armin said have a selected by default cpython comparison so people can have a baseline. I think this speed center could be used for the common mercurial benchmark repository that is being setup, that would be very very cool. On Feb 25, 2010, at 6:18 AM, Armin Rigo wrote: > Hi Miquel, > > May I point out a couple of comments about http://speed.pypy.org/ ? > The first is that it looks great, indeed; thank you very much for doing > such a site! :-) > > The comments are mostly about the graphics in /timeline: > > * they should also display, or allow to display, the speed of CPython > for comparison; > > * they should have a longer maximum history than 100, to see more > clearly the long-term evolution. > > > All in all it looks great! > > A bientot, > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- Leonardo Santagada santagada at gmail.com From tobami at googlemail.com Thu Feb 25 16:10:16 2010 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 25 Feb 2010 16:10:16 +0100 Subject: [pypy-dev] speed.pypy.org launched Message-ID: As some of you already know, there is a new performance site online: speed.pypy.org. In Autumn last year I offered my help with a benchmarking website, and eventually for a new main site for the project. What you can see now is a work in progress, which I hope to improve in the next months. I called it Codespeed, and in time it could become a framework for other open source projects to use (for example javascript implementation development, connected to a forge, etc...). But right now, speed has two goals: - To have a powerful tool for the devs to analyse performance and detect regressions in the different pypy interpreters. - To improve the visibility of the project by letting python developers(users) have a look themselves at current performance and to compare pypy to other implementations, not having to wait for some dev to manually put together the graphs and post them on the blog. To that end, there are currently two views. The Overview does what it's name says, and allows a developer to have a broad view of how last revisions are affecting performance, and immediately spot regressions. If the user wants to closer examine the changes in the performance of a given benchmark, just clicking on it takes you to the the timeline for the selected benchmark. Both views will get some more features, and I have further plans down the pipeline: - A Comparison tab: graph bars directly comparing performance of different releases (pypy 1.2, trunk, cpython 2.x, unladen swallow, Jython, etc.) - A statistics tab: Go through regression test literature to see if we can have good metrics shown. - RSS feed/email alerts (for big regressions) - svn info integration - etc... In the end it is a site for the pypy devs, so I hope to get a lot of feedback from you so that it caters exactly to your needs. Have a nice day! Miquel PD: When it is a bit more complete I'll write a post for the PyPy status blog so it gets wider known. From tobami at googlemail.com Thu Feb 25 16:00:28 2010 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 25 Feb 2010 16:00:28 +0100 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: References: <20091002114028.GK15455@trillke.net> <20100225091804.GA15891@code0.codespeak.net> Message-ID: Thanks for the nice comments. I was waiting for the test hook to be implemented before emailing pypy-dev about it, but seeing as It already got out I may do it now as well ;-) @Armin >* they should also display, or allow to display, the speed of CPython >for comparison; Already planned. >* they should have a longer maximum history than 100, to see more >clearly the long-term evolution. Done @Leonardo phew! those were a lot of suggestions, thanks. I will start implementing some of them, though I would like the changes to be discussed so that they get done right and all agree on what's best (in the cases where it can't be an option). I already implemented a couple of changes suggested by fijal: logarithmic scale for the overview bars and variable width so it fits on small screens, down to 980px width. I hope he is happy now :-) I'll start a new thread for the explanation and plans. Cheers! 2010/2/25 Leonardo Santagada : > I also have a bunch of comments, all are my opinion and should not be taken as demands (or even as a good review). > > - When you first visit the site I think it would be better to be on the timeline like on http://buildbot.pypy.org/plotsummary.html that shows all benchmarks on the same page, but just the last 50 revisions or it becomes too hard to see the recent improvements. (being able to show more history is important too, maybe there is a way to show the trend and the last x revisions clearly). > > - Overview window comments: > ? ? ? ?- rename the column "result" to "time" (or timing?); > ? ? ? ?- create a new column called "previous time" with the last time of the previous revision. > ? ? ? ?- rename "current change" to "improvement" and invert the values (instead of a green -2% you end up with a green 2%); > ? ? ? ?- with the last two ones it will make understanding the improvement and trends a lot easier; > ? ? ? ?- trend should have the number of revisions on the label (and the values reversed like the "improvement" column); > ? ? ? ?- rename the column "times cpython" to something else (eg. "compared to cpython") and maybe have values like "2x slow down" and "2.5x speedup"; > ? ? ? ?- the graph should say what it is graphing; > ? ? ? ?- maybe invert the order of results. > > - Timeline window comments: > ? ? ? ?- showing all benchmarks in the same page would be good > ? ? ? ?- Pypy with jit should come first in the list of interpreters, and while you only test with hybrid gc there is no need to state that explicitly; > ? ? ? ?- see all benchmarks together, so you don't have to hunt around (maybe have a detailed view); > ? ? ? ?- describe all axis; > ? ? ? ?- as armin said have a selected by default cpython comparison so people can have a baseline. > > > I think this speed center could be used for the common mercurial benchmark repository that is being setup, that would be very very cool. > > > On Feb 25, 2010, at 6:18 AM, Armin Rigo wrote: > >> Hi Miquel, >> >> May I point out a couple of comments about http://speed.pypy.org/ ? >> The first is that it looks great, indeed; thank you very much for doing >> such a site! :-) >> >> The comments are mostly about the graphics in /timeline: >> >> * they should also display, or allow to display, the speed of CPython >> ?for comparison; >> >> * they should have a longer maximum history than 100, to see more >> ?clearly the long-term evolution. >> >> >> All in all it looks great! >> >> A bientot, >> Armin. >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > -- > Leonardo Santagada > santagada at gmail.com > > > > From tobami at googlemail.com Thu Feb 25 18:05:16 2010 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 25 Feb 2010 18:05:16 +0100 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: References: <20091002114028.GK15455@trillke.net> <20100225091804.GA15891@code0.codespeak.net> Message-ID: I have implemented some of Leonardo's suggestions: >- Overview window comments: > - rename the column "result" to "time" (or timing?); done > - trend should have the number of revisions on the label (and the values reversed like the "improvement" column); done > - rename the column "times cpython" to something else (eg. "compared to cpython") and maybe have values like "2x slow down" and "2.5x speedup"; > - the graph should say what it is graphing; it is clearer now, plus there are some tooltips on hover. >- Timeline window comments: > - Pypy with jit should come first in the list of interpreters, and while you only test with hybrid gc there is no need to state that explicitly; only interpreter name shown now. (order implemented but not commited). I'll see about the rest when I have time. Miquel 2010/2/25 Leonardo Santagada : > I also have a bunch of comments, all are my opinion and should not be taken as demands (or even as a good review). > > - When you first visit the site I think it would be better to be on the timeline like on http://buildbot.pypy.org/plotsummary.html that shows all benchmarks on the same page, but just the last 50 revisions or it becomes too hard to see the recent improvements. (being able to show more history is important too, maybe there is a way to show the trend and the last x revisions clearly). > > - Overview window comments: > ? ? ? ?- rename the column "result" to "time" (or timing?); > ? ? ? ?- create a new column called "previous time" with the last time of the previous revision. > ? ? ? ?- rename "current change" to "improvement" and invert the values (instead of a green -2% you end up with a green 2%); > ? ? ? ?- with the last two ones it will make understanding the improvement and trends a lot easier; > ? ? ? ?- trend should have the number of revisions on the label (and the values reversed like the "improvement" column); > ? ? ? ?- rename the column "times cpython" to something else (eg. "compared to cpython") and maybe have values like "2x slow down" and "2.5x speedup"; > ? ? ? ?- the graph should say what it is graphing; > ? ? ? ?- maybe invert the order of results. > > - Timeline window comments: > ? ? ? ?- showing all benchmarks in the same page would be good > ? ? ? ?- Pypy with jit should come first in the list of interpreters, and while you only test with hybrid gc there is no need to state that explicitly; > ? ? ? ?- see all benchmarks together, so you don't have to hunt around (maybe have a detailed view); > ? ? ? ?- describe all axis; > ? ? ? ?- as armin said have a selected by default cpython comparison so people can have a baseline. > > > I think this speed center could be used for the common mercurial benchmark repository that is being setup, that would be very very cool. > > > On Feb 25, 2010, at 6:18 AM, Armin Rigo wrote: > >> Hi Miquel, >> >> May I point out a couple of comments about http://speed.pypy.org/ ? >> The first is that it looks great, indeed; thank you very much for doing >> such a site! :-) >> >> The comments are mostly about the graphics in /timeline: >> >> * they should also display, or allow to display, the speed of CPython >> ?for comparison; >> >> * they should have a longer maximum history than 100, to see more >> ?clearly the long-term evolution. >> >> >> All in all it looks great! >> >> A bientot, >> Armin. >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > -- > Leonardo Santagada > santagada at gmail.com > > > > From cfbolz at gmx.de Thu Feb 25 18:38:16 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 25 Feb 2010 18:38:16 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: Message-ID: <4B86B588.4000601@gmx.de> Hi Miquel, On 02/25/2010 04:10 PM, Miquel Torres wrote: > As some of you already know, there is a new performance site online: > speed.pypy.org. [...] I'm quite impressed, this is very cool work and a good improvement over the current plots. Thanks for doing this! One thing that I would really find useful are error bars in the timeline view. This would help us judge whether up-and-down movements are within the margin of randomness, or whether it is a real effect. I don't know how annoying they are to implement though, no clue whether the plot lib supports that. There should be enough information about errors in the json files, as far as I remember. Cheers, Carl Friedrich From tobami at googlemail.com Thu Feb 25 22:10:29 2010 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 25 Feb 2010 22:10:29 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: <4B86B588.4000601@gmx.de> References: <4B86B588.4000601@gmx.de> Message-ID: Hi Carl, error bars are certainly doable and supported by the plot lib. The only problem is that my backend model doesn't include such a field, so at the moment the data (which is certainly included in the json files) is not being saved. I will need to change the DB models in order to acomodate that kind of information. I'll add this to the wish list. Cheers, Miquel 2010/2/25 Carl Friedrich Bolz : > Hi Miquel, > > On 02/25/2010 04:10 PM, Miquel Torres wrote: >> As some of you already know, there is a new performance site online: >> speed.pypy.org. > [...] > > I'm quite impressed, this is very cool work and a good improvement over > the current plots. Thanks for doing this! > > One thing that I would really find useful are error bars in the timeline > view. This would help us judge whether up-and-down movements are within > the margin of randomness, or whether it is a real effect. I don't know > how annoying they are to implement though, no clue whether the plot lib > supports that. There should be enough information about errors in the > json files, as far as I remember. > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From jacob at openend.se Thu Feb 25 23:39:28 2010 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Thu, 25 Feb 2010 23:39:28 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: <4B86B588.4000601@gmx.de> Message-ID: <201002252339.28334.jacob@openend.se> Thursday 25 February 2010 you wrote: > Hi Carl, > > error bars are certainly doable and supported by the plot lib. The > only problem is that my backend model doesn't include such a field, so > at the moment the data (which is certainly included in the json files) > is not being saved. I will need to change the DB models in order to > acomodate that kind of information. > > I'll add this to the wish list. How do you actually determine the error? Multiple runs or sliding average of historic data? Jacob From stephen at thorne.id.au Fri Feb 26 01:04:21 2010 From: stephen at thorne.id.au (Stephen Thorne) Date: Fri, 26 Feb 2010 10:04:21 +1000 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: Message-ID: <20100226000421.GA2304@thorne.id.au> On 2010-02-25, Miquel Torres wrote: > To that end, there are currently two views. > The Overview does what it's name says, and allows a developer to have > a broad view of how last revisions are affecting performance, and > immediately spot regressions. If the user wants to closer examine the > changes in the performance of a given benchmark, just clicking on it > takes you to the the timeline for the selected benchmark. I have a simple question :) On the 'Timeline' view there is a graph. I realise the x axis is 'revision', but what is the y axis? It is not labelled and has no units. -- Regards, Stephen Thorne Development Engineer Netbox Blue From renesd at gmail.com Fri Feb 26 09:26:48 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Fri, 26 Feb 2010 08:26:48 +0000 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: Message-ID: <64ddb72c1002260026i74b49b85xdb4a0789dc526ccb@mail.gmail.com> Very cool :) pypy is showing great progress! From cfbolz at gmx.de Fri Feb 26 10:54:01 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 26 Feb 2010 10:54:01 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: <201002252339.28334.jacob@openend.se> References: <4B86B588.4000601@gmx.de> <201002252339.28334.jacob@openend.se> Message-ID: <4B879A39.2040606@gmx.de> On 02/25/2010 11:39 PM, Jacob Hall?n wrote: > Thursday 25 February 2010 you wrote: >> Hi Carl, >> >> error bars are certainly doable and supported by the plot lib. The >> only problem is that my backend model doesn't include such a field, so >> at the moment the data (which is certainly included in the json files) >> is not being saved. I will need to change the DB models in order to >> acomodate that kind of information. >> >> I'll add this to the wish list. Great, thanks. > How do you actually determine the error? Multiple runs or sliding average of > historic data? We use the Unladen Swallow benchmark runner, which runs each benchmark multiple times and computes the standard deviation. I then computes a 95% confidence interval using, I think, Student's T distribution. Cheers, Carl Friedrich From stefan_ml at behnel.de Fri Feb 26 10:59:49 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Feb 2010 10:59:49 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: <4B86B588.4000601@gmx.de> References: <4B86B588.4000601@gmx.de> Message-ID: Carl Friedrich Bolz, 25.02.2010 18:38: > On 02/25/2010 04:10 PM, Miquel Torres wrote: >> As some of you already know, there is a new performance site online: >> speed.pypy.org. > [...] > > I'm quite impressed, this is very cool work and a good improvement over > the current plots. Thanks for doing this! > > One thing that I would really find useful are error bars in the timeline > view. This would help us judge whether up-and-down movements are within > the margin of randomness, or whether it is a real effect. I don't know > how annoying they are to implement though, no clue whether the plot lib > supports that. There should be enough information about errors in the > json files, as far as I remember. The standard deviation is commonly considered meaningless for benchmark results. All that really matters is the fastest run, everything else is just fog. Read the docs on timeit, for example. Stefan From tobami at googlemail.com Fri Feb 26 11:05:32 2010 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 26 Feb 2010 11:05:32 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: <4B86B588.4000601@gmx.de> Message-ID: You may also consider that a benchmark that varies greatly between runs may be a flawed benchmark. I think it should be considered, but only on the running side, and act accordingly (too high a deviation: discard run, reconsider benchmark, reconsider environment or whatever). Miquel 2010/2/26 Stefan Behnel : > Carl Friedrich Bolz, 25.02.2010 18:38: >> On 02/25/2010 04:10 PM, Miquel Torres wrote: >>> As some of you already know, there is a new performance site online: >>> speed.pypy.org. >> [...] >> >> I'm quite impressed, this is very cool work and a good improvement over >> the current plots. Thanks for doing this! >> >> One thing that I would really find useful are error bars in the timeline >> view. This would help us judge whether up-and-down movements are within >> the margin of randomness, or whether it is a real effect. I don't know >> how annoying they are to implement though, no clue whether the plot lib >> supports that. There should be enough information about errors in the >> json files, as far as I remember. > > The standard deviation is commonly considered meaningless for benchmark > results. All that really matters is the fastest run, everything else is > just fog. > > Read the docs on timeit, for example. > > Stefan > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From cfbolz at gmx.de Fri Feb 26 11:25:50 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 26 Feb 2010 11:25:50 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: <4B86B588.4000601@gmx.de> Message-ID: <4B87A1AE.90903@gmx.de> On 02/26/2010 10:59 AM, Stefan Behnel wrote: > Carl Friedrich Bolz, 25.02.2010 18:38: >> On 02/25/2010 04:10 PM, Miquel Torres wrote: >>> As some of you already know, there is a new performance site online: >>> speed.pypy.org. >> [...] >> >> I'm quite impressed, this is very cool work and a good improvement over >> the current plots. Thanks for doing this! >> >> One thing that I would really find useful are error bars in the timeline >> view. This would help us judge whether up-and-down movements are within >> the margin of randomness, or whether it is a real effect. I don't know >> how annoying they are to implement though, no clue whether the plot lib >> supports that. There should be enough information about errors in the >> json files, as far as I remember. > > The standard deviation is commonly considered meaningless for benchmark > results. All that really matters is the fastest run, everything else is > just fog. I disagree with this, please read the following paper: http://buytaert.net/files/oopsla07-georges.pdf Cheers, Carl Friedrich From stefan_ml at behnel.de Fri Feb 26 11:30:11 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Feb 2010 11:30:11 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: <4B86B588.4000601@gmx.de> Message-ID: Miquel Torres, 26.02.2010 11:05: > You may also consider that a benchmark that varies greatly between > runs may be a flawed benchmark. > > I think it should be considered, but only on the running side, and act > accordingly (too high a deviation: discard run, reconsider benchmark, > reconsider environment or whatever). Right, there might even have been a cron job running at the same time. There are various reasons why benchmark numbers can vary. Especially in a JIT environment, you'd normally expect the benchmark numbers to decrease over a certain time, or to stay constantly high for a while, then show a peak when the compiler kicks in, and then continue at a lower level (e.g. with the Sun JVM's hotspot JIT or incremental JIT compilers in general). I assume that the benchmarking machinery handles this, but it's yet another reason why highly differing timings can occur within a single run, and why it's only the best run that really matters. You could even go one step further: ignore deviating results in the history graph and only present them when they are still reproducible (preferably with the same source revision) an hour later. Stefan From bea at changemaker.nu Fri Feb 26 11:44:20 2010 From: bea at changemaker.nu (Bea During) Date: Fri, 26 Feb 2010 11:44:20 +0100 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: <20100225091804.GA15891@code0.codespeak.net> References: <20091002114028.GK15455@trillke.net> <20100225091804.GA15891@code0.codespeak.net> Message-ID: <4B87A604.9010601@changemaker.nu> Hi there Armin Rigo wrote: > Hi Miquel, > > May I point out a couple of comments about http://speed.pypy.org/ ? > The first is that it looks great, indeed; thank you very much for doing > such a site! :-) > > Yes - it looks both nice and very useful indeed! Do we have a link from PyPys "main page" http://codespeak.net/pypy/dist/pypy/doc/ to this page so people can find it more easily? Or did I miss that? Cheers Bea > The comments are mostly about the graphics in /timeline: > > * they should also display, or allow to display, the speed of CPython > for comparison; > > * they should have a longer maximum history than 100, to see more > clearly the long-term evolution. > > > All in all it looks great! > > A bientot, > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > From anto.cuni at gmail.com Fri Feb 26 12:08:39 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 26 Feb 2010 12:08:39 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: <4B87A1AE.90903@gmx.de> References: <4B86B588.4000601@gmx.de> <4B87A1AE.90903@gmx.de> Message-ID: <4B87ABB7.6050005@gmail.com> Carl Friedrich Bolz wrote: > I disagree with this, please read the following paper: > > http://buytaert.net/files/oopsla07-georges.pdf +1 for this paper. I was about to link it as well, but cf has been faster. I looked at the unladen swallow runner when I was writing my thesis, and I can confirm that it does the "statistically right" thing as described in the paper to compute error bars. ciao, Anto From tobami at googlemail.com Fri Feb 26 12:30:35 2010 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 26 Feb 2010 12:30:35 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: <4B87ABB7.6050005@gmail.com> References: <4B86B588.4000601@gmx.de> <4B87A1AE.90903@gmx.de> <4B87ABB7.6050005@gmail.com> Message-ID: The paper is right, and the unladen swallow runner does the right thing. What I meant was: use the statistically right method (like we are doing now!), but don't show deviation bars if the deviation is acceptable. Check after the run whether the deviation is not "acceptable". If it isn't, rerun later, check that nothing in the background is affecting performance, reevaluate reproducibility of the given benchmark, etc. But it doesn't change the fact that speed could save the deviation data for later use. What do you think? Miquel 2010/2/26 Antonio Cuni : > Carl Friedrich Bolz wrote: > >> I disagree with this, please read the following paper: >> >> http://buytaert.net/files/oopsla07-georges.pdf > > +1 for this paper. I was about to link it as well, but cf has been faster. > I looked at the unladen swallow runner when I was writing my thesis, and I can > confirm that it does the "statistically right" thing as described in the paper > to compute error bars. > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From cfbolz at gmx.de Fri Feb 26 13:13:15 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 26 Feb 2010 13:13:15 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: <4B86B588.4000601@gmx.de> <4B87A1AE.90903@gmx.de> <4B87ABB7.6050005@gmail.com> Message-ID: <4B87BADB.8080708@gmx.de> On 02/26/2010 12:30 PM, Miquel Torres wrote: > The paper is right, and the unladen swallow runner does the right thing. > > What I meant was: use the statistically right method (like we are > doing now!), but don't show deviation bars if the deviation is > acceptable. Check after the run whether the deviation is not > "acceptable". If it isn't, rerun later, check that nothing in the > background is affecting performance, reevaluate reproducibility of the > given benchmark, etc. I think an important point is that the deviations don't need to come from anything in the background. We don't use threads in the benchmarks (yet) which would obviously insert non-determinism, but even currently there is enough randomness in the interpreter itself. The GC can start at different points, the JIT could decide (late in the process) that something else should be compiled, there are cache-effects, etc. This randomness is not a bad thing, but I think we should try to at least evaluate it, by showing the error bars. We should do that even if the errors are small, because that is a good result worth mentioning. I guess around 20 or even 10 years ago you could attribute a "correct" running time to a program, but nowadays there is noise on all levels of the system and it is not really possible to ignore that. Also, there are really a lot more levels too :-). > But it doesn't change the fact that speed could save the deviation > data for later use. Yip. Cheers, Carl Friedrich From stefan_ml at behnel.de Fri Feb 26 13:30:25 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Feb 2010 13:30:25 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: <4B87A1AE.90903@gmx.de> References: <4B86B588.4000601@gmx.de> <4B87A1AE.90903@gmx.de> Message-ID: Carl Friedrich Bolz, 26.02.2010 11:25: > http://buytaert.net/files/oopsla07-georges.pdf It's sad that the paper doesn't try to understand *why* others use different ways to benchmark. They even admit at the end that their statistical approach is only really interesting when the differences are small enough, not mentioning at that point that the system must be complex enough also, such as the Sun JVM. However, if the differences are small and the benchmarked system is complex, it's best to question the benchmark in the first place, rather than the statistics that lead to its results. Anyway, I agree that, given the complexity of at least some of the benchmarks in the suite, and given the requirement to do continuous benchmarking to find both small and large differences, taking statistically relevant run lines makes sense. Stefan From cfbolz at gmx.de Fri Feb 26 13:43:41 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 26 Feb 2010 13:43:41 +0100 Subject: [pypy-dev] speed.pypy.org launched In-Reply-To: References: <4B86B588.4000601@gmx.de> <4B87A1AE.90903@gmx.de> Message-ID: <4B87C1FD.3040806@gmx.de> On 02/26/2010 01:30 PM, Stefan Behnel wrote: > Carl Friedrich Bolz, 26.02.2010 11:25: >> http://buytaert.net/files/oopsla07-georges.pdf > > It's sad that the paper doesn't try to understand *why* others use > different ways to benchmark. I guess those others should write their own papers (or blog posts or whatever) :-). If you know any well-written ones, I would be very interested. > They even admit at the end that their > statistical approach is only really interesting when the differences are > small enough, not mentioning at that point that the system must be complex > enough also, such as the Sun JVM. However, if the differences are small and > the benchmarked system is complex, it's best to question the benchmark in > the first place, rather than the statistics that lead to its results. [...] In my opinion there are probably not really many non-complex systems around nowadays, at least if we are talking about typical "desktop" systems. There is also a lot of noise on the CPU level, with caches, out of order execution, not even talking about the OS. And while PyPy is not quite as complex as a JVM, it is certainly moving in this direction. So even if your benchmark itself is a simple piece of Python code, the whole system that you invoke is still complex. Cheers, Carl Friedrich From fijall at gmail.com Fri Feb 26 16:05:19 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 26 Feb 2010 10:05:19 -0500 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: <4B87A604.9010601@changemaker.nu> References: <20091002114028.GK15455@trillke.net> <20100225091804.GA15891@code0.codespeak.net> <4B87A604.9010601@changemaker.nu> Message-ID: <693bc9ab1002260705o3bc25dd4lc92c3ed5d0e0e0f8@mail.gmail.com> On Fri, Feb 26, 2010 at 5:44 AM, Bea During wrote:> Hi there > > Armin Rigo wrote: >> Hi Miquel, >> >> May I point out a couple of comments about http://speed.pypy.org/ ? >> The first is that it looks great, indeed; thank you very much for doing >> such a site! :-) >> >> > Yes - it looks both nice and very useful indeed! > > Do we have a link from PyPys "main page" > http://codespeak.net/pypy/dist/pypy/doc/ ?to this > page so people can find it more easily? > > Or did I miss that? > Our main website is a mess right now. I would like to have it redesigned, so we have actually interesting links from there, and obviously the one to graphs (we have two of those now) would be great. Note that for now speed.pypy.org is not automatically updated - it requires adjustments to buildbot run. Cheers, fijal From con at samuelreis.de Fri Feb 26 16:21:16 2010 From: con at samuelreis.de (Samuel Reis) Date: Fri, 26 Feb 2010 16:21:16 +0100 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: <693bc9ab1002260705o3bc25dd4lc92c3ed5d0e0e0f8@mail.gmail.com> References: <20091002114028.GK15455@trillke.net> <20100225091804.GA15891@code0.codespeak.net> <4B87A604.9010601@changemaker.nu> <693bc9ab1002260705o3bc25dd4lc92c3ed5d0e0e0f8@mail.gmail.com> Message-ID: <4B87E6EC.6010302@samuelreis.de> Am 26.02.10 16:05, schrieb Maciej Fijalkowski: > On Fri, Feb 26, 2010 at 5:44 AM, Bea During > wrote:> Hi there > >> Armin Rigo wrote: >> >>> Hi Miquel, >>> >>> May I point out a couple of comments about http://speed.pypy.org/ ? >>> The first is that it looks great, indeed; thank you very much for doing >>> such a site! :-) >>> >>> >>> >> Yes - it looks both nice and very useful indeed! >> >> Do we have a link from PyPys "main page" >> http://codespeak.net/pypy/dist/pypy/doc/ to this >> page so people can find it more easily? >> >> Or did I miss that? >> >> > Our main website is a mess right now. I would like to have it > redesigned, so we have actually interesting links from there, and > obviously the one to graphs (we have two of those now) would be great. > Note that for now speed.pypy.org is not automatically updated - it > requires adjustments to buildbot run. > > Cheers, > fijal > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev If there's the need I could to do a redesign of the layout and (if needed) structure. So you guys could focus on pypy development and just answer a few questions I come up with while working. Greetings -- Samuel From fijall at gmail.com Fri Feb 26 16:58:46 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 26 Feb 2010 10:58:46 -0500 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: <4B87E6EC.6010302@samuelreis.de> References: <20091002114028.GK15455@trillke.net> <20100225091804.GA15891@code0.codespeak.net> <4B87A604.9010601@changemaker.nu> <693bc9ab1002260705o3bc25dd4lc92c3ed5d0e0e0f8@mail.gmail.com> <4B87E6EC.6010302@samuelreis.de> Message-ID: <693bc9ab1002260758y558882dapb98c0e37f8048ff7@mail.gmail.com> On Fri, Feb 26, 2010 at 10:21 AM, Samuel Reis wrote: > Am 26.02.10 16:05, schrieb Maciej Fijalkowski: >> On Fri, Feb 26, 2010 at 5:44 AM, Bea During >> wrote:> ?Hi there >> >>> Armin Rigo wrote: >>> >>>> Hi Miquel, >>>> >>>> May I point out a couple of comments about http://speed.pypy.org/ ? >>>> The first is that it looks great, indeed; thank you very much for doing >>>> such a site! :-) >>>> >>>> >>>> >>> Yes - it looks both nice and very useful indeed! >>> >>> Do we have a link from PyPys "main page" >>> http://codespeak.net/pypy/dist/pypy/doc/ ?to this >>> page so people can find it more easily? >>> >>> Or did I miss that? >>> >>> >> Our main website is a mess right now. I would like to have it >> redesigned, so we have actually interesting links from there, and >> obviously the one to graphs (we have two of those now) would be great. >> Note that for now speed.pypy.org is not automatically updated - it >> requires adjustments to buildbot run. >> >> Cheers, >> fijal >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > If there's the need I could to do a redesign of the layout and (if > needed) structure. > So you guys could focus on pypy development and just answer a few > questions I come up with while working. Great! I have tons of ideas and no actual time to do all this (and no skills as well). How about meeting on IRC at some point? It's easier to discuss stuff that on mail I'm fijal on #pypy at freenode. > > Greetings -- Samuel > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From tobami at googlemail.com Fri Feb 26 19:24:51 2010 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 26 Feb 2010 19:24:51 +0100 Subject: [pypy-dev] new speed.pypy.org site? In-Reply-To: References: <20091002114028.GK15455@trillke.net> <20100225091804.GA15891@code0.codespeak.net> Message-ID: I have commited a bunch of changes: - Added cpython baseline to timeline - Changed timeline defaults to both interpreters selected (plus baseline) - Performance improvements in Timeline's ajax: due to above changes, I worried that for a lot of interpreters selected the queries could be too slow. Even though it is known that Premature optimization is the root of all evil (tm), I had a look and optimized django database calls a little bit (or more precisely made my implementation suck less). So instead of nearly 200 ms per interpreter, it now takes less than 100ms (measured locally, add >100ms for network latency). Plot rerendering should feel faster now. I find it interesting how quick django is. Sometimes you may even forget that the data is being pulled from the server each time a different plot is selected. - Timeline: Added labels for the axes - pypy-c-jit appears now listed first - Added tooltips for Host info and for some benchmarks. Copied the descriptions for the benchmarks listed at the unladen site. Cheers, Miquel 2010/2/25 Leonardo Santagada : > I also have a bunch of comments, all are my opinion and should not be taken as demands (or even as a good review). > > - When you first visit the site I think it would be better to be on the timeline like on http://buildbot.pypy.org/plotsummary.html that shows all benchmarks on the same page, but just the last 50 revisions or it becomes too hard to see the recent improvements. (being able to show more history is important too, maybe there is a way to show the trend and the last x revisions clearly). > > - Overview window comments: > ? ? ? ?- rename the column "result" to "time" (or timing?); > ? ? ? ?- create a new column called "previous time" with the last time of the previous revision. > ? ? ? ?- rename "current change" to "improvement" and invert the values (instead of a green -2% you end up with a green 2%); > ? ? ? ?- with the last two ones it will make understanding the improvement and trends a lot easier; > ? ? ? ?- trend should have the number of revisions on the label (and the values reversed like the "improvement" column); > ? ? ? ?- rename the column "times cpython" to something else (eg. "compared to cpython") and maybe have values like "2x slow down" and "2.5x speedup"; > ? ? ? ?- the graph should say what it is graphing; > ? ? ? ?- maybe invert the order of results. > > - Timeline window comments: > ? ? ? ?- showing all benchmarks in the same page would be good > ? ? ? ?- Pypy with jit should come first in the list of interpreters, and while you only test with hybrid gc there is no need to state that explicitly; > ? ? ? ?- see all benchmarks together, so you don't have to hunt around (maybe have a detailed view); > ? ? ? ?- describe all axis; > ? ? ? ?- as armin said have a selected by default cpython comparison so people can have a baseline. > > > I think this speed center could be used for the common mercurial benchmark repository that is being setup, that would be very very cool. > > > On Feb 25, 2010, at 6:18 AM, Armin Rigo wrote: > >> Hi Miquel, >> >> May I point out a couple of comments about http://speed.pypy.org/ ? >> The first is that it looks great, indeed; thank you very much for doing >> such a site! :-) >> >> The comments are mostly about the graphics in /timeline: >> >> * they should also display, or allow to display, the speed of CPython >> ?for comparison; >> >> * they should have a longer maximum history than 100, to see more >> ?clearly the long-term evolution. >> >> >> All in all it looks great! >> >> A bientot, >> Armin. >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > -- > Leonardo Santagada > santagada at gmail.com > > > >