From jnoller at gmail.com Tue Jul 5 13:15:55 2011 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 5 Jul 2011 07:15:55 -0400 Subject: [Speed] Getting the project off the ground Message-ID: FWIW: I have complete admin/root access on the machine - I can setup new users as needed, but I don't want to hand them out willy-nilly. The code running speed.pypy.org: https://github.com/tobami/codespeed/ I know there's been some work abstracting it/improving it - I'm not sure where that work stands at this point. Miquel Torres who has been leading the effort can provide more detail - I do know there are also two GSOC projects going on as well. If anything, I need to get a static page up on the site explaining what it's going to be sometime this week, but I'm leery to install anything like apache that can gobble resources that are better spent on speed :) jesse From alex.gaynor at gmail.com Tue Jul 5 17:01:55 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 5 Jul 2011 08:01:55 -0700 Subject: [Speed] Getting the project off the ground In-Reply-To: References: Message-ID: Ideally the webserver for speed.python should *really* be hosted on a different machine. I know Maciej has been investigating moving speed.pypy's web presence to ep.io, so that could be an easy way to do the static site as well. Alex On Tue, Jul 5, 2011 at 4:15 AM, Jesse Noller wrote: > FWIW: I have complete admin/root access on the machine - I can setup > new users as needed, but I don't want to hand them out willy-nilly. > > The code running speed.pypy.org: > > https://github.com/tobami/codespeed/ > > I know there's been some work abstracting it/improving it - I'm not > sure where that work stands at this point. Miquel Torres who has been > leading the effort can provide more detail - I do know there are also > two GSOC projects going on as well. > > If anything, I need to get a static page up on the site explaining > what it's going to be sometime this week, but I'm leery to install > anything like apache that can gobble resources that are better spent > on speed :) > > jesse > _______________________________________________ > Speed mailing list > Speed at python.org > http://mail.python.org/mailman/listinfo/speed > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Jul 5 17:24:37 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 5 Jul 2011 17:24:37 +0200 Subject: [Speed] Getting the project off the ground In-Reply-To: References: Message-ID: On Tue, Jul 5, 2011 at 5:01 PM, Alex Gaynor wrote: > Ideally the webserver for speed.python should *really* be hosted on a > different machine. ?I know Maciej has been investigating moving speed.pypy's > web presence to ep.io, so that could be an easy way to do the static site as > well. > Alex Yes. There is speed.ep.io which doesn't quite work, but we're getting there. Also - we need some sort of locking mechanism. If you run benchmarks, you don't want anything else to run on the same machine at the same time. This is something DasIch can work on I suppose ;-) > > On Tue, Jul 5, 2011 at 4:15 AM, Jesse Noller wrote: >> >> FWIW: I have complete admin/root access on the machine - I can setup >> new users as needed, but I don't want to hand them out willy-nilly. >> >> The code running speed.pypy.org: >> >> https://github.com/tobami/codespeed/ >> >> I know there's been some work abstracting it/improving it - I'm not >> sure where that work stands at this point. Miquel Torres who has been >> leading the effort can provide more detail - I do know there are also >> two GSOC projects going on as well. >> >> If anything, I need to get a static page up on the site explaining >> what it's going to be sometime this week, but I'm leery to install >> anything like apache that can gobble resources that are better spent >> on speed :) >> >> jesse >> _______________________________________________ >> Speed mailing list >> Speed at python.org >> http://mail.python.org/mailman/listinfo/speed > > > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > > _______________________________________________ > Speed mailing list > Speed at python.org > http://mail.python.org/mailman/listinfo/speed > > From jnoller at gmail.com Tue Jul 5 17:51:05 2011 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 5 Jul 2011 11:51:05 -0400 Subject: [Speed] Getting the project off the ground In-Reply-To: References:

Message-ID: On Tue, Jul 5, 2011 at 11:24 AM, Maciej Fijalkowski wrote: > On Tue, Jul 5, 2011 at 5:01 PM, Alex Gaynor wrote: >> Ideally the webserver for speed.python should *really* be hosted on a >> different machine. ?I know Maciej has been investigating moving speed.pypy's >> web presence to ep.io, so that could be an easy way to do the static site as >> well. >> Alex > > Yes. There is speed.ep.io which doesn't quite work, but we're getting there. > > Also - we need some sort of locking mechanism. If you run benchmarks, > you don't want anything else to run on the same machine at the same > time. This is something DasIch can work on I suppose ;-) > >> >> On Tue, Jul 5, 2011 at 4:15 AM, Jesse Noller wrote: >>> >>> FWIW: I have complete admin/root access on the machine - I can setup >>> new users as needed, but I don't want to hand them out willy-nilly. >>> >>> The code running speed.pypy.org: >>> >>> https://github.com/tobami/codespeed/ >>> >>> I know there's been some work abstracting it/improving it - I'm not >>> sure where that work stands at this point. Miquel Torres who has been >>> leading the effort can provide more detail - I do know there are also >>> two GSOC projects going on as well. >>> >>> If anything, I need to get a static page up on the site explaining >>> what it's going to be sometime this week, but I'm leery to install >>> anything like apache that can gobble resources that are better spent >>> on speed :) >>> >>> jesse >>> _______________________________________________ >>> Speed mailing list >>> Speed at python.org >>> http://mail.python.org/mailman/listinfo/speed >> >> >> I can get us free web hosting (VPS based) from OSU/OSL From jnoller at gmail.com Tue Jul 5 17:54:03 2011 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 5 Jul 2011 11:54:03 -0400 Subject: [Speed] Getting the project off the ground In-Reply-To: References:

Message-ID: On Tue, Jul 5, 2011 at 11:51 AM, Jesse Noller wrote: > On Tue, Jul 5, 2011 at 11:24 AM, Maciej Fijalkowski wrote: >> On Tue, Jul 5, 2011 at 5:01 PM, Alex Gaynor wrote: >>> Ideally the webserver for speed.python should *really* be hosted on a >>> different machine. ?I know Maciej has been investigating moving speed.pypy's >>> web presence to ep.io, so that could be an easy way to do the static site as >>> well. >>> Alex >> >> Yes. There is speed.ep.io which doesn't quite work, but we're getting there. >> >> Also - we need some sort of locking mechanism. If you run benchmarks, >> you don't want anything else to run on the same machine at the same >> time. This is something DasIch can work on I suppose ;-) >> >>> >>> On Tue, Jul 5, 2011 at 4:15 AM, Jesse Noller wrote: >>>> >>>> FWIW: I have complete admin/root access on the machine - I can setup >>>> new users as needed, but I don't want to hand them out willy-nilly. >>>> >>>> The code running speed.pypy.org: >>>> >>>> https://github.com/tobami/codespeed/ >>>> >>>> I know there's been some work abstracting it/improving it - I'm not >>>> sure where that work stands at this point. Miquel Torres who has been >>>> leading the effort can provide more detail - I do know there are also >>>> two GSOC projects going on as well. >>>> >>>> If anything, I need to get a static page up on the site explaining >>>> what it's going to be sometime this week, but I'm leery to install >>>> anything like apache that can gobble resources that are better spent >>>> on speed :) >>>> >>>> jesse >>>> _______________________________________________ >>>> Speed mailing list >>>> Speed at python.org >>>> http://mail.python.org/mailman/listinfo/speed >>> >>> >>> > > I can get us free web hosting (VPS based) from OSU/OSL > Just to add to this: We have resources we do not need to pay for to host the website if determined that we need to separate the host from the results / runner - since this is PSF backed, I can get us the hosting for free, and we should - the board does not enjoy paying for web hosting, and OSU/OSL has been incredibly gracious. jesse From tobami at googlemail.com Tue Jul 5 22:02:45 2011 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 5 Jul 2011 22:02:45 +0200 Subject: [Speed] Getting the project off the ground Message-ID: Hi all!, thanks Jesse for all the good work, now we can get to get some work done! So you are already discussing the different components of the speed.python.org Who is willing to collaborate on the benchmark runner part? I hope Maciej can get some help there, some did say they would help some. A benchmark runner for all Python implementations is not entirely trivial. The ruby guys were also looking for one. Apart from the benchmark run itself, there needs to be a policy of locking, like you mentioned. Does that mean we would be using the whole monster Machine for a single benchmark run? So no VM isolation for parallel runs? On visualization part, like I said we helped the Ruby guys configure a Codespeed instance: speed.rubyspec.org They have some hefty needs, about half a hundred different benchmarks, and all ruby implementations, so they are putting Codespeed to its paces. That is quite fortunate, as wo are working on the same kind of improvements that speed.python.org will need. Any extra help is nevertheless welcome, of course ;-) So how about the runner part? how many people are interested in helping out? Miquel From marty at martyalchin.com Tue Jul 5 23:21:04 2011 From: marty at martyalchin.com (Marty Alchin) Date: Tue, 5 Jul 2011 14:21:04 -0700 Subject: [Speed] Getting the project off the ground In-Reply-To: References: Message-ID: On Tue, Jul 5, 2011 at 1:02 PM, Miquel Torres wrote: > On visualization part, like I said we helped the Ruby guys configure a > Codespeed instance: speed.rubyspec.org > They have some hefty needs, about half a hundred different benchmarks, > and all ruby implementations, so they are putting Codespeed to its > paces. That is quite fortunate, as wo are working on the same kind of > improvements that speed.python.org will need. Any extra help is > nevertheless welcome, of course ;-) I don't have any experience with Codespeed yet, but I joined this list in particular to see if I could help with visualizing the results, so consider me on board for that portion if there's anything I can do. -Marty From tobami at googlemail.com Wed Jul 6 08:45:49 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 6 Jul 2011 08:45:49 +0200 Subject: [Speed] Getting the project off the ground In-Reply-To: References: Message-ID: Hi Marty, we are currently working on several things. Depending on your experience/interests you may want to help in some: * Danilo is implementing error bars and improving multi-project support in the views as a GSoC (no coding help needed, but maybe more ideas on what statistics features would make sense to have) * Frank wants to start implementation of a RESTful API to all data * Together with Stefan (RoarVM) and Chuck (speed.rubyspec.org) I am fixing bugs/polishing/documenting * There is a feature request (PyPy) for implementing a new view to display regressions * Another feature request are benchmark grouping (see the problem on speed.rubyspec.org) * The reports (sort of overall improvement/regression of all results for a revision) need to be redesigned to be less regression biased * Codespeed needs a website, even if just a basic one (can be hosted at Github itself) * Unit test coverage is currently only about 33% for the backend (no UI tests yet) All of these will be useful for speed.python.org, of course. To get learn a bit more, you can go to the wiki: https://github.com/tobami/codespeed/wiki And read the Overview and Roadmap pages (which may need some updating) There are also a lot of open feature issues on https://github.com/tobami/codespeed/issues And we also have a mailing list at groups: http://groups.google.com/group/codespeed So if you are interested on some of the items mentioned, or even have other ideas just add/edit issues on Github or start a discussion thread! Miquel 2011/7/5 Marty Alchin : > On Tue, Jul 5, 2011 at 1:02 PM, Miquel Torres wrote: >> On visualization part, like I said we helped the Ruby guys configure a >> Codespeed instance: speed.rubyspec.org >> They have some hefty needs, about half a hundred different benchmarks, >> and all ruby implementations, so they are putting Codespeed to its >> paces. That is quite fortunate, as wo are working on the same kind of >> improvements that speed.python.org will need. Any extra help is >> nevertheless welcome, of course ;-) > > I don't have any experience with Codespeed yet, but I joined this list > in particular to see if I could help with visualizing the results, so > consider me on board for that portion if there's anything I can do. > > -Marty > From lac at openend.se Wed Jul 6 19:25:19 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 06 Jul 2011 19:25:19 +0200 Subject: [Speed] Getting the project off the ground In-Reply-To: Message from Miquel Torres of "Wed, 06 Jul 2011 08:45:49 +0200." References:

Message-ID: <201107061725.p66HPJZx027315@theraft.openend.se> What I would like is a way to have hg bisect automated for me. So I could induicate a fast build, and a later build that is slow, and then say 'find me the first build in between where things slowed down. I want to be able to ask for this, go to bed, and find out when I wake up the next day exactly whre to go looking for bugs/slowdowns. I realise that this does not have to be done through the codespeak interface, and that in some cases it may be hard to tell the difference between a real slowdown, random noise, and the ever popular 'somebody was trying to compile pypy on tannit when the benchmarks were being run' though I suppose with the new machine we can work out a mechanism for preventing the last. But I still think this would be a neat feature for codespeed to have as a feature. Laura From pypy at pocketnix.org Wed Jul 6 20:47:13 2011 From: pypy at pocketnix.org (Da_Blitz) Date: Wed, 6 Jul 2011 18:47:13 +0000 Subject: [Speed] Getting the project off the ground In-Reply-To: <201107061725.p66HPJZx027315@theraft.openend.se> References:

<201107061725.p66HPJZx027315@theraft.openend.se> Message-ID: <20110706184713.GA6292@pocketnix.org> > to be done through the codespeak interface, and that in some cases it may be > hard to tell the difference between a real slowdown, random noise, and > the ever popular 'somebody was trying to compile pypy on tannit when the > benchmarks were being run' though I suppose with the new machine we can > work out a mechanism for preventing the last. But I still think this would > be a neat feature for codespeed to have as a feature. cgroups and cpuset support could help with this, you can isolate one cpu purely for benchmarking and another for user use (translation and testing for example). you can also avoid some issues such as cache contention (which fijal mentions every time i mention doing multiple translations at once) by splittign the cpus up so that each group gets 4 cpus belonging to one socket stick all processes in one cgroup and limit the cpuset (usergroup), then create another cgroup and give it the remaining cores on the other socket (benchgroup), by default all processes will go to the parent cgroup, in this case, usergroup as ou put all the PID's in there. thenyou can just put the benchmark script in a wrapper to grab its own pid and echo that pid into the benchgroup magic file to have the benchamrks run on the other cpu using the above you can also reserve memory for benchmarking/transtlation if anyone wants more details feel free to ask From anto.cuni at gmail.com Wed Jul 6 21:01:20 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 06 Jul 2011 21:01:20 +0200 Subject: [Speed] Getting the project off the ground In-Reply-To: <20110706184713.GA6292@pocketnix.org> References:

<201107061725.p66HPJZx027315@theraft.openend.se> <20110706184713.GA6292@pocketnix.org> Message-ID: <4E14B100.5080309@gmail.com> On 06/07/11 20:47, Da_Blitz wrote: > cgroups and cpuset support could help with this, you can isolate one > cpu purely for benchmarking and another for user use (translation and > testing for example). that would be really cool. However, I'm not sure it's enough to prevent "random noise". E.g., I know that modern CPUs have a mechanism that turns off the unused cores in order to speed up the active ones (by slightly incrementing the frequency). If this is the case, then I fear that the only way to have a precise benchmark is to run only one process at time. ciao, Anto From debatem1 at gmail.com Wed Jul 6 21:09:06 2011 From: debatem1 at gmail.com (geremy condra) Date: Wed, 6 Jul 2011 15:09:06 -0400 Subject: [Speed] Getting the project off the ground In-Reply-To: <4E14B100.5080309@gmail.com> References:

<201107061725.p66HPJZx027315@theraft.openend.se> <20110706184713.GA6292@pocketnix.org> <4E14B100.5080309@gmail.com> Message-ID: On Wed, Jul 6, 2011 at 3:01 PM, Antonio Cuni wrote: > On 06/07/11 20:47, Da_Blitz wrote: >> cgroups and cpuset support could help with this, you can isolate one >> cpu purely for benchmarking and another for user use (translation and >> testing for example). > > that would be really cool. However, I'm not sure it's enough to prevent > "random noise". > > E.g., I know that modern CPUs have a mechanism that turns off the unused cores > in order to speed up the active ones (by slightly incrementing the frequency). > ?If this is the case, then I fear that the only way to have a precise > benchmark is to run only one process at time. sudo -s "echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor" Geremy Condra From pypy at pocketnix.org Wed Jul 6 21:11:24 2011 From: pypy at pocketnix.org (Da_Blitz) Date: Wed, 6 Jul 2011 19:11:24 +0000 Subject: [Speed] Getting the project off the ground In-Reply-To: <4E14B100.5080309@gmail.com> References:

<201107061725.p66HPJZx027315@theraft.openend.se> <20110706184713.GA6292@pocketnix.org> <4E14B100.5080309@gmail.com> Message-ID: <20110706191124.GB6292@pocketnix.org> On Wed, Jul 06, 2011 at 09:01:20PM +0200, Antonio Cuni wrote: > that would be really cool. However, I'm not sure it's enough to prevent > "random noise". > > E.g., I know that modern CPUs have a mechanism that turns off the unused cores > in order to speed up the active ones (by slightly incrementing the frequency). > If this is the case, then I fear that the only way to have a precise > benchmark is to run only one process at time. Looks like a X5680 cpu so yes it has the turbo boost feature (http://en.wikipedia.org/wiki/Nehalem_(microarchitecture)#Server_.2F_Desktop_Processors_2) i am running an i7 for building pypy and have the same turbo boost feature and in practice have not found it to be an issue aslong as you are only running one translation at a time. as the workload is single threadded it ramps up nicely. a lock or two should prevent the turbo boost enabling/disabling erratic but it is also under kernel control. i havent investigated how much control the kernel has over it but i assume if you switch the cpu speed governers from performance over to user mode and manually set the freqency that should not be much of an issue turbo mode is socket specific so the isolation i talked about in my last post would prevent compiles' from affecting the cpu freqency on the benchmark cpus as a side note this may be a good time for my to push for making the benchmark scores user time + kernel time instead of wall clock time From stutzbach at google.com Wed Jul 6 21:07:48 2011 From: stutzbach at google.com (Daniel Stutzbach) Date: Wed, 6 Jul 2011 12:07:48 -0700 Subject: [Speed] Getting the project off the ground In-Reply-To: <4E14B100.5080309@gmail.com> References:

<201107061725.p66HPJZx027315@theraft.openend.se> <20110706184713.GA6292@pocketnix.org> <4E14B100.5080309@gmail.com> Message-ID: On Wed, Jul 6, 2011 at 12:01 PM, Antonio Cuni wrote: > that would be really cool. However, I'm not sure it's enough to prevent > "random noise". > A little random noise is unavoidable. Sometimes the kernel will preempt the thread, so the kernel can do kernel things. There's no practical way around that. A little random noise is okay, though. Take several measurements and compute confidence intervals. -- Daniel Stutzbach -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Thu Jul 7 16:56:11 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 07 Jul 2011 16:56:11 +0200 Subject: [Speed] Getting the project off the ground In-Reply-To: <20110706191124.GB6292@pocketnix.org> References:

<201107061725.p66HPJZx027315@theraft.openend.se> <20110706184713.GA6292@pocketnix.org> <4E14B100.5080309@gmail.com> <20110706191124.GB6292@pocketnix.org> Message-ID: <4E15C90B.4080106@gmail.com> On 06/07/11 21:11, Da_Blitz wrote: > Looks like a X5680 cpu so yes it has the turbo boost feature > (http://en.wikipedia.org/wiki/Nehalem_(microarchitecture)#Server_.2F_Desktop_Processors_2) > i am running an i7 for building pypy and have the same turbo boost > feature and in practice have not found it to be an issue aslong as you > are only running one translation at a time. as the workload is single > threadded it ramps up nicely. a lock or two should prevent the turbo > boost enabling/disabling erratic but it is also under kernel control. > > i havent investigated how much control the kernel has over it but i > assume if you switch the cpu speed governers from performance over to > user mode and manually set the freqency that should not be much of an > issue I did some benchmarks, trying to understand how much the turbo boost and/or scaling governors affect the performance and, most importantly, if/how much they affect the standard deviation. Since we are talking of benchmarks, the smaller the standard deviation is, the better. I ran the benchmark on Linux on an Intel i7 920 CPU, which has 4 physical cores (8 logical ones with hyperthreading, but we do not want to use them). The benchmark consisted in running richards.py (one of the benchmarks we use in PyPy) using 1, 2, 3 or 4 cores at the same time. I used "taskset" to set the cpu affinity to a specific core. For each number of core, I ran the benchmark 10 times in a row, and the measured the average time spent and the standard deviation (so, with 4 cores I had a total of 40 runs). If the turbo boost theory is true, we expect the benchmarks to be slower when we run 4 in parallel. Here are the results: 1 core: AVG: 1.939 seconds STDEV: 0.016 seconds 2 cores: AVG: 2.020 STDEV: 0.013 3 cores: AVG: 2.022 STDEV: 0.016 4 cores: AVG: 2.033 STDEV: 0.023 We can see that with 4 cores performance drops a bit, but not much (~4% between 1 core and 4 cores). This is using the "ondemand" governor, which is the default on my system. I tried to run it also with "performance", which in theory should give better performance and smaller stdev, but in practice it does not (I don't know why, honestly): 1 core, performance governor: AVG: 1.961 seconds STDEV: 0.027 seconds I also tried to manually set the CPU to the lowest possible frequency, but got even worse results: 1 core, slowest frequency: AVG: 3.532 seconds STDEV: 0.042 seconds > turbo mode is socket specific so the isolation i talked about in my > last post would prevent compiles' from affecting the cpu freqency on > the benchmark cpus Although the turbo boost seems not to affect much the performance, I agree with Da_Blitz in that it makes sense to reserve 1 socket (i.e. 6 cores) for benchmarks. Then, we can use the other 6 cores for general usage (e.g., running tests or translations). But, before doing this we should check that using one socket does not actually affect the performance of the other. As usual, I don't trust the theory too much :-). ciao, Anto From stutzbach at google.com Thu Jul 7 18:55:45 2011 From: stutzbach at google.com (Daniel Stutzbach) Date: Thu, 7 Jul 2011 09:55:45 -0700 Subject: [Speed] Getting the project off the ground In-Reply-To: <4E15C90B.4080106@gmail.com> References:

<201107061725.p66HPJZx027315@theraft.openend.se> <20110706184713.GA6292@pocketnix.org> <4E14B100.5080309@gmail.com> <20110706191124.GB6292@pocketnix.org> <4E15C90B.4080106@gmail.com> Message-ID: On Thu, Jul 7, 2011 at 7:56 AM, Antonio Cuni wrote: > This is using the "ondemand" governor, which is the default on my system. > > I tried to run it also with "performance", which in theory should give > better > performance and smaller stdev, but in practice it does not (I don't know > why, > honestly): > For what it's worth, I ran some similar experiments last year and found that "performance" yielded much lower standard deviations for certain performance tests, compared to "on demand". I no longer recall which ones unfortunately, but in a nutshell "on demand" is just as good only when it's smart enough detect that it should be using the maximum CPU frequency. Sometimes it doesn't, and the standard deviation goes up a few orders of magnitude. -- Daniel Stutzbach -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Mon Jul 11 07:59:12 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jul 2011 07:59:12 +0200 Subject: [Speed] Getting the project off the ground In-Reply-To: References:

<201107061725.p66HPJZx027315@theraft.openend.se> <20110706184713.GA6292@pocketnix.org> <4E14B100.5080309@gmail.com> <20110706191124.GB6292@pocketnix.org> <4E15C90B.4080106@gmail.com> Message-ID: <4E1A9130.5070409@behnel.de> Daniel Stutzbach, 07.07.2011 18:55: > On Thu, Jul 7, 2011 at 7:56 AM, Antonio Cuni wrote: > >> This is using the "ondemand" governor, which is the default on my >> system. I tried to run it also with "performance", which in theory >> should give better performance and smaller stdev, but in practice it >> does not (I don't know why, honestly) > > For what it's worth, I ran some similar experiments last year and found that > "performance" yielded much lower standard deviations for certain performance > tests, compared to "on demand". I no longer recall which ones > unfortunately, but in a nutshell "on demand" is just as good only when it's > smart enough detect that it should be using the maximum CPU frequency. > > Sometimes it doesn't, and the standard deviation goes up a few orders of > magnitude. That matches my benchmarking experience as well. Stefan From pypy at pocketnix.org Mon Jul 11 16:25:58 2011 From: pypy at pocketnix.org (Da_Blitz) Date: Mon, 11 Jul 2011 14:25:58 +0000 Subject: [Speed] Ideas for speed.python.org Message-ID: <20110711142558.GD6292@pocketnix.org> Hi i have been watching speed since it was announced and was expecting some form of rough plan for the server would be discussed but haven't seen one proposed to date. since no one has got the ball rolling i thought i would put some questions out there to get it all started What sort of raid level will be used with the server, raid10? raid5? How is the hard drive going to be divided up? Any opinions on what OS should be installed? What services need to be running What services need to be remotely accessible is there any other infrastructure we need to integrate to? eg ldap for logins (unlikely but worth asking) what are we doing about turbo boost? disabling in the BIOS or just leaving it in there Who will need regular accesses to the box? what is the box for besides benchmarking, will we allow users to log on and run their own compile jobs? to answer some of my own questions: Personally i think raid5 should be sufficient i don't see IO being a significant bottle neck and having 900GB vs 600GB may be an advantage down the road. raid 6 was ruled out as there are only 4 drives in the machine. there should be 2 unused hard drive slots we can use in the future if we need to. IIRC there is online raid5 -> raid6 migration code in the MD device subsystem but i can confirm that if anyone needs me to as for hard drives, i am a fan of LVM as long as its don't right. VG name being - where count is the install number for the day. if you don't do it this way you WILL have issues when the machine crashes and you need to plug those drives into another machine with the same hostname. as for partitioning, 10-15GB for / is normally sufficient. a couple of GB's for log files (/var/log) to avoid a misconfigured logrotate cron job from eating all the disk space (seen this more times than i want to count in production). 5GB for /tmp and the remaining space apart from 20GB allocated to /home the reason for the 20GB of "slosh" space is so that if something does fill up and you cant afford to delete it you can resize the partition and then extend the filesystem while the machine is still running (only works on some filesystems and as such i would recommend only using a filesystem with only resize, online shrinking is not a high priority as one may think). another tactic i have seen is a separate partition to install daily/weekly backups (eg /backups) that are rsyncd to a remote server. cheap and effective. as for OS choice i assume that means ubuntu for 90% of the people on this mailing list, this is basically a matter of personal preference but if anyone has specific constructive comments then i recommend mentioning it that covers most of the presetup work that cant be changed easily at another date is this just a benchmarking server, or a server people can also log into and use as a generic build machine? i always assumed that one socket would be used to benchmarking and the other socket would be given over to normal users who needed a machine with a bit more speed as it is some nice hardware and it would be a shame to see it go to waste. this leads into the next question of who needs access to the machine and what level of accesses do they need. remotely accessible serveries, i know apache was mentioned and someone wanted something a tiny bit more light weight. as long as you tune it apache is fine, we used to run 2000+ domains on machines with a quarter as much ram and only 2 cpus and still had them humming along with no issue. i would doubt you would hit the kind of traffic to that machine that would justify not using the default config and would put this as premature optimization. if you still want to look at alternatives, i have had experience with nginx. on my machine it only uses about 2MB but i would have to recommend lighttpd due to its cgi support and its fcgi process support (no external monitoring app required for fcgi processes unlike nginx) i know someone mentioned doing some virtualisation which i assume by that they meant xen or virtual box. the cgroups support is one half of the light weight virtualisation stack for linux (the resource counting part) with LXC being the other half. but full virtualisation is something that should also be discussed. i will stop here as my original list of "points to be discussed" came to about 50 items and even then i would have considered that a short list for server deployment, hopefully someone has some more input or if people like these ideas i can start putting this together as a more concrete list of things that need to get done Man with a plan Da_Blitz From jnoller at gmail.com Mon Jul 11 16:58:37 2011 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Jul 2011 10:58:37 -0400 Subject: [Speed] Ideas for speed.python.org In-Reply-To: <20110711142558.GD6292@pocketnix.org> References: <20110711142558.GD6292@pocketnix.org> Message-ID: On Mon, Jul 11, 2011 at 10:25 AM, Da_Blitz wrote: > Hi > > i have been watching speed since it was announced and was expecting > some form of rough plan for the server would be discussed but haven't > seen one proposed to date. since no one has got the ball rolling i > thought i would put some questions out there to get it all started > Note - you're asking setup questions which have actually already been done by the folks as OSU/OSL at my request. So, these decisions have already been made - we actually already have the box ready to go and offer accounts to the people who will deploy codespeed and get the benchmarks going. > What sort of raid level will be used with the server, raid10? raid5? This is already done. We have hardware level Raid 5 > How is the hard drive going to be divided up? Filesystem Size Used Avail Use% Mounted on /dev/sda2 19G 3.4G 15G 20% / none 7.9G 196K 7.9G 1% /dev none 7.9G 0 7.9G 0% /dev/shm none 7.9G 48K 7.9G 1% /var/run none 7.9G 0 7.9G 0% /var/lock /dev/sda1 184M 29M 147M 17% /boot root at speed-python:~# lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert swap lvm -wi-ao 14.90g root at speed-python:~# pvs PV VG Fmt Attr PSize PFree /dev/sda3 lvm lvm2 a- 1.07t 1.06t root at speed-python:~# > Any opinions on what OS should be installed? Linux speed-python 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux > What services need to be running Only what we need to host the site and SSH > What services need to be remotely accessible SSH > is there any other infrastructure we need to integrate to? eg ldap for logins (unlikely but worth asking) None > what are we doing about turbo boost? disabling in the BIOS or just leaving it in there I'll leave that to you guys to decide. If you want it shut off, I need to contact OSU/OSL to have it done. > Who will need regular accesses to the box? This is up to the team. Currently, I have root and 1 user, and Fijal has an account. > what is the box for besides benchmarking, will we allow users to log on and run their own compile jobs? Benchmarking and code coverage only. This is not a general shell account box for users. That's not how I sold it to HP who donated the machine. ...snip > is this just a benchmarking server, or a server people can also log > into and use as a generic build machine? i always assumed that one > socket would be used to benchmarking and the other socket would be > given over to normal users who needed a machine with a bit more speed > as it is some nice hardware and it would be a shame to see it go to > waste. this leads into the next question of who needs access to the > machine and what level of accesses do they need. > This server is for benchmarking CPython, PyPy and Jython and potentially running detailed code coverage reports. Until we have that done and in place, I can not in good conscience discuss allowing any other users on the box for general jobs and work. > i know someone mentioned doing some virtualisation which i assume by > that they meant xen or virtual box. the cgroups support is one half of > the light weight virtualisation stack for linux (the resource counting > part) with LXC being the other half. but full virtualisation is > something that should also be discussed. > We should not be using virtualization to generate benchmark numbers; hypervisors are lying bastards :) Deployment is done. We need code. From santagada at gmail.com Mon Jul 11 17:14:06 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Mon, 11 Jul 2011 12:14:06 -0300 Subject: [Speed] Ideas for speed.python.org In-Reply-To: References: <20110711142558.GD6292@pocketnix.org> Message-ID: Forgot to send to the list On Mon, Jul 11, 2011 at 11:25 AM, Da_Blitz wrote: > production). 5GB for /tmp and the remaining space apart from 20GB > allocated to /home pypy uses /tmp to put its source files so I don't know if 5gb is enough. > is this just a benchmarking server, or a server people can also log > into and use as a generic build machine? i always assumed that one > socket would be used to benchmarking and the other socket would be > given over to normal users who needed a machine with a bit more speed Why not use both sockets for benchmarking? Either put 2 interpreters on each socket (cpython+jython in one and pypy+ironpython in the other) or use one of the sockets to run the benchmarks on windows/freebsd. I think more granularity in the benchmarks is interesting to find regressions, and the performance of interpreters in different oses might be very different also (ironpython running on .net, different jvm etc). Besides this two points I think everything looks ok to me (but I don't know much about sysadmin). Some people have a lot of scary stories about RAID5, but I think this is mostly a problem with old RAID controllers. -- Leonardo Santagada -- Leonardo Santagada From stefan_ml at behnel.de Mon Jul 11 18:43:37 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jul 2011 18:43:37 +0200 Subject: [Speed] Ideas for speed.python.org In-Reply-To: References: <20110711142558.GD6292@pocketnix.org> Message-ID: <4E1B2839.8080800@behnel.de> Jesse Noller, 11.07.2011 16:58: > On Mon, Jul 11, 2011 at 10:25 AM, Da_Blitz wrote: >> Who will need regular accesses to the box? > > This is up to the team. Currently, I have root and 1 user, and Fijal > has an account. > > This server is for benchmarking CPython, PyPy and Jython and > potentially running detailed code coverage reports. I'd like to set up Cython eventually. Shouldn't be too hard, there's a "build'n'run" command line interface to it. Can we have ccache installed on the machine? Tends to shave off some factors of build time. Stefan From jnoller at gmail.com Mon Jul 11 18:47:32 2011 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Jul 2011 12:47:32 -0400 Subject: [Speed] Ideas for speed.python.org In-Reply-To: <4E1B2839.8080800@behnel.de> References: <20110711142558.GD6292@pocketnix.org> <4E1B2839.8080800@behnel.de> Message-ID: On Mon, Jul 11, 2011 at 12:43 PM, Stefan Behnel wrote: > Jesse Noller, 11.07.2011 16:58: >> >> On Mon, Jul 11, 2011 at 10:25 AM, Da_Blitz wrote: >>> >>> Who will need regular accesses to the box? >> >> This is up to the team. Currently, I have root and 1 user, and Fijal >> has an account. >> >> This server is for benchmarking CPython, PyPy and Jython and >> potentially running detailed code coverage reports. > > I'd like to set up Cython eventually. Shouldn't be too hard, there's a > "build'n'run" command line interface to it. > > Can we have ccache installed on the machine? Tends to shave off some factors > of build time. > > Stefan > We should track things like this in a tracker or something so we can not lose site of them; I'm essentially looking for someone to step up and be the project manager and get things deployed. I'm currently tapped out. jesse From tobami at googlemail.com Mon Jul 11 21:15:45 2011 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 11 Jul 2011 21:15:45 +0200 Subject: [Speed] Ideas for speed.python.org In-Reply-To: References: <20110711142558.GD6292@pocketnix.org> <4E1B2839.8080800@behnel.de> Message-ID: > We should track things like this in a tracker or something so we can > not lose site of them; I'm essentially looking for someone to step up > and be the project manager and get things deployed. I'm currently > tapped out. Yeah, an issue tracker would be a good thing, just an speed account on Github or BitBucket would suffice. I am also tapped out at the moment. Later maybe, but not now. Miquel 2011/7/11 Jesse Noller : > On Mon, Jul 11, 2011 at 12:43 PM, Stefan Behnel wrote: >> Jesse Noller, 11.07.2011 16:58: >>> >>> On Mon, Jul 11, 2011 at 10:25 AM, Da_Blitz wrote: >>>> >>>> Who will need regular accesses to the box? >>> >>> This is up to the team. Currently, I have root and 1 user, and Fijal >>> has an account. >>> >>> This server is for benchmarking CPython, PyPy and Jython and >>> potentially running detailed code coverage reports. >> >> I'd like to set up Cython eventually. Shouldn't be too hard, there's a >> "build'n'run" command line interface to it. >> >> Can we have ccache installed on the machine? Tends to shave off some factors >> of build time. >> >> Stefan >> > > We should track things like this in a tracker or something so we can > not lose site of them; I'm essentially looking for someone to step up > and be the project manager and get things deployed. I'm currently > tapped out. > > jesse > _______________________________________________ > Speed mailing list > Speed at python.org > http://mail.python.org/mailman/listinfo/speed > From tleeuwenburg at gmail.com Tue Jul 26 13:29:20 2011 From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg) Date: Tue, 26 Jul 2011 21:29:20 +1000 Subject: [Speed] I wonder if I can help in some way Message-ID: Hi all, I feel a bit unconfident about emailing the speed at python mailing list for some reason. I have put together a very small, but fully functional, module called benchmarker.py. It records profile stats to disk and submits them to codespeed. I suspect it could not currently meet all the needs of Python for benchmarking,but I would be more than happy to hear about the requirements and maybe help out it some way. I hope it is clear that I'm not claiming benchmarker.py is anything amazing, just something I have put together which *might* be useful. I have independently spoken to Miquel Torres as I was working on the codespeed integration code. He is curious to see if he has any application for benchmarker.py himself, but that will unfold as it may. It is something I am using for my work, so the project has a life of its own, but of course if it can be of use to anyone else also, that would be exciting for me :) Regards, Tennessee -------------- next part -------------- An HTML attachment was scrubbed... URL: