From solipsis at pitrou.net Sun Nov 1 14:29:44 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 01 Nov 2009 14:29:44 +0100 Subject: [concurrency] Reworking the GIL Message-ID: <1257082184.5485.4.camel@localhost> Hello, I've just noticed that this list exists (not that it seems very active). So I thought I'd send you a link to the recent work I've done to improve the behaviour of the GIL, the results of which you can find in the SVN sandbox : http://mail.python.org/pipermail/python-dev/2009-October/093321.html http://svn.python.org/view/sandbox/trunk/newgil/ (it also gives a name to the "Dave Beazley effect" ;P) Regards Antoine. From aahz at pythoncraft.com Sun Nov 1 14:43:50 2009 From: aahz at pythoncraft.com (Aahz) Date: Sun, 1 Nov 2009 05:43:50 -0800 Subject: [concurrency] Reworking the GIL In-Reply-To: <1257082184.5485.4.camel@localhost> References: <1257082184.5485.4.camel@localhost> Message-ID: <20091101134350.GA20799@panix.com> On Sun, Nov 01, 2009, Antoine Pitrou wrote: > > I've just noticed that this list exists (not that it seems very active). > So I thought I'd send you a link to the recent work I've done to improve > the behaviour of the GIL, the results of which you can find in the SVN > sandbox : > http://mail.python.org/pipermail/python-dev/2009-October/093321.html Nice work! If you're interested in getting broader testing, I suggest you post this to c.l.py as well. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "You could make Eskimos emigrate to the Sahara by vigorously arguing -- at hundreds of screens' length -- for the wonder, beauty, and utility of snow." --PNH to rb in r.a.sf.f From sparks.m at gmail.com Sun Nov 1 18:09:52 2009 From: sparks.m at gmail.com (Michael Sparks) Date: Sun, 1 Nov 2009 17:09:52 +0000 Subject: [concurrency] Reworking the GIL In-Reply-To: <1257082184.5485.4.camel@localhost> References: <1257082184.5485.4.camel@localhost> Message-ID: <20e5c9660911010909r851d6f7h254db8acbd916b85@mail.gmail.com> On Sun, Nov 1, 2009 at 1:29 PM, Antoine Pitrou wrote: > > Hello, > > I've just noticed that this list exists (not that it seems very active). FWIW, It's not been active because the list moved from google groups to python.org. (Rather more annoyingly the original group was also destroyed for no good reason either). As far as I can tell that killed discussion. [1] (It certainly put me off using this list for anything :-( ) > So I thought I'd send you a link to the recent work I've done to improve > the behaviour of the GIL, the results of which you can find in the SVN > sandbox : Can you explain what effect you're aiming to achieve with this change? The reason I'm asking is because using kamaelia[2] it should be possible to craft a bunch of useful test cases or benchmarks - as long as I know what I'm supposed to be testing :-) [2] http://www.kamaelia.org/Home For example, we mix threads in our code safely and happily at the moment, what benefit should I expect to see from this change ? (primarily a message passing setup, though we do use a form of simplified software transactional memory as well) Primarily we use co-operative scheduling of generator based components, but threaded components are common as well to enable use of blocking code, and to also allow the system as a whole to sleep. (For example in network code, select runs in its own thread, which wakes other parts of the system, and can itself block. Similarly the generator scheduler sleeps in a thread - either foreground or background - until something needs running) Similar points go for GUI code etc as well. As a result having mixed workloads is common, so if this reworking should have a benefit, we should be able to practically test for in with kamaelia :-) (The reason for using generators has historically good reasons incidentally, but also has the side effect of encouraging the creation of small reusable things and some handy side effects: primarily because you can put lots of generators in threads, and threads in processes, but doing it the other way is harder :-) If there's anything you'd like me to test, please let me know. I can't promise anything, but willing to help if I can. Regards, Michael. -- [1] I find python.org mailing lists a real pain in the neck because the mail server throws away perfectly valid mails from my mail server, so I have to really want to send something if I want to send anything - because it means I end up having to use google's web gmail interface (rather than a decent mail client) which is a real pain for anything other than occasional discussion. From solipsis at pitrou.net Sun Nov 1 18:50:31 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 01 Nov 2009 18:50:31 +0100 Subject: [concurrency] Reworking the GIL In-Reply-To: <20e5c9660911010909r851d6f7h254db8acbd916b85@mail.gmail.com> References: <1257082184.5485.4.camel@localhost> <20e5c9660911010909r851d6f7h254db8acbd916b85@mail.gmail.com> Message-ID: <1257097831.5485.16.camel@localhost> Le dimanche 01 novembre 2009 ? 17:09 +0000, Michael Sparks a ?crit : > > Can you explain what effect you're aiming to achieve with this change? As the message on python-dev says: make switching latencies more predictable (rather than wildly varying), and reduce the overhead of the GIL itself (in terms of spurious system calls, for example, and therefore of wasted CPU time with multiple threads). > The reason I'm asking is because using kamaelia[2] it should be > possible to craft a bunch of useful test cases or benchmarks - as > long as I know what I'm supposed to be testing :-) Well, it would be nice if you had an existing multithreaded py3k workload so as to test it with the newgil branch. (as far as benchmarks go, I've written ccbench which tries to test the things I was interested in when starting this experiment) > Similarly the generator scheduler sleeps in a thread - either > foreground or background - until something needs running) Similar > points go for GUI code etc as well. As a result having mixed workloads > is common, so if this reworking should have a benefit, we should be > able to practically test for in with kamaelia :-) Well if you are somehow unsatisfied with long (GUI, IO, etc.) latencies in your workload perhaps the new GIL will improve things. If you are satisfied with the current GIL, however, I think it's unlikely you'll notice any difference :-) > [1] I find python.org mailing lists a real pain in the neck because > the mail server throws away perfectly valid mails from my mail server, > so I have to really want to send something if I want to send anything > - because it means I end up having to use google's web gmail interface You should contact the postmaster or mail admin at python.org, IMO. (I don't like the Google stuff either, and I tend to run away from everything Google groups) Regards Antoine. From solipsis at pitrou.net Sun Nov 1 18:53:17 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 01 Nov 2009 18:53:17 +0100 Subject: [concurrency] Reworking the GIL In-Reply-To: <1257097831.5485.16.camel@localhost> References: <1257082184.5485.4.camel@localhost> <20e5c9660911010909r851d6f7h254db8acbd916b85@mail.gmail.com> <1257097831.5485.16.camel@localhost> Message-ID: <1257097997.5485.18.camel@localhost> > As the message on python-dev says: make switching latencies more > predictable (rather than wildly varying), and reduce the overhead of the > GIL itself (in terms of spurious system calls, for example, and > therefore of wasted CPU time with multiple threads). Ah, the third chief weapon is to replace the obscure opcode-based setting (sys.setcheckinterval()) with a time-based setting: sys.setswitchinterval(seconds), the default being 5 ms.