From bradallen137 at gmail.com Fri Nov 11 17:49:44 2011 From: bradallen137 at gmail.com (Brad Allen) Date: Fri, 11 Nov 2011 10:49:44 -0600 Subject: [concurrency] about the concurrency-sig list Message-ID: Python has a rich toolset for taking on concurrency challenges, and many of these tools have their own mailing list. However, concurrency solutions tend to involve working with these tools in combination, and it might be nice to have a place for higher level community discussion across this toolset. I see lots of interesting comments about concurrency scattered across Twitter, G+, and blog postings about gevent, 0mq, celery, multiprocessing, stackless, etc, so I know there is plenty to talk about. I don't want to shut down those scattered discussions; Twitter and G+ are great fun. However, concurrency-sig might be a good home for more in-depth discussions on such topics; currently this list seems dormant. According to the archive, the last postings were in 2009 about the GIL. Why not try to reboot this list, and send out an announcement to call for participation here? I've bcc'd a number of people who might be interested, but may not know about the concurrency-sig list. Maybe there is another such list which I don't know about. If so, please let me know, and I'll post it back to this thread in the concurrency-sig so people looking at the archives will know where they need to go. From jnoller at gmail.com Fri Nov 11 17:59:06 2011 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 11 Nov 2011 11:59:06 -0500 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: Message-ID: I concur On Nov 11, 2011, at 11:49 AM, Brad Allen wrote: > Python has a rich toolset for taking on concurrency challenges, and > many of these tools have their own mailing list. However, concurrency > solutions tend to involve working with these tools in combination, and > it might be nice to have a place for higher level community discussion > across this toolset. > > I see lots of interesting comments about concurrency scattered across > Twitter, G+, and blog postings about gevent, 0mq, celery, > multiprocessing, stackless, etc, so I know there is plenty to talk > about. I don't want to shut down those scattered discussions; Twitter > and G+ are great fun. > > However, concurrency-sig might be a good home for more in-depth > discussions on such topics; currently this list seems dormant. > According to the archive, the last postings were in 2009 about the > GIL. Why not try to reboot this list, and send out an announcement to > call for participation here? > > I've bcc'd a number of people who might be interested, but may not > know about the concurrency-sig list. > > Maybe there is another such list which I don't know about. If so, > please let me know, and I'll post it back to this thread in the > concurrency-sig so people looking at the archives will know where they > need to go. From brian.curtin at gmail.com Fri Nov 11 18:16:42 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Fri, 11 Nov 2011 11:16:42 -0600 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: Message-ID: On Fri, Nov 11, 2011 at 10:49, Brad Allen wrote: > However, concurrency-sig might be a good home for more in-depth > discussions on such topics; currently this list seems dormant. > According to the archive, the last postings were in 2009 about the > GIL. Why not try to reboot this list, and send out an announcement to > call for participation here? I have a number of posts cooking for blog.python.org, but I'll add this to the list of things to put out there. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pete at wearpants.org Fri Nov 11 21:44:24 2011 From: pete at wearpants.org (Peter Fein) Date: Fri, 11 Nov 2011 14:44:24 -0600 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: Message-ID: Hi- I started this list & the concurrency section on the wiki http://wiki.python.org/moin/Concurrency after taking one of Dave Beazley's classes. Other folks & I wanted a place to discuss the kind of issues Brad discusses below, in a general and cross-toolkit context. I don't know why it never really took off, though I'd love to see it have more life. Other possible topic includes: Python under Hadoop, MPI, data processing pipelines (generators). Should we just establish "Threads: you're doing it wrong" as a ground rule and be done with it? ;-P --Pete On Fri, Nov 11, 2011 at 10:49 AM, Brad Allen wrote: > Python has a rich toolset for taking on concurrency challenges, and > many of these tools have their own mailing list. However, concurrency > solutions tend to involve working with these tools in combination, and > it might be nice to have a place for higher level community discussion > across this toolset. > > I see lots of interesting comments about concurrency scattered across > Twitter, G+, and blog postings about gevent, 0mq, celery, > multiprocessing, stackless, etc, so I know there is plenty to talk > about. I don't want to shut down those scattered discussions; Twitter > and G+ are great fun. > > However, concurrency-sig might be a good home for more in-depth > discussions on such topics; currently this list seems dormant. > According to the archive, the last postings were in 2009 about the > GIL. Why not try to reboot this list, and send out an announcement to > call for participation here? > > I've bcc'd a number of people who might be interested, but may not > know about the concurrency-sig list. > > Maybe there is another such list which I don't know about. If so, > please let me know, and I'll post it back to this thread in the > concurrency-sig so people looking at the archives will know where they > need to go. > _______________________________________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/mailman/listinfo/concurrency-sig > From jnoller at gmail.com Fri Nov 11 21:47:31 2011 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 11 Nov 2011 15:47:31 -0500 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: Message-ID: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> On Friday, November 11, 2011 at 3:44 PM, Peter Fein wrote: > Hi- > > I started this list & the concurrency section on the wiki > http://wiki.python.org/moin/Concurrency after taking one of Dave > Beazley's classes. Other folks & I wanted a place to discuss the kind > of issues Brad discusses below, in a general and cross-toolkit > context. I don't know why it never really took off, though I'd love to > see it have more life. > > Other possible topic includes: Python under Hadoop, MPI, data > processing pipelines (generators). > > Should we just establish "Threads: you're doing it wrong" as a ground > rule and be done with it? ;-P > > --Pete > No, given that "Threads: you're doing it wrong" is patently incorrect except for certain cases. I use them more than I use multiprocessing. :) From peter.a.portante at gmail.com Fri Nov 11 21:54:15 2011 From: peter.a.portante at gmail.com (Peter Portante) Date: Fri, 11 Nov 2011 15:54:15 -0500 Subject: [concurrency] about the concurrency-sig list In-Reply-To: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> References: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> Message-ID: Perhaps we would want to avoid dogma, and have this list provide understanding of how things work and pros & cons? On Fri, Nov 11, 2011 at 3:47 PM, Jesse Noller wrote: > > > On Friday, November 11, 2011 at 3:44 PM, Peter Fein wrote: > > > Hi- > > > > I started this list & the concurrency section on the wiki > > http://wiki.python.org/moin/Concurrency after taking one of Dave > > Beazley's classes. Other folks & I wanted a place to discuss the kind > > of issues Brad discusses below, in a general and cross-toolkit > > context. I don't know why it never really took off, though I'd love to > > see it have more life. > > > > Other possible topic includes: Python under Hadoop, MPI, data > > processing pipelines (generators). > > > > Should we just establish "Threads: you're doing it wrong" as a ground > > rule and be done with it? ;-P > > > > --Pete > > > No, given that "Threads: you're doing it wrong" is patently incorrect > except for certain cases. I use them more than I use multiprocessing. :) > > > _______________________________________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/mailman/listinfo/concurrency-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pete at wearpants.org Fri Nov 11 22:17:25 2011 From: pete at wearpants.org (Peter Fein) Date: Fri, 11 Nov 2011 15:17:25 -0600 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> Message-ID: Just stirring the pot a little. ;-) On Fri, Nov 11, 2011 at 2:54 PM, Peter Portante wrote: > Perhaps we would want to avoid dogma, and have this list provide > understanding of how things work and pros & cons? > > On Fri, Nov 11, 2011 at 3:47 PM, Jesse Noller wrote: >> >> >> On Friday, November 11, 2011 at 3:44 PM, Peter Fein wrote: >> >> > Hi- >> > >> > I started this list & the concurrency section on the wiki >> > http://wiki.python.org/moin/Concurrency after taking one of Dave >> > Beazley's classes. Other folks & I wanted a place to discuss the kind >> > of issues Brad discusses below, in a general and cross-toolkit >> > context. I don't know why it never really took off, though I'd love to >> > see it have more life. >> > >> > Other possible topic includes: Python under Hadoop, MPI, data >> > processing pipelines (generators). >> > >> > Should we just establish "Threads: you're doing it wrong" as a ground >> > rule and be done with it? ;-P >> > >> > --Pete >> > >> No, given that "Threads: you're doing it wrong" is patently incorrect >> except for certain cases. I use them more than I use multiprocessing. :) >> >> >> _______________________________________________ >> concurrency-sig mailing list >> concurrency-sig at python.org >> http://mail.python.org/mailman/listinfo/concurrency-sig > > > _______________________________________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/mailman/listinfo/concurrency-sig > > From aahz at pythoncraft.com Sun Nov 13 17:27:59 2011 From: aahz at pythoncraft.com (Aahz) Date: Sun, 13 Nov 2011 08:27:59 -0800 Subject: [concurrency] Threads are good In-Reply-To: References: Message-ID: <20111113162751.GA29755@panix.com> On Fri, Nov 11, 2011, Peter Fein wrote: > > Should we just establish "Threads: you're doing it wrong" as a ground > rule and be done with it? ;-P No. I know you're trolling and trying to be humorous about it, so I'll respond seriously: One case where a thread is the perfect mechanism is when you want to poll for a particular condition periodically. For example, if you're synchronizing files between machines, but you don't want to sync open files, you put the open files into a list that a thread monitors. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From gbelotsky at gmail.com Sun Nov 13 18:18:05 2011 From: gbelotsky at gmail.com (George Belotsky) Date: Sun, 13 Nov 2011 09:18:05 -0800 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> Message-ID: Yes, this sig is an excellent idea. I would certainly appreciate all the information that I could get my hands on, since concurrency is central to the work that I am doing. Getting periodic emails, with more information on the wiki, would be most welcome. All the best, George. On Fri, Nov 11, 2011 at 1:17 PM, Peter Fein wrote: > Just stirring the pot a little. ;-) > > On Fri, Nov 11, 2011 at 2:54 PM, Peter Portante > wrote: > > Perhaps we would want to avoid dogma, and have this list provide > > understanding of how things work and pros & cons? > > > > On Fri, Nov 11, 2011 at 3:47 PM, Jesse Noller wrote: > >> > >> > >> On Friday, November 11, 2011 at 3:44 PM, Peter Fein wrote: > >> > >> > Hi- > >> > > >> > I started this list & the concurrency section on the wiki > >> > http://wiki.python.org/moin/Concurrency after taking one of Dave > >> > Beazley's classes. Other folks & I wanted a place to discuss the kind > >> > of issues Brad discusses below, in a general and cross-toolkit > >> > context. I don't know why it never really took off, though I'd love to > >> > see it have more life. > >> > > >> > Other possible topic includes: Python under Hadoop, MPI, data > >> > processing pipelines (generators). > >> > > >> > Should we just establish "Threads: you're doing it wrong" as a ground > >> > rule and be done with it? ;-P > >> > > >> > --Pete > >> > > >> No, given that "Threads: you're doing it wrong" is patently incorrect > >> except for certain cases. I use them more than I use multiprocessing. :) > >> > >> > >> _______________________________________________ > >> concurrency-sig mailing list > >> concurrency-sig at python.org > >> http://mail.python.org/mailman/listinfo/concurrency-sig > > > > > > _______________________________________________ > > concurrency-sig mailing list > > concurrency-sig at python.org > > http://mail.python.org/mailman/listinfo/concurrency-sig > > > > > _______________________________________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/mailman/listinfo/concurrency-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at dabeaz.com Mon Nov 14 13:58:03 2011 From: dave at dabeaz.com (David Beazley) Date: Mon, 14 Nov 2011 06:58:03 -0600 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> Message-ID: <49492659-F1CE-4D84-BBF1-143365BFCADD@dabeaz.com> Pete's wiseguy thread comment not withstanding, I'll freely admit that I'm usually not interested in participating in online discussions about concurrency due to excessive religious dogma backed by a lack of empirical data or experiments. IMHO, there needs to be more level-minded experimentation, analysis, and discussion. If this is what this list might be about, then it might be worthwhile. Cheers, Dave On Nov 11, 2011, at 2:54 PM, Peter Portante wrote: > Perhaps we would want to avoid dogma, and have this list provide understanding of how things work and pros & cons? > > On Fri, Nov 11, 2011 at 3:47 PM, Jesse Noller wrote: > > > On Friday, November 11, 2011 at 3:44 PM, Peter Fein wrote: > > > Hi- > > > > I started this list & the concurrency section on the wiki > > http://wiki.python.org/moin/Concurrency after taking one of Dave > > Beazley's classes. Other folks & I wanted a place to discuss the kind > > of issues Brad discusses below, in a general and cross-toolkit > > context. I don't know why it never really took off, though I'd love to > > see it have more life. > > > > Other possible topic includes: Python under Hadoop, MPI, data > > processing pipelines (generators). > > > > Should we just establish "Threads: you're doing it wrong" as a ground > > rule and be done with it? ;-P > > > > --Pete > > > No, given that "Threads: you're doing it wrong" is patently incorrect except for certain cases. I use them more than I use multiprocessing. :) > > > _______________________________________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/mailman/listinfo/concurrency-sig > > _______________________________________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/mailman/listinfo/concurrency-sig From gbelotsky at gmail.com Mon Nov 14 19:51:18 2011 From: gbelotsky at gmail.com (George Belotsky) Date: Mon, 14 Nov 2011 10:51:18 -0800 Subject: [concurrency] about the concurrency-sig list In-Reply-To: <49492659-F1CE-4D84-BBF1-143365BFCADD@dabeaz.com> References: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> <49492659-F1CE-4D84-BBF1-143365BFCADD@dabeaz.com> Message-ID: Thanks, Dave! I would love to see this be entirely data-driven. Mere arguing, is, indeed, not useful. All the best, George. On Mon, Nov 14, 2011 at 4:58 AM, David Beazley wrote: > Pete's wiseguy thread comment not withstanding, I'll freely admit that I'm > usually not interested in participating in online discussions about > concurrency due to excessive religious dogma backed by a lack of empirical > data or experiments. IMHO, there needs to be more level-minded > experimentation, analysis, and discussion. If this is what this list might > be about, then it might be worthwhile. > > Cheers, > Dave > > > On Nov 11, 2011, at 2:54 PM, Peter Portante wrote: > > > Perhaps we would want to avoid dogma, and have this list provide > understanding of how things work and pros & cons? > > > > On Fri, Nov 11, 2011 at 3:47 PM, Jesse Noller wrote: > > > > > > On Friday, November 11, 2011 at 3:44 PM, Peter Fein wrote: > > > > > Hi- > > > > > > I started this list & the concurrency section on the wiki > > > http://wiki.python.org/moin/Concurrency after taking one of Dave > > > Beazley's classes. Other folks & I wanted a place to discuss the kind > > > of issues Brad discusses below, in a general and cross-toolkit > > > context. I don't know why it never really took off, though I'd love to > > > see it have more life. > > > > > > Other possible topic includes: Python under Hadoop, MPI, data > > > processing pipelines (generators). > > > > > > Should we just establish "Threads: you're doing it wrong" as a ground > > > rule and be done with it? ;-P > > > > > > --Pete > > > > > No, given that "Threads: you're doing it wrong" is patently incorrect > except for certain cases. I use them more than I use multiprocessing. :) > > > > > > _______________________________________________ > > concurrency-sig mailing list > > concurrency-sig at python.org > > http://mail.python.org/mailman/listinfo/concurrency-sig > > > > _______________________________________________ > > concurrency-sig mailing list > > concurrency-sig at python.org > > http://mail.python.org/mailman/listinfo/concurrency-sig > > _______________________________________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/mailman/listinfo/concurrency-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pete at wearpants.org Mon Nov 14 21:09:53 2011 From: pete at wearpants.org (Peter Fein) Date: Mon, 14 Nov 2011 14:09:53 -0600 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> <49492659-F1CE-4D84-BBF1-143365BFCADD@dabeaz.com> Message-ID: I think usability (from a developer's perspective) is important as well as pure performance. Take the mixed feelings people have about Twisted for example. On Mon, Nov 14, 2011 at 12:51 PM, George Belotsky wrote: > Thanks, Dave! > > I would love to see this be entirely data-driven.? Mere arguing, is, indeed, > not useful. > > All the best, > > George. > > On Mon, Nov 14, 2011 at 4:58 AM, David Beazley wrote: >> >> Pete's wiseguy thread comment not withstanding, I'll freely admit that I'm >> usually not interested in participating in online discussions about >> concurrency due to excessive religious dogma backed by a lack of empirical >> data or experiments. ? IMHO, there needs to be more level-minded >> experimentation, analysis, and discussion. ?If this is what this list might >> be about, then it might be worthwhile. >> >> Cheers, >> Dave >> >> >> On Nov 11, 2011, at 2:54 PM, Peter Portante wrote: >> >> > Perhaps we would want to avoid dogma, and have this list provide >> > understanding of how things work and pros & cons? >> > >> > On Fri, Nov 11, 2011 at 3:47 PM, Jesse Noller wrote: >> > >> > >> > On Friday, November 11, 2011 at 3:44 PM, Peter Fein wrote: >> > >> > > Hi- >> > > >> > > I started this list & the concurrency section on the wiki >> > > http://wiki.python.org/moin/Concurrency after taking one of Dave >> > > Beazley's classes. Other folks & I wanted a place to discuss the kind >> > > of issues Brad discusses below, in a general and cross-toolkit >> > > context. I don't know why it never really took off, though I'd love to >> > > see it have more life. >> > > >> > > Other possible topic includes: Python under Hadoop, MPI, data >> > > processing pipelines (generators). >> > > >> > > Should we just establish "Threads: you're doing it wrong" as a ground >> > > rule and be done with it? ;-P >> > > >> > > --Pete >> > > >> > No, given that "Threads: you're doing it wrong" is patently incorrect >> > except for certain cases. I use them more than I use multiprocessing. :) >> > >> > >> > _______________________________________________ >> > concurrency-sig mailing list >> > concurrency-sig at python.org >> > http://mail.python.org/mailman/listinfo/concurrency-sig >> > >> > _______________________________________________ >> > concurrency-sig mailing list >> > concurrency-sig at python.org >> > http://mail.python.org/mailman/listinfo/concurrency-sig >> >> _______________________________________________ >> concurrency-sig mailing list >> concurrency-sig at python.org >> http://mail.python.org/mailman/listinfo/concurrency-sig > > > _______________________________________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/mailman/listinfo/concurrency-sig > > From solipsis at pitrou.net Tue Nov 15 20:06:19 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 15 Nov 2011 20:06:19 +0100 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: <68A439E2A2A6453B95C5535CADA35A6A@gmail.com> <49492659-F1CE-4D84-BBF1-143365BFCADD@dabeaz.com> Message-ID: <1321383979.3272.10.camel@localhost.localdomain> Le lundi 14 novembre 2011 ? 14:09 -0600, Peter Fein a ?crit : > I think usability (from a developer's perspective) is important as > well as pure performance. But even usability benefits from data (surveys for example, or side-by-side comparisons of solutions to real-world problems). "Mixed feelings" doesn't mean anything. Some people love Twisted and some people hate it (but some people also hate Python). (not to mention that Twisted's "concurrency" doesn't aim to solve the same problems as, e.g., multiprocessing's "concurrency") From aphor at me.com Tue Nov 15 22:29:23 2011 From: aphor at me.com (Jeremy McMillan) Date: Tue, 15 Nov 2011 15:29:23 -0600 Subject: [concurrency] about the concurrency-sig list In-Reply-To: References: Message-ID: <809DECB2-1E7A-455E-80C1-C11BAD3E9CF7@me.com> In the olden days, someone on the list would offer to curate a FAQ. The purpose of a FAQ is to squelch debate that is not productive, and I think specifying some guidelines would add value. Here's my suggestion. 1) What sort of discussion is appropriate for concurrency-sig? a: How to generalize parallel problems (is a discussion topic adequate covered by more general prior discussion)? b: How to generalize Python approaches/components/solutions to parallel problems (frameworks, multicore, MPP clustering, coroutine/pipeline idiom, event processing, etc.) c: How to hypothesize performance of an approach to a problem via Amdahl's Law and/or difficulty getting a good implementation d: How to design and conduct robust testing e: Analysis of test data f: Discussion of implications to general knowledge (goto a:) Discussion should, IMHO, progress through a..f states and then the particular subject can be closed. If you want to know how to get more concurrency in a Python solution to X, that falls under a. If you want to compare X and Y toolkits, that's probably b, but the discussion needs to point to c. Trouble with any given system are under c and d. Bring data everyone loves e! With new data, we can cut loose a little bit and prune our collective dogma together :) 2) What sort of discussion is inappropriate for concurrency-sig? If it doesn't fit one of the states in 1), or it prevents discussion from progressing through those states, we're not interested. On Nov 15, 2011, at 5:00 AM, concurrency-sig-request at python.org wrote: > From: David Beazley > To: py concurrency sig > Subject: Re: [concurrency] about the concurrency-sig list > Message-ID: <49492659-F1CE-4D84-BBF1-143365BFCADD at dabeaz.com> > Content-Type: text/plain; charset=us-ascii > > Pete's wiseguy thread comment not withstanding, I'll freely admit that I'm usually not interested in participating in online discussions about concurrency due to excessive religious dogma backed by a lack of empirical data or experiments. IMHO, there needs to be more level-minded experimentation, analysis, and discussion. If this is what this list might be about, then it might be worthwhile. > > Cheers, > Dave > > > On Nov 11, 2011, at 2:54 PM, Peter Portante wrote: > >> Perhaps we would want to avoid dogma, and have this list provide understanding of how things work and pros & cons? >> >> On Fri, Nov 11, 2011 at 3:47 PM, Jesse Noller wrote: >> >> >> On Friday, November 11, 2011 at 3:44 PM, Peter Fein wrote: >> >>> Hi- >>> >>> I started this list & the concurrency section on the wiki >>> http://wiki.python.org/moin/Concurrency after taking one of Dave >>> Beazley's classes. Other folks & I wanted a place to discuss the kind >>> of issues Brad discusses below, in a general and cross-toolkit >>> context. I don't know why it never really took off, though I'd love to >>> see it have more life. >>> >>> Other possible topic includes: Python under Hadoop, MPI, data >>> processing pipelines (generators). >>> >>> Should we just establish "Threads: you're doing it wrong" as a ground >>> rule and be done with it? ;-P >>> >>> --Pete >>> >> No, given that "Threads: you're doing it wrong" is patently incorrect except for certain cases. I use them more than I use multiprocessing. :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: