From brian at sweetapp.com Fri Jan 15 07:21:21 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 15 Jan 2010 17:21:21 +1100 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> References: <4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com> <1257605583.3437.3.camel@localhost> <4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com> <1257607273.3437.13.camel@localhost> <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com> <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com> <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> Message-ID: Hi all, I've updated the implementation based on the feedback that I've received and the updated documentation of the API is here: http://sweetapp.com/futures-pep/ If you are still interested, please take a look and let me know what you think.. Cheers, Brian From anh.hai.trinh at gmail.com Fri Jan 15 11:50:14 2010 From: anh.hai.trinh at gmail.com (Anh Hai Trinh) Date: Fri, 15 Jan 2010 17:50:14 +0700 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com> <1257607273.3437.13.camel@localhost> <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com> <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com> <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> Message-ID: Hello all, I'd like to point out an alternative module with respect to asynchronous computation: `stream` (which I wrote) supports ThreadPool, ProcessPool and Executor with a simpler API and implementation. My module takes a list-processing oriented view in which a ThreadPool/ProcessPool is simply a way of working with each stream element concurrently and output results possibly in out of order. A trivial example is: from stream import map range(10) >> ThreadPool(map(lambda x: x*x)) >> sum # returns 285 The URLs retrieving example is: import urllib2 from stream import ThreadPool URLs = [ 'http://www.cnn.com/', 'http://www.bbc.co.uk/', 'http://www.economist.com/', 'http://nonexistant.website.at.baddomain/', 'http://slashdot.org/', 'http://reddit.com/', 'http://news.ycombinator.com/', ] def retrieve(urls, timeout=10): for url in urls: yield url, urllib2.urlopen(url, timeout=timeout).read() if __name__ == '__main__': retrieved = URLs >> ThreadPool(retrieve, poolsize=len(URLs)) for url, content in retrieved: print '%r is %d bytes' % (url, len(content)) for url, exception in retrieved.failure: print '%r failed: %s' % (url, exception) Note that the main argument to ThreadPool is an iterator-processing function: one that takes an iterator and returns an iterator. A ThreadPool/Process simply distributes the input to workers running such function and gathers their output as a single stream. One important different between `stream` and `futures` is the order of returned results. The pool object itself is an iterable and the returned iterator's `next()` calls unblocks as soon as there is an output value. The order of output is the order of job completion, whereas for `futures.run_to_results()`, the order of the returned iterator is based on the submitted FutureList --- this means if the first item takes a long time to complete, subsequent processing of the output can not benefit from other results already available. The other difference is that there is no absolutely no abstraction but two bare iterables for client code to deal with: one iterable over the results, and one iterable over the failure; both are thread-safe. If delicate job control is necessary, an Executor can be used. It is implemented on top of the pool, and offers submit(*items) which returns job ids to be used for cancel() and status(). Jobs can be submitted and canceled concurrently. The documentation is available at . The code repository is located at . The implementation of ThreadPool, ProcessPool and Executor is little more than 300 lines of code. Peace, -- // aht http://blog.onideas.ws From brian at sweetapp.com Fri Jan 15 12:28:44 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 15 Jan 2010 22:28:44 +1100 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com> <1257607273.3437.13.camel@localhost> <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com> <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com> <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> Message-ID: <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> On 15 Jan 2010, at 21:50, Anh Hai Trinh wrote: > Hello all, > > I'd like to point out an alternative module with respect to > asynchronous computation: `stream` (which I wrote) supports > ThreadPool, ProcessPool and Executor with a simpler API and > implementation. Neat! I'm not sure that I'd agree with the simpler API part though :-) > My module takes a list-processing oriented view in which a > ThreadPool/ProcessPool is simply a way of working with each stream > element concurrently and output results possibly in out of order. > > A trivial example is: > > from stream import map > range(10) >> ThreadPool(map(lambda x: x*x)) >> sum > # returns 285 I think that you are probably missing an import. The equivalent using futures would be: from futures import ThreadPoolExecutor sum(ThreadPoolExecutor.map(lambda x: x*x, range(10)) > > The URLs retrieving example is: > > import urllib2 > from stream import ThreadPool > > URLs = [ > 'http://www.cnn.com/', > 'http://www.bbc.co.uk/', > 'http://www.economist.com/', > 'http://nonexistant.website.at.baddomain/', > 'http://slashdot.org/', > 'http://reddit.com/', > 'http://news.ycombinator.com/', > ] > > def retrieve(urls, timeout=10): > for url in urls: > yield url, urllib2.urlopen(url, timeout=timeout).read() > > if __name__ == '__main__': > retrieved = URLs >> ThreadPool(retrieve, poolsize=len(URLs)) > for url, content in retrieved: > print '%r is %d bytes' % (url, len(content)) > for url, exception in retrieved.failure: > print '%r failed: %s' % (url, exception) > > > Note that the main argument to ThreadPool is an iterator-processing > function: one that takes an iterator and returns an iterator. A > ThreadPool/Process simply distributes the input to workers running > such function and gathers their output as a single stream. "retrieve" seems to take multiple url arguments. Does ThreadPool using some sort of balancing strategy if poolsize where set to < len(URLs)? > One important different between `stream` and `futures` is the order of > returned results. The pool object itself is an iterable and the > returned iterator's `next()` calls unblocks as soon as there is an > output value. The order of output is the order of job completion, > whereas for `futures.run_to_results()`, the order of the returned > iterator is based on the submitted FutureList --- this means if the > first item takes a long time to complete, subsequent processing of the > output can not benefit from other results already available. Right, which is why futures has a as_completed() function. One difference is between the two implementations is that streamed remembers the arguments that it is processing while futures discards them when it doesn't need them. This was done for memory consumption reasons but the streamed approach seems to lead to simpler code. > The other difference is that there is no absolutely no abstraction but > two bare iterables for client code to deal with: one iterable over the > results, and one iterable over the failure; both are thread-safe. > > If delicate job control is necessary, an Executor can be used. It is > implemented on top of the pool, and offers submit(*items) which > returns job ids to be used for cancel() and status(). Jobs can be > submitted and canceled concurrently. What type is each "item" supposed to be? Can I wait on several items? What if they are created by different executors? Cheers, Brian > The documentation is available at >. > > The code repository is located at . > The implementation of ThreadPool, ProcessPool and Executor is little > more than 300 lines of code. > > > Peace, > > -- > // aht > http://blog.onideas.ws From anh.hai.trinh at gmail.com Fri Jan 15 14:56:47 2010 From: anh.hai.trinh at gmail.com (Anh Hai Trinh) Date: Fri, 15 Jan 2010 20:56:47 +0700 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> References: <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com> <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com> <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> Message-ID: > I'm not sure that I'd agree with the simpler API part though :-) I was referring to your old API. Still, we are both obviously very biased here :-p > Does ThreadPool using some > sort of balancing strategy if poolsize where set to < len(URLs)? Yes, of course! Otherwise it wouldn't really qualify as a pool. > "retrieve" seems to take multiple url arguments. Correct. `retrieve` is simply a generator that retrieve URLs sequentially, the ThreadPool distributes the input stream so that each workers get an iterator over its work load. >> If delicate job control is necessary, an Executor can be used. It is >> implemented on top of the pool, and offers submit(*items) which >> returns job ids to be used for cancel() and status(). Jobs can be >> submitted and canceled concurrently. > > What type is each "item" supposed to be? Whatever your iterator-processing function is supposed to process. The URLs example can be written using an Executor as: e = Executor(ThreadPool, retrieve) e.submit(*URLs) e.close() print list(e.result) > Can I wait on several items? Do you mean wait for several particular input values to be completed? As of this moment, yes but rather inefficiently. I have not considered it is a useful feature, especially when taking a wholesale, list-processing view: that a worker pool process its input stream _out_of_order_. If you just want to wait for several particular items, it means you need their outputs _in_order_, so why do you want to use a worker pool in the first place? However, I'd be happy to implement something like Executor.submit(*items, wait=True). Cheers, aht From brett at python.org Fri Jan 15 22:19:12 2010 From: brett at python.org (Brett Cannon) Date: Fri, 15 Jan 2010 13:19:12 -0800 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <1257607273.3437.13.camel@localhost> <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com> <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com> <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> Message-ID: On Fri, Jan 15, 2010 at 02:50, Anh Hai Trinh wrote: > Hello all, > > I'd like to point out an alternative module with respect to > asynchronous computation: `stream` (which I wrote) supports > ThreadPool, ProcessPool and Executor with a simpler API and > implementation. > > My module takes a list-processing oriented view in which a > ThreadPool/ProcessPool is simply a way of working with each stream > element concurrently and output results possibly in out of order. > > A trivial example is: > > ?from stream import map > ?range(10) >> ThreadPool(map(lambda x: x*x)) >> sum > ?# returns 285 I have not looked at the code at all, but the overloading of binary shift is not going to be viewed as a good thing. I realize there is an analogy to C++ streams, but typically Python's stdlib frowns upon overloading operators like this beyond what a newbie would think an operator is meant to do. -Brett > > > The URLs retrieving example is: > > ?import urllib2 > ?from stream import ThreadPool > > ?URLs = [ > ? ? 'http://www.cnn.com/', > ? ? 'http://www.bbc.co.uk/', > ? ? 'http://www.economist.com/', > ? ? 'http://nonexistant.website.at.baddomain/', > ? ? 'http://slashdot.org/', > ? ? 'http://reddit.com/', > ? ? 'http://news.ycombinator.com/', > ?] > > ?def retrieve(urls, timeout=10): > ? ? for url in urls: > ? ? ? ?yield url, urllib2.urlopen(url, timeout=timeout).read() > > ?if __name__ == '__main__': > ? ? retrieved = URLs >> ThreadPool(retrieve, poolsize=len(URLs)) > ? ? for url, content in retrieved: > ? ? ? ?print '%r is %d bytes' % (url, len(content)) > ? ? for url, exception in retrieved.failure: > ? ? ? ?print '%r failed: %s' % (url, exception) > > > Note that the main argument to ThreadPool is an iterator-processing > function: one that takes an iterator and returns an iterator. A > ThreadPool/Process simply distributes the input to workers running > such function and gathers their output as a single stream. > > One important different between `stream` and `futures` is the order of > returned results. ?The pool object itself is an iterable and the > returned iterator's `next()` calls unblocks as soon as there is an > output value. ?The order of output is the order of job completion, > whereas for `futures.run_to_results()`, the order of the returned > iterator is based on the submitted FutureList --- this means if the > first item takes a long time to complete, subsequent processing of the > output can not benefit from other results already available. > > The other difference is that there is no absolutely no abstraction but > two bare iterables for client code to deal with: one iterable over the > results, and one iterable over the failure; both are thread-safe. > > If delicate job control is necessary, an Executor can be used. It is > implemented on top of the pool, and offers submit(*items) which > returns job ids to be used for cancel() and status(). ?Jobs can be > submitted and canceled concurrently. > > The documentation is available at . > > The code repository is located at . > The implementation of ThreadPool, ProcessPool and Executor is little > more than 300 lines of code. > > > Peace, > > -- > // aht > http://blog.onideas.ws > _______________________________________________ > stdlib-sig mailing list > stdlib-sig at python.org > http://mail.python.org/mailman/listinfo/stdlib-sig > From brian at sweetapp.com Sat Jan 16 00:06:39 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 16 Jan 2010 10:06:39 +1100 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com> <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com> <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> Message-ID: <97A57947-CD95-49AF-AC6A-96843536266E@sweetapp.com> On 16 Jan 2010, at 00:56, Anh Hai Trinh wrote: >> I'm not sure that I'd agree with the simpler API part though :-) > > I was referring to your old API. Still, we are both obviously very > biased here :-p For sure. I'm definitely used to looking at Future-style code so I find the model intuitive. >> Does ThreadPool using some >> sort of balancing strategy if poolsize where set to < len(URLs)? > > Yes, of course! Otherwise it wouldn't really qualify as a pool. > >> "retrieve" seems to take multiple url arguments. > > Correct. `retrieve` is simply a generator that retrieve URLs > sequentially, the ThreadPool distributes the input stream so that each > workers get an iterator over its work load. That's a neat idea - it saves you the overhead of a function call. >>> If delicate job control is necessary, an Executor can be used. It is >>> implemented on top of the pool, and offers submit(*items) which >>> returns job ids to be used for cancel() and status(). Jobs can be >>> submitted and canceled concurrently. >> >> What type is each "item" supposed to be? > > Whatever your iterator-processing function is supposed to process. > The URLs example can be written using an Executor as: > > e = Executor(ThreadPool, retrieve) > e.submit(*URLs) > e.close() > print list(e.result) There are two common scenarios where I have seen Future-like things used: 1. Do the same operation on different data e.g. copy some local files to several remote servers 2. Do several different operations on different data e.g. parallelizing code like this: db = setup_database(host, port) data = parse_big_xml_file(request.body) save_data_in_db(data, db) I'm trying to get a handle on how streams accommodates the second case. With futures, I would write something like this: db_future = executor.submit(setup_database, host, port) data_future = executor.submit(parse_big_xml_file, data) # Maybe do something else here. wait( [db_future, data_future], timeout=10, # If either function raises then we can't complete the operation so # there is no reason to make the user wait. return_when=FIRST_EXCEPTION) db = db_future.result(timeout=0) data = data.result(timeout=0) save_data_in_db(data, db) Cheers, Brian > >> Can I wait on several items? > > Do you mean wait for several particular input values to be completed? > As of this moment, yes but rather inefficiently. I have not considered > it is a useful feature, especially when taking a wholesale, > list-processing view: that a worker pool process its input stream > _out_of_order_. If you just want to wait for several particular > items, it means you need their outputs _in_order_, so why do you want > to use a worker pool in the first place? > > However, I'd be happy to implement something like > Executor.submit(*items, wait=True). > > Cheers, > aht > _______________________________________________ > stdlib-sig mailing list > stdlib-sig at python.org > http://mail.python.org/mailman/listinfo/stdlib-sig From anh.hai.trinh at gmail.com Sat Jan 16 15:44:20 2010 From: anh.hai.trinh at gmail.com (Anh Hai Trinh) Date: Sat, 16 Jan 2010 21:44:20 +0700 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: <97A57947-CD95-49AF-AC6A-96843536266E@sweetapp.com> References: <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> <97A57947-CD95-49AF-AC6A-96843536266E@sweetapp.com> Message-ID: > 2. Do several different operations on different data e.g. parallelizing code > like this: > > db = setup_database(host, port) > data = parse_big_xml_file(request.body) > save_data_in_db(data, db) > > I'm trying to get a handle on how streams accommodates the second case. With > futures, I would write something like this: > > db_future = executor.submit(setup_database, host, port) > data_future = executor.submit(parse_big_xml_file, data) > # Maybe do something else here. > wait( > ? ?[db_future, data_future], > ? ?timeout=10, > ? ?# If either function raises then we can't complete the operation so > ? ?# there is no reason to make the user wait. > ? ?return_when=FIRST_EXCEPTION) > > db = db_future.result(timeout=0) > data = data.result(timeout=0) > save_data_in_db(data, db) For this kind of scenario, I feel `futures` and friends are not needed. My solution is to explicit use different threads for different operations then use join() thread to wait for a particular operation. Threading concurrency means memory is shared and thread.join() can be used to synchronize events. Generally, I would be doubtful about any library that support parallelization of code that "do several different operations on different data". One could have put it as "write concurrent programs", to which the answer must be a complete concurrency model: threading, multiprocessing, Erlang, Goroutines and CSP channels, etc. Cheers, -- // aht http://blog.onideas.ws From brian at sweetapp.com Sat Jan 16 23:22:13 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 17 Jan 2010 09:22:13 +1100 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> <97A57947-CD95-49AF-AC6A-96843536266E@sweetapp.com> Message-ID: On 17 Jan 2010, at 01:44, Anh Hai Trinh wrote: >> 2. Do several different operations on different data e.g. >> parallelizing code >> like this: >> >> db = setup_database(host, port) >> data = parse_big_xml_file(request.body) >> save_data_in_db(data, db) >> >> I'm trying to get a handle on how streams accommodates the second >> case. With >> futures, I would write something like this: >> >> db_future = executor.submit(setup_database, host, port) >> data_future = executor.submit(parse_big_xml_file, data) >> # Maybe do something else here. >> wait( >> [db_future, data_future], >> timeout=10, >> # If either function raises then we can't complete the operation >> so >> # there is no reason to make the user wait. >> return_when=FIRST_EXCEPTION) >> >> db = db_future.result(timeout=0) >> data = data.result(timeout=0) >> save_data_in_db(data, db) > > For this kind of scenario, I feel `futures` and friends are not > needed. My solution is to explicit use different threads for different > operations then use join() thread to wait for a particular operation. > Threading concurrency means memory is shared and thread.join() can be > used to synchronize events. It is definitely true that you can roll your own implementation using threads but the purpose of the futures library is to make that unnecessary. > Generally, I would be doubtful about any library that support > parallelization of code that "do several different operations on > different data". One could have put it as "write concurrent programs", > to which the answer must be a complete concurrency model: threading, > multiprocessing, Erlang, Goroutines and CSP channels, etc. I don't understand your doubts. To me the example that I gave is simple and useful. Cheers, Brian From jnoller at gmail.com Sun Jan 17 00:17:25 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 16 Jan 2010 18:17:25 -0500 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> <97A57947-CD95-49AF-AC6A-96843536266E@sweetapp.com> Message-ID: <4222a8491001161517o2af00a5fted48c4353c86a383@mail.gmail.com> On Sat, Jan 16, 2010 at 5:22 PM, Brian Quinlan wrote: > It is definitely true that you can roll your own implementation using > threads but the purpose of the futures library is to make that unnecessary. I'd like to stress this; futures/pools/etc are common enough patterns (and I get requests to add more to multiprocessing) that it makes sense as an add-on to the stdlib. This is sugar; not magic. jesse From anh.hai.trinh at gmail.com Sun Jan 17 05:53:10 2010 From: anh.hai.trinh at gmail.com (Anh Hai Trinh) Date: Sun, 17 Jan 2010 11:53:10 +0700 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> <97A57947-CD95-49AF-AC6A-96843536266E@sweetapp.com> Message-ID: On Sun, Jan 17, 2010 at 5:22 AM, Brian Quinlan wrote: >>> db_future = executor.submit(setup_database, host, port) >>> data_future = executor.submit(parse_big_xml_file, data) >>> # Maybe do something else here. >>> wait( >>> ? [db_future, data_future], >>> ? timeout=10, >>> ? # If either function raises then we can't complete the operation so >>> ? # there is no reason to make the user wait. >>> ? return_when=FIRST_EXCEPTION) >>> >>> db = db_future.result(timeout=0) >>> data = data.result(timeout=0) >>> save_data_in_db(data, db) > > It is definitely true that you can roll your own implementation using > threads but the purpose of the futures library is to make that unnecessary. > > I don't understand your doubts. To me the example that I gave is simple and > useful. What I mean is that your example is simple enough to do with threads. Here: [...] def setup_db(): nonlocal db; db = setup_database(host, port) def parse_xml(): nonlocal data; data = parse_big_xml(file) db_thread = threading.Thread(target=setup_db) db_thread.start() parse_thread = threading.Thread(target=parse_xml) parse_thread.start() [...] # Do something else here. db_thread.join() parse_thread.join() save_data_in_db(data, db) I used "nonlocal" here but you usually do this within a method and refer to self.db, self.data. > I don't understand your doubts. To me the example that I gave is simple and useful. My doubt is about the usefulness of futures' constructs for the kind of code that "Do several different operations on different data". I think ThreadPool/ProcessPool is really useful when you do 1. Same operation on different data 2. Different operations on same datum But 3. Different operations on different data is perhaps misusing it. It is a too general use case because dependency comes into play. What if the different operations depend on each other? A useful thread pool construct for this would be at a more fundamental level, e.g. Grand Central Dispatch. Perhaps you would give another example? Cheers, -- // aht http://blog.onideas.ws From brett at python.org Sun Jan 17 06:44:00 2010 From: brett at python.org (Brett Cannon) Date: Sat, 16 Jan 2010 21:44:00 -0800 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> <5B433E1E-DEC2-4B4A-8F24-926107BC7EDA@sweetapp.com> <97A57947-CD95-49AF-AC6A-96843536266E@sweetapp.com> Message-ID: Do you guys mind taking this discussion off-list? As of right now neither of your projects are old enough or well known enough to be considered for inclusion in the stdlib at this time so this is really not relevant to the stdlib SIG to continue here. -Brett On Sat, Jan 16, 2010 at 20:53, Anh Hai Trinh wrote: > On Sun, Jan 17, 2010 at 5:22 AM, Brian Quinlan wrote: > >>>> db_future = executor.submit(setup_database, host, port) >>>> data_future = executor.submit(parse_big_xml_file, data) >>>> # Maybe do something else here. >>>> wait( >>>> ? [db_future, data_future], >>>> ? timeout=10, >>>> ? # If either function raises then we can't complete the operation so >>>> ? # there is no reason to make the user wait. >>>> ? return_when=FIRST_EXCEPTION) >>>> >>>> db = db_future.result(timeout=0) >>>> data = data.result(timeout=0) >>>> save_data_in_db(data, db) >> >> It is definitely true that you can roll your own implementation using >> threads but the purpose of the futures library is to make that unnecessary. >> >> I don't understand your doubts. To me the example that I gave is simple and >> useful. > > What I mean is that your example is simple enough to do with threads. Here: > > [...] > > def setup_db(): > ?nonlocal db; > ?db = setup_database(host, port) > > def parse_xml(): > ?nonlocal data; > ?data = parse_big_xml(file) > > db_thread = threading.Thread(target=setup_db) > db_thread.start() > > parse_thread = threading.Thread(target=parse_xml) > parse_thread.start() > > [...] # Do something else here. > > db_thread.join() > parse_thread.join() > save_data_in_db(data, db) > > I used "nonlocal" here but you usually do this within a method and > refer to self.db, self.data. > > >> I don't understand your doubts. To me the example that I gave is simple and useful. > > My doubt is about the usefulness of futures' constructs for the kind > of code that "Do several different operations on different data". I > think ThreadPool/ProcessPool is really useful when you do > > 1. Same operation on different data > 2. Different operations on same datum > > But > > 3. Different operations on different data > > is perhaps misusing it. It is a too general use case because > dependency comes into play. What if the different operations depend on > each other? A useful thread pool construct for this would be at a more > fundamental level, e.g. Grand Central Dispatch. > > Perhaps you would give another example? > > Cheers, > -- > // aht > http://blog.onideas.ws > _______________________________________________ > stdlib-sig mailing list > stdlib-sig at python.org > http://mail.python.org/mailman/listinfo/stdlib-sig > From brian at sweetapp.com Fri Jan 29 03:22:06 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 29 Jan 2010 13:22:06 +1100 Subject: [stdlib-sig] futures - a new package for asynchronous execution In-Reply-To: References: <4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com> <1257605583.3437.3.camel@localhost> <4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com> <1257607273.3437.13.camel@localhost> <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com> <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com> <37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com> <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com> <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com> Message-ID: <3BB0755A-2D85-46AF-89A6-CFC4A56744AF@sweetapp.com> I've updated the PEP and included it inline. The interesting changes start in the "Specification" section. Cheers, Brian PEP: XXX Title: futures - execute computations asynchronously Version: $Revision$ Last-Modified: $Date$ Author: Brian Quinlan Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 16-Oct-2009 Python-Version: 3.2 Post-History: ======== Abstract ======== This PEP proposes a design for a package that facilitates the evaluation of callables using threads and processes. ========== Motivation ========== Python currently has powerful primitives to construct multi-threaded and multi-process applications but parallelizing simple operations requires a lot of work i.e. explicitly launching processes/threads, constructing a work/ results queue, and waiting for completion or some other termination condition (e.g. failure, timeout). It is also difficult to design an application with a global process/thread limit when each component invents its own parallel execution strategy. ============= Specification ============= Check Prime Example ------------------- :: import futures import math PRIMES = [ 112272535095293, 112582705942171, 112272535095293, 115280095190773, 115797848077099, 1099726899285419] def is_prime(n): if n % 2 == 0: return False sqrt_n = int(math.floor(math.sqrt(n))) for i in range(3, sqrt_n + 1, 2): if n % i == 0: return False return True with futures.ProcessPoolExecutor() as executor: for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)): print('%d is prime: %s' % (number, is_prime)) Web Crawl Example ----------------- :: import futures import urllib.request URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] def load_url(url, timeout): return urllib.request.urlopen(url, timeout=timeout).read() with futures.ThreadPoolExecutor(max_threads=5) as executor: future_to_url = dict((executor.submit(load_url, url, 60), url) for url in URLS) for future in futures.as_completed(future_to_url): url = future_to_url[future] if future.exception() is not None: print('%r generated an exception: %s' % (url, future.exception())) else: print('%r page is %d bytes' % (url, len(future.result()))) Interface --------- The proposed package provides two core classes: `Executor` and `Future`. An `Executor` receives asynchronous work requests (in terms of a callable and its arguments) and returns a `Future` to represent the execution of that work request. Executor '''''''' `Executor` is an abstract class that provides methods to execute calls asynchronously. `submit(fn, *args, **kwargs)` Schedules the callable to be executed as fn(*\*args*, *\*\*kwargs*) and returns a `Future` instance representing the execution of the function. `map(func, *iterables, timeout=None)` Equivalent to map(*func*, *\*iterables*) but executed asynchronously and possibly out-of-order. The returned iterator raises a `TimeoutError` if `__next__()` is called and the result isn't available after *timeout* seconds from the original call to `run_to_results()`. If *timeout* is not specified or ``None`` then there is no limit to the wait time. If a call raises an exception then that exception will be raised when its value is retrieved from the iterator. `Executor.shutdown(wait=False)` Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to `Executor.run_to_futures`, `Executor.run_to_results` and `Executor.map` made after shutdown will raise `RuntimeError`. If wait is `True` then the executor will not return until all the pending futures are done executing and the resources associated with the executor have been freed. ProcessPoolExecutor ''''''''''''''''''' The `ProcessPoolExecutor` class is an `Executor` subclass that uses a pool of processes to execute calls asynchronously. `__init__(max_processes)` Executes calls asynchronously using a pool of a most *max_processes* processes. If *max_processes* is ``None`` or not given then as many worker processes will be created as the machine has processors. ThreadPoolExecutor '''''''''''''''''' The `ThreadPoolExecutor` class is an `Executor` subclass that uses a pool of threads to execute calls asynchronously. `__init__(max_threads)` Executes calls asynchronously using a pool of at most *max_threads* threads. Future Objects '''''''''''''' The `Future` class encapsulates the asynchronous execution of a function or method call. `Future` instances are returned by `Executor.submit`. `cancel()` Attempt to cancel the call. If the call is currently being executed then it cannot be cancelled and the method will return `False`, otherwise the call will be cancelled and the method will return `True`. `Future.cancelled()` Return `True` if the call was successfully cancelled. `Future.done()` Return `True` if the call was successfully cancelled or finished running. `result(timeout=None)` Return the value returned by the call. If the call hasn't yet completed then this method will wait up to *timeout* seconds. If the call hasn't completed in *timeout* seconds then a `TimeoutError` will be raised. If *timeout* is not specified or ``None`` then there is no limit to the wait time. If the future is cancelled before completing then `CancelledError` will be raised. If the call raised then this method will raise the same exception. `exception(timeout=None)` Return the exception raised by the call. If the call hasn't yet completed then this method will wait up to *timeout* seconds. If the call hasn't completed in *timeout* seconds then a `TimeoutError` will be raised. If *timeout* is not specified or ``None`` then there is no limit to the wait time. If the future is cancelled before completing then `CancelledError` will be raised. If the call completed without raising then ``None`` is returned. `index` int indicating the index of the future in its `FutureList`. Module Functions '''''''''''''''' `wait(fs, timeout=None, return_when=ALL_COMPLETED)` Wait for the `Future` instances in the given sequence to complete. Returns a 2-tuple of sets. The first set contains the futures that completed (finished or were cancelled) before the wait completed. The second set contains uncompleted futures. This method should always be called using keyword arguments, which are: *fs* is the sequence of Future instances that should be waited on. *timeout* can be used to control the maximum number of seconds to wait before returning. If timeout is not specified or None then there is no limit to the wait time. *return_when* indicates when the method should return. It must be one of the following constants: ============================= ================================================== Constant Description ============================= ================================================== `FIRST_COMPLETED` The method will return when any call finishes. `FIRST_EXCEPTION` The method will return when any call raises an exception or when all calls finish. `ALL_COMPLETED` The method will return when all calls finish. `RETURN_IMMEDIATELY` The method will return immediately. ============================= ================================================== `as_completed(fs, timeout=None)` Returns an iterator over the Future instances given by *fs* that yields futures as they complete (finished or were cancelled). Any futures that completed before `as_completed()` was called will be yielded first. The returned iterator raises a `TimeoutError` if `__next__()` is called and the result isn?t available after *timeout* seconds from the original call to `as_completed()`. If *timeout* is not specified or `None` then there is no limit to the wait time. ========= Rationale ========= The proposed design of this module was heavily influenced by the the Java java.util.concurrent package [1]_. The conceptual basis of the module, as in Java, is the Future class, which represents the progress and result of an asynchronous computation. The Future class makes little commitment to the evaluation mode being used e.g. it can be be used to represent lazy or eager evaluation, for evaluation using threads, processes or remote procedure call. Futures are created by concrete implementations of the Executor class (called ExecutorService in Java). The reference implementation provides classes that use either a process a thread pool to eagerly evaluate computations. Futures have already been seen in Python as part of a popular Python cookbook recipe [2]_ and have discussed on the Python-3000 mailing list [3]_. The proposed design is explicit i.e. it requires that clients be aware that they are consuming Futures. It would be possible to design a module that would return proxy objects (in the style of `weakref`) that could be used transparently. It is possible to build a proxy implementation on top of the proposed explicit mechanism. The proposed design does not introduce any changes to Python language syntax or semantics. Special syntax could be introduced [4]_ to mark function and method calls as asynchronous. A proxy result would be returned while the operation is eagerly evaluated asynchronously, and execution would only block if the proxy object were used before the operation completed. ======================== Reference Implementation ======================== The reference implementation [5]_ contains a complete implementation of the proposed design. It has been tested on Linux and Mac OS X. ========== References ========== .. [1] `java.util.concurrent` package documentation `http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/package-summary.html ` .. [2] Python Cookbook recipe 84317, "Easy threading with Futures" `http://code.activestate.com/recipes/84317/` .. [3] `Python-3000` thread, "mechanism for handling asynchronous concurrency" `http://mail.python.org/pipermail/python-3000/2006-April/ 000960.html` .. [4] `Python 3000` thread, "Futures in Python 3000 (was Re: mechanism for handling asynchronous concurrency)" `http://mail.python.org/pipermail/python-3000/2006-April/ 000970.html` .. [5] Reference `futures` implementation `http://code.google.com/p/pythonfutures ` ========= Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From tim at ksu.edu Sun Jan 31 23:46:28 2010 From: tim at ksu.edu (tim at ksu.edu) Date: Sun, 31 Jan 2010 14:46:28 -0800 (PST) Subject: [stdlib-sig] socket.makefile() questions In-Reply-To: <1199139348.5692621264977965383.JavaMail.root@mbs107.c101.zcs.mail.ac4.yahoo.net> Message-ID: <1126304848.5692691264977988530.JavaMail.root@mbs107.c101.zcs.mail.ac4.yahoo.net> My apologies if this message is to the wrong group. I have been experimenting with socket.makefile()from Python 3.1.1. I have not had much difficulty reading from the returned file object, but I don't understand the behavior when trying to write (send on the socket). I'm hoping that someone can explain how this is supposed to work. I find that this works for an established connection on socket s: fd = s.makefile('wb', buffering = 0) fd.write("This is a test message\n".encode('ascii')) A mode of 'rwb' also works. The object fd is of type SocketIO. fd = s.makefile('w', buffering = 0) -> ValueError exception fd = s.makefile('w') -> io.BufferedWriter, which does not send data. fd = s.makefile('wb') -> io.TextIOWrapper, which does not send data. The default value of the "buffering" parameter is None, which from my testing has a different result than 0 (zero). So, questions: 1) Why does buffering = None result in a buffered file object? 2) Are there bugs or incomplete work with socket.makefile(), io.BufferedWriter and io.TextIOWrapper in terms of why the latter two objects are returned, but fail to send data? Thank you, -- Tim Bower Assistant Professor Kansas State University at Salina Computer Systems Technology tim at ksu.edu