[Python-Dev] parallelizing
Matthieu Bec
mdcb808 at gmail.com
Wed Sep 13 15:11:29 EDT 2017
Thank you, I'll take your advice.
Regarding your example, I think it gives the illusion to work because
sleep() is GIL aware under the hood.
I don't think it works for process() that mainly runs bytecode, because
of the GIL.
Sorry if I wrongly thought that was a language level discussion.
Regards,
Matthieu
On 9/13/17 10:49 AM, Chris Barker wrote:
> This really isn't the place to ask this kind of question.
>
> If you want to know how to do something with python, try python-users
> , stack overflow, etc.
>
> If you have an idea about a new feature you think python could have,
> then the python-ideas list is the place for that. But if you want
> anyone to take it seriously, it should be a better formed idea before
> you post there.
>
> But:
>
> On Tue, Sep 12, 2017 at 4:43 PM, Matthieu Bec <mdcb808 at gmail.com
> <mailto:mdcb808 at gmail.com>> wrote:
>
> There are times when you deal with completely independent
> input/output 'pipes' - where parallelizing would really help speed
> things up.
>
> Can't there be a way to capture that idiom and multi thread it in
> the language itself?
>
> Example:
>
> loop:
>
> read an XML
>
> produce a JSON like
>
>
> Regular old threading works fine for this:
>
> import time
> import random
> import threading
>
>
> def process(infile, outfile):
> "fake function to simulate a process that takes a random amount of
> time"
> time.sleep(random.random())
> print("processing: {} to make {}".format(infile, outfile))
>
>
> for i in range(10):
> threading.Thread(target=process, args=("file%i.xml" % i,
> "file%i.xml" % i)).start()
>
>
> It gets complicated if you need to pass information back and forth, or
> worry about race conditions, or manage a queue, or ....
>
> But just running a nice self-contained thread safe function in another
> thread is pretty straightforward.
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris.Barker at noaa.gov <mailto:Chris.Barker at noaa.gov>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170913/4c56e26f/attachment.html>
More information about the Python-Dev
mailing list