[Python-ideas] Better integration of multiprocessing with asyncio

[Dan O'Reilly]

Right - I was just calling it out in the example to highlight that the
aioprocessing calls wouldn't block the event loop, as opposed to the
equivalent multiprocessing calls, which would. Though I suppose that should
be obvious from the use of "yield from". I'll just remove the comments from
the example and add a sentence or two afterwards that clarifies the event
loop is never blocked.

On Thu, Oct 9, 2014 at 12:35 AM, Guido van Rossum <guido at python.org> wrote:

> It takes time to get used to it, but eventually you don't have to think
> about the event loop. There really is no need for any comment of that kind
> at all. The beauty is that *nothing* blocks the event loop...
>
>
> On Wednesday, October 8, 2014, Dan O'Reilly <oreilldf at gmail.com> wrote:
>
>> Ah, right. I find the terminology confusing myself, and I probably got it
>> wrong. I meant "non-blocking" as in - "this won't block the event loop",
>> not "this won't block the coroutine". I'll try to clarify that. Thanks!
>>
>> On Thu, Oct 9, 2014 at 12:03 AM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> This looks neat. I skimmed the README, and I noticed you marked up most
>>> "yield from" epressions with "# non blocking". That feels confusing to me,
>>> because when I read asyncio code, I think fo "yield from" as blocking (the
>>> task, if not the world :-).  What do you think?
>>>
>>> --Guido
>>>
>>>
>>> On Tuesday, October 7, 2014, Dan O'Reilly <oreilldf at gmail.com> wrote:
>>>
>>>> For what it's worth, I ended up writing a module that takes the
>>>> "threads on top of regular multiprocessing" approach to integrate the two:
>>>> https://github.com/dano/aioprocessing.
>>>>
>>>> Re-implementing multiprocessing to use asyncio internally, while an
>>>> interesting exercise, would require a very large amount of effort both to
>>>> implement and maintain alongside the current multiprocessing module. I'm
>>>> not sure it's really worth it when using threads on top of multiprocessing
>>>> gives you the same effect without requiring you to basically re-implement
>>>> large parts of the multiprocessing module.
>>>>
>>>> Anyway, we'll see if it ends up getting much use...
>>>>
>>>> On Sat, Jul 26, 2014 at 11:48 PM, Ryan Hiebert <ryan at ryanhiebert.com>
>>>> wrote:
>>>>
>>>>>
>>>>> On Jul 26, 2014, at 10:43 PM, Guido van Rossum <guido at python.org>
>>>>> wrote:
>>>>>
>>>>> I'm going to go out on a limb here and say that it feels too early to
>>>>> me. First someone has to actually solve this problem well as a 3rd party
>>>>> package before we can talk about adding it to the asyncio package. It
>>>>> doesn't actually sound like Billiards has adapted to asyncio yet (not that
>>>>> I have any idea what Billiards is -- it sounds like a fork of
>>>>> multiprocessing actually?).
>>>>>
>>>>>
>>>>> Yep, Billiard is a fork of multiprocessing:
>>>>> https://pypi.python.org/pypi/billiard
>>>>>
>>>>
>>>>
>>>
>>> --
>>> --Guido van Rossum (on iPad)
>>>
>>
>>
>
> --
> --Guido van Rossum (on iPad)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141009/6ac3776b/attachment.html>