[Python-ideas] fork

[Andrew Barnert]

On Aug 3, 2015, at 10:11, Sven R. Kunze <srkunze at mail.de> wrote:
> 
>> On 02.08.2015 02:02, Andrew Barnert wrote:
>> Your idea of having a single global "pool manager" object, where you could submit tasks and, depending on how they're marked, they get handled differently might have merit. But that's something you could build pretty easily on top of concurrent.futures (at least for threads vs. processes; you can add in coroutines later, because they're not quite as easy to integrate), upload to PyPI,
> 
> You mean something like this?
> 
> https://pypi.python.org/pypi/xfork

Did you just write this today? Then yes, that proves my point about how easy it is to write it. Now you just have to get people using it, get some experience with it, etc. and you can come back with a proposal to put something like this in the stdlib, add syntactic support, etc. that it will be hard for anyone to disagree with. (Or to discover that it has flaws that need to be fixed, or fundamental flaws that can't be fixed, before making the proposal.)

One quick comment: from my experience (mostly with other languages that are very different from Python, so I can't promise how well it applies here...), implicit futures without implicit laziness or even an explicit delay mechanism are not as useful as they look at first glance. Code that forks off 8 Fibonacci calls, but waits for each one's result before forking off the next one, might as well have just stayed sequential. And if you're going to use the result by forking off another job, then it's actually more convenient to use explicit futures like the ones in the stdlib.

One slightly bigger idea: If you really want to pursue your implicit-as-possible design further, you might want to consider making the decorators replace the function with an object whose __call__ method just implicitly submits it to the pool. Then you can use normal function-calling syntax and pretend everything is magic. You can even add operator dunder methods to your future class that do the same thing (so "result * 2" just builds a new future out of "self.get() * 2", either submitted to the pool, probably better, tacked on as an add_done_callback). I think there's a limit to how far you can push this without some mechanism to mark when you need to actual value (in ML-derived languages and C++, static types make this easier: a cast, implicit or explicit, forces a wait; in Python, that doesn't work), but it might be worth exploring that limit. Or it might be better to just stop at the magic function calls and leave the futures alone.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150803/7c496e48/attachment.html>