ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?

[Ian Kelly]

On Thu, Sep 25, 2014 at 8:47 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> Ian Kelly <ian.g.kelly at gmail.com>:
>
>> On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>>> Example (pseudocode):
>>>
>>>    def callback(self):
>>>       with self.lock:
>>>           ...
>>>
>>>    def xyz(self, f):
>>>       with self.lock:
>>>           ...
>>>           f.add_done_callback(self.callback)
>>>
>>> The code will deadlock if the callback is invoked immediately.
>>
>> Easily solved using a re-entrant lock.
>
> Not easily. Your state transition might not be complete at the time of
> the preemption. Many such issues can be resolved by moving the
> preemptive function calls to the end of the critical section, but that
> is not always possible, easy or taken into account.

Fair enough. In the simple example above, I see no harm in simply
moving the add_done_callback call outside (after) the "with self.lock"
block altogether. I can imagine more complex examples where the call
is further down the stack and harder to isolate from the lock,
although I would argue that's probably a code smell.

> If preemption needs to be prepared for, the real solution is:
>
>
>    def xyz(self, f):
>        with self.lock:
>            ...
>            def add_callback():
>                f.add_done_callback(self.callback)
>            self.executor.schedule(add_callback)
>
> which is ugly, of course.

YMMV, but I think I would prefer to make the executor.schedule call be
the callback:

    def callback(self, f):
        with self.lock:
            ...

    def xyz(self, f):
        with self.lock:
            ...
            f.add_done_callback(
                functools.partial(self.executor.schedule, self.callback))