ThreadPoolExecutor - callback guaranteed to run in same thread as the submitted function?

[Marko Rauhamaa]

Ian Kelly <ian.g.kelly at gmail.com>:

> On Thu, Sep 25, 2014 at 3:46 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> Example (pseudocode):
>>
>>    def callback(self):
>>       with self.lock:
>>           ...
>>
>>    def xyz(self, f):
>>       with self.lock:
>>           ...
>>           f.add_done_callback(self.callback)
>>
>> The code will deadlock if the callback is invoked immediately.
>
> Easily solved using a re-entrant lock.

Not easily. Your state transition might not be complete at the time of
the preemption. Many such issues can be resolved by moving the
preemptive function calls to the end of the critical section, but that
is not always possible, easy or taken into account.

If preemption needs to be prepared for, the real solution is:


   def xyz(self, f):
       with self.lock:
           ...
           def add_callback():
               f.add_done_callback(self.callback)
           self.executor.schedule(add_callback)

which is ugly, of course.

> Alternatively, one could use synchronization to make sure the future
> can't complete until all callbacks have been added. That would have
> the benefit of keeping the callback processing on the same thread as
> the future, as the OP wanted.

You can still keep the callback in the same thread if your executor is
tied to the the same thread as the calling function. That's the classic
event loop. The executor API should be independent of the implementation
(single vs multithreaded), but it should be able to guarantee that it
never makes callbacks preemptively.


Marko