Nesting concurrent.futures.ThreadPoolExecutor

[Rob Gaddi]

On 07/20/2017 12:44 PM, Andrew McLean wrote:
> I have a program where I am currently using a 
> concurrent.futures.ThreadPoolExecutor to run multiple tasks 
> concurrently. These tasks are typically I/O bound, involving access to 
> local databases and remote REST APIs. However, these tasks could 
> themselves be split into subtasks, which would also benefit from 
> concurrency.
> 
> What I am hoping is that it is safe to use a 
> concurrent.futures.ThreadPoolExecutor within the tasks. I have coded up 
> a toy example, which seems to work. However, I'd like some confidence 
> that this is intentional. Concurrency is notoriously tricky.
> 
> I very much hope this is safe, because otherwise it would not be safe to 
> use a ThreadPoolExecutor to execute arbitrary code, in case it also used 
> concurrent.futures to exploit concurrency.
> 

Well that last statement is clearly false.  It's not safe to use any 
multiple access mechanism (threading, processes, async stuff) to execute 
arbitrary code; so it's by definition not safe to use a ThreadPoolExecutor.

I'm not being cute and using "safe" in some Turing sense, I'm talking 
specifically about multiple accesses.  Whenever you have multiple access 
you have interlock issues, which you resolve with mutexes or message 
queues or however you decide to do so.  Those issues don't go away when 
you use the ThreadPoolExecutor.  There is every possibility, especially 
if you start recursively spawning threads, that A spawns B, A blocks on 
something that B is supposed to do (such as completing a Future), but 
due to the thread limit of the pool, the mere existence of A is 
preventing B from being executed, and you have a deadlock.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.