[issue14119] Ability to adjust queue size in Executors

[Michael Hrivnak]

Michael Hrivnak <mhrivnak at hrivnak.org> added the comment:

My project also has a use case for this, very similar to the others described. Here's what we want:

with ThreadPoolExecutor(queue_size=500) as executor:
  for item in parse_a_long_list_of_work(somefile.xml):
    executor.submit(Job(item))

I do not want to parse the entire list of work items and load them into memory at once. It is preferable for the main thread running the above code to block on submit() when the queue size is above some threshold.

It's a classic case of the producer and consumer operating at different speeds. In the past, a Queue object has been the way to connect such a producer and consumer. The various Executor classes do not provide an easy way to consume from a provided Queue object, so giving them that capability would be a reasonable alternative to having the submit() method block.

----------
nosy: +mhrivnak

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue14119>
_______________________________________