multiprocessing.Pool, its queue, and pre-emption

[John Ladasky]

On Sep 15, 3:14 pm, Chris Angelico <ros... at gmail.com> wrote:
> On Fri, Sep 16, 2011 at 6:52 AM, John Ladasky <lada... at my-deja.com> wrote:
> > Suppose that I have a second, parallelizable, long-running task T2
> > that I want to address in REAL TIME when the need arises.  Using Pool,
> > is there a way for me to insert the chunks of T2 at the HEAD of the
> > task queue, instead of at its TAIL?
>
> That's a self-contradiction there. Even if you insert a task into the
> head of the queue, it won't be addressed in real time; the only way to
> do that would be to have a dedicated process, always ready to handle
> the real-time events, or else some kind of interrupt system.
>
> But if you just want them handled when there's a worker available,
> what you're after is a priority queue system. I don't know if one
> exists in Python already or not, but if not, it'd be a worthwhile
> addition.
>
> ChrisA

Hi, Chris,

Sorry if I'm not quite familiar with the proper terminology for
queues.  Let me try to define what I need as precisely as I can.

1) I aim to keep all five child Processes as busy as possible at all
times, crunching numbers.

2) I break my data for the low-priority task, T1, into large enough
chunks to benefit from multiprocessing, but small enough so that any
given Process should become available fairly frequently -- say, every
50 milliseconds.  Starting 50 milliseconds late would be close enough
to "real time" for my purposes.

3) But let's say that T1 still has twenty chunks in the Process queue
when T2 comes along.  I could be waiting 300 milliseconds for the
queue to empty.  If I could just ensure that T2 gets the next
available Processes, I would be happy.  Since T2 is also
parallelizable and I am using five child Processes, I would divide T2
into exactly five chunks.  These chunks should get inserted at
positions 0-4 in the queue rather than at 20-24.  T2 would briefly
commandeer all five Processes and then clear out.

If that's not putting T2 at the head of the queue, I guess I don't
know a better way to describe it.