[Python-ideas] Add an introspection API to Executor

[Ram Rachum]

Sounds good to me. Having to specify `IntrospectableQueue` to the executor
is a bit of a chore, but not too bad to get this functionality. I also bet
that the performance difference wouldn't be an issue for most uses.


On Mon, Aug 25, 2014 at 11:05 PM, 'Andrew Barnert' via python-ideas <
python-ideas at googlegroups.com> wrote:

> I don't think there's any issue with letting people introspect the
> executor. The problem is that the main thing you get is a queue, and
> there's a limit to how introspectable a queue can be.
>
> In particular, if you want to iterate the waiting tasks, you have to
> iterate the queue, and there's no safe way to do that.
>
> Since CPython's queue.Queue happens to be just a deque and a mutex, you
> could make it iterable at the cost of blocking all producers and consumers
> (which might be fine for many uses, like debugging or exploratory
> programming), or provide a snapshot API to return a copy of the deque.
>
> But do you want to make that a requirement on all subclasses of Queue, and
> all other implementations' queue modules? Does ProirityQueue have to
> nondestructively iterate a heap in order? Does Jython have to use a mutex
> and a deque instead of a more efficient (and possibly lock-free) queue from
> the Java stdlib? What does multiprocessing.Queue do on each implementation?
>
> I don't think the costs are worth the benefit. And I assume that's why the
> existing queue API doesn't provide an iteration or snapshot mechanism.
>
> But there's an option that might be worth doing:
>
> Provide a queue.IntrospectableQueue type that _is_ defined to have such a
> mechanism, but to otherwise work like a Queue (except maybe less
> efficiently). Then provide an optional parameter for the Executor that lets
> you specify an alternate queue constructor in place of the default. So,
> when exploring or debugging, you could pass queue.IntrospectableQueue (or
> multiprocessing.IntrospectableQueue for ProcessPoolExecutor).
>
> Whether the interface is "lock_and_return_iterator" or "snapshot", this
> would be trivial to implement in CPython, and other Pythons could just copy
> the CPython version instead of extending their native queue types.
>
> Sent from a random iPhone
>
> On Aug 25, 2014, at 11:54, Guido van Rossum <guido at python.org> wrote:
>
> It might be worth it to make the implementation somewhat more complicated
> if it serves a good purpose, for example giving the user of the program
> insights into how well the executor is performing. Without such insight you
> may be attempting to tune parameters (like the pool size) without being
> able to evaluate their effect.
>
>
> On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou <antoine at python.org>
> wrote:
>
>> Le 25/08/2014 14:16, Ram Rachum a écrit :
>>
>>  Maybe I'm missing something, but I don't think that's something that
>>> should block implementation.
>>>
>>> Information not available? Change the executor code to make that
>>> information available.
>>>
>>
>> Not if that would make the implementation much more complicated, or
>> significantly slower.
>>
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/pl3r5SsbLLU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140826/44c51d62/attachment.html>