[py-dev] execnet hangs

[holger krekel]

Hi Armin, 

On Fri, Apr 08, 2005 at 09:52 +0100, Armin Rigo wrote:
> On Thu, Apr 07, 2005 at 11:45:33PM +0200, holger krekel wrote:
> > rather think about an adaptive scheme where we allow to
> > specify "maxthreads" and default it to 10 or so. Would that 
> > make sense to you?
> 
> Not really.  Anything that, like my example above, appears to work until some
> small number (2 or 10) and completely deadlocks above that number is bound to
> create surprises and bugs.

Full ack.  However, nobody forces us to deadlock if we are 
exceeding the number of maximum threads. 

> Why not create new 'executor' threads when none is available, with no upper
> bound?  I'd much prefer buggy programs to trash the system with resource
> consumption than nicely written programs deadlock because of an internal
> limit.

Not so sure about this.  I want to do things on production machines
(codespeak.net among others) with execnet and i wouldn't want to starve
resources there because of a casual code bug :-) 

> It should be clear that remote_exec() creates a new thread on the
> remote side, and that if you are going to do really a lot of remove_exec() you
> should do only one and use channels.

Using channels and doing your own multiplexing is painful. 
I'd say we should have a default of 10 for maxthreads but allow 
it to be None, meaning no bounds.  If the bound is exceeded a 
Gateway.ThreadsExhausted (subclass of RuntimeError) would be
raised on remote_exec() side. The point is that it is reasonably hard 
for a user of py.execnet to restrict the number of threads themselves 
(would probably invole subclassing etc.pp.).  Is that agreeable? 

cheers, 

    holger