Better Threading, starting with Events

[Rick Lee]

I like the idea of being able to wait for several events at once.   That's
something I can't do with queues in the Queue module.  So my wish list would
extend to queues as well.  Another thing on my wish list is to be able to join
to several threads at once.  And then finally, to be able to select from any
combination of events, queues, joins, and maybe anything else that blocks.

Maybe you also want to keep on eye on compatibility with Stackless.

- Rick Lee

Warren Postma wrote:

> Here is my little 'event' module __doc__ string:
>
> event module 1.0
>
>   Provides an efficient way for threads to go to sleep while waiting for
>   work to do, freeing up the python interpreter to run the other threads,
>   and yet waking up immediately upon signalling the worker thread
>
>   Uses CreateEvent/SetEvent/WaitForMultipleObjects Win32 functions.
>
>   Usage Sample:
>
>      import event
>      sd = event.new('Shutdown')
>      sd.MyAttribute = 1 # stores something in event dictionary
>      work = event.new('DoSomeWork',1) # 'one-shot'
>      work .work_to_do = func # pass the function to the thread
>      ret = event.select( (sd,work), 5000 ) # wait 5 seconds for signal
>      if ret:  # Got signal?
>          print repr(ret)
>
> ------------------------------------------------------------
>                 Here are the module methods:
> ------------------------------------------------------------
>
> event.new( 'name', manualResetFlag=1, initialStateFlag=0)
>    -> Creates and returns a new event object
>         manualResetFlag = does event reset automatically when received
>         (default=1)
>         initialStateFlag = event signalled or not signalled initially
>         (default=0)
>
>  e = event.new( 'ASampleEvent')
>
>  Using a WinEvent:
>         e.name       name of event <string-attribute>
>         e.wait()     wait indefinitely (until python shutdown) for event
>         e.wait(500)  wait up to 500 milliseconds (0.5 seconds) for event
>         e.state()    query state (non blocking) returns 0 or 1
>         e.set()      set signalled state
>         e.reset()    reset to non-signalled state
>         e.handle()   get internal handle value (as integer)
>
> To wait for several events in parallel, use event.select( (e1,e2,..),
> timeout )
>
> ------------------------------------------------------------
>
> event.select( <event tuple>, timeout=None)
>    -> Waits for a single event or any of a group of events,
>     returns None if the timeout occurs.
>       Note that all waits are interruptable by shutdown,
>       which will cause an exception to be raised.
>
>   Example:
>      ev1 = event.new('DoSomething')
>      ev2 = event.new('DoSomethingElse')
>      ret=event.select( (ev1,ev2), 5000 ) # wait up to 5000 millisec.
>      if ret:
>           print 'got event', ret.name  # which is it? DoSomething, or
> DoSomethingElse
>      else: # none?
>           print 'timeout'
>
> --------------------------- End ----------------------------
>
> Now, for a discussion on my Evil Plan:
>
> Anyone interested in native C Type for Event Signalling? I have written a
> 1.0, or 0.1, that works, it just ain't portable off of Win32. I'll make it
> portable if there is some agreement on the basic idea behind what I'm doing.
>
> I'd like feedback on whether my Evil Plan calls for a PEP.  Before I go
> PEP'ing, I'd like to get some initial comments on other people who have
> implemented worker threads. Is anyone else dissatisfied with threading.py?
>
> My first problem with the existing facilities of threading.py is its
> existence. I don't like that the Primitives are not Primitive, the interface
> is great, the implementation is in Python. Urgh!  Great for Jython/JPython.
> Not so great for CPython guys like me.
>
> Why should Locks, Events, and Signals and Semaphores and so on all live in a
> py file?  Their efficiency is therefore sub-par, and it will negatively
> affect the performance of all apps that use it.  I also feel that a
> consistent API and underlying implementation ought to permeate downwards
> into the core of CPython.  Anyone with me? At that stage, you can see my
> Evil Plan is beginning to get into swing.
>
> But back to reality for a minute. The first thing I needed was extremely
> efficient Event Objects, which could make a thread be "sleepy", waking up
> periodically every 5 seconds to do some housekeeping, then going back to
> sleep, but to be instantly interruptable at any time and immediately start
> doing whatever work is required. They let go of the python interpreter lock,
> and don't make the interpreter do any work, running functions in
> threading.py while the thread ought to be idle, etcetera.
>
> Also, my eventual evil plan involves thread-by-thread signalling in ways not
> currently supported by the current frame objects, and the code in ceval.c.
> It should also involve making various blocking functions "interruptable".
> For example, time.sleep, which calls the C function floatsleep in
> timemodule.c:
>       Py_BEGIN_ALLOW_THREADS
>       Sleep((unsigned long)millisecs);
>       Py_END_ALLOW_THREADS
>
> This should be, IMHO:
>       Py_BEGIN_ALLOW_THREADS
>       ret=WaitForSingleObject( hSystemShutdown, INIFINITE);
>       Py_END_ALLOW_THREADS
>       if (ret != WAIT_TIMEOUT)
>             return NULL; // system is shutting down.
>
> I am basically doing my R&D here anyways to get threading maximally
> efficient on my embedded systems.  But I am more than willing, excited even,
> to try my hand at proposing a PEP.   Not for my little event module, but for
> a whole bunch of threading primitives, in C. Perhaps the CPython version of
> threading.py could be adapted to use them.
>
> The PEP would probably be for an expanded  "thread" module, and a revised
> threading system that improves "pervasive threading" inside CPython.
>
> Anyone? Anyone? Bueller?
>
> Warren