Better Threading, starting with Events

[Warren Postma]

Here is my little 'event' module __doc__ string:

event module 1.0

  Provides an efficient way for threads to go to sleep while waiting for
  work to do, freeing up the python interpreter to run the other threads,
  and yet waking up immediately upon signalling the worker thread

  Uses CreateEvent/SetEvent/WaitForMultipleObjects Win32 functions.

  Usage Sample:

     import event
     sd = event.new('Shutdown')
     sd.MyAttribute = 1 # stores something in event dictionary
     work = event.new('DoSomeWork',1) # 'one-shot'
     work .work_to_do = func # pass the function to the thread
     ret = event.select( (sd,work), 5000 ) # wait 5 seconds for signal
     if ret:  # Got signal?
         print repr(ret)

------------------------------------------------------------
                Here are the module methods:
------------------------------------------------------------

event.new( 'name', manualResetFlag=1, initialStateFlag=0)
   -> Creates and returns a new event object
        manualResetFlag = does event reset automatically when received
        (default=1)
        initialStateFlag = event signalled or not signalled initially
        (default=0)

 e = event.new( 'ASampleEvent')

 Using a WinEvent:
        e.name       name of event <string-attribute>
        e.wait()     wait indefinitely (until python shutdown) for event
        e.wait(500)  wait up to 500 milliseconds (0.5 seconds) for event
        e.state()    query state (non blocking) returns 0 or 1
        e.set()      set signalled state
        e.reset()    reset to non-signalled state
        e.handle()   get internal handle value (as integer)

To wait for several events in parallel, use event.select( (e1,e2,..),
timeout )

------------------------------------------------------------

event.select( <event tuple>, timeout=None)
   -> Waits for a single event or any of a group of events,
    returns None if the timeout occurs.
      Note that all waits are interruptable by shutdown,
      which will cause an exception to be raised.

  Example:
     ev1 = event.new('DoSomething')
     ev2 = event.new('DoSomethingElse')
     ret=event.select( (ev1,ev2), 5000 ) # wait up to 5000 millisec.
     if ret:
          print 'got event', ret.name  # which is it? DoSomething, or
DoSomethingElse
     else: # none?
          print 'timeout'


--------------------------- End ----------------------------

Now, for a discussion on my Evil Plan:

Anyone interested in native C Type for Event Signalling? I have written a
1.0, or 0.1, that works, it just ain't portable off of Win32. I'll make it
portable if there is some agreement on the basic idea behind what I'm doing.

I'd like feedback on whether my Evil Plan calls for a PEP.  Before I go
PEP'ing, I'd like to get some initial comments on other people who have
implemented worker threads. Is anyone else dissatisfied with threading.py?

My first problem with the existing facilities of threading.py is its
existence. I don't like that the Primitives are not Primitive, the interface
is great, the implementation is in Python. Urgh!  Great for Jython/JPython.
Not so great for CPython guys like me.

Why should Locks, Events, and Signals and Semaphores and so on all live in a
py file?  Their efficiency is therefore sub-par, and it will negatively
affect the performance of all apps that use it.  I also feel that a
consistent API and underlying implementation ought to permeate downwards
into the core of CPython.  Anyone with me? At that stage, you can see my
Evil Plan is beginning to get into swing.

But back to reality for a minute. The first thing I needed was extremely
efficient Event Objects, which could make a thread be "sleepy", waking up
periodically every 5 seconds to do some housekeeping, then going back to
sleep, but to be instantly interruptable at any time and immediately start
doing whatever work is required. They let go of the python interpreter lock,
and don't make the interpreter do any work, running functions in
threading.py while the thread ought to be idle, etcetera.

Also, my eventual evil plan involves thread-by-thread signalling in ways not
currently supported by the current frame objects, and the code in ceval.c.
It should also involve making various blocking functions "interruptable".
For example, time.sleep, which calls the C function floatsleep in
timemodule.c:
      Py_BEGIN_ALLOW_THREADS
      Sleep((unsigned long)millisecs);
      Py_END_ALLOW_THREADS

This should be, IMHO:
      Py_BEGIN_ALLOW_THREADS
      ret=WaitForSingleObject( hSystemShutdown, INIFINITE);
      Py_END_ALLOW_THREADS
      if (ret != WAIT_TIMEOUT)
            return NULL; // system is shutting down.

I am basically doing my R&D here anyways to get threading maximally
efficient on my embedded systems.  But I am more than willing, excited even,
to try my hand at proposing a PEP.   Not for my little event module, but for
a whole bunch of threading primitives, in C. Perhaps the CPython version of
threading.py could be adapted to use them.

The PEP would probably be for an expanded  "thread" module, and a revised
threading system that improves "pervasive threading" inside CPython.

Anyone? Anyone? Bueller?

Warren