Turning a callback function into a generator

[Alex Martelli]

Kirk McDonald <kirklin.mcdonald at gmail.com> wrote:
   ...
> > def func(callback):
> >      for i in [1, 2, 3, 4, 5]:
> >          callback(i)
   ...
> Threads are probably a weak point of mine. I have little experience with
> them, and so I am inclined to paranoia while using them. What 

Paranoia is the correct state of mind to start from when threads are
involved.

> implications does this have for "func"? In my case, it is a function in
> an extension module. Does it have to be thread safe? If it isn't, what
> limitations does this impose on the use of this wrapper?

The 'func' you gave as an example IS threadsafe (if callback is): it
does not access any global object that might be used by other threads at
the same time.  If the real func *DOES* access (or, even worse, modify)
global objects -- or more generally objects that might be modified by
other threads -- then you must try to make things safe again by adding
locks (with a strong risk of deadlocking, of course).

This problem is basically (to a large extent) tied to your very specs:
you want the whole callstack up to the point where 'func' calls
'callback' to be essentially ``frozen'' while somehow the 'callback'
(the only point in which you allow intervention in the code, since you
forbid alterations of 'func'!) magically hands over the item to a yield
statement which provides the item to calling-code.  Now calling code can
do anything it wants (to globals or any other objects that, for all
you've told us, 'func' might be accessing or altering too) -- and this
must not damage func's behavior.

If there are no other threads in your process, beyond the main one (on
which the generator-wrapper is called) and the specialized one which the
generator-wrapper created (to run func in), then the issues are pretty
much the same as if the generator-wrapper was able to avoid using
threads -- it's just slightly less deterministic where control may shift
between the two threads, but I believe (without being able to prove it)
that this is unlikely to matter... the key issue remains ensuring that
the *caller* of the wrapper, and func itself, don't tread on each
other's toes, regarding all that's "suspended" on the stack while the
caller does whatever it wants with the just-yielded item.

If there ARE other threads in your process, and func is not threadsafe
with respect to them, then of course you should not be calling it
(wrapped or not) without precautions in terms of locking &c -- the
wrapper doesn't matter.

> All that aside, I read the previous threads but it took until just now
> to understand how this works. :-) I'll probably have a go at 
> implementing this in D, as that will be more convenient for the library...

OK, sounds good.  D looks interesting (I found Walter Bright's short
presentation of it at Google quite engaging), and I might like to give
it a try if it was as handy to interface to Python as C++ (or Java or C#
or Objective C) already are.


Alex