Design thought for callbacks

[Cem Karan]

On Feb 21, 2015, at 12:27 PM, Steven D'Aprano <steve+comp.lang.python at pearwood.info> wrote:

> Cem Karan wrote:
> 
>> 
>> On Feb 21, 2015, at 8:15 AM, Chris Angelico <rosuav at gmail.com> wrote:
>> 
>>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>>>> OK, so it would violate the principle of least surprise for you. 
>>>> Interesting.  Is this a general pattern in python?  That is, callbacks
>>>> are owned by what they are registered with?
>>>> 
>>>> In the end, I want to make a library that offers as few surprises to the
>>>> user as possible, and no matter how I think about callbacks, they are
>>>> surprising to me.  If callbacks are strongly-held, then calling 'del
>>>> foo' on a callable object may not make it go away, which can lead to
>>>> weird and nasty situations.
> 
> How?
> 
> The whole point of callbacks is that you hand over responsibility to another
> piece of code, and then forget about your callback. The library will call
> it, when and if necessary, and when the library no longer needs your
> callback, it is free to throw it away. (If I wish the callback to survive
> beyond the lifetime of your library's use of it, I have to keep a reference
> to the function.)

Marko mentioned it earlier; if you think you've gotten rid of all references to some chunk of code, and it is still alive afterwards, that can be surprising.

>>>> Weakly-held callbacks mean that I (as the 
>>>> programmer), know that objects will go away after the next garbage
>>>> collection (see Frank's earlier message), so I don't get 'dead'
>>>> callbacks coming back from the grave to haunt me.
> 
> I'm afraid this makes no sense to me. Can you explain, or better still
> demonstrate, a scenario where "dead callbacks rise from the grave", so to
> speak?

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg = msg
    def callback(self, stuff):
        print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks = list()
    def register_callback(self, callback):
        self._callbacks.append(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ == "__main__":
    foo = Callback_object("Evil Zombie")
    lib = Fake_library()
    lib.register_callback(foo.callback)

    # Way later, after the user forgot all about the callback above
    foo = Callback_object("Your Significant Other")
    lib.register_callback(foo.callback)

    # And finally getting around to running all those callbacks.
    lib.execute_callbacks()
"""

Output:
From Evil Zombie: Surprise!
From Your Significant Other: Surprise!

In this case, the user made an error (just as Marko said in his earlier message), and forgot about the callback he registered with the library.  The callback isn't really rising from the dead; as you say, either its been garbage collected, or it hasn't been.  However, you may not be ready for a callback to be called at that moment in time, which means you're surprised by unexpected behavior.

>>>> So, what's the consensus on the list, strongly-held callbacks, or
>>>> weakly-held ones?
>>> 
>>> I don't know about Python specifically, but it's certainly a general
>>> pattern in other languages. They most definitely are owned, and it's
>>> the only model that makes sense when you use closures (which won't
>>> have any other references anywhere).
>> 
>> I agree about closures; its the only way they could work.
> 
> *scratches head* There's nothing special about closures. You can assign them
> to a name like any other object.
> 
> def make_closure():
>    x = 23
>    def closure():
>        return x + 1
>    return closure
> 
> func = make_closure()
> 
> Now you can register func as a callback, and de-register it when your done:
> 
> register(func)
> unregister(func)
> 
> 
> Of course, if you thrown away your reference to func, you have no (easy) way
> of de-registering it. That's no different to any other object which is
> registered by identity. (Registering functions by name is a bad idea, since
> multiple functions can have the same name.)
> 
> As an alternative, your callback registration function might return a ticket
> for the function:
> 
> ticket = register(func)
> del func
> unregister(ticket)
> 
> but that strikes me as over-kill. And of course, the simplest ticket is to
> return the function itself :-)

Agreed on all points; closures are just ordinary objects.  The only difference (in my opinion) is that they are 'fire and forget'; if you are registering or tracking them then you've kind of defeated the purpose.  THAT is what I meant about how you handle closures.

> 
>> When I was 
>> originally thinking about the library, I was trying to include all types
>> of callbacks, including closures and callable objects.  The callable
>> objects may pass themselves, or one of their methods to the library, or
>> may do something really weird.
> 
> I don't think they can do anything too weird. They have to pass a callable
> object. Your library just calls that object. You shouldn't need to care
> whether it is a function, a method, a type, a callable instance, or
> something else. You just call it, and when you're done calling it forever,
> you just throw it away.

That doesn't quite solve the problem, but it comes close.  The headache (as shown in my earlier code) is that you think you've gotten rid of something before it is called, but it turns out you haven't.  I'm starting to think that there isn't a solution to this other than telling the programmer "Don't do that".  

>> Although I just realized that closures may cause another problem.  In my
>> code, I expect that many different callbacks can be registered for the
>> same event.  Unregistering means you request to be unregistered for the
>> event. How do you do that with a closure?  Aren't they anonymous?
> 
> Not unless you create them using lambda. Using the make_closure function
> above:
> 
> 
> py> func = make_closure()
> py> func.__name__
> 'closure'
> 
> Of course, if you call make_closure twice, both functions will have the same
> internal name. You can set the function __name__ and __qualname__ to fix
> that. This is how the functools.wraps decorator works.
> 
> But that's a red herring. Don't register functions by name! Not all callable
> objects have names, and those that do, you may have multiple *distinct*
> callbacks with the same name.
> 
> There are two reasonable approaches: unregister by identity, or by returning
> a ticket which uniquely identifies the callback. The user is responsible
> for keeping track of their own ticket. If I lose it, I can't unregister my
> callback any more. So sad, sucks to be me.
> 
> 
> The simplest possible identity-based scheme would be something like this:
> 
> 
> # don't hate me for using a global variable
> CALLBACKS = []
> 
> def register(func):
>    if func not in CALLBACKS:
>         CALLBACKS.append(func)
> 
> def unregister(func):
>    try:
>        CALLBACKS.remove(func)
>    except ValueError:
>        pass
> 
> 
> That's probably a bit too simple, since it won't behave as expected with
> bound methods. The problem is that bound methods are generated on the fly,
> so this won't work:
> 
> register(instance.spam)
> # later
> unregister(instance.spam)  # a different instance!
> 
> I would have to do this:
> 
> bound_method = instance.spam
> register(bound_method)
> unregister(bound_method)
> 
> 
> But a more sophisticated unregister function should work:
> 
> # Untested
> def unregister(func):
>    for i, f in enumerate(CALLBACKS):
>        if (f is func) or (isinstance(f, types.MethodType) 
>                and f.__wrapped__ is func):
>            del CALLBACKS[i]
>            return

Are you sure about that?  I just tested out the following code, and it appears to work correctly:

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg = msg
    def callback(self, stuff):
        print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks = set()
    def register_callback(self, callback):
        self._callbacks.add(callback)
    def unregister_callback(self, callback):
        self._callbacks.discard(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ == "__main__":
    foo = Callback_object("Evil Zombie")
    lib = Fake_library()
    lib.register_callback(foo.callback)
    lib.unregister_callback(foo.callback)
    lib.execute_callbacks()
"""

I'll admit though, I don't know if it worked because I got lucky, or if python guarantees it works...

> The simplest possible ticket-based system is probably something like this:
> 
> CALLBACKS = {}
> NEXT_TICKET = 1
> 
> def register(func):
>    global NEXT_TICKET
>    ticket = NEXT_TICKET
>    NEXT_TICKET += 1
>    callbacks[ticket] = func
>    return ticket
> 
> def unregister(ticket):
>    if ticket in CALLBACKS:
>        del CALLBACKS[ticket]
> 

I'd probably go with something similar to this, except that I'd use UUIDs for the tickets.  I know me and my users, and somewhere along the line I'd use a ticket to unregister from the wrong callback dictionary!

>>> If you're expecting 'del foo' to destroy the object, then you have a
>>> bigger problem than callbacks, because that's simply not how Python
>>> works. You can't _ever_ assume that deleting something from your local
>>> namespace will destroy the object, because there can always be more
>>> references. So maybe you need a more clear way of saying "I'm done
>>> with this, get rid of it".
>> 
>> Agreed about 'del', and I don't assume that the object goes away at the
>> point.  The problem is debugging and determining WHY your object is still
>> around.  I know a combination of logging and gc.get_referrers() will
>> probably help you figure out why something is still around, but I'm trying
>> to avoid that headache.
> 
> Why do you care? Surely all your library should care about is whether or not
> they have a reference to the callback.If they do, they should call it (when
> appropriate). If they don't, they aren't responsible for it.

I care because I care about anyone using my code.  Telling them 'tough, its your problem' doesn't get you many friends.  Making a library that performs as expected and where it is easy to debug what went wrong makes everyone (including me!) happy.

>> I guess the real problem is how this creates cycles in the call graph. 
>> User code effectively owns the library code, which via callbacks owns the
>> user code.  I have no idea what the best point the cycle is to break it,
>> and not surprise someone down the road.  The only idea I have is to
>> redesign the library a little, and make anything that accepts a callback
>> actually be a subclass of collections.abc.Container, or even
>> collections.abc.MutableSet.  That makes it very obvious that the object
>> owns the callback, and that you will need to remove your object to
>> unregister it.
> 
> My brain hurts from the complexity of your solution. What is the actual
> problem you are trying to solve? I would like to see an example of an
> actual failure before trying to solve a purely theoretical failure mode.

If I'm making your head hurt, then my solution is a bad solution.  The whole reason I started this discussion was to figure out if an alternative method would make more sense to my potential endusers (fellow programmers).  If strongly-held callbacks cause fewer headaches, then that is what I'll go with.

Thanks,
Cem Karan