Design thought for callbacks

[Steven D'Aprano]

Cem Karan wrote:

> 
> On Feb 21, 2015, at 8:15 AM, Chris Angelico <rosuav at gmail.com> wrote:
> 
>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>>> OK, so it would violate the principle of least surprise for you. 
>>> Interesting.  Is this a general pattern in python?  That is, callbacks
>>> are owned by what they are registered with?
>>> 
>>> In the end, I want to make a library that offers as few surprises to the
>>> user as possible, and no matter how I think about callbacks, they are
>>> surprising to me.  If callbacks are strongly-held, then calling 'del
>>> foo' on a callable object may not make it go away, which can lead to
>>> weird and nasty situations.

How?

The whole point of callbacks is that you hand over responsibility to another
piece of code, and then forget about your callback. The library will call
it, when and if necessary, and when the library no longer needs your
callback, it is free to throw it away. (If I wish the callback to survive
beyond the lifetime of your library's use of it, I have to keep a reference
to the function.)


>>> Weakly-held callbacks mean that I (as the 
>>> programmer), know that objects will go away after the next garbage
>>> collection (see Frank's earlier message), so I don't get 'dead'
>>> callbacks coming back from the grave to haunt me.

I'm afraid this makes no sense to me. Can you explain, or better still
demonstrate, a scenario where "dead callbacks rise from the grave", so to
speak?


>>> So, what's the consensus on the list, strongly-held callbacks, or
>>> weakly-held ones?
>> 
>> I don't know about Python specifically, but it's certainly a general
>> pattern in other languages. They most definitely are owned, and it's
>> the only model that makes sense when you use closures (which won't
>> have any other references anywhere).
> 
> I agree about closures; its the only way they could work.

*scratches head* There's nothing special about closures. You can assign them
to a name like any other object.

def make_closure():
    x = 23
    def closure():
        return x + 1
    return closure

func = make_closure()

Now you can register func as a callback, and de-register it when your done:

register(func)
unregister(func)


Of course, if you thrown away your reference to func, you have no (easy) way
of de-registering it. That's no different to any other object which is
registered by identity. (Registering functions by name is a bad idea, since
multiple functions can have the same name.)

As an alternative, your callback registration function might return a ticket
for the function:

ticket = register(func)
del func
unregister(ticket)

but that strikes me as over-kill. And of course, the simplest ticket is to
return the function itself :-)



> When I was 
> originally thinking about the library, I was trying to include all types
> of callbacks, including closures and callable objects.  The callable
> objects may pass themselves, or one of their methods to the library, or
> may do something really weird.

I don't think they can do anything too weird. They have to pass a callable
object. Your library just calls that object. You shouldn't need to care
whether it is a function, a method, a type, a callable instance, or
something else. You just call it, and when you're done calling it forever,
you just throw it away.


> Although I just realized that closures may cause another problem.  In my
> code, I expect that many different callbacks can be registered for the
> same event.  Unregistering means you request to be unregistered for the
> event. How do you do that with a closure?  Aren't they anonymous?

Not unless you create them using lambda. Using the make_closure function
above:


py> func = make_closure()
py> func.__name__
'closure'

Of course, if you call make_closure twice, both functions will have the same
internal name. You can set the function __name__ and __qualname__ to fix
that. This is how the functools.wraps decorator works.

But that's a red herring. Don't register functions by name! Not all callable
objects have names, and those that do, you may have multiple *distinct*
callbacks with the same name.

There are two reasonable approaches: unregister by identity, or by returning
a ticket which uniquely identifies the callback. The user is responsible
for keeping track of their own ticket. If I lose it, I can't unregister my
callback any more. So sad, sucks to be me.


The simplest possible identity-based scheme would be something like this:


# don't hate me for using a global variable
CALLBACKS = []

def register(func):
    if func not in CALLBACKS:
         CALLBACKS.append(func)

def unregister(func):
    try:
        CALLBACKS.remove(func)
    except ValueError:
        pass


That's probably a bit too simple, since it won't behave as expected with
bound methods. The problem is that bound methods are generated on the fly,
so this won't work:

register(instance.spam)
# later
unregister(instance.spam)  # a different instance!

I would have to do this:

bound_method = instance.spam
register(bound_method)
unregister(bound_method)


But a more sophisticated unregister function should work:

# Untested
def unregister(func):
    for i, f in enumerate(CALLBACKS):
        if (f is func) or (isinstance(f, types.MethodType) 
                and f.__wrapped__ is func):
            del CALLBACKS[i]
            return


The simplest possible ticket-based system is probably something like this:

CALLBACKS = {}
NEXT_TICKET = 1

def register(func):
    global NEXT_TICKET
    ticket = NEXT_TICKET
    NEXT_TICKET += 1
    callbacks[ticket] = func
    return ticket

def unregister(ticket):
    if ticket in CALLBACKS:
        del CALLBACKS[ticket]




>> If you're expecting 'del foo' to destroy the object, then you have a
>> bigger problem than callbacks, because that's simply not how Python
>> works. You can't _ever_ assume that deleting something from your local
>> namespace will destroy the object, because there can always be more
>> references. So maybe you need a more clear way of saying "I'm done
>> with this, get rid of it".
> 
> Agreed about 'del', and I don't assume that the object goes away at the
> point.  The problem is debugging and determining WHY your object is still
> around.  I know a combination of logging and gc.get_referrers() will
> probably help you figure out why something is still around, but I'm trying
> to avoid that headache.

Why do you care? Surely all your library should care about is whether or not
they have a reference to the callback.If they do, they should call it (when
appropriate). If they don't, they aren't responsible for it.


> I guess the real problem is how this creates cycles in the call graph. 
> User code effectively owns the library code, which via callbacks owns the
> user code.  I have no idea what the best point the cycle is to break it,
> and not surprise someone down the road.  The only idea I have is to
> redesign the library a little, and make anything that accepts a callback
> actually be a subclass of collections.abc.Container, or even
> collections.abc.MutableSet.  That makes it very obvious that the object
> owns the callback, and that you will need to remove your object to
> unregister it.

My brain hurts from the complexity of your solution. What is the actual
problem you are trying to solve? I would like to see an example of an
actual failure before trying to solve a purely theoretical failure mode.

If I register something as a callback, I expect that callback will stay
alive for as long as the callbacks are needed. If I might want to
unregister it, then I have to keep a reference to the function, otherwise
how will I know what I am unregistering?

# this makes no sense and cannot work
register(func)
del func
# later
unregister(func)

So if I do that, (1) it won't work; (2) I'll probably get an exception; (3)
the solution is "don't do that"; and (4) solving this problem is not YOUR
responsibility.

When your code is done with the callbacks, you can just remove them, no
questions asked. If I still have a reference to the callback, that
reference will still be valid no matter what you do. If I don't have a
reference to it, presumably that's because I don't need it any more. I
can't access the callback anyway.


> The only problem is how to handle closures; since they are 
> anonymous, how do you decide which one to remove?

You identify them by identity, or by a ticket, the same as for any other
object.



-- 
Steven