Design thought for callbacks

[Steven D'Aprano]

Chris Angelico wrote:

> On Sun, Feb 22, 2015 at 1:04 PM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:
>> Marko Rauhamaa wrote:
>>
>>> Grant Edwards <invalid at invalid.invalid>:
>>>
>>>> the polite thing to do is to delete references to it when you're done
>>>> with it.
>>>
>>> I disagree with that recommendation. You should do the natural thing and
>>> not care who holds references to who.
>>
>> I don't understand this. What is "the natural thing" if not to delete
>> references to an object when you are done with it? Normally you just let
>> things go out of scope, but if that won't happen, you have to take active
>> steps, such as calling del or setting the reference to None.
> 
> I think the disagreement here is over the interpretation of "done with
> it". If you drop all references to a connected socket object, Python
> can rightly assume that you're done with the file and want to close
> it; but what if you drop all references to a listening socket that's
> been configured to call a function whenever someone connects?
> 
> def client_connected(sock):
>     sock.send("Hello!\r\n")
>     # whatever
> 
> listener = socket(23, on_accept=client_connected)
> 
> What should happen if that main socket isn't bound to a name? In my
> opinion, the fact that it's configured for callback mode should mean
> that it's kept alive. But it's also understandable to want to treat it
> as "done", that it can be disposed of. It seems weird to me that you
> should have to have a name somewhere that you'll never use, though.

But you are using it. You might not be using it by name, but you are using
it via the callback function. What did you expect, that Python should read
your mind and somehow intuit that you still care about this socket
listener, but not some other socket listener that you are done with?

If you have a servant that follows you around everywhere, throwing objects
away when you stop using them, then naturally if you stop using something
it will be thrown away. What did you expect?

You don't have to bind the listener to a name. Any reference will do. You
can dump it in a bucket:

bucket_of_stuff = []
bucket_of_stuff.append(some_function(a, b, c))
bucket_of_stuff.append(make_web_server())
bucket_of_stuff.append(socket(23, on_accept=client_connected))


So long as your bucket is still alive, the garbage collector won't collect
it or its contents.

Hypothetically we could have a system in place where you instruct the
garbage collector to not collect an object, then drop all references to it:

gc.never_collect(socket(23, on_accept=client_connected))

but that's a potential memory leak, because now you have no way of telling
the GC to collect it again once you've finished listening. If you never
finish listening, or at least not until your application shuts down, it
doesn't count as a leak. But in general, if listeners might come and go,
but you have no way for them to be garbage collected once they are done,
then that's a leak.

What's the easiest way for the GC to flag an object as "never collect this"?
It can keep a reference to it. Keeping a list of things you want to be kept
alive is simple, easy and obvious:

# hypothetically inside gc.py
_ALIVE = []
def never_collect(obj):
    _ALIVE.append(obj)


but that's so trivial that it's not worth putting it in the gc module.
Besides, that means any other code could reach into the gc and remove your
listener from the "keep alive list" without your knowledge or permission.

It's simpler and safer to just keep it alive yourself:

alive = []
alive.append(socket(...))

but of course that's just my bucket of stuff under a different name.

Using the idiom "keep objects alive by keeping a reference to them" (e.g.
bind them to a name, or stick them in a list which you keep) makes things
much simpler. You can trivially flag objects as "collect" or "don't
collect" as needed, without having to import the gc module or memorise some
obscure API or worry about implementation details ("if I flag an object
as 'never collect' *twice*, do I have to unflag it twice to undo it?"). You
just use the regular interface to the garbage collector: the existence of a
reference, any reference, keeps an object alive.




-- 
Steven