[Python-ideas] Weak-referencing/weak-proxying of (bound) methods

[Jan Kaliszewski]

Jan Kaliszewski dixit (2012-06-16, 01:41):

> Tal Einat dixit (2012-06-15, 15:41):
> 
> > On Mon, Jun 11, 2012 at 2:16 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> [snip]
> > >    >>> import weakref
> > >    >>> class A:
> > >    ...     def method(self): print(self)
> > >    ...
> > >    >>> A.method
> > >    <function method at 0xb732926c>
> > >    >>> a = A()
> > >    >>> a.method
> > >    <bound method A.method of <__main__.A object at 0xb7326bec>>
> > >    >>> r = weakref.ref(a.method)  # creating a weak reference
> > >    >>> r                          # ...but it appears to be dead
> > >    <weakref at 0xb7327d9c; dead>
> > >    >>> w = weakref.proxy(a.method)  # the same with a weak proxy
> > >    >>> w
> > >    <weakproxy at 0xb7327d74 to NoneType at 0x829f7d0>
> > >    >>> w()
> > >    Traceback (most recent call last):
> > >      File "<stdin>", line 1, in <module>
> > >    ReferenceError: weakly-referenced object no longer exists
> > >
> > > This behaviour is perfectly correct -- but still surprising,
> > > especially for people who know little about method creation
> > > machinery, descriptors etc.
> > >
> > > I think it would be nice to make this 'trap' less painful --
> [snip]
> > > A prototype implementation:
> > >
> > >    class InstanceCachedMethod(object):
> > >
> > >        def __init__(self, func):
> > >            self.func = func
> > >            (self.instance_attr_name
> > >            ) = '__{0}_method_ref'.format(func.__name__)
> > >
> > >        def __get__(self, instance, owner):
> > >            if instance is None:
> > >                return self.func
> > >            try:
> > >                return getattr(instance, self.instance_attr_name)
> > >            except AttributeError:
> > >                method = types.MethodType(self.func, instance)
> > >                setattr(instance, self.instance_attr_name, method)
> > >                return method
> [snip]
> > I was bitten by this issue a while ago as well. It made working with
> > weakref proxies much more involved than I expected it would be.
> > 
> > Wouldn't it be better to approach the issue from the opposite end, and
> > improve/wrap/replace weakref.proxy with something that can handle bound
> > methods?
> 
> Indeed, probably could it be done by wrapping weakref.ref()/proxy()
> with something like the following:
> 
>     # here `obj` is the object that is being weak-referenced...
>     if isinstance(obj, types.MethodType):
>         try:
>             cache = obj.__self__.__method_cache__
>         except AttributeError:
>             cache = obj.__self__.__method_cache__ = WeakKeyDictionary()
>         method_cache.setdefault(obj.__func__, set()).add(obj)
> 
> (Using WeakKeyDictionary with corresponding function objects as weak
> keys -- to provide automagic cleanup when a function is deleted, e.g.
> replaced with another one.  In other words: the actual weak ref/proxy
> to a method lives as long as the corresponding function does).

On second thought -- no, it shouldn't be done on the side of
weakref.ref()/proxy().

Why?  My last idea described just above has such a bug: each time
you create a new weak reference to the method another method object
is cached (added to __method_cache__[func] set).

You could think that caching only one object (just in
__method_cache__[func]) would be a better idea, but it wouldn't:
such a behaviour would be strange and unstable: after creating
a new weakref to the method, the old weakref would became invalid...

And yes, we can prevent it by ensuring that each time you take
the method from a class instance you get the same object (per class
instance) -- but then we come back to my previous idea of a
descriptor-decorator.  And IMHO such a decorator should not be
applied on the class dictionary implicitly by weakref.ref()/proxy()
but explicitly in the class body with the decorator syntax
(applying such a decorater, i.e. replacing a function with a
caching descriptor is a class dict, is too invasive operation to
be done silently).

So I renew (and update) my previous descriptor-decorator that
could be added to functools (or to weakref as a helper?) and
applied explicitly by programmers, when needed:

    class CachedMethod(object):

        def __init__(self, func):
            self.func = func

        def __get__(self, instance, owner):
            if instance is None:
                return self.func
            try:
                cache = instance.__method_cache__
            except AttributeError:
                # not thread-safe :-(
                cache = instance.__method_cache__ = WeakKeyDictionary()
            return cache.setdefault(
                  self.func,
                  types.MethodType(self.func, instance))

Usage:

    class MyClass(object):
        @CachedMethod
        def my_method(self):
            ...

    instance = MyClass()
    method_weak_proxy = weakref.proxy(instance.my_method)
    method_weak_proxy()  # works!

It should be noted that caching a reference to a method in an
instance causes circular referencing (class <-> instance).
However, ofter it is not a problem and can help avoiding
circular references involving other objects which we want to
have circular-ref-free (typical use case: passing a bound
method as a callback).

Cheers.
*j