Infinite recursion in __reduce__ when calling original base class reduce, why?

[Chris Torek]

In article <4df669ea$0$49182$e4fe514c at news.xs4all.nl>
Irmen de Jong  <irmen.NOSPAM at xs4all.nl> wrote:
>I've pasted my test code below. It works fine if 'substitute' is True,
>but as soon as it is set to False, it is supposed to call the original
>__reduce__ method of the base class. However, that seems to crash
>because of infinite recursion on Jython and IronPython and I don't
>know why. It works fine in CPython and Pypy.

In this particular case (no fancy inheritance going on), the base
__reduce__ method would be object.__reduce__.  Perhaps in those
implementations, object.__reduce__ goes back to TestClass.__reduce__,
rather than being appropriately magic.

>I wonder if my understanding of __reduce__ is wrong, or that I've
>hit a bug in IronPython and Jython?  Do I need to do something with
>__reduce_ex__ as well?

You should not *need* to; __reduce_ex__ is just there so that you
can do something different for different versions of the pickle
protocol (I believe).

Nonetheless, there is something at least slightly suspicious here:

>import pickle
>
>class Substitute(object):
>    def __init__(self, name):
>        self.name=name
>    def getname(self):
>        return self.name
>
>class TestClass(object):
>    def __init__(self, name):
>        self.name=name
>        self.substitute=True
>    def getname(self):
>        return self.name
>    def __reduce__(self):
>        if self.substitute:
>            return Substitute, ("SUBSTITUTED:"+self.name,)
>        else:
>            # call the original __reduce__ from the base class
>            return super(TestClass, self).__reduce__()  #  crashes on
>ironpython/jython
[snip]

In general, the way __reduce__ is written in other class implementations
(as distributed with Python2.5 at least) boils down to the very
simple:

    def __reduce__(self):
        return self.__class__, (arg, um, ents)

For instance, consider a class with a piece that looks like this:

    def __init__(self, name, value):
        self.name = name
        self.value = value
        self.giant_cached_state = None

    def make_parrot_move(self):
        if self.giant_cached_state is None:
            self._do_lots_of_computation()
        return self._quickstuff_using_cache()

Here, the Full Internal State is fairly long but the part that
needs to be saved (or, for copy operations, copied -- but you can
override this with __copy__ and __deepcopy__ members, if copying
the cached state is a good idea) is quite short.  Pickled instances
need only save the name and value, not any of the computed cached
stuff (if present).  So:

    def __reduce__(self):
        return self.__class__, (name, value)

If you define this (and no __copy__ and no __deepcopy__), the
pickler will save the name and value and call __init__ with the
name and value arguments.  The copy.copy and copy.deepcopy operations
will also call __init__ with these arguments (unless you add
__copy__(self) and __deepcopy__(self) functions).

So, it seems like in this case, you would want:

    def __reduce__(self):
        if self.substitute:
            return Substitute, ("SUBSTITUTED:"+self.name,)
        else:
            return self.__class__, (self.name,)

or if you want to be paranoid and only do a Substitute if
self.__class__ is your own class:

    if type(self) == TestClass and self.substitute:
            return Substitute, ("SUBSTITUTED:"+self.name,)
        else:
            return self.__class__, (self.name,)

In CPython, if I import your code (saved in foo.py):

    >>> x = foo.TestClass("janet")
    >>> x
    <foo.TestClass object at 0x66290>
    >>> x.name
    'janet'
    >>> x.__reduce__()
    (<class 'foo.Substitute'>, ('SUBSTITUTED:janet',))
    >>> x.substitute=False
    >>> x.__reduce__()
    (<function _reconstructor at 0x70bf0>, (<class 'foo.TestClass'>, <type 'object'>, None), {'name': 'janet', 'substitute': False})

which is of course the same as:

    >>> object.__reduce__(x)
    (<function _reconstructor at 0x70bf0>, (<class 'foo.TestClass'>, <type 'object'>, None), {'name': 'janet', 'substitute': False})

which means that CPython's object.__reduce__() uses a "smart" fallback
reconstructor.  Presumably IronPython and Jython lack this.
-- 
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html