Generator using item[n-1] + item[n] memory

[Peter Otten]

Chris Angelico wrote:

> On Sat, Feb 15, 2014 at 6:27 PM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
>> On Fri, Feb 14, 2014 at 8:31 PM, Nick Timkovich <prometheus235 at gmail.com>
>> wrote:
>>> OK, now the trick; adding `data = None` inside the generator works, but
>>> in my actual code I wrap my generator inside of `enumerate()`, which
>>> seems to
>>> obviate the "fix".  Can I get it to play nice or am I forced to count
>>> manually. Is that a feature?
>>
>> Yeah, looks like enumerate also doesn't release its reference to the
>> previous object until after it gets the next one.  You'll just have to
>> make do without.
> 
> You could write your own enumerate function.
> 
> def enumerate(it, i=0):
>     it = iter(it)
>     while True:
>         yield i, next(it)
>         i += 1
> 
> That shouldn't keep any extra references around.

An alternative approach ist to yield weak refs and thus have the generator 
control the object lifetime. This doesn't work with the built-in list type 
though:

import weakref

try:
    from itertools import imap # py2
except ImportError:
    imap = map # py3

N = 0

def log_deleted(*args):
    global N
    N -= 1
    print("deleted, new N: {}".format(N))

def log_created():
    global N
    N += 1
    print("created, new N: {}".format(N))

def weakrefs(f):
    def weakrefs(*args, **kw):
        return imap(lambda x: weakref.proxy(x, log_deleted), f(*args, **kw))
    return weakrefs

class List(list):
    def __str__(self):
        s = str(self[:5])
        if len(self) > 10:
            s = s[:-1] + ", ... ]"
        return s

@weakrefs
def biggen():
    sizes = 1, 1, 10, 1, 1, 10, 10, 1, 1, 10, 10, 20, 1, 1, 20, 20, 1, 1
    for size in sizes:
        data = List([1] * int(size * 1e4))
        log_created()
        yield data
        data = None

if __name__ == "__main__":
    for i, x in enumerate(biggen()):
        print("{} {}".format(i, x))