[Python-ideas] Deterministic iterator cleanup

[Nathaniel Smith]

On Fri, Oct 21, 2016 at 3:48 PM, Amit Green <amit.mixie at gmail.com> wrote:
> NOTE: This is my first post to this mailing list, I'm not really sure
>       how to post a message, so I'm attempting a reply-all.
>
> I like Nathaniel's idea for __iterclose__.
>
> I suggest the following changes to deal with a few of the complex issues
> he discussed.
>
> 1.  Missing __iterclose__, or a value of none, works as before,
>     no changes.
>
> 2.  An iterator can be used in one of three ways:
>
>     A. 'for' loop, which will call __iterclose__ when it exits
>
>     B.  User controlled, in which case the user is responsible to use the
>         iterator inside a with statement.
>
>     C.  Old style.  The user is responsible for calling __iterclose__
>
> 3.  An iterator keeps track of __iter__ calls, this allows it to know
>     when to cleanup.
>
>
> The two key additions, above, are:
>
>     #2B. User can use iterator with __enter__ & __exit cleanly.
>
>     #3.  By tracking __iter__ calls, it makes complex user cases easier
>          to handle.

These are interesting ideas! A few general comments:

- I don't think we want the "don't bother to call __iterclose__ on
exhaustion" functionality --it's actually useful to be able to
distinguish between

    # closes file_handle
    for line in file_handle:
        ...

and

    # leaves file_handle open
    for line in preserve(file_handle):
        ...

To be able to distinguish these cases, it's important that the 'for'
loop always call __iterclose__ (which preserve() might then cancel
out).

- I think it'd be practically difficult and maybe too much magic to
add __enter__/__exit__/nesting-depth counts to every iterator
implementation. But, the idea of using a context manager for repeated
partial iteration is a great idea :-). How's this for a simplified
version that still covers the main use cases?

@contextmanager
def reuse_then_close(it):   # TODO: come up with a better name
    it = iter(it)
    try:
        yield preserve(it)
    finally:
        iterclose(it)

with itertools.reuse_then_close(some_generator(...)) as it:
    for obj in it:
        ...
    # still open here, because our reference to the iterator is
wrapped in preserve(...)
    for obj in it:
        ...
# but then closed here, by the 'with' block

-n

-- 
Nathaniel J. Smith -- https://vorpus.org