[Python-ideas] Map and filter should also convert StopIteration to RuntimeError

[Guido van Rossum]

On Fri, Dec 12, 2014 at 10:10 PM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:
>
>
> > On Dec 12, 2014, at 4:36 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
> wrote:
> >
> > Terry Reedy wrote:
> >> PEP 479 reverses this acceptance by having generator.__next turn
> StopIteration raised in a user-written generator function body into a
> RuntimeError.  I propose that other builtin iterator.__next__ methods that
> execute a passed in function do the same.
> >
> > Maybe also any user-written function whose
> > name isn't "__next__"?
>
> That does seem to be where this is headed :-(
>
> I'm -1 on this proposal.
>

Me too. PEP 479 was a clear win. __next__ methods are a much murkier area
and we should not mess with them.


> Generators are part of the language internals and language spec, so Guido
> can reasonably decide to take this in whatever direction he wants.   In
> contrast, now you're moving on to parts of the language library that merely
> call functions and return results.
>
> It is not the responsibility of accumulate(), filter(), map(), or any
> other higher-order functions to impose rules about what those functions are
> allowed to do  -- they can take as much time as they want, they can hold
> the GIL, they can manipulate signals, they can use tons of memory, they can
> return scalar values or complex objects, and they can raise any exception,
> but now StopIteration would become an event that gets special treatment.
>
> Further, this new made-up rule (for which there is zero demonstrated need
> in any language I know) would have to be applied throughout the standard
> library and possibly be extended to third-party code. It would be yet
> another language idiosyncrasy that would have to be learned, remembered,
> and StackOverflowed about for an eternity.
>
> In the case of generators, the PEP 479 rule is easily applied (not hard to
> search for or to mitigate).  In contrast, the "fix" in this case would need
> to be applied to the *called* functions or their callees, possibly far
> removed from the map/filter/accumulate call.  If that function is in
> third-party code or in a C-library, then the mitigation would require
> redesigning the call logic completely (not fun) or to wrap the function in
> something transforms a StopIteration into a custom exception and then
> re-catches the custom exception upstream from the higher-order function
> (also not fun).
>
> For anyone who has tested code that is currently working correctly but
> becomes broken by this proposal, the simplest mitigation will be for them
> to write their own variants of
> map/filter/accumulate/dropwhile/takewhile/groupby/starmap that just ignore
> this proposal and restore the behavior that has been happily in place for
> ages.
>

When it comes to map() and filter() and friends the situation is murkier
yet. In Python 2, raising StopIteration in the function passed to
map/filter was just bubbled out. In Python 3, where these have become lazy,
a StopIteration raised by the function terminates the iteration. Was that
intentional? I doubt it. Is it now a feature? Maybe (though it would make
more sense to use something like takewhile). Could it mask a subtle bug?
Probably (but it doesn't sound like a common situation -- map/filter
functions are usually small and simple). Should we "fix" it? I don't think
so. It's not sufficiently broken, the fix would lead us onto a slippery
slope. Enough is enough.


> Raymond
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141213/6ce1e082/attachment-0001.html>