Iterators everywhere in Python 3000 (was: Why is dictionary.keys() a list and not a set?)

[Robert Brewer]

Alex Martelli wrote:
> The typical reason I'm asking for .values() is because each
> value is a count (e.g. of how many time the key has occurred)
> and I want the total, so the typical idiom is sum(d.values())
> -- getting a result set would be an utter disaster, and should
> I ever want it, set(d.values()) is absolutely trivial to code.
> 
> Note that in both cases I don't need a LIST, just an ITERATOR,
> so itervalues would do just as well (and probably faster)
> except it looks more cluttered and by a tiny margin less
> readable -- "the sum of the values" rolls off the tongue,
> "the sum of the itervalues" doesn't!-) So, the direction
> of change for Python 3000, when backwards compatibility
> can be broken, is to return iterators rather than lists.
> At that time, should you ever want a list, you'll say so
> explicitly, as you do now when you want a set, i.e.
> list(d.values())

I'd just like to throw in my anecdotal opinion regarding interfaces based on iterators. I wrote Dejavu, a pure-Python ORM, starting at about Python 2.3. I thought iterators were a Pretty Neat Idea, and used/generated iterators "all the way down": SQL results were iterated over, ORM Units were created and yielded to a StorageManager interface, which yielded them to a Sandbox interface, which yielded them to the caller. It worked wonderfully!

But it was a nightmare to use. Yes, half the calls were "for thing in recall(cls, expr):", but the half that were not (slicing, popping, len, and most of all: sort) required "list(things)" or "[x for x in things]" to be written again and again. It seems trivial until you've done it many, many times. When one of my users prompted me to switch back to lists for the public interface, I jumped at the idea, and have been much happier since then as an end-user of my own library. I still have a top-level "xrecall" method, for those times when I really *need* an iterator (4.5% of the time in my current apps), and I use iterators "behind the scenes" still to support that.

It would be even worse for Python: Dejavu might require a half-dozen list() calls in any given module. Some Python 3000 code could require hundreds. Because I wrote Dejavu in Python, I could separate the public, list-like interface from the internal, iterative ones pretty easily. Python can't do that for itself; if dict.values became an iterator, then a huge swath of end-user code, and even the stdlib, would be affected, with no interface layer in-between to call list() for you. [I wonder how the PyPy folks feel about "iterators everywhere"...?]

Given my experience with Dejavu, I'd rather not see Python 3000 move to using iterators with abandon. IMO, it fixes a poor interface (iter) for 5% of code by introducing a poor interface (list) for 50% of code.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20051124/1aed758f/attachment.html>