[Python-ideas] __len__() for map()

[E. Madison Bray]

On Thu, Nov 29, 2018 at 9:36 PM Terry Reedy <tjreedy at udel.edu> wrote:
> >> https://docs.python.org/3/library/functions.html#map says
> >> "map(function, iterable, ...)
> >>       Return an iterator [...]"
> >>
> >> The wording is intentional.  The fact that map is a class and the
> >> iterator an instance of the class is a CPython implementation detail.
> >> Another implementation could use the generator function equivalent given
> >> in the Python 2 itertools doc, or a translation thereof.  I don't know
> >> what pypy and other implementations do.  The fact that CPython itertools
> >> callables are (now) C-coded classes instead Python-coded generator
> >> functions, or C translations thereof (which is tricky) is for
> >> performance and ease of maintenance.
> >
> > Exactly how intentional is that wording though?
>
> The use of 'iterator' is exactly intended, and the iterator protocol is
> *intentionally minimal*, with one iterator specific __next__ method and
> one boilerplate __iter__ method returning self.  This is more minimal
> than some might like.  An argument against the addition of length_hint
> and __length_hint__ was that it might be seen as extending at least the
> 'expected' iterator protocol.  The docs were written to avoid this.

You still seem to be confusing my point.  I'm not advocating even for
__length_hint__ (I think there are times that would be useful but it's
still pretty problematic).

I admit one thing I'm a little stuck on though is that map() currently
just immediately calls iter() on its arguments to get their iterators,
and does not store references to the original iterables.  It would be
nice if more iterators could have an exposed reference to the objects
they're iterating, in cases where that's even meaningful.  For some
reason I thought, for example, that a list_iterator could give me a
reference back to the list itself.  This was probably omitted
intentionally but it still feels pretty limiting :(

> >  Point being, I don't think it's a massive leap or
> > imposition on any implementation to go from "Return an iterator [...]"
> > to "Return an iterator that has these attributes [...]"
>
> Do you propose exposing the inner struct members of *all* C-coded
> iterators?  (And would you propose that all Python-coded iterators
> should use public names for the equivalents?)  Some subset thereof?
> (What choice rule?)  Or only for map?  If the latter, why do you
> consider map so special?

Not necessarily, no.  But certainly a few: I'm using map() as an
example but at the very least map() and filter().  An exact choice
rule is something worth thinking about but I don't think you're going
to find an "objective" rule.  I think it goes without saying that
map() is special in a way: It's one of the most basic extensions to
function application and is a fundamental construct in functional
programming and from a category-theortical perspective.  I'm not
saying Python's built-in map() needs to represent anything
mathematically formal, but it's certainly quite fundamental which is
why it's a built-in in the first place.

> >>> This is necessary because if I have a function that used to take, say,
> >>> a list as an argument, and it receives a `map` object, I now have to
> >>> be able to deal with map()s,
>
> In both 2 and 3, the function has to deal with iterator inputs one way
> or another.  In both 2 and 3, possible interator inputs includes maps
> passed as generator comprehensions, '(<expression with x> for x in
> iterable)'.

Yes, but those are still less common, and generator expressions were
not even around when Sage was first started: I've been around long
enough to remember when they were added to the language, and were well
predated by map() and filter().  The Sagebook [1] introduces them
around page 60.  I'm not sure if it even introduces generators
expressions at all.

I think a lot of Python and C++ experts don't realize that the
"iterator" concept is not at all immediately obvious to a lot of
non-programmers.  Most iterator inputs supplied by users are things
like sized collections for which it's easy to think about "going over
them one by one" and not more abstract iterators.  This is true
whether the user is a Python expert or not.

> >> If a function is documented as requiring a list, or a sequence, or a
> >> length object, it is a user bug to pass an iterator.  The only thing
> >> special about map and filter as errors is the rebinding of the names
> >> between Py2 and Py3, so that the same code may be good in 2.x and bad in
> >> 3.x.
> >
> > It's not a user bug if you're porting a massive computer algebra
> > application that happens to use Python as its implementation language
> > (rather than inventing one from scratch) and your users don't need or
> > want to know too much about Python 2 vs Python 3.
>
> As a former 'scientist who programs' I can understand the desire for
> ignorance of such details.  As a Python core developer, I would say that
> if you want Sage to allow and cater to such ignorance, you have to
> either make Sage a '2 and 3' environment, without burdening Python 3, or
> make future Sage a strictly Python 3 environment (as many scientific
> stack packages are doing or planning to do).

"ignorance" is not a word I would use here, frankly.

> ...
> > That said, I regret bringing up Sage; I was using it as an example but
> > I think the point stands on its own.
>
> Yes, the issues of hiding versus exposing implementation details, and
> that of saving versus deleting and, when needed, recreating 'redundant'
> information, are independent of Sage and 2 versus 3.

I agree there, that this is not really an argument about Sage or
Python 2/3.  Though I don't think this is an "implementation detail".
In an abstract sense a map is a special container for a function and a
sequence that has special semantics.  As far as I'm concerned this is
what it *is* in some ontological sense, and this fact is not a mere
implementation detail.

[1] http://dl.lateralis.org/public/sagebook/sagebook-ba6596d.pdf