Problem of function calls from map()

[Paul McGuire]

"Georg Brandl" <g.brandl-nospam at gmx.net> wrote in message
news:ecemdl$qd5$1 at news.albasani.net...
> Paul McGuire wrote:
> > "Dasn" <dasn at bluebottle.com> wrote in message
> > news:mailman.9606.1156169593.27775.python-list at python.org...
> >>
> >> Hi, there.
> >>
> >> 'lines' is a large list of strings each of which is seperated by '\t'
> >> >>> lines = ['bla\tbla\tblah', 'bh\tb\tb', ... ]
> >>
> >> I wanna split each string into a list. For speed, using map() instead
> >> of 'for' loop.
> >
<snip>
> >
> > def splitUsing(chars):
> >     def tmp(s):
> >         return s.split(chars)
> >     return tmp
> >
> > for d in map(splitUsing('\t'), data):
> >     print d
>
> And why is this better than
>
> map(lambda t: t.split('\t'), data)
>
> ?
>
> Georg

Hmm, "better" is a funny word.  My posting was definitely more verbose, but
verbosity isn't always bad.

In defense of brevity:
- often (but not always) runs faster
- usually easier to understand as a single gestalt (i.e., you don't have to
jump around in the code, or grasp the intent of a dozen or more lines, when
one or a few lines do all the work), but this can be overdone

In defense of verbosity:
- usually more explicit, as individual bits of logic are exposed as separate
functions or statements, and anonymous functions can be given more
descriptive names
- usually easier to understand, especially for language newcomers
- separate functions can be compiled by psyco

Of course, such generalizations invite obvious extremes and counterexamples.
Prime number algorithms compacted into one-liners are anything but quick to
understand; conversely, I've seen a 40-line database function exploded into
>100 classes (this was in Java, so each was also a separate file!) in
pursuit of implementing a developer's favorite GOF pattern.

This idiom (as used in the splitUsing case) of returning a callable from a
function whose purpose is to be a factory for callables seems to be a common
one in Python, I think I've seen it go by different names: currying, and
closures being most common, and decorators are another flavor of this idea.
Perhaps these idioms ("idia"?) emerged when "lambda" was on Guido's Py3K
chopping block.

So I wouldn't really hold these two routines up for "betterness" - the OP's
performance test shows them to be about the same.  To summarize his
performance results (times in CPU secs):
- explicit "for" loop - 20.510  (309130 total function calls; 154563 to
split and 154563 to append)
- list comprehension - 12.240 (154567 total function calls; 154563 to split)
- map+lambda - 20.480   (309130 total function calls; 154563 to <lambda> and
154563 to split)
- map+splitUsing - 21.900  (309130 total function calls; 154563 to tmp and
154563 to split)

The big winner here is the list comprehension, and it would seem it outdoes
the others by halving the number of function calls.  Unfortunately, most of
our currying/closure/decorator idioms are implemented using some sort of
"function-calls-an-embedded-function" form, and function calls are poison to
performance in Python (and other languages, too, but perhaps easier to
observe in Python).  Even the anonymous lambda implementation has this same
issue.

So the interesting point here is to go back to the OP's OP, in which he
states, "For speed, [I'm] using map() instead of 'for' loop."  As it turns
out, map() isn't much of a win in this case.  The real, "best" solution is
the list comprehension, not only for speed, but also for ease of readability
and understanding.  It's tough to beat this:

    return [s.split('\t') for s in lines]

for clarity, explicity, brevity, and as it happens, also for speed.

-- Paul