[Python-Dev] iterzip()

[Tim Peters]

[Raymond Hettinger]
> ...
> I just timed it and am shocked.  iterzip() has exactly the same code
> as zip() except for the final append result to list.  So, I expected only
> a microscopic speed-up.  I don't see where the huge performance
> improvement came from.

Well, in the zip case you're allocating 4 million bytes worth of integers, 4
bytes at a time, and don't get to reuse any of the space for the duration.
Likewise you're allocating space for a million 3-element tuples, and 4MB
worth of contiguous vector 4 bytes at a time.  That's all a real drag.
Cute:  I can make the Python genzip run faster than the builtin zip with
"the usual" speed tricks:

from time import clock as now

def run(f, iterables):
    start = now()
    for tup in f(*iterables):
        pass
    return now() - start

def genzip(*iterables):
    getters = [iter(i).next for i in iterables]
    t = tuple
    while 1:
        yield t([g() for g in getters])

n = 1000000
for f in [zip, genzip]:
    print run(f, [xrange(n), xrange(n), xrange(n)]), f.__name__

On my box under current CVS, that yields

C:\Code\python\PCbuild>python -O temp.py
12.3587522419 zip
10.4161551484 genzip

Better, I can get genzip down to 8.34 seconds by skipping the tuple()
call -- in real life, it doesn't really matter whether it returns a tuple or
a list.

One reason zip is faster for me than for you is that pymalloc is enabled by
default in 2.3, and can likely pump out space for a million small tuples
significantly faster than your platform malloc can do it.

> First timing:
> ----------------------
> 19.439999938 zip
> 1.43000006676 iterzip
> 30.1000000238 genzip

If only I had a use for iterzip <wink>.