built in zip function speed

[Steven D'Aprano]

On Tue, 04 Jul 2006 07:18:29 -0700, mart.franklin at gmail.com wrote:

> I hope I am not being too ignorant :p but here goes... my boss has
> written a bit of python code and asked me to speed it up for him...
> I've reduced the run time from around 20 minutes to 13 (not bad I think
> ;) to speed it up further I asked him to replace a loop like this:-
> 
> 
> index = 0
> 
> for element in a:
>    av = a[index]
>    bv = b[index]
>    cv = c[index]
>    dv = d[index]
>    avbv = (av-bv) * (av-bv)
>    diff = cv - dv
>    e.append(diff - avbv)
>    index = index + 1

This is, I think, a good case for an old-fashioned for-with-index loop:

for i in len(a):
    e.append(c[i] - d[i] - (a[i] - b[i])**2)

Python doesn't optimize away lines of code -- you have to do it yourself.
Every line of Python code takes a bit of time to execute. My version uses
34 lines disassembled; yours takes 60 lines, almost twice as much code.

(See the dis module for further details.)

It's too much to hope that my code will be twice as fast as yours, but it
should be a little faster.

> (where a, b, c and d are 200,000 element float arrays)
> to use the built in zip function.. it would seem made for this problem!
> 
> for av, bv, cv, dv in zip(a, b, c, d):
>    avbv = (av-bv) * (av - bv)
>    diff = cv - dv
>    e.append(diff - avbv)
> 
> however this seems to run much slower than *I* thought it would
> (and in fact slower than slicing) I guess what I am asking is.. would
> you expect this?

Yes. zip() makes a copy of your data. It's going to take some time to copy
4 * 200,000 floats into one rather large list. That list is an ordinary
Python list of objects, not an array of bytes like the array module
uses. That means zip has to convert every one of those 800,000 floats
into rich Python float objects. This won't matter for small sets of data,
but with 800,000 of them, it all adds up.


-- 
Steven.