performance degradation when looping through lists

[Peter Otten]

Joachim Worringen wrote:

> I need to process large lists (in my real application, this is to parse
> the content of a file). I noticed that the performance to access the
> individual list elements degrades over runtime.
> 
> This can be reproduced easily using this code:
> 
> import time
> 
> N=100000
> p=10000
> 
> A=[]
> for i in range(N):
>      A.append(str(i))
> 
> j = 0
> t = time.clock()
> for i in range(len(A)):
>     j += int(A[i])
>     if i % p == 0:
>         t = time.clock() - t
>         print t
> 
> (the string conversion only servers to increase the duration of each
> iteration; you can observer the same effect with ints, too).
> 
> When running this, I get output like this:
> 0.0
> 0.37
> 0.03
> 0.4
> 0.06
> 0.43
> 0.09
> 0.46
> 0.13
> 0.49
> 
> I use Python 2.3.4 (#1, Sep  3 2004, 12:08:45)
> [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-110)] on linux2
> 
> I wonder why
> 1. The execution times alternate between "fast" and "slow" (I observe the
> same effect in my much more complex application)

Your timing code is buggy. Change it to

import time

N=100000
p=10000

A=[]
for i in range(N):
     A.append(str(i))

j = 0
start = time.clock()
for i in range(len(A)):
    j += int(A[i])
    if i % p == 0:
        end = time.clock()
        print end - start
        start = end

Does the problem persist? I hope not.

Peter