performance degradation when looping through lists

[Joachim Worringen]

bruno at modulix wrote:
> Joachim Worringen wrote:
>> I need to process large lists (in my real application, this is to parse
>> the content of a file). 
> 
> Then you probably want to use generators instead of lists. The problem
> with large lists is that they eat a lot of memory - which can result in
> swapping .

The effect also shows up in tiny examples (as the one posted) which surely don't 
swap on a 512MB machine.

Also, I only read parts of the file into memory to avoid that memory becomes 
exhausted.

Of course, using less memory is always a good idea - do you have a pointer on 
how to use generators for this application (basically, buffering file content in 
memory for faster access)? BTW, the effect also shows up with the linecache module.

>> I noticed that the performance to access the
>> individual list elements degrades over runtime.
> 
> I leave this point to gurus, but it may have to do with swapping. Also,
> this is not real-time, so variations may have to do with your OS tasks
> scheduler.

See above for the swapping. And the OS scheduler may create variations in 
runtime, but not monotone degradation. I don't think these two effect come into 
play here.

-- 
Joachim - reply to joachim at domain ccrl-nece dot de

Opinion expressed is personal and does not constitute
an opinion or statement of NEC Laboratories.