[Tutor] endless processing through for loop

[Alan Gauld]

"Dinesh B Vadhia" <dineshbvadhia at hotmail.com> wrote 

fw = open(newLine.txt, 'w')
for i in xrange(0, 700,000, 1):
    read a file fname from folder
    for line in open(fname, 'r'):
        do some simple string processing on line
        fw.write(newline)
fw.close()

>  From 550,000 to 580,000 takes an additional 4 hours.

Sounds like a memory problem,. Can you look in Task 
manager (assuming Windows) or top(Linux) to see what 
the memory usage looks like? 

In theory the read files should get closed automatically but
being extra cautious I might try the inner loop as:

    fr = open(fname, 'r')
    for line in fr:
        do some simple string processing on line
        fw.write(newline)
    fr.close()

Just to be sure.
I might also try adding an fw.flush() after the write to 
ensure it goes to disk. If that isn't enough I might try 
opening and closing the write file each time using 
append mode. In theiry that shouldn't make any 
difference but it might be that its trying to hold the 
entire output file (which must be huge!) in memory...

Not certain any of those will help but its something to try!

Alan G.