Most efficient way to write data out to a text file?

[Bengt Richter]

On Thu, 27 Jun 2002 01:28:22 GMT, candiazoo at attbi.com wrote:

>I assume some sort of block i/o or preallocating a large block of file space
>would be the best way to do it?  I am outputting about 700,000 records to a text
>file and my program only spits out about 20 records per second.  This is using
>standard file open and write...
>
>This is on a W2K machine... perhaps I should use the win32file interface?  Any
>thoughts/opinions?
>
I doubt that that is the solution. Where are your "records" coming from and/or
how are they being generated/mdified, and how big are they?

If you are building string records by s='' followed by s+='chunk' or the like many times,
this is a typical newbie performance killer. Try accumulating your chunks in a list,
like sl=[] followed by sl.append('chunk') instead, and then do f.write(''.join(sl))
instead of the f.write(s) you would have done.

Just guessing...

Try timing generation of output data vs writing it, and see what's happening.
import time and use clock() to get an accurate floating point seconds time reading. E.g.,

from time import clock
tstart = clock()
# ... generate an output record here
tendgen=clock()
# ... write the output  something like f.write(record) here
tendwrite=clock()
print """\
generating took %f sec
outputting took %f sec""" % (tendgen-tstart, tendwrite-tendgen)

Regards,
Bengt Richter