Simple file writing techiques ...

Ant antroy at gmail.com
Wed Jul 19 08:15:58 EDT 2006


> fout = open('somefile','w')
> for line in convertedData:
>   fout.write("%s\n" % line)
> fout.close()
>
>  -- or --
>
> fout = open('somefile','w')
> fout.write("%s" % '\n'.join(convertedData))
> fout.close()

I shouldn't think it matters too much which of these you use - time
them and see what happens.

> An issue that I'm probably most concerned with is scalabitiy, what if
> the file was huge, like some sort of log file.

Sucking in the entire file into a list won't scale well, as a huge log
file could quickly eat all of your available memory. You'd be better
off processing each line as you go, and writing it to a temp file,
renaming the temp file once you have finished. Something like:

in_f = "access.log"
out_f = "access.log.tmp"

infile = open(in_f)
outfile = open(out_f)

for line in infile:
    outfile.write(process(line))

infile.close()
outfile.close()

os.remove(in_f)
os.rename(out_f, in_f)

(Not tested, but you get the idea...)




More information about the Python-list mailing list