optomizations

[Chris Angelico]

On Tue, Apr 23, 2013 at 11:19 AM, Rodrick Brown <rodrick.brown at gmail.com> wrote:
>             with gzip.open(args.inputfile) as datafile:
>                 for line in datafile:
>                     outfile = '{}{}{}_combined.log'.format(dateobj.year,
> dateobj.month, dateobj.day)
>                     outdir = (args.outputdir + os.sep + siteurl)
>
>                     with open(outdir + os.sep + outfile, 'w+') as outf:
>                         outf.write(line)

You're opening files and closing them again for every line. This
wouldn't cause you to spin the CPU (more likely it'd thrash the hard
disk - unless you have an SSD), but it is certainly an optimization
target.

Can you know in advance what files you need? If not, I'd try something
like this:

outf = {} # Might want a better name though

.....
               outfile = ...
               if outfile not in outf:
                   os.makedirs(...)
                   outf[outfile] = open(...)
               outf[outfile].write(line)

for f in outf.values():
  f.close()

Open files only as needed, close 'em all at the end.

ChrisA