writing to file very slow

[John Machin]

"Moritz Lennert" <mlennert at club.worldonline.be> wrote in message news:<mailman.1048690542.21767.python-list at python.org>...

> 
> So what is wrong ? Is there a limit to the size of a pgqueryobject in
> PyGreSQL ? Is my writing routine not very efficient ? Should I use another
> module to connect to PostgreSQL ?

Your writing routine is not very efficient. Apart from the obvious
"string append instead of using join" problem which has already been
mentioned, you call results.getresult() MxN times when it needs to be
called only once. This is probably the killer, not the first suspect.
Below I have rewritten the whole function, using list comprehensions
and avoiding range() as much as possible. Together with some cosmetics
like spaces around "=" and more than 2 chars of indent, plus caching
some attribute look-ups, it should now be fast enough and legible
enough -- even looks like it was written in a HLL like Python :-)

You may even be able to avoid one or both of the list comprehensions;
see comments in the code.

BTW, according to pychecker, "tempfile.mktemp is deprecated". It will
also tell you that temdir is not used.

Hope this helps,
John

# compiled but untested
def fichier_resultats(results):
   temdir = "/tmp" # not used !!!
   tfilename = tempfile.mktemp('rec.txt')
   f = open(tfilename,'w')
   write_record = f.write

   numfields = len(results.listfields()) 
   # This was being obtained many times instead of once.

   bar_join = "|".join

   varnames = [str(results.fieldname(z)) for z in xrange(numfields)]
   # (1) Do you REALLY need the str()?? What does fieldname() return
if not a string???
   # (2) I suggest changing the API so that fieldname is a list
instead of a method.
   write_record(bar_join(varnames))
   write_record("\n")

   whole_result = results.getresult()
   # above is an MxN list of lists which was being got MxN times!!!
   for x_result in whole_result:
      var = [str(x_result[y]) for y in x_result]
      write_record(bar_join(var))
      # If you don't need str(), above 2 lines become simply:
      # write_record(bar_join(x_result))
      write_record("\n")

   f.close()
   return(tfilename)