[Python-Dev] Re: Optional separatorargument for file.writelines()and StringIO.writelines()

[Bob Ippolito]

On Feb 26, 2004, at 10:03 PM, Raymond Hettinger wrote:

>>> I had been led astray because I was experimenting with using
>>> cStringIO.writelines() as a basis for implementing str.join() for
>>> general iterables without creating an intermediate tuple.  Right
> now,
>>> ''.join(it) will unexpectedly consume much more memory than really
>>> needed.
>
> [Jeremy Fincher]
>> This sounds like a cool idea, are you still going to implement it?
>
>
> One way or another, I'll make str.join() smarter and faster than it is
> now.  Which way proves to be better is still open.

In trying to time this in pure python, I discovered that 
cStringIO.writelines doesn't take generators.. apparently only objects 
that support len(...) are allowed

Currently, it seems that ''.join is faster than cStringIO.writelines 
anyway, even if you just pass it a big list.

from cStringIO import StringIO
def join2(sep, seq):
     sio = StringIO()
     if not sep:
         sio.writelines(seq)
     else:
         # does not work
         def join2():
             yield seq.next()
             for s in seq:
                 yield sep
                 yield s
         sio.writelines(join2())
     return sio.getvalue()

import time
from itertools import repeat
import operator
def timeit(fn, *args, **kwargs):
     lst = []
     for ig in xrange(100):
         t0 = time.time()
         fn(*args, **kwargs)
         lst.append(time.time() - t0)
     print fn.__name__
     print '', 'max:', max(lst)
     print '', 'min:', min(lst)
     print '', 'avg:', reduce(operator.add, lst)/len(lst)


if __name__ == '__main__':
     strings = [' ' * 1000] * 1000
     print 'no separator'
     print '------------'
     timeit(join2, '', strings)
     timeit(''.join, strings)


[crack:~] bob% python strio.py
no separator
------------
join2
  max: 0.0453701019287
  min: 0.0157110691071
  avg: 0.0181557154655
join
  max: 0.0425598621368
  min: 0.00337600708008
  avg: 0.00432039260864