[New-bugs-announce] [issue5445] codecs.StreamWriter.writelines problem when passed generator

[Daniel Lescohier]

New submission from Daniel Lescohier <daniel.lescohier at cbs.com>:

This is the implementation of codecs.Streamwriter.writelines for all 
Python versions that I've checked:

    def writelines(self, list):

        """ Writes the concatenated list of strings to the stream
            using .write().
        """
        self.write(''.join(list))

This may be a problem if the 'list' parameter is a generator. The 
generator may be returning millions of values, which the join will 
concatenate in memory. It can surprise the programmer with large 
memory use. I think join should only be used when you know the size of 
your input, and this method does not know this. I think the safe 
implementation of this method would be:

    def writelines(self, list):

        """ Writes the concatenated list of strings to the stream
            using .write().
        """
        write = self.write
        for value in list:
            write(value)

If a caller knows that it's input list would use a reasonable amount 
of memory, it can get the same functionality as before by doing 
stream.write(''.join(list)).

----------
components: Library (Lib)
message_count: 1.0
messages: 83322
nosy: dlesco
nosy_count: 1.0
severity: normal
status: open
title: codecs.StreamWriter.writelines problem when passed generator
versions: Python 2.5, Python 2.6, Python 2.7, Python 3.0

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5445>
_______________________________________