"".join(string_generator()) fails to be magic

[Diez B. Roggisch]

Matt Mackal schrieb:
> I have an application that occassionally is called upon to process
> strings that are a substantial portion of the size of memory. For
> various reasons, the resultant strings must fit completely in RAM.
> Occassionally, I need to join some large strings to build some even
> larger strings.
> 
> Unfortunately, there's no good way of doing this without using 2x the
> amount of memory as the result. You can get most of the way there with
> things like cStringIO or mmap objects, but when you want to actually
> get the result as a Python string, you run into the copy again.
> 
> Thus, it would be nice if there was a way to join the output of a
> string generator so that I didn't need to keep the partial strings in
> memory. <subject> would be the obvious way to do this, but it of
> course converts the generator output to a list first.

You can't built a contiguous string of bytes without copying them.

The question is: what do you need the resulting strings for? Depending 
on the use-case, it might be that you could spare yourself the actual 
concatenation, but instead use a generator like this:


def charit(strings):
     for s in strings:
         for c in s:
             yield c

Diez