"".join(string_generator()) fails to be magic

[Carl Banks]

On Oct 11, 2:26 am, Matt Mackal <m... at selenic.com> wrote:
> I have an application that occassionally is called upon to process
> strings that are a substantial portion of the size of memory. For
> various reasons, the resultant strings must fit completely in RAM.
> Occassionally, I need to join some large strings to build some even
> larger strings.

Do you really need a Python string?  Some functions work just fine on
mmap or array objects, for example regular expressions:

>>> import array
>>> import re
>>> a = array.array('c','hello, world')
>>> a
array('c', 'hello, world')
>>> m = re.search('llo',a)
>>> m
<_sre.SRE_Match object at 0x009DCB80>
>>> m.group(0)
array('c', 'llo')

I would look to see if there's a way to use an array or mmap instead.
If you have an upper bound for the total size, then you can reserve
the needed number of bytes.

If you really need a Python string, you might have to resort to a C
solution.


Carl Banks