What are the limitations of cStringIO.StringIO() ?

[Stephen Hansen]

>
>
>    1. Anyone knows whet's the limitation on cStringIO.StringIO() objects ?
>    2. Could you suggest a better way to create a string that contains the
>    concatenation of several big files ?
>
>
This isn't a limit in cStringIO.StringIO, but a limit on the amount of
memory a process is able to allocate from the OS. There's no real way
around it-- but you can take a different approach.

You have to get the memory somewhere, but if what you're combining
can't fit into memory (generally, 2 GB, but it depends on OS and
platform), you can use memory from a different source-- the hard-drive
for instance.

I would suggest you create a temporary file somewhere, and instead of
reading the contents into memory, read it in chunks and write it to
this temporary file, then go on and do the rest. That way you'll never
have so much in memory that you hit the limits. When this is done, you
can move this temporary file to where you want it, or read from it and
pass its output somewhere.

--S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20090701/36980e11/attachment-0001.html>