What are the limitations of cStringIO.StringIO() ?

[Simon Forman]

On Wed, Jul 1, 2009 at 7:48 AM, Barak, Ron<Ron.Barak at lsi.com> wrote:
> Hi,
>
> I think I'm up against a limitation in cStringIO.StringIO(), but could not
> find any clues on the web, especially not in
> http://docs.python.org/library/stringio.html.
>
> What I have is the following function:
>
>     def concatenate_files(self,filename_array):
>         import types
>
>         f = cStringIO.StringIO()
>         for filename in filename_array:
>             log_stream = LogStream(filename)
>             if not isinstance(log_stream, types.NoneType):
>                 try:
>                     string_ = log_stream.input_file.read()
>                 except AttributeError, e:
>                         sys.stderr.write("AttributeError: "+str(e)+"\n")
>                         sys.stderr.write("log_stream:
> "+str(log_stream)+"\n")
>             else:
>                 return(None)
>
>             f.write(string_)
>         return (f)
>
> And, when the list of files - in filename_array - is pointing to long/large
> enough files, I get the exception:
>
> Traceback (most recent call last):
>   File "svm_ts_tool_in_progress.py", line 244, in OnSelectCell
>     self.InspectorViewController(row,col)
>   File "svm_ts_tool_in_progress.py", line 1216, in InspectorViewController
>     self.InspectorePaneGetRecords(self.req_table, rec)
>   File "svm_ts_tool_in_progress.py", line 1315, in InspectorePaneGetRecords
>     log_stream =
> ConcatenatedLogStream(self.events_dict[req_table]["db_dict"].keys())
>   File "c:\Documents and
> Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 31, in
> __init__
>     self.input_file = self.concatenate_files(filename_array)
>   File "c:\Documents and
> Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 47, in
> concatenate_files
>     string_ = log_stream.input_file.read()
> MemoryError
>
> Anyone knows whet's the limitation on cStringIO.StringIO() objects ?
> Could you suggest a better way to create a string that contains the
> concatenation of several big files ?
>
> Thanks,
> Ron.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>

MemoryError means you're out of memory.  Try using the operating
system commands for file concatenation, i.e. "cat" on *nix, dunno how
on windows.  Or just do whatever it is you're doing in a different way
that doesn't require having all the data in memory at once.

Also, this:

not isinstance(log_stream, types.NoneType)

should just be this:

log_stream is not None

and return is not a function, leave out the ()'s around the expression
you're returning, and if you're returning None, you can leave out the
None as it is the default return value anyway.

Generally speaking don't do module imports in functions.