Making a file-like object for manipulating a large file

Sean Davis seandavi at gmail.com
Thu Aug 23 22:00:25 EDT 2007


This should be a relatively simple problem, but I haven't quite got
the idea of how to go about it.  I have a VERY large file that I would
like to load a line at a time, do some manipulations on it, and then
make it available to as a file-like object for use as input to a
database module (psycopg2) that wants a file-like object (with read
and readlines methods).  I could write the manipulated file out to
disk and then read it back in, but that seems wasteful.  So, it seems
like I need a buffer, a way to fill the buffer and a way to have read
and readlines use the buffer.  What I can't do is to load the ENTIRE
file into a stringio object, as the file is much too large.  Any
suggestions?

Thanks,
Sean




More information about the Python-list mailing list