Making a file-like object for manipulating a large file

[Steve Holden]

Sean Davis wrote:
> This should be a relatively simple problem, but I haven't quite got
> the idea of how to go about it.  I have a VERY large file that I would
> like to load a line at a time, do some manipulations on it, and then
> make it available to as a file-like object for use as input to a
> database module (psycopg2) that wants a file-like object (with read
> and readlines methods).  I could write the manipulated file out to
> disk and then read it back in, but that seems wasteful.  So, it seems
> like I need a buffer, a way to fill the buffer and a way to have read
> and readlines use the buffer.  What I can't do is to load the ENTIRE
> file into a stringio object, as the file is much too large.  Any
> suggestions?
> 
The general approach would be (something like the following untested code):

def filter_lines(f):
     for line in f:
         if to_be_included(line):
             yield line

fil = open("somefile.big.txt", "r")\

filegen = filter_lines(fil)

You can then iterate over the filegen generator, or write your own class 
that makes it file-like. At least the generator manages to throw away 
the unwanted content without buffering the whole file in memory.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------