huge file -> memory problem

[Jørn Eilertsen]

Hi Fred,

Thanks for taking the time, but what you describe is basically what I've done,
but it still seems to swallow the whole file into memory.

Regards,
Jorn

Fred Gansevles wrote:

> Jørn Eilertsen wrote:
>
> > Hi,
> >
> > I have this program that's supposed to wade through a rather large file
> > (700MB and growing) line by line searching for a pattern. The problem
> > is, when i open the file to read it using f = open(file, 'r'), it just
> > takes all available memory which obviously isn't acceptable. And now the
> > question; is there a way to open and read a file which isn't doesn't
> > require that much memory. It seems by doing it the way I am the whole
> > file is read into memory.
> >
> > Any help would be greatly apreciated.
> >
> > Jørn
>
> In the case of a BIG file, reading the whole file consumes all the
> available memory.
> Since you search the file on a line-by-line basis, using 'readline()' is
> the simplest solution.
>
> Your loop looks like:
>
>         fp = open (file, 'r')
>         while 1:
>                 line = fp.readline ()
>                 if not line:    # EOF
>                         break
>                 # if a trailing '\n' is a problem; chop it here
>                 if line[-1:] == '\n': line = line[:-1]
>                 process (line)