huge file -> memory problem
Fred Gansevles
gansevle at cs.utwente.nl
Mon Mar 13 07:59:21 EST 2000
Jørn Eilertsen wrote:
> Hi,
>
> I have this program that's supposed to wade through a rather large file
> (700MB and growing) line by line searching for a pattern. The problem
> is, when i open the file to read it using f = open(file, 'r'), it just
> takes all available memory which obviously isn't acceptable. And now the
> question; is there a way to open and read a file which isn't doesn't
> require that much memory. It seems by doing it the way I am the whole
> file is read into memory.
>
> Any help would be greatly apreciated.
>
> Jørn
In the case of a BIG file, reading the whole file consumes all the
available memory.
Since you search the file on a line-by-line basis, using 'readline()' is
the simplest solution.
Your loop looks like:
fp = open (file, 'r')
while 1:
line = fp.readline ()
if not line: # EOF
break
# if a trailing '\n' is a problem; chop it here
if line[-1:] == '\n': line = line[:-1]
process (line)
More information about the Python-list
mailing list