How can I verify if the regex exist in a file without reading ?

[Steven D'Aprano]

On Fri, 15 Jun 2018 01:01:03 -0700, francois.rabanel wrote:

> I work with a file which contains millions lines, a simply file.read()
> and I'm running out of memory

Assuming each line is on average a hundred characters long, a million 
lines is (approximately) 100 MB. Even on a computer with only 2GB of 
memory, you should be able to read 100 MB.

But you shouldn't: it is much better to process the file line by line.


# Don't do this:
with open(pathname) as f:
    text = f.read()  # Slurp the entire file into memory at once.
    ...

# Do this instead
with open(pathname) as f:
    for line in f:
        # process one line at a time


You said you are running out of memory, earlier you said the computer was 
crashing... please describe exactly what happens. If you get a Traceback, 
copy and paste the entire message.

(Not just the last line.)




-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson