memory error

[Benjamin Niemann]

Bart Nessux wrote:
> def windows():
>    import os
>    excludes = ['hiberfil.sys', 'ipnathlp.dll', 'helpctr.exe', 'etc', 
> 'etc', 'etc']
>    size_list = []
>    for root, dirs, files in os.walk('/'):
>       total = [x for x in files if x not in excludes]
>       for t in total:
>          s = file(os.path.join(root,t))
>          size = s.read()
>          size_list.append(size)
>          s.close()
> 
> windows()
> 
> The above function crashes with a memory error on Windows XP Pro at the 
> 'size = s.read()' line. Page File usage (normally ~120 MB) will rise to 
> 300+ MB and pythonw.exe will consume about 200 MB of actual ram right 
> before the crash. The machine has 512 MB of ram and is doing nothing 
> else while running the script.
> 
> I've written the script several ways, all with the same result. I've 
> noticed that a binary read 'rb' consumes almost twice as much physical 
> memory and causes the crash to happen quicker, but that's about it.
> 
> Initially, I wanted to use Python to open every file on the system (that 
> could be opened) and read the contents so I'd know the size of the file 
> and then add all of the reads to a list that I'd sum up. Basically 
> attempt to add up all bytes on the machine's disk drive.
> 
> Any ideas on what I'm doing wrong or suggestions on how to do this 
> differently?
Your building an array containing the *contents* of all your files.
If you really need to use read(), use "size = len(s.read())", but this 
still requires to read and hold a complete file at a time in memory (and 
probably chokes when it stumbles over your divx collection ;)
I think using os.stat() should be better...