memory leak in Python 2.1/FreeBSD? [was Re: Garbage collection on strike?]

[greg jorgensen]

On 10 May 2001, Aahz Maruch wrote:

> * Much more likely, though, is that you're simply not deleting object
> references.  Garbage collection in Python only works on circular
> references, where two or more objects refer to each other, but none has
> any external references pointing in.  If you've got a dict, for example,
> that has a reference to the objects that aren't disappearing, your
> memory usage will keep going up as you allocate new objects.

When you wrote "GC in Python only works on circular references" you are
referring to the garbage collector and not the reference counting
mechanism, right? I've been working on some code that fails on FreeBSD but
works on Linux (different PCs, different amount of RAM/swap, both using
Python 2.1). I haven't had time to work on the problem but I was
suspecting a memory/garbage collection problem.

The suspect code is a loop:

  f = open('filename', 'rb')
  while 1:
      buf = f.read(blocksize)
      if not buf: break
      # buf is sent over open ftp connection
      conn.send(buf)

(It's an overriden version of ftplib.FTP.storbinary that writes a '#' to
stdout for each block transferred).

On the FreeBSD machine transferring a 19 meg file the Python program quits
with no errors after transferring about 10% of the file. If I change the
block size from 8192 (default) to 16384 it quits after sending only 5% of
the file. There's no error; the loop above terminates because of the "if
not buf:" test. The file I'm transferring is a .zip archive; it verifies
and unzips without any problems.

This same script on my Linux box can successfully transfer the entire 19
meg file to the same ftp host, though it is a lot slower than using ftp
from the console.

Do I need to explicitly delete "buf" in my loop after I use it? My
understanding of Python's reference counting model tells me no, it will go
away automatically when all references to it are gone; simply binding buf
again to the bytes returned by read() should make the memory buf was
holding go away. Right?

Any ideas on why this works on my Linux box (128 megs of RAM) but fails
reliably in the same place on the FreeBSD box (64 megs)?

Thanks!

Greg Jorgensen
gregj at pobox.com