Please help with MemoryError

[Stephen Hansen]

On Thu, Feb 11, 2010 at 3:39 PM, Jeremy <jlconlin at gmail.com> wrote:

> My Python program now consumes over 2 GB of memory and then I get a
> MemoryError.  I know I am reading lots of files into memory, but not
> 2GB worth.  I thought I didn't have to worry about memory allocation
> in Python because of the garbage collector.  On this note I have a few
> questions.  FYI I am using Python 2.6.4 on my Mac.
>

Is this pure-python, or are you using extension modules? Its possible a C
extension has a memory leak. Its also possible you're keeping references to
objects far longer then you need.


> 1.    When I pass a variable to the constructor of a class does it
> copy that variable or is it just a reference/pointer?  I was under the
> impression that it was just a pointer to the data.
>

The question doesn't precisely make sense in terms of Python, but I think I
see what you're getting at. Python doesn't implicitly copy anything: you
have to explicitly copy an object to end up with two of them. That said, it
sort of depends on what you're doing. When you pass objects into a class, if
that class stores a reference to those objects, they'll live as long as the
class does. More details would require you showing some examples of what
you're doing.


> 2.    When do I need to manually allocate/deallocate memory and when
> can I trust Python to take care of it?
>

Generally, you can "trust" Python-- and you can't manually
allocate/deallocate even if you wanted to.

If you're suffering a memory leak, it almost certainly comes from one of two
places:

1. Your code is keeping references somewhere to objects that you don't need
anymore, or
2. Some C extension you are using has some ref-counting bugs.

Once upon a time, reference cycles would also cause memory leaks, but the
cyclic GC breaks them now.


> 3.    Any good practice suggestions?
>

Look over your code: look at places where you bind objects to any names that
aren't local. Look for lists or dictionaries that you're putting stuff into
without really meaning to keep it forever, and maybe you aren't purging.
Global vars are worth a close look too.

This may be useful:
http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks

--S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20100211/105fd670/attachment-0001.html>