[IPython-dev] Nasty, nasty references bug

[Fernando Perez]

Hi all,

https://bugs.launchpad.net/ipython/+bug/269966 has proven to be a
rather unpleasant little beast.

I've attached here a file that explains in great detail the problem
and illustrates it even without using ipyton at all.  The issue, at
heart, has to do with how the Python VM tears down module objects when
they go out of scope.  I suspect the module teardown machinery is
allowed to be so aggressive (it takes a rather Jack the Ripper
approach to the objects contained in a module, without any regard for
their reference count) because in normal use, modules are only deleted
at interpreter shutdown.  By that point all politeness tends to be
cast overboard and the interpreter just goes on a rampage on anything
it finds in memory.

IPython, as usual, finds a way to clash with this (we have a good
track record of finding dark corners of Python's behavior).  So
basically we have the following:

- if we cache all execution modules, we leak memory.  For certain use
patterns this can be a major problem, and it has been reported several
times.

- but if we don't, there are cases that become totally broken.  The
test script attached may seem artificial, but with matplotlib it
becomes easy to get the problem.

I do have two ideas on what to do, but I won't be able to try them
until Monday (limited net  access this weekend):

- go back to having FakeModule NOT inherit from a real module.  Now we
have real tests and we can try to build a robust test harness for this
to work without being a true module subclass.

- I'm going to try to keep a single instance of the execution module,
but replacing its __dict__ with a fresh one each run.  This might give
us all the behavior we need, without triggering the Python Module Mad
Butcher.

I tend to favor the latter, so I'll give it a try on Monday, once I
have access to my normal computer.  If anyone has any insights on this
and knows the reasons behind this part of Python's internal machinery,
I'm quite interested.

I might also ping python-dev at least to query them for their thoughts on this.

Cheers,

f


ps - Many thanks to Matthew Brett for his patience when I needed
someone to discuss this and clarify things out.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: modrefs.py
Type: text/x-python
Size: 7192 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090404/51255bac/attachment.py>