[SciPy-Dev] Circular reference in interp1d causing memory errors

[josef.pktd at gmail.com]

On Sat, May 18, 2013 at 8:24 AM, Pauli Virtanen <pav at iki.fi> wrote:

> 18.05.2013 14:14, josef.pktd at gmail.com kirjoitti:
> [clip]
> > Does anyone know a more explicit explanation?
> > Or, is it really just the cyclical reference, even though it's a
> > method (which always hold self)
>
> Python gc is capable of breaking most reference cycles automatically.
> The point however is that it does not run on every allocation. If you
> allocate big objects that are kept around by a reference cycle at a too
> rapid rate, you run into a memoryerror before the gc gets around to
> breaking the cycles.
>

That's the problem with applications with large numpy arrays, especially in
loops, cyclic garbage collection doesn't kick in soon enough.

We didn't realize this in statsmodels until a user pointed out that we have
an (unnecessary) cyclical reference.

scipy.stats.distributions are safe, there shouldn't be memory problems,
since they reuse the instance and don't hold any data.
But we need to check this in statsmodels, where we still use instance
method assignment in the __init__ of one or three models.

Thanks,

Josef


>
> --
> Pauli Virtanen
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20130518/f44b9a41/attachment.html>