[SciPy-Dev] Circular reference in interp1d causing memory errors

[josef.pktd at gmail.com]

On Sat, May 18, 2013 at 9:27 AM, <josef.pktd at gmail.com> wrote:

>
>
> On Sat, May 18, 2013 at 8:24 AM, Pauli Virtanen <pav at iki.fi> wrote:
>
>> 18.05.2013 14:14, josef.pktd at gmail.com kirjoitti:
>> [clip]
>> > Does anyone know a more explicit explanation?
>> > Or, is it really just the cyclical reference, even though it's a
>> > method (which always hold self)
>>
>> Python gc is capable of breaking most reference cycles automatically.
>> The point however is that it does not run on every allocation. If you
>> allocate big objects that are kept around by a reference cycle at a too
>> rapid rate, you run into a memoryerror before the gc gets around to
>> breaking the cycles.
>>
>
> That's the problem with applications with large numpy arrays, especially
> in loops, cyclic garbage collection doesn't kick in soon enough.
>
> We didn't realize this in statsmodels until a user pointed out that we
> have an (unnecessary) cyclical reference.
>
> scipy.stats.distributions are safe, there shouldn't be memory problems,
> since they reuse the instance and don't hold any data.
> But we need to check this in statsmodels, where we still use instance
> method assignment in the __init__ of one or three models.
>

Follow-up question:

Is there a way to check whether a class gets garbage collected by reference
counting, and gets immediately collected when the reference count is zero?
Something that can be used in unittests.

I never went over the details of gc.

Josef


>
> Thanks,
>
> Josef
>
>
>>
>> --
>> Pauli Virtanen
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20130518/3748107f/attachment.html>