getting memory usage of varaibles

[Larry Martell]

On Wed, May 3, 2017 at 8:29 AM, Ned Batchelder <ned at nedbatchelder.com> wrote:
> On Tuesday, May 2, 2017 at 11:49:37 PM UTC-4, Larry.... at gmail.com wrote:
>> On Tue, May 2, 2017 at 7:01 PM, Erik <python at lucidity.plus.com> wrote:
>> > On 02/05/17 23:28, Larry Martell wrote:
>> >>>>
>> >>>> Anyone have any thoughts on how I can monitor the variables' memory
>> >>>> usage as the script runs?
>> >>>
>> >>>
>> >>> This is application-specific, but sometimes it helps to look at the
>> >>> objects' types, or even their values.
>> >>
>> >>
>> >> The types are dict and list, so they are not very useful, nor were the
>> >> values.
>> >
>> >
>> > Given your description, I would hazard a guess that you have a circular
>> > reference somewhere (such that some objects may remain referenced after you
>> > believe they should have been discarded). You could _probably_ break that
>> > circle with some carefully placed 'del' statements somewhere - but you'd
>> > need to analyze your data structures (and closures) first to see if that's
>> > the case.
>>
>> This script is part of a django project, but it is not part of the
>> core web django app. It's a stand alone script that is run from the
>> command line. I added some profiling to the script and I found that
>> the object count was increasing repidly with each loop (and of course
>> with that the memory usage). I originally wrote it using the django
>> ORM but now I have rewritten it using MySQLdb and embedded SQL. That
>> stopped the increase in the object count and drastically reduced the
>> memory leakage, but did not stop it. But at least now I can run it
>> without it using all of the memory and getting killed. I'd still love
>> to know where it's leaking memory, but I may not get the time for
>> that.
>
> When you were using the Django ORM, did you have settings.DEBUG=True?
> If so, Django stores the SQL queries for debugging.

Yes, I did have DEBUG=True, but I wasn't trying to see the SQL Django
was generating. But looking at the objects I found that when the
script entered its main loop there were around 28K objects, and each
pass through the loop would add around 10K new objects. When I looked
at their content most of them were lists of the column names of the
table I was selecting from - many, many objects with the same values.
When I switched to not using the ORM there were 28K objects throughout
the entire run of the script (which was 100's of millions of
iterations of its loop).