[Mailman-Developers] re: Much cpu/memory load

[Dan Mick]

Dan Mick wrote:
> Simone Piunno wrote:
> 
>> On Thu, Oct 31, 2002 at 07:03:50PM -0800, Dan Mick wrote:
>>
>>
>>> The problem's affecting me badly right now too.  So far it seems that 
>>> the holdup is in the 'date.html' index file processing; everything 
>>> else is finished, but the "bin/qrunner -r Arch -o" process has lost 
>>> its little mind trying to write the date index.  It's in a loop, 
>>> chewing up 409600 bytes of memory (brk), and then opening, reading, 
>>> and closing archidxentry.html,
>>> hundreds of times in a row.
>>
>>
>>
>> Ok, I believe this is because of my patch for i18n in archives.
>> Do you feel like trying this patch?  It should speed up things a lot....
>>
>> Index: HyperArch.py
>> ===================================================================
>> RCS file: /cvsroot/mailman/mailman/Mailman/Archiver/HyperArch.py,v
>> retrieving revision 2.22
>> diff -u -r2.22 HyperArch.py
>> --- HyperArch.py        19 Oct 2002 20:59:27 -0000      2.22
>> +++ HyperArch.py        1 Nov 2002 10:13:21 -0000
>> @@ -923,10 +923,14 @@
>>              'sequence': article.sequence,
>>              'author':   author
>>          }
>> -        print Utils.maketext(
>> -            'archidxentry.html', d, raw=1,
>> -            lang=self.maillist.preferred_language,
>> -            mlist=self.maillist)
>> +        print """<LI><A HREF="%(filename)s">%(subject)s
>> +        </A><A NAME="%(sequence)i">&nbsp;</A>
>> +        <I>%(author)s
>> +        </I>""" % d
>> +        #print Utils.maketext(
>> +        #    'archidxentry.html', d, raw=1,
>> +        #    lang=self.maillist.preferred_language,
>> +        #    mlist=self.maillist
>>
> 
> That's definitely the cause of the many opens of archidxentry.html, but it
> doesn't explain the 400K-per-iteration growth.
> 
> Barry, wasn't there some Python instrumentation trick for using the GC 
> to find unreferenced objects and complain, or invoking GC manually in a 
> loop to try to alleviate such problems?....was it in the gc module?...

yes, with that patch the opens of archidxentry.html go away, and now its 
only system call activity is brk() after brk() after brk() (with an 
occasional flush of buffers to date.html).  Something is wrong in the 
date-processing loop that's leaking memory hand-over-fist.