[Mailman-Users] Problem with archrunner using large %'s of cpu (read faq & archives)

[Richard Barrett]

On Friday, October 31, 2003, at 11:59  pm, Brad Knowles wrote:

> At 6:21 PM -0500 2003/10/31, Scott Lambert wrote:
>
>>  I haven't looked at the code yet, and probably won't (ENOTIME), but  
>> it
>>  almost sounds to me like it's not pruning it's list of handled  
>> messages
>>  and has to walk all of them each time.  I would have expected queue
>>  handling to get faster as the queue got smaller due to fewer files
>>  in the directory that it needs to search through.  Maybe it's just a
>>  function of the python datastructure being used.
>
> 	If it's using files as the queue mechanism, then deleting a file  
> simply marks the entry in the directory as "available", and it still  
> takes just at long to scan the directory afterwards as it did before.
>
> 	This is a known problem with many MTAs handling large amounts of  
> messages, and is one reason why you should use a hashed directory  
> scheme for your mail queue (a la postfix), or you should periodically  
> stop the MTA, move the mail queue directory aside, create a new mail  
> queue directory (with appropriate ownership and permissions), then  
> move what messages may remain from the old queue back into the new one  
> (or fire up queue runners to clear the old queue while the new one is  
> being used for new mail).
>

In MM 2.1.3, the relevant code is in  
$prefix/Mailman/Queue/Switchboard.py function files() starting at line  
204 which is called from $prefix/Mailman/Queue/Runner.py line 89 when  
subclassed from $prefix/Mailman/Queue/ArchRunner.py

Rather than just theorize, feel free to make specific suggestions about  
the deficiencies and appropriate remedies based on the code being  
executed. Dare I say it, you could even submit a patch to fix any  
obvious errors in the code.

> 	Mailman could very easily be suffering from the same sort of problem  
> -- once you get a directory with a large number of entries in it, it  
> takes a long time to scan it even if there are only a few files that  
> are currently visible.  Same problem, perhaps the same solution?
>
> -- Brad Knowles, <brad.knowles at skynet.be>
>
> "They that can give up essential liberty to obtain a little temporary
> safety deserve neither liberty nor safety."
>     -Benjamin Franklin, Historical Review of Pennsylvania.
>
> GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---)  
> W+++(--) N+
> !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++)  
> R+(+++)
> tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)*  
> z(+++)
>
> ------------------------------------------------------
> Mailman-Users mailing list
> Mailman-Users at python.org
> http://mail.python.org/mailman/listinfo/mailman-users
> Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
> Searchable Archives:  
> http://www.mail-archive.com/mailman-users%40python.org/
>
> This message was sent to: r.barrett at openinfo.co.uk
> Unsubscribe or change your options at
> http://mail.python.org/mailman/options/mailman-users/ 
> r.barrett%40openinfo.co.uk
>