[Mailman-Users] Memory usage

[Brad Knowles]

On 12/4/07, Grigory Batalov wrote:

>>  See
>>  <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.056.htp>.
>
>   Sorry, not much help.
>   Can you explain me, why qrunners take more and more memory (RES)?

You seem to have fewer lists and fewer numbers of members per list 
than some of the sites I'm familiar with, but your lists may be 
higher in hourly or daily traffic.

>   It was 25Mb maximum in my previous letter, now it is 36Mb:
>
>  $ top
>    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  17660 mailman   15   0  101M  36M 2668 S  0.0  1.8   0:41.33 qrunner
>  32356 mailman   15   0  100M  35M 2668 S  0.0  1.7   0:38.30 qrunner
>  17584 mailman   15   0  100M  35M 2668 S  0.0  1.7   0:40.04 qrunner
>  32739 mailman   18   0 99.7M  34M 2660 S  0.0  1.7   0:33.94 qrunner
>   3182 mailman   15   0 99.5M  34M 2668 S  0.0  1.7   0:39.10 qrunner
>  ....
>
>   Some of them took up to 200Mb (!) before I had to restart them.
>   All this looks like slow and fast memory leak.

That's not so different from what we've got on python.org (see 
<http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.015.htp>), 
and the RSS for our qrunners is between 11MB and 41MB, depending on 
the specific runner.  Note that neither yours nor ours are sucking up 
any CPU time, so they're primed for being paged or swapped out if you 
do run into any memory pressure.  Also note that all that Linux stats 
quoted on 
<http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.056.htp> 
are from the python.org machines.

I'm not a Linux performance tuning expert, but I'm not seeing any 
real problems in what you've shown us so far.  If you are seeing 
problems, then you might want to consult a Linux performance tuning 
expert.

>   My vmstat, if you are interested in:
>
>$ vmstat -a 1 3
>procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
>  r  b   swpd   free  inact active   si   so    bi    bo   in    cs us sy id wa
>  1  0      0 1302408      0      0    0    0     0     0    0     1 
>1  0 94  5
>  0  0      0 1302408      0      0    0    0     0     0    0  1625 
>0  0 99  1
>  0  0      0 1303084      0      0    0    0     0     0    0  1800 
>0  0 88 12
>
>   This is OpenVZ VE on Linux.

You've got over 1GB of memory that is marked as "free".  I'm not 
seeing any memory pressure here.

However, I would wonder why your command isn't showing you how much 
memory is inactive or active.  This would seem to me to be a system 
problem that you probably want to get resolved, although it doesn't 
have anything to do with Mailman.


That said, MTAs and mailing lists really, really want direct access 
to their disk subsystems where they handle all their messages, and 
they are likely to perform much worse in a virtual server environment 
than most other types of applications.

The types of applications that will tend to perform well under 
virtualization are those which are CPU-bound, but are infrequently 
used.  The I/O-bound systems, especially those that are disk 
I/O-bound on very specific issues like synchronous meta-data updates 
(which also involves lots of filesystem overhead, as well as physical 
disk I/O), will tend to perform poorly under virtualization.

-- 
Brad Knowles <brad at shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>