[Mailman-Users] Python process size grows 30x in 8 hours (memory

[Mark Sapiro]

Fletcher Cocquyt wrote:
>
>Here is the current leaked state since the the cron 13:27 restart only 3
>hours ago:
>last pid: 20867;  load averages:  0.53,  0.47,  0.24
>16:04:15
>91 processes:  90 sleeping, 1 on cpu
>CPU states: 99.1% idle,  0.3% user,  0.6% kernel,  0.0% iowait,  0.0% swap
>Memory: 1640M real, 77M free, 1509M swap in use, 1699M swap free
>
>   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
> 24167 mailman    1  59    0  311M  309M sleep    0:28  0.02% python
> 24158 mailman    1  59    0  308M  305M sleep    0:30  0.01% python
> 24169 mailman    1  59    0  303M  301M sleep    0:28  0.01% python
> 24165 mailman    1  59    0   29M   27M sleep    0:09  0.03% python
> 24161 mailman    1  59    0   29M   27M sleep    0:12  0.07% python
> 24164 mailman    1  59    0   28M   26M sleep    0:07  0.01% python
> 24172 mailman    1  59    0   26M   24M sleep    0:04  0.01% python
> 24160 mailman    1  59    0   26M   24M sleep    0:08  0.01% python
> 24162 mailman    1  59    0   26M   23M sleep    0:10  0.01% python
> 24166 mailman    1  59    0   26M   23M sleep    0:04  0.01% python
> 24171 mailman    1  59    0   25M   23M sleep    0:04  0.02% python
> 24163 mailman    1  59    0   24M   22M sleep    0:04  0.01% python
> 24168 mailman    1  59    0   19M   17M sleep    0:03  0.02% python
> 24170 mailman    1  59    0 9516K 6884K sleep    0:01  0.01% python
> 24159 mailman    1  59    0 9500K 6852K sleep    0:00  0.00% python
>
>And the mapping to the runners:
>god at irt-smtp-02:mailman-2.1.11 4:16pm 66 # /usr/ucb/ps auxw | egrep mailman
>| awk '{print $2 " " $11}'
>24167 --runner=IncomingRunner:5:8
>24165 --runner=BounceRunner:0:1
>24158 --runner=IncomingRunner:7:8
>24162 --runner=VirginRunner:0:1
>24163 --runner=IncomingRunner:1:8
>24166 --runner=IncomingRunner:0:8
>24168 --runner=IncomingRunner:4:8
>24169 --runner=IncomingRunner:2:8
>24171 --runner=IncomingRunner:6:8
>24172 --runner=IncomingRunner:3:8
>24160 --runner=CommandRunner:0:1
>24161 --runner=OutgoingRunner:0:1
>24164 --runner=ArchRunner:0:1
>24170 /bin/python
>24159 /bin/python


What are these last 2? Presumably they are the missing NewsRunner and
RetryRunner, but what is the extra stuff in the ps output causing $11
to be the python command and not the runner option? And again, why are
these two, which presumably have done nothing, seemingly the biggest.

Here's some additional thought.

Are you sure there is an actual leak? Do you know that if you just let
them run, they don't reach some stable size and remain there as
opposed to growing so large that they eventually throw a MemoryError
exception and get restarted by mailmanctl.

If you allowed them to do that once, the MemoryError traceback might
provide a clue.

Caveat! I know very little about Python's memory management. Some of
what follows may be wrong.

Here's what I think - Python allocates more memory (from the OS) as
needed to import additional modules and create new objects. Imports
don't go away, but objects that are destroyed or become unreachable
(eg a file object that is closed or a message object whose only
reference gets assigned to something else) become candidates for
garbage collection and ultimately the memory allocated to them is
collected and reused (assuming no leaks). I *think* however, that no
memory is ever actually freed back to the OS. Thus, Python processes
that run for a long time can grow, but don't shrink.

Now, IncomingRunner in particular can get very large if large messages
are arriving, even if those messages are ultimately not processed very
far. Incoming runner reads the entire message into memory and then
parses it into a message object which is even bigger than the message
string. So, if someone happens to send a 100MB attachment to a list,
IncomingRunner is going to need over 200MB before it ever looks at the
message itself. This memory will later become available for other use
within that IncomingRunner instance, but I don't think it is ever
freed back to the OS.

Also, I see very little memory change between the 3 hour old snapshot
above and the 8 hour old one from your prior post. If this is really a
memory leak, I'd expect the 8 hour old ones to be perhaps twice as big
as the 3 hour old ones.

Also, do you have any really big lists with big config.pck files. If
so, Runners will grow as they instantiate that (those) big list(s).

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan