[Mailman-Users] Python process size grows 30x in 8 hours (memory

[Barry Warsaw]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 2, 2008, at 11:15 PM, Mark Sapiro wrote:

> Yes, I know. I was just about to resend. It is attached here. The  
> MUA I
> used to send the previous message gives any attachment without an
> extension Content-Type: application/octet-stream, so the list's  
> content
> filtering removed it.

Ah, np.

> |> It does the following compared to the normal script.
> |
> |> The normal script reads the message from the pipe from the MTA and
> |> queues it in the 'in' queue for processing by an IncomingRunner.  
> This
> |> script receives the message and instead queues it in the 'bad'  
> queue.
> |> It then looks at the size of the 'bad' queue entry (a Python pickle
> |> that will be just slightly larger than the message text). If the  
> size
> |> is less than MAXSIZE bytes (a parameter near the beginning of the
> |> script, currently set to 1000000, but which you can change as you
> |> desire), it moves the queue entry from the 'bad' queue to the 'in'
> |> queue for processing.
> |
> | I'm not sure 'bad' should be used.  Perhaps a separate queue called
> | 'raw'?  It is nice that files > MAXSIZE need only be left in 'bad'.
>
>
> If we're going to do something like this going forward, we can  
> certainly
> change the queue. For this 'debug' effort, I wanted to keep it simple
> and use an existing mm_cfg queue name.

Excellent point.  A couple of very minor comments on the file, but  
other than that, it looks great.  (I know you copied this from the  
original file, but still I can't resist. ;)

> # Copyright (C) 1998,1999,2000,2001,2002 by the Free Software  
> Foundation, Inc.

1998-2008

>
> #
> # This program is free software; you can redistribute it and/or
> # modify it under the terms of the GNU General Public License
> # as published by the Free Software Foundation; either version 2
> # of the License, or (at your option) any later version.
> #
> # This program is distributed in the hope that it will be useful,
> # but WITHOUT ANY WARRANTY; without even the implied warranty of
> # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> # GNU General Public License for more details.
> #
> # You should have received a copy of the GNU General Public License
> # along with this program; if not, write to the Free Software
> # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
> 02110-1301, USA.
>
> """Accept posts to a list and handle them properly.
>
> The main advertised address for a list should be filtered to this  
> program,
> through the mail wrapper.  E.g. for list `test at yourdomain.com', the  
> `test'
> alias would deliver to this script.
>
> Stdin is the mail message, and argv[1] is the name of the target  
> mailing list.
>
> """
>
> import os
> import sys
>
> import paths
> from Mailman import mm_cfg
> from Mailman import Utils
> from Mailman.i18n import _
> from Mailman.Queue.sbcache import get_switchboard
> from Mailman.Logging.Utils import LogStdErr
>
> LogStdErr("error", "post")
>
> MAXSIZE = 1000000
>
>
>
> def main():
>    # TBD: If you've configured your list or aliases so poorly as to  
> get
>    # either of these first two errors, there's little that can be  
> done to
>    # save your messages.  They will be lost.  Minimal testing of new  
> lists
>    # should avoid either of these problems.
>    try:
>        listname = sys.argv[1]
>    except IndexError:
>        print >> sys.stderr, _('post script got no listname.')
>        sys.exit(1)
>    # Make sure the list exists
>    if not Utils.list_exists(listname):
>        print >> sys.stderr, _('post script, list not found: % 
> (listname)s')
>        sys.exit(1)
>    # Immediately queue the message for the incoming qrunner to  
> process.  The
>    # advantage to this approach is that messages should never get  
> lost --
>    # some MTAs have a hard limit to the time a filter prog can run.   
> Postfix
>    # is a good example; if the limit is hit, the proc is SIGKILL'd  
> giving us
>    # no chance to save the message.
>    bdq = get_switchboard(mm_cfg.BADQUEUE_DIR)
>    filebase = bdq.enqueue(sys.stdin.read(),
>                           listname=listname,
>                           tolist=1, _plaintext=1)

Should probably use True there instead of 1.

>    frompath= os.path.join(mm_cfg.BADQUEUE_DIR, filebase + '.pck')
>    topath= os.path.join(mm_cfg.INQUEUE_DIR, filebase + '.pck')

Space in front of the =

>
>
>    if os.stat(frompath).st_size < MAXSIZE:
>        os.rename(frompath,topath)

Space after the comma.

>
> if __name__ == '__main__':
>    main()

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkhsTbQACgkQ2YZpQepbvXGGigCfe+w4Ynz/qvFEp6VmurbySv42
b6cAoJa9wuSaql8dLUo8/VXT/Sxiu9pW
=Ywsy
-----END PGP SIGNATURE-----