[Mailman-Developers] Listing Lists Faster in 2.0?

[Barry A. Warsaw]

>>>>> "RU" == Roberto Ullfig <rullfig@uchicago.edu> writes:

    RU> So, in 1.0rc2, displaying the list of lists for 529 lists
    RU> requires 529**2 = 279841 system stat calls and takes over one
    RU> and a half minutes on our Ultra-2 2x296 processor system! Is
    RU> this because of Python, Mailman, or both? Has this been
    RU> "fixed" in 2.0? You really should only need to make one stat
    RU> call per list.

Uh, it's because of Mailman :)

I implemented a list_lists scripts which does on the command line what
listinfo.py does in HTML (see attached).  Here's what truss -c gives
me:

-------------------- snip snip --------------------
Portal - [no description available]
Postal - [no description available]
 Stage - Staging new Mailman releases
  Test - [no description available]
syscall      seconds   calls  errors
_exit            .00       1
read             .00     102
write            .00       8
open             .11     607    474
close            .01     143
time             .00       3
brk              .03     227
stat             .03     201    157
getpid           .00      10
fstat            .00      66
ioctl            .02      63     61
execve           .00      10      8
umask            .00       2
fcntl            .00       7
readlink         .00       2      2
sigprocmask      .00       2
sigaction        .00      50
sigpending       .00       1
mmap             .00      42
mprotect         .00      10
munmap           .00      11
uname            .00       4
sysconfig        .00       1
lwp_create       .00       6
lwp_continue     .00       2
lwp_self         .00       3
llseek           .00     114
door             .00       5
lwp_schedctl     .01       5
getdents64       .01      15
fstat64          .00      67
open64           .00       7
                ----     ---    ---
sys totals:      .22    1797    702
usr time:        .51
elapsed:        1.19
-------------------- snip snip --------------------

Getting the list of list names, requires at least a listdir() and an
exists() for every directory found there.

Nothing about this will change for 2.0.

-Barry

-------------------- snip snip --------------------
#! /usr/bin/env python
#
# Copyright (C) 1998,1999,2000 by the Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software 
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

"""List all mailing lists.

Usage: %(program)s [options]

Where:

    --advertised
    -a
        List only those mailing lists that are publically advertised

    --virtual-host-overview=domain
    -V domain
        List only those mailing lists that are homed to the given virtual
        domain.  This only works if the VIRTUAL_HOST_OVERVIEW variable is
        set.

    --help
    -h
        Print this text and exit.

"""

import sys
import getopt
import paths

from Mailman import mm_cfg
from Mailman import MailList
from Mailman import Utils
from Mailman import Errors

program = sys.argv[0]

def usage(status, msg=''):
    print __doc__ % globals()
    if msg:
        print msg
    sys.exit(status)



def main():
    try:
        opts, args = getopt.getopt(sys.argv[1:], 'aV:h',
                                   ['advertised', 'virtual-host-overview=',
                                    'help'])
    except getopt.error, msg:
        usage(1, msg)

    advertised = 0
    vhost = None

    for opt, arg in opts:
        if opt in ('-h', '--help'):
            usage(0)
        elif opt in ('-a', '--advertised'):
            advertised = 1
        elif opt in ('-V', '--virtual-host-overview'):
            vhost = arg

    names = Utils.list_names()
    names.sort()

    mlists = []
    longest = 0
    for n in names:
        mlist = MailList.MailList(n, lock=0)
        if advertised and not mlist.advertised:
            continue
        if vhost and mm_cfg.VIRTUAL_HOST_OVERVIEW and \
               string.find(vhost, mlist.web_page_url) == -1 and \
               string.find(mlist.web_page_url, vhost) == -1:
            continue
        mlists.append(mlist)
        longest = max(len(mlist.real_name), longest)

    if not mlists:
        print 'No matching mailing lists found'
        return

    format = '%%%ds - %%.%ds' % (longest, 77 - longest)
    for mlist in mlists:
        description = mlist.description or '[no description available]'
        print format % (mlist.real_name, description)
            

if __name__ == '__main__':
    main()