[Mailman-Developers] a patch to scale Cgi/admin.py

[Fil]

I'm knee-deep in Mailman/Gui/admin.py and it really doesn't scale. 

I use a test-list of 300k addresses, and it's a bit more than 5 minutes to
get it to answer (if the connection holds that long, of course).

It's particularly true when using the MySQLMemberAdaptor, where many things
are not taken from memory but are reprocessed with MySQL queries. 
For instance, the part that checks if the member is regular/digest fetches
fetches all the data for each subscriber (more plainly said, it's in N^2).

Another bottleneck is the list of chunks that is computed and displayed (and
sent to the client) - his list is quite long to compute, and as a user it's
not that useful in general.

Last but not least, the search facility calls the mysql-db for each member,
in order to extract her name and regexp it; and that's very long. Is wasn't
able to find how to speed this up, and just disabled it in my system (but
not in the patch provided below)

So here are a few small changes, that make a radical improvement (down to 45
seconds from 4 minutes):

--- /home/fil/src_mailman/mailman/Mailman/Cgi/admin.py  2005-02-12 21:22:55.000000000 +0100
+++ Mailman/Cgi/admin.py        2005-10-29 16:43:56.116988176 +0200
@@ -876,6 +876,7 @@ def membership_options(mlist, subcat, cg
             doc.addError(_('Bad regular expression: ') + regexp)
         else:
             # BAW: There's got to be a more efficient way of doing this!
+            # yes please... this doesn't scale at all
             names = [mlist.getMemberName(s) or '' for s in all]
             all = [a for n, a in zip(names, all)
                    if cre.search(n) or cre.search(a)]
@@ -978,6 +979,8 @@ def membership_options(mlist, subcat, cg
                   MemberAdaptor.BYADMIN : _('A'),
                   MemberAdaptor.BYBOUNCE: _('B'),
                   }
+    # memorize the regular-or-digest list
+    regular_or_digest = mlist.getRegularMemberKeys()
     # Now populate the rows
     for addr in members:
         link = Link(mlist.GetOptionsURL(addr, obscure=1),
@@ -1021,8 +1024,8 @@ def membership_options(mlist, subcat, cg
         # This code is less efficient than the original which did a has_key on
         # the underlying dictionary attribute.  This version is slower and
         # less memory efficient.  It points to a new MemberAdaptor interface
-        # method.
-        if addr in mlist.getRegularMemberKeys():
+        # method. (Modified by Fil to "cache" the result - useful for MySQLMemberAdaptor)
+        if addr in regular_or_digest:
             cells.append(Center(CheckBox(addr + '_digest', 'off', 0).Format()))
         else:
             cells.append(Center(CheckBox(addr + '_digest', 'on', 1).Format()))
@@ -1113,7 +1116,7 @@ def membership_options(mlist, subcat, cg
         range listed below:</em>''')
         chunkmembers = buckets[bucket]
         last = len(chunkmembers)
-        for i in range(numchunks):
+        for i in range(min(10,numchunks)):
             if i == chunkindex:
                 continue
             start = chunkmembers[i*chunksz]


-- Fil