Apache/2.0.40 Server at www.stat.math.ethz.ch Port 443

tags). Ideally our page would be wrapped by the browser, yet still
preserve paragraph boundaries and indenting. I've failed to find an HTML way
to do this. My current plan is to modify the archiving mechanism to word
wrap to a fixed 80 chars, perhaps using the Unix command-line utility
'fold'.

Many thanks,
Chris


From chris at jellybaby.net  Mon Jan 19 12:43:04 2004
From: chris at jellybaby.net (Chris Boulter)
Date: Mon Jan 19 12:43:09 2004
Subject: [Mailman-Developers] List locks not getting relinquished
Message-ID: <20040119174304.GH51157@jellybaby.net>

Hi,

I'm having some problems with lists not getting unlocked. I don't really
know how to set about debugging this and I haven't found anything in this
list's archive or the documentation to help. My Mailman installation is
quite non-standard, so it may be hard to provide help, but I'd appreciate
hints about how to debug the locking mechanism (for instance, can I tell
which process created a lock?).

I'm using Mailman 2.1.2. We have sync_members running in a loop from a
daemon, syncing ~500 lists every few hours. Of these, half a dozen lists are
failing (and my daemon has to forcibly kill the sync_members process when it
detects a timeout). There seems to be no common feature to the lists which
fail. When it happens, I can forcibly break the locks and restore correct
behaviour by manually deleting Mailman/locks/*, but then the problem recurs
eventually. However, once I've broken the locks, I can successfully run
sync_members on the lists from the command line hundreds or thousands of
times without failure, either making changes to the subscribers or not each
time.

I tried setting
        LIST_LOCK_DEBUGGING = On
but didn't find the result enlightening (I can post some here if anyone
thinks this is a good place to look).

So maybe it's something about the daemon which doesn't relinquish a lock.
What the daemon does is mostly to run sync_members but occasionally to run
newlist or a Python script of my devising (to change a user's password on a
list, or change the list owner). Possibly my own scripts are bad: I don't
know much about Python, but I've checked them as carefully as I can. I've
posted them here in case any kind individual could cast an eye over them
(mostly kludged together from distributed Mailman code):
        http://chris.jellybaby.net/mailman/

The command being run by the daemon, and the stacktrace when I ran the
command manually and hit ^C are below (nb I've modified sync_members and
added an '--ignore-invalid' option, so the line numbers below aren't correct
for the standard distro).


/usr/local/mailman/bin/sync_members --ignore-invalid --welcome-msg=no
--goodbye-msg=no --notifyadmin=no --file /var/tmp/mlmsync124897.tmp
latin.america
^CTraceback (most recent call last):
  File "/usr/local/mailman/bin/sync_members", line 301, in ?
    main()
  File "/usr/local/mailman/bin/sync_members", line 234, in main
    mlist = MailList.MailList(listname)
  File "/usr/local/mailman/Mailman/MailList.py", line 122, in __init__
    self.Lock()
  File "/usr/local/mailman/Mailman/MailList.py", line 155, in Lock
    self.__lock.lock(timeout)
  File "/usr/local/mailman/Mailman/LockFile.py", line 312, in lock
    self.__sleep()
  File "/usr/local/mailman/Mailman/LockFile.py", line 496, in __sleep
    time.sleep(interval)
KeyboardInterrupt

Many thanks,
Chris

From my at freexp.de  Mon Jan 19 14:54:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Mon Jan 19 15:03:44 2004
Subject: [Mailman-Developers] Achieving better formatting of archives
In-Reply-To: <20040119170244.GG51157@jellybaby.net>
References: <20040119170244.GG51157@jellybaby.net>
Message-ID: <919gaRA$pwB@my.freexp.de>

Chris Boulter  wrote on 19.01.04:

> 2. HTML emails are not dealt with nicely in the archive. [...]

> 3. Long lines in emails. No word wrapping is done on plain text
> emails; sensible enough in a way. However, we think most of our users
> will send emails with long lines. [...]

Both HTML emails and arbitrary long lines are *bad*.  Educate your
users. :-)

Just my 2c.


        Michael

From somuchfun at atlantismail.com  Mon Jan 19 15:42:09 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Mon Jan 19 15:46:06 2004
Subject: [Mailman-Developers] Question about mailman performance
Message-ID: <20423201512973@smtp.westbay.net>

Whenever I have a mailing list send out 30,000 emails python takes up about
95% of cpu time and it is impossible to get into the mailman admin
interface.
The apache server responds just fine to other requests.
Is this normal? I hope not, ;-)
Any advice?


From PieterB at gewis.nl  Tue Jan 20 04:31:37 2004
From: PieterB at gewis.nl (PieterB)
Date: Tue Jan 20 04:31:42 2004
Subject: [Mailman-Developers] explaining bounce_score_threshold
Message-ID: <20040120093137.GA33020@gewis.win.tue.nl>

I'm having some trouble with a bouncing e-mailadress (still
investigating). I didn't really understand the bounce_score_threshold
help text. Mailman 2.1.3 help text for the item is:

bounce_score_threshold (bounce): The maximum member bounce score before
the member's subscription is disabled. This value can be a floating
point number.

I think the explanation on 
http://www.washington.edu/computing/mailman/owners/maintenance.html
is much clearer. Can the text be changed to something like:

bounce_score_threshold (bounce): The maximum member bounce score
before the member's subscription is disabled. Each subscriber is
assigned a bounce score (floating point number) and every time
Mailman encounters a bounce from this member, the score goes up.
Hard bounces (meaning a fatal error) increase the score by 1, while
soft bounces (meaning a temporary error occurred) increase the score
by 0.5. Note that the bounce scores are only increased once per day
so even if ten hard bounces are received for the same subscriber
in one day, the subscriber's score will increase by only 1 for that
day.

PieterB

--
http://zwiki.org



From barry at python.org  Tue Jan 20 08:03:10 2004
From: barry at python.org (Barry Warsaw)
Date: Tue Jan 20 08:03:19 2004
Subject: [Mailman-Developers] [Fwd: Linux User & Developer Expo 2004
	(Correction)]
Message-ID: <1074603789.14688.16.camel@anthem>

Are any of our UK or Euro Mailman folks interested in representing the
project at this expo?  I won't be able to make it.

-Barry

-------------- next part --------------
An embedded message was scrubbed...
From: Brian Teeman 
Subject: Linux User & Developer Expo 2004 (Correction)
Date: Tue, 20 Jan 2004 10:21:09 +0000 (GMT)
Size: 2961
Url: http://mail.python.org/pipermail/mailman-developers/attachments/20040120/ac4b6d9e/attachment.mht
From brad.knowles at skynet.be  Tue Jan 20 10:15:29 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Tue Jan 20 10:16:08 2004
Subject: [Mailman-Developers] [Fwd: Linux User & Developer
	Expo 2004 	(Correction)]
In-Reply-To: <1074603789.14688.16.camel@anthem>
References: <1074603789.14688.16.camel@anthem>
Message-ID: 

At 8:03 AM -0500 2004/01/20, Barry Warsaw wrote:

>  Are any of our UK or Euro Mailman folks interested in representing the
>  project at this expo?  I won't be able to make it.

	Keep in mind that there is also FOSDEM on 20-22 Feb (see 
), the USENIX Annual Technical Conference on 
Jun 27-Jul 2 (), SANE 2004 on 
Sep 27 - Oct 1 (), and LISA 
2004 Nov 14-19 ().

	See also , among others.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From my at freexp.de  Tue Jan 20 11:14:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Tue Jan 20 12:08:28 2004
Subject: [Mailman-Developers] Mixed Languages in response messages
In-Reply-To: <90HFbCDppwB@my.freexp.de>
References: <90HFbCDppwB@my.freexp.de>
Message-ID: <91DhVzrppwB@my.freexp.de>

Michael Heydekamp  wrote on 05.01.04:

> the response to the 'help' request of a subscriber delivers a German
> output except for the command 'confirm ' which
> comes in English.

> The mailman.po file looks OK to me and compiles fine, so what can be
> the reason?

As there was no response so far I'm wondering if this an extremely
difficult question and/or if there's any better place to ask?


        Michael

From andyk at spunge.org  Tue Jan 20 13:14:11 2004
From: andyk at spunge.org (Andrzej Kasperowicz)
Date: Tue Jan 20 13:14:12 2004
Subject: [Mailman-Developers] Mixed Languages in response messages
In-Reply-To: <91DhVzrppwB@my.freexp.de>
References: <90HFbCDppwB@my.freexp.de>
Message-ID: <400D7E03.9042.13F9A64@localhost>

> > The mailman.po file looks OK to me and compiles fine, so what can be
> > the reason?
> 
> As there was no response so far I'm wondering if this an extremely
> difficult question and/or if there's any better place to ask?

I also noticed mixed languages response in various situations regarding 
Polish language. 
Also in the list-owner administrative interface when set to Polish some 
text is in Polish, and some in English, and what's more interesting, 
sometimes it can be in English, and the next time it can be in Polish. 
Maybe it depends on the phase of the Moon or strength of the wind or 
something... ;)

Cheers!
ak

PS. Who did the translation to Polish? As there are some style errors. 
Like sentence shouldn't start with "Aby" because "aby" is conjunctive.
I found also one mistyping.



From somuchfun at atlantismail.com  Tue Jan 20 15:30:07 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Tue Jan 20 15:35:05 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
Message-ID: <20302643201593@smtp.westbay.net>

I have a problem that might affect lots of people who send emails to large
lists.
AOL requires an additional header in the emails identifying the recipient
for the feedback loops to unsubscribe AOL users who complain.
So how do I add an additional header with a field?

Thanks for all the help!


From pioppo at ferrara.linux.it  Tue Jan 20 16:04:06 2004
From: pioppo at ferrara.linux.it (Simone Piunno)
Date: Tue Jan 20 16:30:05 2004
Subject: [Mailman-Developers] Mixed Languages in response messages
In-Reply-To: <400D7E03.9042.13F9A64@localhost>
References: <90HFbCDppwB@my.freexp.de> <400D7E03.9042.13F9A64@localhost>
Message-ID: <200401202204.06918.pioppo@ferrara.linux.it>

On Tuesday 20 January 2004 19:14, Andrzej Kasperowicz wrote:

> I also noticed mixed languages response in various situations regarding
> Polish language.
> Also in the list-owner administrative interface when set to Polish some
> text is in Polish, and some in English, and what's more interesting,
> sometimes it can be in English, and the next time it can be in Polish.
> Maybe it depends on the phase of the Moon or strength of the wind or
> something... ;)

Maybe the polish translation is still incomplete or a bit outdated.

> PS. Who did the translation to Polish? As there are some style errors.
> Like sentence shouldn't start with "Aby" because "aby" is conjunctive.
> I found also one mistyping.

You should better ask on the mailman-i18n@python.org list, there's a good 
chance the polish translator will read your message.

Cheers,
  Simone


From pioppo at ferrara.linux.it  Tue Jan 20 16:01:53 2004
From: pioppo at ferrara.linux.it (Simone Piunno)
Date: Tue Jan 20 16:32:33 2004
Subject: [Mailman-Developers] Mixed Languages in response messages
In-Reply-To: <91DhVzrppwB@my.freexp.de>
References: <90HFbCDppwB@my.freexp.de> <91DhVzrppwB@my.freexp.de>
Message-ID: <200401202201.53482.pioppo@ferrara.linux.it>


> > the response to the 'help' request of a subscriber delivers a German
> > output except for the command 'confirm ' which
> > comes in English.

"confirm" is not translatable, because it's really a command for the interface 
minilanguage.  E.g. you don't transalate "print" when you're writing a 
programming manual.

Of course "" can be translated, and in fact in italian it 
is, but maybe the german translator didn't think this was a good idea.

> if there's any better place to ask?

There's a list devoted to Mailman translations:

   mailman-i18n@python.org

Cheers,
  Simone


From barry at python.org  Tue Jan 20 17:28:15 2004
From: barry at python.org (Barry Warsaw)
Date: Tue Jan 20 17:28:33 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: <20302643201593@smtp.westbay.net>
References: <20302643201593@smtp.westbay.net>
Message-ID: <1074637694.26872.17.camel@anthem>

On Tue, 2004-01-20 at 15:30, Somuchfun wrote:
> I have a problem that might affect lots of people who send emails to large
> lists.
> AOL requires an additional header in the emails identifying the recipient
> for the feedback loops to unsubscribe AOL users who complain.
> So how do I add an additional header with a field?

Can you describe this additional header in more detail?  Better yet
would be some documentation on the web describing this header.  I.e. if
it's a standard (or maybe even a proposed standard), perhaps Mailman
should support it out of the box.

It's actually pretty easy to add such a header.  One option is to just
include the header in the original message.  Or you can get Mailman to
add it to every message by modifying the code in the pipeline.  The
place to start would be Mailman/Handlers/CookHeaders.py.  That's where
the RFC 2369 headers are added for example.

-Barry



From my at freexp.de  Tue Jan 20 17:39:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Tue Jan 20 17:42:21 2004
Subject: [Mailman-Developers] Redirect list to list?
Message-ID: <91DiOxyppwB@my.freexp.de>

Hi,

we want all messages sent by moderated members or non-members to our CVS
list be automatically redirected to our Dev list (rather than really
moderating or discarding them).

Only a specific address (in fact the list address itself) and the non-
moderated members shall be allowed to post to the list without being
redirected.

Is there any patch or other solution which does that job?


        Michael

From my at freexp.de  Tue Jan 20 17:41:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Tue Jan 20 17:42:24 2004
Subject: [Mailman-Developers] Mixed Languages in response messages
In-Reply-To: <200401202201.53482.pioppo@ferrara.linux.it>
References: <90HFbCDppwB@my.freexp.de> <91DhVzrppwB@my.freexp.de>
	<200401202201.53482.pioppo@ferrara.linux.it>
Message-ID: <91DiPUCKpwB@my.freexp.de>

Simone Piunno  wrote on 20.01.04:

>>> the response to the 'help' request of a subscriber delivers a
>>> German output except for the command 'confirm
>>> ' which comes in English.

> "confirm" is not translatable, because it's really a command for the
> interface minilanguage.  E.g. you don't transalate "print" when
> you're writing a programming manual.

I think you got me wrong. ;)  The problem is that there *is* a
translation in mailman.po already but it's not used due to some reason.


This is what I get:

----------8<----------
>   confirm 
>       Confirm an action.  The confirmation-string is required and should be
>       supplied by a mailback confirmation notice.
        English paragraph -> ^^^^^^^^^^^^^^^^^^^^^
>   end
>       Beendet die Liste der Kommandos. Sinnvoll, falls Ihr
>       Mailprogramm eine Signatur an den Nachrichtentext anh?ngt.
        German paragraph -> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
----------8<----------


So a mix of English (for the "confirm" command) and German (for the
"end" command).  And this is what I should get (because it's in
mailman.po, and yes, I re-compiled mailman.po after I made some
corrections, and the behaviour is the same also with the original file):

----------8<----------
>   confirm 
>       Best?tigt einen Befehl. Der Best?tigungscode ist zwingend
>       erforderlich und mu? in der R?ckbest?tigung ?bermittelt werden.
        German paragraph -> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   end
>       Beendet die Liste der Kommandos. Sinnvoll, falls Ihr
>       Mailprogramm eine Signatur an den Nachrichtentext anh?ngt.
        German paragraph -> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
----------8<----------


See?

>> if there's any better place to ask?

> There's a list devoted to Mailman translations:

>    mailman-i18n@python.org

Well, I know that this list exists but having had a quick look into it
it didn't make the impression to me as if this would be the right place
for this particular question.  It's not a translation issue (as I can
translate the stuff myself), it does more look like a code and thus a
developer issue to me.

I'm not too familiar with the format and the logic of mailman.po
(although I made some successful changes and corrections), but it might
have to do with the 'fuzzy' line here:

----------8<----------
> #: Mailman/Commands/cmd_confirm.py:17
> #, fuzzy
     ^^^^^
> msgid ""
----------8<----------

I'm not sure what that does and when I remove it and re-compile the
file, it doesn't change anything.

Also do I find the English text in fact in line 17 of cmd_confirm.py,
but I dont't know what that means either.


        Michael

From pioppo at ferrara.linux.it  Tue Jan 20 18:06:05 2004
From: pioppo at ferrara.linux.it (Simone Piunno)
Date: Tue Jan 20 17:59:44 2004
Subject: [Mailman-Developers] Mixed Languages in response messages
In-Reply-To: <91DiPUCKpwB@my.freexp.de>
References: <90HFbCDppwB@my.freexp.de>
	<200401202201.53482.pioppo@ferrara.linux.it>
	<91DiPUCKpwB@my.freexp.de>
Message-ID: <200401210006.05977.pioppo@ferrara.linux.it>

On Tuesday 20 January 2004 23:41, Michael Heydekamp wrote:

> Well, I know that this list exists but having had a quick look into it
> it didn't make the impression to me as if this would be the right place
> for this particular question.  It's not a translation issue (as I can
> translate the stuff myself), it does more look like a code and thus a
> developer issue to me.
> I'm not too familiar with the format and the logic of mailman.po
> (although I made some successful changes and corrections), but it might
> have to do with the 'fuzzy' line here:
>
> ----------8<----------
>
> > #: Mailman/Commands/cmd_confirm.py:17
> > #, fuzzy

that "fuzzy" tag means this translation is *disabled* (the software wil 
pretend it's not there).
The tag is automatically inserted by the msgmerge tool whenever a translation 
is associated to a new original string using a best effort guess.
So, until the translator edits the file, confirming the fuzzy association is 
good (e.g removing the fuzzy tag) and/or adding improvements, this 
translation won't be used.
And yes, this *is* a question for the -i18n list, because the internal format 
of a .po file is more familiar to translators than to coders.
BTW, try to add a -v when you compile the .po file, e.g.

 msgfmt -v -o mailman.mo mailman.po

You should see something like this:

 1196 translated messages, 18 fuzzy translations, 65 untranslated messages.

So you can easily know the status of the translation (the line above is 
related to the most recent copy of the german po file I have on my machine).

Cheers,
  Simone


From barry at python.org  Tue Jan 20 18:22:38 2004
From: barry at python.org (Barry Warsaw)
Date: Tue Jan 20 18:22:50 2004
Subject: [Mailman-Developers] explaining bounce_score_threshold
In-Reply-To: <20040120093137.GA33020@gewis.win.tue.nl>
References: <20040120093137.GA33020@gewis.win.tue.nl>
Message-ID: <1074640958.26872.30.camel@anthem>

On Tue, 2004-01-20 at 04:31, PieterB wrote:
> I'm having some trouble with a bouncing e-mailadress (still
> investigating). I didn't really understand the bounce_score_threshold
> help text. Mailman 2.1.3 help text for the item is:
> 
> bounce_score_threshold (bounce): The maximum member bounce score before
> the member's subscription is disabled. This value can be a floating
> point number.
> 
> I think the explanation on 
> http://www.washington.edu/computing/mailman/owners/maintenance.html
> is much clearer. Can the text be changed to something like:
> 
> bounce_score_threshold (bounce): The maximum member bounce score
> before the member's subscription is disabled. Each subscriber is
> assigned a bounce score (floating point number) and every time
> Mailman encounters a bounce from this member, the score goes up.
> Hard bounces (meaning a fatal error) increase the score by 1, while
> soft bounces (meaning a temporary error occurred) increase the score
> by 0.5. Note that the bounce scores are only increased once per day
> so even if ten hard bounces are received for the same subscriber
> in one day, the subscriber's score will increase by only 1 for that
> day.

I probably won't use that verbatim, but I will try to flesh out the
details for bounce_score_threshold.  I've also asked the UW folks if
they mind me linking to their site.  While sometimes specific to the UW
installation, that site has a lot of good information.

-Barry



From barry at python.org  Tue Jan 20 19:02:54 2004
From: barry at python.org (Barry Warsaw)
Date: Tue Jan 20 19:03:14 2004
Subject: [Mailman-Developers] List locks not getting relinquished
In-Reply-To: <20040119174304.GH51157@jellybaby.net>
References: <20040119174304.GH51157@jellybaby.net>
Message-ID: <1074643374.26872.58.camel@anthem>

On Mon, 2004-01-19 at 12:43, Chris Boulter wrote:

> I'm having some problems with lists not getting unlocked. I don't really
> know how to set about debugging this and I haven't found anything in this
> list's archive or the documentation to help. My Mailman installation is
> quite non-standard, so it may be hard to provide help, but I'd appreciate
> hints about how to debug the locking mechanism (for instance, can I tell
> which process created a lock?).

I was going to suggest LIST_LOCK_DEBUGGING, but that only turns on
logging for the lock that the MailList object creates.  That /ought/ to
be sufficient to get gobs of output, but you can also try hacking
LockFile.py to set the default withlogging argument on the constructor
to True.  That way, you'll see all the output from all lock operations. 
Probably won't help much though. ;)

> I'm using Mailman 2.1.2. We have sync_members running in a loop from a
> daemon, syncing ~500 lists every few hours. Of these, half a dozen lists are
> failing (and my daemon has to forcibly kill the sync_members process when it
> detects a timeout). 

I want to know what "failing" means.  Do you get tracebacks?  Does the
process just hang?  Does it work but none of the changes persist?

Now, if the process is hanging, and you're running them on a live site,
with messages being processed, cron jobs running, people hitting the web
site, well, that's not unexpected!  The locks are there specifically so
that only one process can modify a list's configuration at a time. 
Mailman tries to be careful to only acquire a list lock when it needs
write access to the mailing list.  Say your sync_members script tries to
lock a list at the same time someone is sending a message through the
list.  It's entirely possible the incoming qrunner process will get the
lock, causing sync_members to block until the lock is available.

Lock acquisition order is completely non-deterministic too.  If multiple
processes lays claim to the same list lock, you've no way of determining
in which order or when each process will acquire it.  The default
setting is for no timeouts on the lock acquisition.

> There seems to be no common feature to the lists which
> fail. When it happens, I can forcibly break the locks and restore correct
> behaviour by manually deleting Mailman/locks/*, but then the problem recurs
> eventually. However, once I've broken the locks, I can successfully run
> sync_members on the lists from the command line hundreds or thousands of
> times without failure, either making changes to the subscribers or not each
> time.

This jives with the above scenario.

> So maybe it's something about the daemon which doesn't relinquish a lock.

Or maybe it just takes a long time to release the lock, or for the
sync_members process to acquire it.  Turning on debugging should give
you output as to which process is laying claim to the lock, which
process gets the lock, and when the process releases it.  You can always
find the process number by looking at the lock files.  The one with the
lock will have a hard link to the generic lock file.  The pid will also
be in the lock file.

> The command being run by the daemon, and the stacktrace when I ran the
> command manually and hit ^C are below (nb I've modified sync_members and
> added an '--ignore-invalid' option, so the line numbers below aren't correct
> for the standard distro).
> 
> 
> /usr/local/mailman/bin/sync_members --ignore-invalid --welcome-msg=no
> --goodbye-msg=no --notifyadmin=no --file /var/tmp/mlmsync124897.tmp
> latin.america
> ^CTraceback (most recent call last):
>   File "/usr/local/mailman/bin/sync_members", line 301, in ?
>     main()
>   File "/usr/local/mailman/bin/sync_members", line 234, in main
>     mlist = MailList.MailList(listname)
>   File "/usr/local/mailman/Mailman/MailList.py", line 122, in __init__
>     self.Lock()
>   File "/usr/local/mailman/Mailman/MailList.py", line 155, in Lock
>     self.__lock.lock(timeout)
>   File "/usr/local/mailman/Mailman/LockFile.py", line 312, in lock
>     self.__sleep()
>   File "/usr/local/mailman/Mailman/LockFile.py", line 496, in __sleep
>     time.sleep(interval)
> KeyboardInterrupt

You've interrupted the process while it's sleeping trying to acquire the
lock.  This has to mean either another live process has the lock
(perhaps it's taking a long time to do something), or there's a stale
lock around.  Mailman is very careful to release the lock when it's no
longer necessary, but if something happens like a process gets kill -9'd
or Python core dumps, or your machine crashes, then a stale lock can
result.  It's easy to find out if the lock is stale by ps'ing the pid
that last acquired it.

HTH,
-Barry



From jeffw at chebucto.ns.ca  Tue Jan 20 23:40:58 2004
From: jeffw at chebucto.ns.ca (Jeff Warnica)
Date: Tue Jan 20 23:38:44 2004
Subject: [Mailman-Developers] Redirect list to list?
In-Reply-To: <91DiOxyppwB@my.freexp.de>
References: <91DiOxyppwB@my.freexp.de>
Message-ID: <20040121004058.nceo8csowkggoc8w@george.warnica>


I dont know of any, and this sugestion is off the cuff. (but it would be fun to
try :P)... Do it at the MTA level, ie: (just invented pseudo alias-code)

dontsendstuffhere@freexp.de: |$MAILMAN cvs
cvs@freexp.de:               |$MAILMAN dev
dev@freexp.de:               |$MAILMAN dev

The cvs list may require tweeking to allow messages in that dont seem to be for
it. There is some kind of AKA-ish option, details of which I dont recall.

Quoting Michael Heydekamp :

> Hi,
>
> we want all messages sent by moderated members or non-members to our CVS
> list be automatically redirected to our Dev list (rather than really
> moderating or discarding them).
>
> Only a specific address (in fact the list address itself) and the non-
> moderated members shall be allowed to post to the list without being
> redirected.
>
> Is there any patch or other solution which does that job?
>

From juanen at metropoli2000.com  Wed Jan 21 03:01:47 2004
From: juanen at metropoli2000.com (Juan Enrique =?ISO-8859-1?Q?G=F3mez?=)
Date: Wed Jan 21 03:02:23 2004
Subject: [Mailman-Developers] Question about mailman performance
In-Reply-To: <20423201512973@smtp.westbay.net>
References: <20423201512973@smtp.westbay.net>
Message-ID: <1074672107.9305.59.camel@amspoke>

El lun, 19-01-2004 a las 21:42, Somuchfun escribi?:

Hi!

I'm afraid it's normal, my smallest list is about 20.000 and it's almost
impossible to handle, and i dont wanna talk about others with more than
200.000 users....

Best,

> Whenever I have a mailing list send out 30,000 emails python takes up about
> 95% of cpu time and it is impossible to get into the mailman admin
> interface.
> The apache server responds just fine to other requests.
> Is this normal? I hope not, ;-)
> Any advice?
> 
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers@python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
-- 
Saludos,
--------------------------------------------------------------------------
|Juan Enrique Gomez Perez
|Metropoli2000 Networks, S.L.
| Phone: +34 914250023 Fax: +34 914250136
| Mobile: +34 615 922 820
| email: juan.enrique.gomez@metropoli2000.com
--------------------------------------------------------------------------
PGP Fingerprint: 6B39 3A2B A17B 1E8E CFFD  FC14 678E 0A22 BD80 C486
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBD80C486

If this message hasn't a correct signature please notify 
it to juanen@metropoli2000.com. To get the public key please
use the above url. Thanks for your help.
--------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Esta parte del mensaje =?ISO-8859-1?Q?est=E1?= firmada
	digitalmente
Url : http://mail.python.org/pipermail/mailman-developers/attachments/20040121/1be661e7/attachment.bin
From juanen at metropoli2000.com  Wed Jan 21 03:57:35 2004
From: juanen at metropoli2000.com (Juan Enrique =?ISO-8859-1?Q?G=F3mez?=)
Date: Wed Jan 21 03:58:11 2004
Subject: [Mailman-Developers] Shunt
Message-ID: <1074675454.9300.61.camel@amspoke>

Hi!

could any one explain me what's intended for the shunt directory in
mailman?.. just curiosity...

Thanks in advance,
-- 
Saludos,
--------------------------------------------------------------------------
|Juan Enrique Gomez Perez
|Metropoli2000 Networks, S.L.
| Phone: +34 914250023 Fax: +34 914250136
| Mobile: +34 615 922 820
| email: juan.enrique.gomez@metropoli2000.com
--------------------------------------------------------------------------
PGP Fingerprint: 6B39 3A2B A17B 1E8E CFFD  FC14 678E 0A22 BD80 C486
Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBD80C486

If this message hasn't a correct signature please notify 
it to juanen@metropoli2000.com. To get the public key please
use the above url. Thanks for your help.
--------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Esta parte del mensaje =?ISO-8859-1?Q?est=E1?= firmada
	digitalmente
Url : http://mail.python.org/pipermail/mailman-developers/attachments/20040121/944c824d/attachment.bin
From my at freexp.de  Tue Jan 20 18:59:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Wed Jan 21 04:09:18 2004
Subject: [Mailman-Developers] Mixed Languages in response messages
In-Reply-To: <200401210006.05977.pioppo@ferrara.linux.it>
References: <90HFbCDppwB@my.freexp.de>
	<200401202201.53482.pioppo@ferrara.linux.it>
	<91DiPUCKpwB@my.freexp.de>
	<200401210006.05977.pioppo@ferrara.linux.it>
Message-ID: <91HicwPppwB@my.freexp.de>

Simone Piunno  wrote on 21.01.04:

> On Tuesday 20 January 2004 23:41, Michael Heydekamp wrote:

>> I'm not too familiar with the format and the logic of mailman.po
>> (although I made some successful changes and corrections), but it
>> might have to do with the 'fuzzy' line here:

[...]

> that "fuzzy" tag means this translation is *disabled* (the software
> wil pretend it's not there).
> The tag is automatically inserted by the msgmerge tool whenever a
> translation is associated to a new original string using a best
> effort guess. So, until the translator edits the file, confirming the
> fuzzy association is good (e.g removing the fuzzy tag) and/or adding
> improvements, this translation won't be used.

Ah, thanks for the explanation.

But as I said: Even if I remove the tag and recompile the file, the
translation is still not used.  What's wrong?

> And yes, this *is* a question for the -i18n list, because the
> internal format of a .po file is more familiar to translators than to
> coders.

Ok.  Probably we can clarify this particular question here, but I will
have more ones which I will then raise in -i18n.  OK?  :)

> BTW, try to add a -v when you compile the .po file, e.g.

Thx for the hint, I had just used -o so far.


        Michael

From maechler at stat.math.ethz.ch  Wed Jan 21 08:13:09 2004
From: maechler at stat.math.ethz.ch (Martin Maechler)
Date: Wed Jan 21 08:13:22 2004
Subject: [Mailman-Developers] Bug in Mailman 2.1.4 {Subscription problem}
In-Reply-To: 
References: 
Message-ID: <16398.31461.201314.57938@gargle.gargle.HOWL>

Dear Mailman developers,

A user ("ADBONT@...", cc'ed) who wanted to subscribe to one of
our mailing lists received a 

  >> Bug in Mailman version 2.1.4
  >> 
  >> We're sorry, we hit a bug!

and sent me the full message, I've found the same message in
mailman's  data/logs/error file, 
and append it below.

Note that this is quite an active mailing list, 
---> https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor/

and many people do subscribe and unsubscribe successfully both
through the web and the email interface.

Though -- I've got reports more than once that subscription
through the web interface produced a "you are subscribed ...."
confirmation (web) and welcome email  and the person was *not*
subscribed after all.
That is really another problem, but maybe related...

Note that our setup is a bit non-standard maybe:
Web  server ("hubble")  is Redhat9
Mail server ("hypatia") is Solaris 8 -- mailman installed on both

Many thanks for mailman,
and for your consideration of this report.

Martin Maechler 	http://stat.ethz.ch/~maechler/
Seminar fuer Statistik, ETH-Zentrum  LEO C16	Leonhardstr. 27
ETH (Federal Inst. Technology)	8092 Zurich	SWITZERLAND
phone: x-41-1-632-3408		fax: ...-1228			<><


Here the excerpt from /data/logs/error :

Jan 21 11:03:34 2004 admin(31707): @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 
admin(31707): [----- Mailman Version: 2.1.4 -----] 
admin(31707): [----- Traceback ------] 
admin(31707): Traceback (most recent call last):
admin(31707):   File "/sfs/s/solaris/8/app/mailman-sfs/2.1.4/scripts/driver", line 87, in run_main
admin(31707):     main()
admin(31707):   File "/sfs/s/solaris/8/app/mailman-sfs/2.1.4/Mailman/Cgi/subscribe.py", line 69, in main
admin(31707):     language = cgidata.getvalue('language')
admin(31707):   File "/scratch/local/app/python/2.3/lib/python2.3/cgi.py", line 560, in getvalue
admin(31707):     if key in self:
admin(31707):   File "/scratch/local/app/python/2.3/lib/python2.3/cgi.py", line 611, in __contains__
admin(31707):     raise TypeError, "not indexable"
admin(31707): TypeError: not indexable
admin(31707): [----- Python Information -----] 
admin(31707): sys.version     =   2.3 (#1, Oct 16 2003, 05:04:34) 
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] 
admin(31707): sys.executable  =   /scratch/local/bin/python 
admin(31707): sys.prefix      =   /scratch/local/app/python/current 
admin(31707): sys.exec_prefix =   /scratch/local/app/python/current 
admin(31707): sys.path        =   /scratch/local/app/python/current 
admin(31707): sys.platform    =   linux2 
admin(31707): [----- Environment Variables -----] 
admin(31707): 	force_response_1_0: 1 
admin(31707): 	SSL_SERVER_I_DN_OU: Department of Mathematics 
admin(31707): 	SSL_CIPHER_EXPORT: false 
admin(31707): 	SSL_SERVER_S_DN_Email: webmaster@math.ethz.ch 
admin(31707): 	SERVER_SOFTWARE: Apache/2.0.40 (Red Hat Linux) 
admin(31707): 	SSL_VERSION_INTERFACE: mod_ssl/2.0.40 
admin(31707): 	SCRIPT_NAME: /mailman/subscribe 
admin(31707): 	SSL_SERVER_A_KEY: rsaEncryption 
admin(31707): 	SSL_SERVER_S_DN_ST: Switzerland 
admin(31707): 	SERVER_SIGNATURE: Apache/2.0.40 Server at www.stat.math.ethz.ch Port 443
admin(31707): 
admin(31707): 	REQUEST_METHOD: POST 
admin(31707): 	PATH_INFO: /bioconductor 
admin(31707): 	SERVER_PROTOCOL: HTTP/1.1 
admin(31707): 	SSL_SERVER_S_DN: /C=CH/ST=Switzerland/L=Zurich/O=ETH Zurich/OU=Seminar for Statistics/CN=www.stat.math.ethz.ch/emailAddress=webmaster@math.ethz.ch 
admin(31707): 	SSL_CIPHER: RC4-MD5 
admin(31707): 	SSL_SERVER_V_START: Feb  7 16:20:08 2003 GMT 
admin(31707): 	SSL_SESSION_ID: 319B2196DF311D397B6A4EBE58D8F8307D04AE1C58665697A02A9A59325182B9 
admin(31707): 	CONTENT_LENGTH: 96 
admin(31707): 	SSL_CLIENT_VERIFY: NONE 
admin(31707): 	SSL_SERVER_I_DN_ST: Switzerland 
admin(31707): 	SSL_SERVER_S_DN_O: ETH Zurich 
admin(31707): 	HTTP_REFERER: https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor, https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor 
admin(31707): 	SERVER_NAME: www.stat.math.ethz.ch 
admin(31707): 	REMOTE_ADDR: 148.177.129.213 
admin(31707): 	SSL_CIPHER_ALGKEYSIZE: 128 
admin(31707): 	SSL_SERVER_I_DN: /C=CH/ST=Switzerland/L=Zurich/O=ETH Zurich/OU=Department of Mathematics/CN=ISG D-MATH/emailAddress=webmaster@math.ethz.ch 
admin(31707): 	SSL_SERVER_I_DN_C: CH 
admin(31707): 	SSL_SERVER_M_VERSION: 1 
admin(31707): 	SSL_SERVER_I_DN_O: ETH Zurich 
admin(31707): 	SERVER_ADDR: 129.132.148.136 
admin(31707): 	DOCUMENT_ROOT: /var/www/vhosts/sfs/htdocs 
admin(31707): 	SSL_SERVER_S_DN_OU: Seminar for Statistics 
admin(31707): 	SERVER_PORT: 443 
admin(31707): 	SSL_SERVER_I_DN_Email: webmaster@math.ethz.ch 
admin(31707): 	SSL_VERSION_LIBRARY: OpenSSL/0.9.7a 
admin(31707): 	PYTHONPATH: /sfs/s/solaris/8/app/mailman-sfs/2.1.4 
admin(31707): 	SCRIPT_FILENAME: /var/www/vhosts/sfs/mailman/subscribe 
admin(31707): 	SERVER_ADMIN: webmaster@math.ethz.ch 
admin(31707): 	SCRIPT_URI: https://www.stat.math.ethz.ch/mailman/subscribe/bioconductor 
admin(31707): 	SSL_SERVER_A_SIG: md5WithRSAEncryption 
admin(31707): 	SSL_SERVER_S_DN_L: Zurich 
admin(31707): 	HTTP_HOST: www.stat.math.ethz.ch 
admin(31707): 	SCRIPT_URL: /mailman/subscribe/bioconductor 
admin(31707): 	HTTPS: on 
admin(31707): 	SSL_SERVER_I_DN_L: Zurich 
admin(31707): 	HTTP_CONNECTION: Keep-Alive 
admin(31707): 	PATH_TRANSLATED: /var/www/vhosts/sfs/htdocs/bioconductor 
admin(31707): 	HTTP_CACHE_CONTROL: no-cache 
admin(31707): 	SSL_SERVER_S_DN_C: CH 
admin(31707): 	REQUEST_URI: /mailman/subscribe/bioconductor 
admin(31707): 	HTTP_ACCEPT: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, application/x-shockwave-flash, */* 
admin(31707): 	SSL_SERVER_M_SERIAL: 07 
admin(31707): 	SSL_SERVER_S_DN_CN: www.stat.math.ethz.ch 
admin(31707): 	HTTP_USER_AGENT: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; JP Rev04) 
admin(31707): 	nokeepalive: 1 
admin(31707): 	GATEWAY_INTERFACE: CGI/1.1 
admin(31707): 	SSL_PROTOCOL: SSLv3 
admin(31707): 	REMOTE_PORT: 19699 
admin(31707): 	HTTP_ACCEPT_LANGUAGE: en-au, en-au 
admin(31707): 	HTTP_ACCEPT_ENCODING: gzip, deflate, gzip, deflate 
admin(31707): 	ssl_unclean_shutdown: 1 
admin(31707): 	SSL_SERVER_V_END: Feb  4 16:20:08 2013 GMT 
admin(31707): 	CONTENT_TYPE: application/x-www-form-urlencoded, application/x-www-form-urlencoded 
admin(31707): 	SSL_SERVER_I_DN_CN: ISG D-MATH 
admin(31707): 	QUERY_STRING:  
admin(31707): 	SSL_CIPHER_USEKEYSIZE: 128 
admin(31707): 	downgrade_1_0: 1 
admin(31707): 	UNIQUE_ID: TLeriIGElIIAAHklESEAAAAI 

From chris at jellybaby.net  Wed Jan 21 12:01:41 2004
From: chris at jellybaby.net (Chris Boulter)
Date: Wed Jan 21 12:01:49 2004
Subject: [Mailman-Developers] List locks not getting relinquished
In-Reply-To: <1074643374.26872.58.camel@anthem>
References: <20040119174304.GH51157@jellybaby.net>
	<1074643374.26872.58.camel@anthem>
Message-ID: <20040121170141.GI3573@jellybaby.net>

On Tue 2004-01-20 19:02:54 -0500, Barry Warsaw wrote:
> On Mon, 2004-01-19 at 12:43, Chris Boulter wrote:
> > I'm using Mailman 2.1.2. We have sync_members running in a loop from a
> > daemon, syncing ~500 lists every few hours. Of these, half a dozen lists are
> > failing (and my daemon has to forcibly kill the sync_members process when it
> > detects a timeout). 
> 
> I want to know what "failing" means.  Do you get tracebacks?  Does the
> process just hang?  Does it work but none of the changes persist?

The sync_members process on the affected lists just hangs. I assume it gets
to the same point as it does when I manually run sync_members, i.e. it's
sleeping waiting for the lock. However, it never actually obtains the lock.
Currently I have my daemon kill the sync_members process five minutes after
starting it if it hasn't finished. Before I introduced this, I saw
sync_members hanging for several hours (at which point I would intervene and
kill it manually). So I don't think it's just down to a slow-running process
(unless it's reeeeally slow).

When sync_members hangs, I also find that I can't log on to the web
interface for that list (browser sits spinning after I enter the site
password). I think this further points to an unrelinquished lock, so that
the CGI program can't get the lock on the list it needs.

> Now, if the process is hanging, and you're running them on a live site,
> with messages being processed, cron jobs running, people hitting the web
> site, well, that's not unexpected!

Certainly that could explain an occasional hang, but I see it every time
with certain lists, until I manually break the locks by deleting the lock
file. This happens both on our live site and on a development machine (which
rarely has any users except me).

> You've interrupted the process while it's sleeping trying to acquire the
> lock.  This has to mean either another live process has the lock
> (perhaps it's taking a long time to do something), or there's a stale
> lock around.  Mailman is very careful to release the lock when it's no
> longer necessary, but if something happens like a process gets kill -9'd
> or Python core dumps, or your machine crashes, then a stale lock can
> result.  It's easy to find out if the lock is stale by ps'ing the pid
> that last acquired it.

This is interesting. Presumably a lock named
        entrepreneurship.club.lock.pavo.9897.0
would have been acquired by pid 9897. Looking at my locks, they do seem to
have been acquired by processes which no longer exist (possibly sync_members
processes started by my daemon then killed after the timeout).

One way I hoped Mailman might resolve this would be to ignore locks older
than a certain age, or delete these locks. I guess it doesn't have this
function though (I couldn't find it in Defaults.py).

Also, I've boldly gone in and deleted Mailman/locks/*. If Mailman expects to
find a master lock file and make links to it, could this approach to
breaking locks be dangerous?

> HTH,

Yes indeed. Thanks for your detailed comments. It sounds like I might have
to do penance with the logger though.

Chris

From andyk at spunge.org  Wed Jan 21 14:23:34 2004
From: andyk at spunge.org (Andrzej Kasperowicz)
Date: Wed Jan 21 14:23:41 2004
Subject: [Mailman-Developers] Option to send also list-owner/moderator
	comments to list-owners address.
Message-ID: <400EDFC6.4606.137410F@localhost>

I suggest to add an option in 'pending moderator requests' to send not 
only a copy of the message to list-owner (as it is now), but a copy of 
the message with list-owners/moderators comments in case the message is 
rejected.

Regards
ak

From my at freexp.de  Wed Jan 21 17:51:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Wed Jan 21 17:56:17 2004
Subject: [Mailman-Developers] Redirect list to list?
In-Reply-To: <20040121004058.nceo8csowkggoc8w@george.warnica>
References: <91DiOxyppwB@my.freexp.de>
	<20040121004058.nceo8csowkggoc8w@george.warnica>
Message-ID: <91HkDxO$pwB@my.freexp.de>

Jeff Warnica  wrote on 21.01.04:

>> we want all messages sent by moderated members or non-members to our
>> CVS list be automatically redirected to our Dev list (rather than
>> really moderating or discarding them).

>> Only a specific address (in fact the list address itself) and the
>> non- moderated members shall be allowed to post to the list without
>> being redirected.

>> Is there any patch or other solution which does that job?

> I dont know of any, and this sugestion is off the cuff. (but it would
> be fun to try :P)... Do it at the MTA level, ie: (just invented
> pseudo alias-code)

> dontsendstuffhere@freexp.de: |$MAILMAN cvs
> cvs@freexp.de:               |$MAILMAN dev
> dev@freexp.de:               |$MAILMAN dev

Might work, but I'm reluctant to tweak on MTA level (plus that I have no
idea how to do it in detail).

> The cvs list may require tweeking to allow messages in that dont seem
> to be for it. There is some kind of AKA-ish option, details of which
> I dont recall.

Anybody else here around who can help?  Or is the users group the better
place to ask?  I mean, we're talking about Python code, right?


        Michael

From somuchfun at atlantismail.com  Wed Jan 21 17:56:14 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Wed Jan 21 17:56:27 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: <1074637694.26872.17.camel@anthem>
Message-ID: <22561752503557@smtp.westbay.net>

Hello Barry,
Let me try to explain why the additional header is so important.
When you use large lists with lots of traffic to AOL they can set you on
something that is called an "feedback loop". This loop creates automated
emails from AOL's postmaster about people on one of your list (as an ISP)
who have clicked the "spam" button in regards to one of the messages
originating from you.
But due to privacy reasons the recipient of the email will be stripped out
of the feedback loop message. So AOL advises to use secret headers to
identify who the person who complained is.
This is crucial because it only takes 1 complaint per 1000 recipients for
AOL to put an automated temporary block on your ip address that results in
bouncing of all traffic.
Of course I could just add a mail merge code in the footer of the message
but that only seems to work with full VERP enabled in mailman and the
slowdown is so dramatic that it is no longer feasible for a list of 50,000
or more.
So what I would like to see are two things:
1. One make the codes like %(user_delivered_to)s in the footer work without
VERP enabled
2. Have the option in the GUI to add headers and use for example
%(user_delivered_to)s in it

Kind of connected to this is the problem that the GUI does not allow to show
all suspended members of a list and on a list with 50,000 members it makes
no sense to go through them page by page to find out who got suspended
because of a bounce.

And then an additional problem is that mailman does not take out
x-AuthenticatedSender headers from the poster of the message. And this
header added by auth smtp reveals very clearly who the sender is even when
the list is set to anonymous posting!

I hope this helps make mailman better and stronger!

> -----Original Message-----
> From: Barry Warsaw [mailto:barry@python.org] 
> Sent: Tuesday, January 20, 2004 2:28 PM
> To: Somuchfun
> Cc: mailman-developers@python.org
> Subject: Re: [Mailman-Developers] Adding headers to mailman 
> generated mails
> 
> On Tue, 2004-01-20 at 15:30, Somuchfun wrote:
> > I have a problem that might affect lots of people who send 
> emails to large
> > lists.
> > AOL requires an additional header in the emails identifying 
> the recipient
> > for the feedback loops to unsubscribe AOL users who complain.
> > So how do I add an additional header with a field?
> 
> Can you describe this additional header in more detail?  Better yet
> would be some documentation on the web describing this 
> header.  I.e. if
> it's a standard (or maybe even a proposed standard), perhaps Mailman
> should support it out of the box.
> 
> It's actually pretty easy to add such a header.  One option is to just
> include the header in the original message.  Or you can get Mailman to
> add it to every message by modifying the code in the pipeline.  The
> place to start would be Mailman/Handlers/CookHeaders.py.  That's where
> the RFC 2369 headers are added for example.
> 
> -Barry
> 
> 
> 


From brad.knowles at skynet.be  Wed Jan 21 19:23:55 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Wed Jan 21 19:26:49 2004
Subject: [Mailman-Developers] Adding headers to mailman
 generated mails
In-Reply-To: <22561752503557@smtp.westbay.net>
References: <22561752503557@smtp.westbay.net>
Message-ID: 

At 2:56 PM -0800 2004/01/21, Somuchfun wrote:

>  Of course I could just add a mail merge code in the footer of the message
>  but that only seems to work with full VERP enabled in mailman and the
>  slowdown is so dramatic that it is no longer feasible for a list of 50,000
>  or more.

	If you're not using VERP and you need per-recipient data in the 
headers, then there is absolutely nothing that mailman can do to help 
you.  Mailman will pass the message to the MTA in chunks of 50 or 100 
(or whatever you specify), and you could not encode all those 
recipient names in the headers without exposing a great deal of 
privacy information about your recipients.

	Moreover, once the message was delivered to the user's mailbox, 
just by looking at the message and header contents there would be no 
way to distinguish between any of the 50 or 100 recipients.

	You could configure your MTA to add per-recipient information 
after splitting incoming envelopes so that it delivers a separate 
message for each, but this would be as bad as enabling VERP (for the 
same reasons) and would not give you the benefit of managing bounces 
much more easily, etc....

>  So what I would like to see are two things:
>  1. One make the codes like %(user_delivered_to)s in the footer work without
>  VERP enabled

	No can do.  When you have 50 recipients, which one should have 
their name inserted into this field?

>  2. Have the option in the GUI to add headers and use for example
>  %(user_delivered_to)s in it

	Again, not possible.  Mailman doesn't have the control at that 
point -- the MTA does.  And if you want to keep your network traffic 
to a reasonable level and keep the MTA from beating the hell out of 
your disk drives as it delivers each copy of the message, then 
there's not much you can do.

	I don't understand how AOL expects people to accomplish this sort 
of thing (and I used to be their Sr. Internet Mail Systems 
Administrator).  Maybe I need to talk to Carl Hutzler.

>  And then an additional problem is that mailman does not take out
>  x-AuthenticatedSender headers from the poster of the message. And this
>  header added by auth smtp reveals very clearly who the sender is even when
>  the list is set to anonymous posting!

	Mailman has taken a pretty strong stance towards not munging the 
message any more than absolutely necessary.  Message body content may 
be filtered or converted, but in particular the headers are 
considered sacrosanct and will not be touched.  This same approach 
can be found in all major MTAs that I know of.

	If you want to configure your MTA to strip certain headers, that 
should be possible, and you should have the option of doing that. 
But I don't think you should be expecting Mailman to do this job for 
you.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From somuchfun at atlantismail.com  Wed Jan 21 19:46:22 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Wed Jan 21 19:46:34 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: 
Message-ID: <00462660303848@smtp.westbay.net>

Brad,
While I understand part of your rationale this is really creating a big
problem for people running large lists and try to handle their email traffic
in an ethical way.
Is there a problem to send out the emails one by one with individual To:
addresses and then add a header or a mail merge field in the footer without
creating bounce addresses that most MTA do not allow or understand?
I do not mind the additional CPU time for sending out the messages one by
one if it solves the problem for right now. But since the VERP bounce back
addresses do not really exist an Exim with verify = recipient will always
have a problem.
So is there a way to tweak Exim into sending the messages individually and
allow the addition of a personalized footer without creating personalized
bounce-back addresses?

Thanks!

> -----Original Message-----
> From: Brad Knowles [mailto:brad.knowles@skynet.be] 
> Sent: Wednesday, January 21, 2004 4:24 PM
> To: Somuchfun
> Cc: mailman-developers@python.org; 'Barry Warsaw'
> Subject: RE: [Mailman-Developers] Adding headers to mailman 
> generated mails
> 
> At 2:56 PM -0800 2004/01/21, Somuchfun wrote:
> 
> >  Of course I could just add a mail merge code in the footer 
> of the message
> >  but that only seems to work with full VERP enabled in 
> mailman and the
> >  slowdown is so dramatic that it is no longer feasible for 
> a list of 50,000
> >  or more.
> 
> 	If you're not using VERP and you need per-recipient data in the 
> headers, then there is absolutely nothing that mailman can do to help 
> you.  Mailman will pass the message to the MTA in chunks of 50 or 100 
> (or whatever you specify), and you could not encode all those 
> recipient names in the headers without exposing a great deal of 
> privacy information about your recipients.
> 
> 	Moreover, once the message was delivered to the user's mailbox, 
> just by looking at the message and header contents there would be no 
> way to distinguish between any of the 50 or 100 recipients.
> 
> 	You could configure your MTA to add per-recipient information 
> after splitting incoming envelopes so that it delivers a separate 
> message for each, but this would be as bad as enabling VERP (for the 
> same reasons) and would not give you the benefit of managing bounces 
> much more easily, etc....
> 
> >  So what I would like to see are two things:
> >  1. One make the codes like %(user_delivered_to)s in the 
> footer work without
> >  VERP enabled
> 
> 	No can do.  When you have 50 recipients, which one should have 
> their name inserted into this field?
> 
> >  2. Have the option in the GUI to add headers and use for example
> >  %(user_delivered_to)s in it
> 
> 	Again, not possible.  Mailman doesn't have the control at that 
> point -- the MTA does.  And if you want to keep your network traffic 
> to a reasonable level and keep the MTA from beating the hell out of 
> your disk drives as it delivers each copy of the message, then 
> there's not much you can do.
> 
> 	I don't understand how AOL expects people to accomplish 
> this sort 
> of thing (and I used to be their Sr. Internet Mail Systems 
> Administrator).  Maybe I need to talk to Carl Hutzler.
> 
> >  And then an additional problem is that mailman does not take out
> >  x-AuthenticatedSender headers from the poster of the 
> message. And this
> >  header added by auth smtp reveals very clearly who the 
> sender is even when
> >  the list is set to anonymous posting!
> 
> 	Mailman has taken a pretty strong stance towards not 
> munging the 
> message any more than absolutely necessary.  Message body content may 
> be filtered or converted, but in particular the headers are 
> considered sacrosanct and will not be touched.  This same approach 
> can be found in all major MTAs that I know of.
> 
> 	If you want to configure your MTA to strip certain 
> headers, that 
> should be possible, and you should have the option of doing that. 
> But I don't think you should be expecting Mailman to do this job for 
> you.
> 
> -- 
> Brad Knowles, 
> 
> "They that can give up essential liberty to obtain a little temporary
> safety deserve neither liberty nor safety."
>      -Benjamin Franklin, Historical Review of Pennsylvania.
> 
> GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ 
> !E-(---) W+++(--) N+
> !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) 
> X++(+++) R+(+++)
> tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- 
> r---(+++)* z(+++)
> 


From colinp at waikato.ac.nz  Wed Jan 21 20:18:17 2004
From: colinp at waikato.ac.nz (Colin Palmer)
Date: Wed Jan 21 20:18:27 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: <22561752503557@smtp.westbay.net>
References: <22561752503557@smtp.westbay.net>
Message-ID: <1074734297.3011.211.camel@firefox.cc.waikato.ac.nz>

On Thu, 2004-01-22 at 11:56, Somuchfun wrote:
> Hello Barry,
> Let me try to explain why the additional header is so important.
> When you use large lists with lots of traffic to AOL they can set you on
> something that is called an "feedback loop". This loop creates automated
> emails from AOL's postmaster about people on one of your list (as an ISP)
> who have clicked the "spam" button in regards to one of the messages
> originating from you.

If this is really important to AOL, seeing their 'official'
documentation on this would be nice, to make sure Mailman does implement
it right.

Just turning on VERP may work,  and would be less overhead than full
personalisation if you use one of the Mailman patches that gets postfix
or qmail to take on some of the VERP overhead.

> And then an additional problem is that mailman does not take out
> x-AuthenticatedSender headers from the poster of the message. And this
> header added by auth smtp reveals very clearly who the sender is even when
> the list is set to anonymous posting!

Maybe if your list wasn't anonymous, you'd get less spam complaints? :)

As an unrelated feature though, being able to give Mailman a
configurable list of headers to strip out of messages would be useful.

-- 
Colin Palmer 
University of Waikato, ITS Division


From brad.knowles at skynet.be  Wed Jan 21 21:21:05 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Wed Jan 21 21:21:17 2004
Subject: [Mailman-Developers] Adding headers to mailman
 generated mails
In-Reply-To: <00462660303848@smtp.westbay.net>
References: <00462660303848@smtp.westbay.net>
Message-ID: 

At 4:46 PM -0800 2004/01/21, Somuchfun wrote:

>  Is there a problem to send out the emails one by one with individual To:
>  addresses and then add a header or a mail merge field in the footer without
>  creating bounce addresses that most MTA do not allow or understand?

	No MTA ever created should ever have a problem with VERPs.  Users 
might be confused by how they look, but the mail server should be 
able to deal with them just fine.  That's the entire point of VERPs.

	That said, if you don't want to do VERPs but you do want a single 
recipient per message generated by Mailman, you should just need to 
adjust the maximum number of recipients per message as specified by 
SMTP_MAX_RCPTS, and enable message personalization (see 
 
and 
). 
Just make sure that you don't enable actual VERP'ing along with these 
other changes.

	As part of the message personalization, add the appropriate 
per-user information in the template footer for the list.  That 
should hopefully deal with the problem.

>  I do not mind the additional CPU time for sending out the messages one by
>  one if it solves the problem for right now. But since the VERP bounce back
>  addresses do not really exist an Exim with verify = recipient will always
>  have a problem.

	This is almost certainly never a CPU time issue.  It's a disk I/O 
capacity issue.  See 
 
and the related "performance" entries in the FAQ.

	Not only can you kill your disk drive(s) on the server by 
specifying too few recipients per message, you can seriously reduce 
your message throughput and totally backlog the machine.  If you have 
a large enough list, you can easily make the machine bury itself to 
the point where it can not ever possibly recover from what was done 
to it.

>  So is there a way to tweak Exim into sending the messages individually and
>  allow the addition of a personalized footer without creating personalized
>  bounce-back addresses?

	I am not familiar with Exim.  I do not know what configuration 
changes would be required to get it to add personalized information 
in the headers.  You would need to talk to someone better acquainted 
with Exim, presumably on an Exim-specific mailing list or newsgroup.

	I don't remember whether or not postfix has a facility that would 
allow you to have it perform per-message/recipient header 
modifications.  I'd have to check the latest documentation, logs, 
etc....

	I can tell you that the default standard installation of sendmail 
will do this for you, automatically.  If there is one and only one 
recipient of a message, the "$u" macro will be defined, and the 
identified username will be shown in the "Received:" headers.  If 
there is more than one recipient for the message, then this macro 
will not be defined, and no usernames will be displayed.


	Unfortunately, most headers tend to be missing from most 
complaints, so your best bet would probably be to get Mailman to put 
the message personalization information into the footer of the 
message, which is more likely to survive.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From barry at python.org  Thu Jan 22 14:37:01 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 22 14:37:36 2004
Subject: [Mailman-Developers] Redirect list to list?
In-Reply-To: <91HkDxO$pwB@my.freexp.de>
References: <91DiOxyppwB@my.freexp.de>
	<20040121004058.nceo8csowkggoc8w@george.warnica>
	<91HkDxO$pwB@my.freexp.de>
Message-ID: <1074800220.31285.36.camel@anthem>

On Wed, 2004-01-21 at 17:51, Michael Heydekamp wrote:

> Anybody else here around who can help?  Or is the users group the better
> place to ask?  I mean, we're talking about Python code, right?

Well, since this feature doesn't exist in Mailman currently, you're
talking about some development being necessary.  Which means
mailman-developers is the right place to post it.

-Barry



From barry at python.org  Thu Jan 22 14:39:19 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 22 14:39:27 2004
Subject: [Mailman-Developers] Shunt
In-Reply-To: <1074675454.9300.61.camel@amspoke>
References: <1074675454.9300.61.camel@amspoke>
Message-ID: <1074800358.31285.39.camel@anthem>

On Wed, 2004-01-21 at 03:57, Juan Enrique G?mez wrote:

> could any one explain me what's intended for the shunt directory in
> mailman?.. just curiosity...

It's where messages that trigger errors in Mailman end up.  The goal is
to never lose a message.  If some unexpected exception occurs while
processing a message, the error gets logged and the message gets moved
to shunt.  The idea is that once the bug is fixed, you can run
bin/unshunt to move them back into the processing queue from whence they
came.

-Barry



From hesco at greens.org  Sun Jan 11 17:09:31 2004
From: hesco at greens.org (Hugh Esco)
Date: Thu Jan 22 14:47:39 2004
Subject: [Mailman-Developers] Two requests.  Hugh
Message-ID: <20040111170931.0a04b902.hesco@greens.org>

Greetings:  

I first started using Mailman during the 2000 campaign cycle, when the Nader campaign put it on their servers, subscribed me to a few of its lists and gave me administrative responsibilities for [ga-vols] to support my work on that campaign.  I came to request a couple of additional lists from the nader2000 server administrator before November's election.  

Based on that experience, I then recommended it to Cameron Spitzer, administrator for the Green Internet Society, which provides a few servers at greens.org to the Green Party movement in this country and around the world.  By 2001 or so, he had worked through his issues with Python and Mailman installation and left me with administrative authority over a brand new ga.greens.org/mailman installation.  I now provide administartive support for close to twenty lists supporting our local affiliates, committees and candidate campaigns.  We are creating additional lists all the time.  

I write with two specific questions today:  

	(1)  The administrative page for a list permits someone to hold the list back as private for list subscribers only, or as available to the browsing public, with anyone able to access its contents.  I would love to see a bit more gradiation here.  We have lists that are set up as Members only lists.  Our By-laws guarantee sunshine in the operation of our Party for Party members.  I would like to be able to create a Members only page, where an authenticated member would be able to review and access lists that are designated as members-only, and which would be as simple to construct as the ga.greens.org/mailman/listinfo page now is.  I imagine that our local affiliates would also appreciate such a service so that we might have a ga.greens.org/mailman/members/listinfo and a ga.greens.org/mailman/members/dekalb/listinfo page as well where authenticated members of the Dekalb County affiliate might find their local -dx, -annc and various committee lists, as well as the -news list and any others that would be listed on the public site.  

The idea here is that we wouldn't have to go to the effort of creating another instance of the server at gagpmembers.greens.org/ or dekalb.gagpmembers.greens.org/, that we could hide internal lists from the browsing public, but not from our members and that the state Party's IT team could manage the admin details for the locals, while still providing additional organization for folks directly accessing the lists that most interest them.  

I would be happy to provide support with review of algorythms, testing and documentation.  Don't know I could be much help with coding.  I'm pretty busy as it is and have only a few thousand lines of perl, php and bash under my belt, with no python experience whatsoever.  Is there any work to provide this sort of functionality already under way?  Might there be any interest in undertaking such a project?  

	(2)  Between administering the ga.greens.org/mailman server itself and administering perhaps a third or more of the lists hosted there, I get plenty of email from bounced addresses.  It is far more than I can keep up with.  I'm looking for a script that will process an mbox of returned mail, extracting a report of what addresses bounced what subject lines from what lists on what dates and for what reasons.  Such a report would permit me to quickly parse the results and send them to individual list administrators and help all of us to know which addresses to unsubscribe and which addresses to endure until their mail boxes got cleaned out and started accepting our email again.  Including the subject lines would also permit us to know when an individual needed a phone call to keep them in the loop about a subject addressed in a bounced message.  

I have toyed with scripting this in perl, but have thought that surely I am not the first person to realize this need, that someone has to have dealt with this issue in the past and have something I can simply download and install.  Any leads on something which already exists or a project already underway to develop something like this would be greatly appreciated.  

I look forward to hearing back from you soon on this.  The 2004 election cycle is upon us and I expect my email overload to only increase as the months go by.  The Mailman project has already done much to make my life easier with respect to this work load.  But I'd really appreciate any leads on how to address these two issues.  

I just added myself to this list, on the daily digest.  I would appreciate it if folks replying to this thread would cc: it directly to me, as well as responding to the list itself, to help me focus on the list traffic most pertinent to how I can get involved in supporting Mailman.  Thank you all who have contributed so much to building such a great tool which has been so helpful to our work, building the Georgia Green Party.  

-- Hugh Esco 
Political Coordinator 
Georgia Green Party 

From ryan at ipa.darktech.org  Sun Jan 18 19:19:57 2004
From: ryan at ipa.darktech.org (ryan)
Date: Thu Jan 22 14:50:04 2004
Subject: [Mailman-Developers] images embedded in html: inline vs. attached
Message-ID: 

Hi,

I've tried searching the list archives, and posting to the mailman-users
list, but I can't seem to find an answer to my problem.  Hopefully someone
on this list can shed some light...

A quick background: At the non-profit museum where I work, we use Mailman
to inform our paying members (and other general members of the public who
signup to our lists)  about our current exhibitions and events.  We like
to use HTML formatted messages with in-body (inline?) images for this
purpose.  It looks nice, and conforms with the overall look of our website
and other publications in print.  I realize sending out bulk HTML mailings
is probably not the most typical use of Mailman, but it works great for
us, and allows people who don't want our emails to quickly remove
themselves without requiring our intervention.

Here's the problem: Today I sent out 2 test messages to a pre-configured
list.  For the two tests, the list had only 6 staff members subscribed.  
The message looked great, everyone saw the content with the image
correctly aligned in the body of the message, between the two blocks of
text where it belonged.

However, after I added the 6700+ actual email addresses to the SAME list,
and sent out the SAME email, the image was removed from the body of the
message and included as an attachment instead.  This essentially made our
pretty little email look like crap.  This behavior occurred on all the
computers of the 6 staff members where I had previously tested, plus on
additional accounts that I was able to confirm through phone calls, etc.

Again, the same list (same configuration), and the same email were used
for the successful test and the unsuccessful real mail-out.  I cannot
stress enough that these were the *exact* same emails, for both the test
with 6 users, and the later mailing with 6700+; the filesizes were the
exact same, and the HTML was unchanged.

So... my only theory is: at some cut-off point, Mailman decides that there
are two many users on a list to embed an image in every email.  Perhaps
because you thought it more efficient to store the image once and then
reference it as an attachment when each member is emailed?

Or could this be some behavior of my mailer daemon, Sendmail?

Any thoughts?

Thanks in advance,
Ryan

From sbackman at askmoses.com  Fri Jan 16 12:13:46 2004
From: sbackman at askmoses.com (S. Backman)
Date: Thu Jan 22 14:50:17 2004
Subject: [Mailman-Developers] Combining Archives / Deleting Archives
Message-ID: 

1. What are the procedures for combining Archives - say 12 monthly archive
folders into one yearly archive folder?

2. How does one eliminate old archive folders?

Thanks

S. Backman




From shigeno at iij-mc.co.jp  Thu Jan  8 06:04:19 2004
From: shigeno at iij-mc.co.jp (SHIGENO Kazutaka)
Date: Thu Jan 22 14:50:50 2004
Subject: [Mailman-Developers] BounceRunner.py patch
Message-ID: <200401081104.UAA11830@sh.iij-mc.co.jp>

Hi,

Please use if the following patch is required for Mailman.

--- Mailman/Queue/BounceRunner.py-	Sat Nov 22 07:45:04 2003
+++ Mailman/Queue/BounceRunner.py	Thu Jan  8 19:15:56 2004
@@ -68,7 +68,7 @@
         # All messages to list-owner@vdom.ain have their envelope sender set
         # to site-owner@dom.ain (no virtual domain).  Is this a bounce for a
         # message to a list owner, coming to the site owner?
-        if msg.get('to', '') == Utils.get_site_email(extra='-owner'):
+        if msg.get('to', '') == Utils.get_site_email(extra='owner'):
             # Send it on to the site owners, but craft the envelope sender to
             # be the -loop detection address, so if /they/ bounce, we won't
             # get stuck in a bounce loop.

-- shigeno

From sthapa at site5.com  Mon Jan 12 14:18:09 2004
From: sthapa at site5.com (Suchandra Thapa)
Date: Thu Jan 22 14:50:55 2004
Subject: [Mailman-Developers] mailman problem/possible bug?
Message-ID: <1073935088.4330.146.camel@hepcat>

I tried posting this in mailman-users but didn't get a reply.  

After a recent upgrade to mailman 2.1.3, I came across a possible bug in
mailman.  The bug is triggered by a character being out of range at line
799 in Mailman/Cgi/admin.py.  The cvs commit for that line and the
comments indicate that the error should appear for non ascii
characters.   The code at that point is encoding the member list (?)
into the ascii character set.  First, are the comments correct? I.e. is
having an email address such as "Andr?"  is correct or is
this a valid email address?  Also is there any way I can clean up the
database if that email address is invalid?  

Finally, an install of mailman 2.1.2 was attempted to temporarily avoid
the problem but an error about a downgrade being dangerous was given. 
Is a switch from mailman 2.1.3 to 2.1.2 safe?

-- 

Suchandra Thapa
System Administration
Site5 Internet Solutions, Inc.


From barry at python.org  Thu Jan 22 14:55:31 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 22 14:55:39 2004
Subject: [Mailman-Developers] Bug in Mailman 2.1.4 {Subscription problem}
In-Reply-To: <16398.31461.201314.57938@gargle.gargle.HOWL>
References: 
	<16398.31461.201314.57938@gargle.gargle.HOWL>
Message-ID: <1074801331.31285.42.camel@anthem>

On Wed, 2004-01-21 at 08:13, Martin Maechler wrote:
> Dear Mailman developers,
> 
> A user ("ADBONT@...", cc'ed) who wanted to subscribe to one of
> our mailing lists received a 
> 
>   >> Bug in Mailman version 2.1.4
>   >> 
>   >> We're sorry, we hit a bug!
> 
> and sent me the full message, I've found the same message in
> mailman's  data/logs/error file, 
> and append it below.
> 
> Note that this is quite an active mailing list, 
> ---> https://www.stat.math.ethz.ch/mailman/listinfo/bioconductor/
> 
> and many people do subscribe and unsubscribe successfully both
> through the web and the email interface.
> 
> Though -- I've got reports more than once that subscription
> through the web interface produced a "you are subscribed ...."
> confirmation (web) and welcome email  and the person was *not*
> subscribed after all.
> That is really another problem, but maybe related...
> 
> Note that our setup is a bit non-standard maybe:
> Web  server ("hubble")  is Redhat9
> Mail server ("hypatia") is Solaris 8 -- mailman installed on both
> 
> Many thanks for mailman,
> and for your consideration of this report.

That's very strange.  It's almost as if the form data didn't get
through, or something else weird is happening in the CGI environment. 
I'm not sure I have much more of an idea than that though.

-Barry



From barry at python.org  Thu Jan 22 15:03:45 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 22 15:03:53 2004
Subject: [Mailman-Developers] List locks not getting relinquished
In-Reply-To: <20040121170141.GI3573@jellybaby.net>
References: <20040119174304.GH51157@jellybaby.net>
	<1074643374.26872.58.camel@anthem>
	<20040121170141.GI3573@jellybaby.net>
Message-ID: <1074801825.31285.49.camel@anthem>

On Wed, 2004-01-21 at 12:01, Chris Boulter wrote:

> This is interesting. Presumably a lock named
>         entrepreneurship.club.lock.pavo.9897.0
> would have been acquired by pid 9897. Looking at my locks, they do seem to
> have been acquired by processes which no longer exist (possibly sync_members
> processes started by my daemon then killed after the timeout).

So some other process is hanging onto the lock, or leaving a stale lock
before your first sync_members process gets killed.  The trick for you
is to figure out which process that is.  Try disabling your daemon's
killing of sync_members, but instead try to print the pid for the
process that thinks it has the lock.  Is that process still around?  Can
you trace it to see if it's still making progress?

> One way I hoped Mailman might resolve this would be to ignore locks older
> than a certain age, or delete these locks. I guess it doesn't have this
> function though (I couldn't find it in Defaults.py).

Mailman does break locks, but the default time out is 5 hours for list
locks.  Breaking locks is generally bad so it's not a good idea to lower
this a whole lot.  OTOH, no properly functioning Mailman process should
hold a list lock for that long (or even close).

> Also, I've boldly gone in and deleted Mailman/locks/*. If Mailman expects to
> find a master lock file and make links to it, could this approach to
> breaking locks be dangerous?

Breaking list locks is dangerous because it's the list lock that ensures
database consistency.  Mailman 2.1 uses a very simple
write-to-Python-pickle persistency mechanism which most definitely is
not multi-process safe.  Breaking the master lock won't completely hose
you since that just protects from multiple mailmanctl processes
starting.  If you don't run that command more than once, you should be
fine.

> Yes indeed. Thanks for your detailed comments. It sounds like I might have
> to do penance with the logger though.

Try not to have too much fun! :)

-Barry



From my at freexp.de  Thu Jan 22 15:09:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Thu Jan 22 15:24:04 2004
Subject: [Mailman-Developers] Redirect list to list?
In-Reply-To: <1074800220.31285.36.camel@anthem>
References: <1074800220.31285.36.camel@anthem>
Message-ID: <91LlnyS4pwB@my.freexp.de>

Hi Barry,

Barry Warsaw  to Michael Heydekamp on 22.01.04:

> On Wed, 2004-01-21 at 17:51, Michael Heydekamp wrote:

>> Anybody else here around who can help?  Or is the users group the
>> better place to ask?  I mean, we're talking about Python code,
>> right?

> Well, since this feature doesn't exist in Mailman currently, you're
> talking about some development being necessary.  Which means
> mailman-developers is the right place to post it.

Ok, now that we have clarified that :) we're "just" still missing the
answer which would put us in the position to do this redirection *now*.
I hoped one of the python experts would be able to deliver some code.

Or probably Jeff was right and we should try it on MTA level?  Geez,
after Mailman and INN more stuff to look at...

Anybody an idea how to do this redirection (except for specific
addresses) with Exim?  Response by personal mail would be fine.


        Michael

From rousse at ccr.jussieu.fr  Thu Jan 22 15:51:33 2004
From: rousse at ccr.jussieu.fr (Guillaume Rousse)
Date: Thu Jan 22 16:05:03 2004
Subject: [Mailman-Developers] mailman setup
In-Reply-To: <200312291650.40147.rousse@ccr.jussieu.fr>
References: <200312291650.40147.rousse@ccr.jussieu.fr>
Message-ID: <200401222151.33691.rousse@ccr.jussieu.fr>

On Monday 29 December 2003 16:50, Guillaume Rousse wrote:
> Hello.
>
> I'm one of the guy managing the mailman package for mandrake-linux. I've
> currently trying to make setup a bit more standard and FHS compliant. I
> need developpers help however, first to make sure my changes won't cause
> trouble, second as I think some of these change should get merged upstream,
> as they would benefit more people.
I didn't received any answer to my post. Does this means this kind of problem 
is considered secondary for developpers ?
-- 
The number of witnesses available is inversely proportional to the skill you 
demonstrate
There will never be anyone around to see you do something brilliant
When you really screw up, you will get network coverage with a 40 share
		-- Murphy's Laws of Locksmithing n?15

From brad.knowles at skynet.be  Thu Jan 22 16:30:56 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Thu Jan 22 16:31:17 2004
Subject: [Mailman-Developers] mailman setup
In-Reply-To: <200401222151.33691.rousse@ccr.jussieu.fr>
References: <200312291650.40147.rousse@ccr.jussieu.fr>
	<200401222151.33691.rousse@ccr.jussieu.fr>
Message-ID: 

At 9:51 PM +0100 2004/01/22, Guillaume Rousse wrote:

>  I didn't received any answer to my post.

	Sorry, I don't recall seeing the previous post.

>                                          Does this means this kind of problem
>  is considered secondary for developpers ?

	You can propose whatever changes you think will be beneficial.

	However, if the developers feel that those changes will be 
detrimental to Mailman or to using Mailman on other platforms, I 
believe that they're highly unlikely to accept them.


	If the changes are felt to be beneficial, then they are more 
likely to be accepted.  If they are minor, they may be slipped in 
earlier during the development and maintenance process.  If they are 
larger, then you are likely to have to wait longer, as more code 
restructuring is likely to be necessary to accommodate them.

	In any event, you'll have to wait for Barry (or one of the other 
developers) to find the time necessary to incorporate them, and where 
you get in this process will depend on their personal priorities.


	At this point, I would suggest getting into the code and start 
mucking about.  Post your suggested changes and why you think they 
should be made, and let's discuss.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From barry at python.org  Thu Jan 22 17:50:33 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 22 17:50:42 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: 
References: <00462660303848@smtp.westbay.net>
	
Message-ID: <1074811832.31285.125.camel@anthem>

On Wed, 2004-01-21 at 21:21, Brad Knowles wrote:

> 	As part of the message personalization, add the appropriate 
> per-user information in the template footer for the list.  That 
> should hopefully deal with the problem.

I should mention that Mailman 2.1's full personalization support (as
opposed to VERP header support) isn't terribly efficient.  I have what I
think will be a very nice scheme to do this about as fast as you can do
in Python, but it requires Python 2.3 so it's slated for Mailman 3.  The
nice upside is that you could conceivably support the templatizing and
personalization of any content, not limited to the footer, header, or
mail headers.  I believe I can make it efficiently configurable so a
list owner could disable content personalization of say the original
message for discussion lists, but enable it for newsletters.  We'll
ignore for now the question of where the personalized content comes
from.

In a previous message, Brad gave great answers and links that are well
worth re-reading every few months.  So I won't rehash anything I agree
with.

> >  So is there a way to tweak Exim into sending the messages individually and
> >  allow the addition of a personalized footer without creating personalized
> >  bounce-back addresses?
> 
> 	I am not familiar with Exim.  I do not know what configuration 
> changes would be required to get it to add personalized information 
> in the headers.  You would need to talk to someone better acquainted 
> with Exim, presumably on an Exim-specific mailing list or newsgroup.

Exim has some very nice capabilities which can be used to embed an
interpreter like Python in your MTA.  For example, we use this on
incoming messages on python.org to filter everything throw spambayes and
do other programmatically interesting checks on the message.  Yes, it
slows down message acceptance a bit, but it's worth it for us.

Nigel can provide details, but I think the same embedding feature could
be used to have the MTA do the final stitching of content template and
personalized data.  It would be A Project to hack together, but I think
it could be a neat idea to play with, although I'm not sure how much it
would help.  Certainly, pushing the stitching down into the MTA and
closer to the external socket connection would reduce disk i/o on the
mail server, because then Mailman could go back to handing one copy of
the message to the MTA, plus some job description of where to get the
personalized information.  Imagine a SQL select statement for instance. 
If the MTA could do what Mailman does here -- not creating a disk image
for each instance of the message, but stitching it together in member as
it's going out on the wire -- I think you'd greatly improve disk
contention.  You wouldn't help bandwidth, but then if JC's evaluation is
accurate, that penalty is a mere  doubling of bandwidth.

I don't think Postfix has the same embedding capabilities, although I
haven't looked at what Postfix 2.1 may provide.

> 	I can tell you that the default standard installation of sendmail 
> will do this for you, automatically.  If there is one and only one 
> recipient of a message, the "$u" macro will be defined, and the 
> identified username will be shown in the "Received:" headers.  If 
> there is more than one recipient for the message, then this macro 
> will not be defined, and no usernames will be displayed.

In a sense, that's what we've talked about before.  If there were a
standard language that the mail server and list manager could agree on
for both defining the template, and defining the per-recipient data
source, we could have a more efficient mechanism, with perhaps a hope of
mta agnosticism.

> 	Unfortunately, most headers tend to be missing from most 
> complaints, so your best bet would probably be to get Mailman to put 
> the message personalization information into the footer of the 
> message, which is more likely to survive.

As for stripping headers, I do think there's some value in being able to
more easily configure the headers to strip for both regular messages and
anonymized messages.  OTOH, it's easy to hack the source.  Cleanse.py in
Mailman/Handlers is the place to look.

-Barry



From pioppo at ferrara.linux.it  Thu Jan 22 18:03:53 2004
From: pioppo at ferrara.linux.it (Simone Piunno)
Date: Thu Jan 22 17:57:16 2004
Subject: [Mailman-Developers] mailman setup
In-Reply-To: <200312291650.40147.rousse@ccr.jussieu.fr>
References: <200312291650.40147.rousse@ccr.jussieu.fr>
Message-ID: <200401230003.53691.pioppo@ferrara.linux.it>

On Thursday 22 January 2004 21:51, Guillaume Rousse wrote:

> I didn't received any answer to my post. Does this means this kind of
> problem is considered secondary for developpers ?

Hi Guillame, 

sorry for the late answer, here are my comments.
Please note that I think none of the suggested changes should make it in the 
standard Mailman tree, at least until 3.0.  Instead, they should be diffs in 
your src.rpm, or on-the-fly patches performed by the spec.

On Monday 29 December 2003 16:50, Guillaume Rousse wrote:

> Then i move logs from /var/lib/mailman/logs to /var/log/mailman through a
> symlink.

You can define LOG_DIR in mm_cfg.py (or overwrite the value in Defaults.py)

> The config file location, however, is still puzzling me. Having a
> configuration file (mm_cfg.py) under /usr is non-standard, as /usr is
> supposed to be read-only. Moreover, it is a pain for us packagers, as we
> have to include everything but this file as normal files, then tag it as
> config file. I'd really think it should be moved in /etc instead, either by
> me using a symlink, but much better by an upstream change.

I'd do the following:

1. after "from Defaults import *" and maybe the LOG_DIR override, add this
    code to the standard mm_cfg.py (without moving it out of the tree!)

   try:
      execfile('/etc/mailman.conf')
   except SyntaxError:
      # I don't know how to write better info, like the offending line number.
      print "Syntax error in the config file!"
      import sys
      sys.exit(1)
   except IOError, e:
      import errno
      errcode, errdesc = e
      if errcode != errno.ENOENT:
         # this shouldn't happen
         raise

2. put your config in /etc/mailman.conf.
    You can use Defaults.py as a template (commenting everything)

3. document in detail whatever you've done, because all information that 
people can google only mentions mm_cfg.py.  People will be confused.

> The second issue are perms. The default install make everything owned by
> the uid/gid determined at configure stage (mail/mail for me), and make all
> directories setgid, even for non-variables files, which is quite really a
> non-standard practice. So i tried to clean it a bit.
>
> I made everything under /usr/lib/mailman owned by root.root, with standard
> perms (644 for files, 755 for directories) except the binaries
> in /usr/lib/mailman/cgi-bin and usr/lib/mailman/mail which are setgid, and
> owned by mail group. I tested, it seems to work.

it's OK, as long as you pack in you rpm the .pyc files too.
Also note that a possible different choice could be to install the Mailman 
directory (the one with mm_cfg.py) in /usr/list/pythonXXX/site-packages, so 
that 3rd party apps can use it as a library.
In this case you should better split in two packages: library and app.

And note the check_perms script... 
you should either patch it or remove it from the package.

> I also made top-level directory for variable files (/var/lib/mailman) owned
> by root/root with standard 755 perms, but i didn't tried yet further perm
> modifications there, as mailman need write permissions in subdirectories.

I believe all the /var/lib/mailman should be sgid.  

the most challenge will be for bin scripts and the like... note the paths.py 
common include.... it must be in the same directory, or you'll have to copy 
the relevant paths on top of all scripts.

Cheers
  Simone


From brad.knowles at skynet.be  Thu Jan 22 18:21:04 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Thu Jan 22 18:21:26 2004
Subject: [Mailman-Developers] Adding headers to mailman
 generated mails
In-Reply-To: <1074811832.31285.125.camel@anthem>
References: <00462660303848@smtp.westbay.net>	
	
	<1074811832.31285.125.camel@anthem>
Message-ID: 

At 5:50 PM -0500 2004/01/22, Barry Warsaw wrote:

>  Nigel can provide details, but I think the same embedding feature could
>  be used to have the MTA do the final stitching of content template and
>  personalized data.  It would be A Project to hack together, but I think
>  it could be a neat idea to play with, although I'm not sure how much it
>  would help.

	This is similar to what Eric Allman (at that time, before 
Sendmail Inc. existed), Bryan Costales (at the time, working for 
InfoBeat/Mercury Mail), and I (working at AOL) were discussing back 
in 1996, in the creation of a Mail-Merge Transport Protocol (MMTP) 
server, based on a modified version of sendmail along with a standard 
language for transmitting that content.  With MMTP servers on both 
ends, it would not matter how many thousands or millions of 
recipients you might have, only one copy of the message body would be 
transmitted, and all the rest would be filled in on the remote end.

	We ultimately gave up on this idea because we realized that it 
would make the spam problem much, much worse.  The same things that 
help regular MTAs transmit millions of customized messages per hour 
to their paying customers would probably allow spammers to transmit 
billions of messages per hour to everyone in the universe.


	Certainly, before any serious discussion of creating something 
like an MMTP server, and trying to make that a standard which you 
would expect programs like sendmail, postfix, and Exim to implement, 
I believe that the spam issue needs to be addressed.  You need to be 
able to prove how this cannot be abused to generate spam instead.

>  If the MTA could do what Mailman does here -- not creating a disk image
>  for each instance of the message, but stitching it together in member as
>  it's going out on the wire -- I think you'd greatly improve disk
>  contention.

	I'm not sure that the MTA could safely do that in memory.  At 
least, it would be difficult to ensure that the MTA gets this done 
right.  This would be akin to handling the entire message queue in 
memory for all messages, something which can't really be done safely 
except under very strict circumstances.

	The only MTA I know of that is capable of doing things like this 
is the latest release of version 8 sendmail, and even then it 
defaults to handling messages in memory that are no larger than 4KB.


	Yes, filesystem I/O is the number one killer, specifically 
synchronous meta-data updates.

	But then people on this list have heard me harping on this 
subject for a long time, and should know by now that I will refer 
them to Nick Christenson's book _Sendmail Performance Tuning_ (see 
), or my own slides from 
an invited talk entitled "Sendmail Performance Tuning for Large 
Systems" at 
.

>  I don't think Postfix has the same embedding capabilities, although I
>  haven't looked at what Postfix 2.1 may provide.

	I'm not aware of anything like this, but I'd have to check.

>  In a sense, that's what we've talked about before.  If there were a
>  standard language that the mail server and list manager could agree on
>  for both defining the template, and defining the per-recipient data
>  source, we could have a more efficient mechanism, with perhaps a hope of
>  mta agnosticism.

	That would be nice.  However, I fear that we have much more basic 
problems that are much more serious, and which need to be resolved 
before we can expect to start worrying about such subjects as 
increasing efficiency in the interfaces between MLMs and MTAs.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From wheakory at isu.edu  Thu Jan 22 18:36:25 2004
From: wheakory at isu.edu (Kory Wheatley)
Date: Thu Jan 22 18:36:26 2004
Subject: [Mailman-Developers] Mailman and CVS
Message-ID: <40105E79.5040201@isu.edu>

This question is not really related to Mailman, but I'm sure with the 
knowledge of the developers it can easily be answered about CVS.

I've been asked to setup a CVS Repository  on a dedicated machine just 
for this purpose, of course users will need to be able to add their 
updates over the network.

My question is where is good instructions  about using CVS (commands),  
and  installing it on one center Repository, and setting up rights for 
individuals.

Thank you in advance.

-- 
Kory Wheatley
Academic Computing Analyst Sr.
Phone 282-3874
#########################################
Everything must point to him.


From barry at python.org  Thu Jan 22 18:56:24 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 22 18:56:34 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: 
References: <00462660303848@smtp.westbay.net>
	 <1074811832.31285.125.camel@anthem>
	
Message-ID: <1074815784.31285.181.camel@anthem>

On Thu, 2004-01-22 at 18:21, Brad Knowles wrote:

> 	This is similar to what Eric Allman (at that time, before 
> Sendmail Inc. existed), Bryan Costales (at the time, working for 
> InfoBeat/Mercury Mail), and I (working at AOL) were discussing back 
> in 1996, in the creation of a Mail-Merge Transport Protocol (MMTP) 
> server, based on a modified version of sendmail along with a standard 
> language for transmitting that content.  With MMTP servers on both 
> ends, it would not matter how many thousands or millions of 
> recipients you might have, only one copy of the message body would be 
> transmitted, and all the rest would be filled in on the remote end.
> 
> 	We ultimately gave up on this idea because we realized that it 
> would make the spam problem much, much worse.  The same things that 
> help regular MTAs transmit millions of customized messages per hour 
> to their paying customers would probably allow spammers to transmit 
> billions of messages per hour to everyone in the universe.

Very interesting.  Some thoughts: there would still be some benefit here
for Mailman if we simply limited access to the MMTP to the localhost
interface.  That's how Mailman hands stuff off to the MTA, and in our
Exim configuration, we do quite a bit of special casing for localhost
connections.  The advantage here is that Mailman could go back to
batching deliveries to its worker mail server, reducing both the
bandwidth between the two processes and the disk i/o on the worker mta.

(Read 'localhost' as privileged connection, e.g. Mailman feeding a smurf
farm.)

I had been thinking along the lines of the language for specifying the
data source as a db connection and a SQL command.  I wouldn't want to do
that across the Internet!  OTOH, with a protocol like MMTP, I suppose
you'd have to send all the data for all the recipients in the same
transaction, and the bandwidth trade-off would depend on the size of the
recipient-centric data.

> 	Certainly, before any serious discussion of creating something 
> like an MMTP server, and trying to make that a standard which you 
> would expect programs like sendmail, postfix, and Exim to implement, 
> I believe that the spam issue needs to be addressed.  You need to be 
> able to prove how this cannot be abused to generate spam instead.

That's certainly tricky, but I think it's got to boil down to privilege
or authentication.  It would still make me nervous to accept such jobs
from other than sites I control.

As an aside: the spam issue is already a huge nightmare for list
servers.  For example, every once in a while we get spamcop reports
targeting python.org.  Why is that?  Well, we filter all email destined
to our lists through various levels of spam defenses, but crap does slip
through.  And then /we/ get flagged as the originator of the spam. 
That's just one issue related to spam we have to deal with.

> >  If the MTA could do what Mailman does here -- not creating a disk image
> >  for each instance of the message, but stitching it together in member as
> >  it's going out on the wire -- I think you'd greatly improve disk
> >  contention.
> 
> 	I'm not sure that the MTA could safely do that in memory.  At 
> least, it would be difficult to ensure that the MTA gets this done 
> right.  This would be akin to handling the entire message queue in 
> memory for all messages, something which can't really be done safely 
> except under very strict circumstances.

What I was thinking was something along the lines of storing the
template and the 'job description', a concise definition of how to get
the recipient-centric data.  The jobs would have to be small enough so
that they could be reliably dequeued, stitched and sent while still
making the guarantees an MTA has to make.  I'm just hand-waving here of
course, and the rest is left as a simple matter of engineering .

> >  In a sense, that's what we've talked about before.  If there were a
> >  standard language that the mail server and list manager could agree on
> >  for both defining the template, and defining the per-recipient data
> >  source, we could have a more efficient mechanism, with perhaps a hope of
> >  mta agnosticism.
> 
> 	That would be nice.  However, I fear that we have much more basic 
> problems that are much more serious, and which need to be resolved 
> before we can expect to start worrying about such subjects as 
> increasing efficiency in the interfaces between MLMs and MTAs.

I should mention that I'm specifically interested in increasing the
efficiency between Mailman and its local worker MTAs.  These are all
systems under my control so I should be able to tune them, set up
privileges, common data source access, etc. to make things work as
smoothly as possible until the message hits the external outgoing
interface.  After that, we have to play nice and standard.

I'm not even touching the 3rd rail of putting the MTA /in/ Mailman any
more :).

-Barry



From barry at python.org  Thu Jan 22 19:03:43 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 22 19:04:00 2004
Subject: [Mailman-Developers] images embedded in html: inline vs. attached
In-Reply-To: 
References: 
Message-ID: <1074816223.31285.186.camel@anthem>

On Sun, 2004-01-18 at 19:19, ryan wrote:

> So... my only theory is: at some cut-off point, Mailman decides that there
> are two many users on a list to embed an image in every email.  Perhaps
> because you thought it more efficient to store the image once and then
> reference it as an attachment when each member is emailed?
> 
> Or could this be some behavior of my mailer daemon, Sendmail?

I have a hard time imagining that it's Mailman.  If it is, it's a bug
I'm unaware of.  Mailman doesn't munge the message differently based on
the size of the recipient list.  In Mailman 2.1, you /can/ do content
filtering but it should work the same regardless of the number of people
receiving the message.

-Barry



From barry at python.org  Thu Jan 22 19:26:15 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 22 19:26:22 2004
Subject: [Mailman-Developers] BounceRunner.py patch
In-Reply-To: <200401081104.UAA11830@sh.iij-mc.co.jp>
References: <200401081104.UAA11830@sh.iij-mc.co.jp>
Message-ID: <1074817574.31285.188.camel@anthem>

On Thu, 2004-01-08 at 06:04, SHIGENO Kazutaka wrote:
> Hi,
> 
> Please use if the following patch is required for Mailman.

Thanks,
-Barry



From brad.knowles at skynet.be  Thu Jan 22 19:31:26 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Thu Jan 22 19:33:28 2004
Subject: [Mailman-Developers] Adding headers to mailman
 generated mails
In-Reply-To: <1074815784.31285.181.camel@anthem>
References: <00462660303848@smtp.westbay.net>	
	
	<1074811832.31285.125.camel@anthem>	
	
	<1074815784.31285.181.camel@anthem>
Message-ID: 

At 6:56 PM -0500 2004/01/22, Barry Warsaw wrote:

>  I had been thinking along the lines of the language for specifying the
>  data source as a db connection and a SQL command.

	We were working on a method to send a specially MIME-formatted 
message to identify the various potential bodyparts, a list of 
recipients and who gets which bodyparts, then a language for 
specifying the template which pulls in the appropriate bodyparts for 
the appropriate recipients (and ways to insert bits of information 
about the user themselves into the message).

>  That's certainly tricky, but I think it's got to boil down to privilege
>  or authentication.  It would still make me nervous to accept such jobs
>  from other than sites I control.

	Yup.

>                                             I'm just hand-waving here of
>  course, and the rest is left as a simple matter of engineering .

	Yeah.  Hand-waving.  Right.

>  I should mention that I'm specifically interested in increasing the
>  efficiency between Mailman and its local worker MTAs.

	The problem is that this is a very small piece of the overall 
puzzle, and you're talking about a lot of things that others might 
also want in other areas.  MTA authors are going to be primarily 
concerned about overall optimization of the MTA in general, and 
certain employers of certain authors might be concerned about things 
like WAN efficiency over other things.

	I can tell you that with some relatively minor modifications to 
sendmail, the folks at InfoBeat/Mercury Mail found that they got 
things to the point where sendmail was no longer the bottleneck -- 
pulling the data out of the database was the big problem, and much 
more difficult to solve.

>                                                         These are all
>  systems under my control so I should be able to tune them, set up
>  privileges, common data source access, etc. to make things work as
>  smoothly as possible until the message hits the external outgoing
>  interface.

	You're talking about picking up a small piece of the puzzle which 
is likely to have significant components which will be difficult or 
nearly impossible to solve.

	I think there are other areas in which I would be inclined to 
focus my attentions within Mailman, at least as far as efficiency is 
concerned.  Instead of using pickles, try Berkeley db b-trees, and 
use that as your "queue" to be processed.  The reason here is that 
b-trees are designed for lightning-fast cursor access, and all you 
need to do is make sure it's indexed on certain key fields.  Let 
Berkeley db take care of the reliability issues (what happens if 
there's a crash), efficiently caching information in memory for 
maximum performance, etc....


>  I'm not even touching the 3rd rail of putting the MTA /in/ Mailman any
>  more :).

	That solution is called L-Soft Listserv, with LSMTP.  In fact, 
you're talking about some of the same sorts of things that they do, 
only you don't want to own both pieces of code that are implementing 
the desired standard, which will make solving the problem orders of 
magnitude more difficult.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From larryt at winfirst.com  Thu Jan 22 12:40:31 2004
From: larryt at winfirst.com (larryt@winfirst.com)
Date: Thu Jan 22 20:02:11 2004
Subject: [Mailman-Developers] Two requests.  Hugh
In-Reply-To: <20040111170931.0a04b902.hesco@greens.org>
References: <20040111170931.0a04b902.hesco@greens.org>
Message-ID: 


Please, nobody help this guy until the greens promise not to campaign
for Bush again. 

-larry


From ryan at ipa.darktech.org  Thu Jan 22 21:33:15 2004
From: ryan at ipa.darktech.org (ryan)
Date: Thu Jan 22 21:33:17 2004
Subject: [Mailman-Developers] embedded html problem with large lists
Message-ID: 

No one has been able to answer this on mailman-users for over a week.  
I'm hoping one of you can help diagnose this problem.

On one of my Mailman lists, I send out messages with embedded HTML.  
These messages contain inline images.

I sent out 2 test messages to a pre-configured list.  For the two tests,
the list had only 6 staff members subscribed.  The message looked great,
everyone saw the content with the image correctly aligned in the body of
the message, between the two blocks of text where it belonged.

However, after I added the 6700+ actual email addresses to the SAME list,
and sent out the SAME email, the image was removed from the body of the
message and included as an attachment instead.  This essentially made my
pretty little email look like crap.

Again, the same list (same configuration), and the same email were used
for the successful test, and the unsuccessful real mail-out.  I cannot
stress enough that these were the *exact* same emails, for both the test
with 6 users, and the later mailing with 6700+; the filesizes were the
exact same, and the HTML was unchanged.

So... my only theory is: at some cut-off point, Mailman decides that there
are two many users on a list to embed html/images into every email.  
Perhaps because it is more efficient to store the html once and then
reference it as an attachment when each member is emailed?

Or could this be some behavior of my mailer daemon, Sendmail?

Any thoughts?

Thanks in advance,
Ryan

From brad.knowles at skynet.be  Thu Jan 22 22:16:17 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Thu Jan 22 22:24:19 2004
Subject: [Mailman-Developers] embedded html problem with large
 lists
In-Reply-To: 
References: 
Message-ID: 

At 6:33 PM -0800 2004/01/22, ryan wrote:

>  No one has been able to answer this on mailman-users for over a week.
>  I'm hoping one of you can help diagnose this problem.

	Read Barry's reply, which I believe was sent earlier today.

	In short, Mailman doesn't do anything different in this respect, 
regardless of whether there is one recipient or a million.  Something 
else must have happened.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From ryan at ipa.darktech.org  Thu Jan 22 23:49:35 2004
From: ryan at ipa.darktech.org (ryan)
Date: Thu Jan 22 23:49:36 2004
Subject: [Mailman-Developers] embedded html problem with large lists
Message-ID: 

On Fri, 23 Jan 2004, Brad Knowles wrote:

> Read Barry's reply, which I believe was sent earlier today.
> 
> In short, Mailman doesn't do anything different in this respect,
> regardless of whether there is one recipient or a million.  Something
> else must have happened.

I apologize -- I missed Barry's reply, and coincidentally didn't think my
first post to this list ever made it past the moderator.  Thank you to
Brad and Berry for replying.

Both of you suggested it was not Mailman which caused this behavior.  
>From your experience, do you think that it could be my MTA, Sendmail
8.12.6?  It is a fairly stock configuration.  I know this is a bit off
topic, but if anyone has any ideas here, I'd deeply appreciate a response
(on or off list).

Keep up the good work with Mailman, it's a great piece of software!

Thanks again,
Ryan


From pioppo at ferrara.linux.it  Fri Jan 23 05:56:41 2004
From: pioppo at ferrara.linux.it (Simone Piunno)
Date: Fri Jan 23 05:56:59 2004
Subject: [Mailman-Developers] mailman setup
In-Reply-To: <200401230003.53691.pioppo@ferrara.linux.it>
References: <200312291650.40147.rousse@ccr.jussieu.fr>
	<200401230003.53691.pioppo@ferrara.linux.it>
Message-ID: <200401231156.41654.pioppo@ferrara.linux.it>

Alle 00:03, venerd? 23 gennaio 2004, Simone Piunno ha scritto:

>    try:
>       execfile('/etc/mailman.conf')
>    except SyntaxError:
>       # I don't know how to write better info, like the offending line

OK, this is better:

    except SyntaxError:
        import sys
        et, ev, tb = sys.exc_info()
        print 'Configuration Error:', ev
        sys.exit(1)

it will print:

  Configuration Error: invalid syntax (/etc/mailman.conf, line NNN)

Cheers,
  Simone

-- 
This signature intentionally left blank



From my at freexp.de  Fri Jan 23 07:54:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Fri Jan 23 07:55:06 2004
Subject: [Mailman-Developers] prevent qp as qp routine broken
Message-ID: <91PnKWtKpwB@my.freexp.de>

Hi,

as Mailman's qp routine seems to be completely broken I'd like to know
how to prevent Mailman from qp'ing messages (for instance when sending
out subscribeack.txt) at all.

I don't see the benefit of qp'ing anyway but wouldn't mind if Mailman
would do it correctly.  But that's not the case ...

Any hint is appreciated.


        Michael

From brad.knowles at skynet.be  Fri Jan 23 11:11:41 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 23 11:34:15 2004
Subject: [Mailman-Developers] embedded html problem with large
 lists
In-Reply-To: 
References: 
Message-ID: 

At 8:49 PM -0800 2004/01/22, ryan wrote:

>  From your experience, do you think that it could be my MTA, Sendmail
>  8.12.6?  It is a fairly stock configuration.  I know this is a bit off
>  topic, but if anyone has any ideas here, I'd deeply appreciate a response
>  (on or off list).

	I'm pretty familiar with sendmail, and I can't think of anything 
it might do that could cause this sort of problem.


	In this case, don't interpret silence at ignoring you.  Instead, 
consider silence to be an acknowledgement that we have no clue as to 
what could have caused this problem, and we're not inclined to try to 
guess.

	At best, we can tell you where we think the problem is not (i.e., 
not mailman, not sendmail).  But beyond that, we have no clue.  Well, 
I have no clue -- I really shouldn't be speaking for anyone else.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From jam at jamux.com  Fri Jan 23 11:50:28 2004
From: jam at jamux.com (John A. Martin)
Date: Fri Jan 23 11:50:38 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: <1074811832.31285.125.camel@anthem> (Barry Warsaw's message of
	"Thu, 22 Jan 2004 17:50:33 -0500")
References: <00462660303848@smtp.westbay.net>
	 <1074811832.31285.125.camel@anthem>
Message-ID: <87oesur4mj.fsf@athene.jamux.com>

>>>>> "baw" == Barry Warsaw
>>>>> "RE: [Mailman-Developers] Adding headers to mailman generated mails"
>>>>>  Thu, 22 Jan 2004 17:50:33 -0500

    baw> I don't think Postfix has the same embedding capabilities,
    baw> although I haven't looked at what Postfix 2.1 may provide.

Hmm... dunno, but perhaps XFORWARD as in postfix->filter->postfix.
Found in recent snapshots and promised with improved documentation for
2.1.

        jam


        
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 154 bytes
Desc: not available
Url : http://mail.python.org/pipermail/mailman-developers/attachments/20040123/0825f4a2/attachment.bin
From somuchfun at atlantismail.com  Fri Jan 23 16:04:13 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Fri Jan 23 16:07:39 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: 
Message-ID: <21042380613285@smtp.westbay.net>

Brad,
I did some testing with a 50,000 members list and the time it takes to send
when using personalization is twice as long (8hrs instead of 4).
Of course that is bad news. I also tested some other mailing list software
like Gordano's Communicator and Lyris Listmanager and they both seem to be
able to send in batches but still add or include personalized fields to
trace the messages.
So if they can do it my simple question is why cannot mailman do it but
still send in batches?

> -----Original Message-----
> From: Brad Knowles [mailto:brad.knowles@skynet.be] 
> Sent: Thursday, January 22, 2004 3:21 PM
> To: Barry Warsaw
> Cc: Brad Knowles; Somuchfun; mailman-developers@python.org
> Subject: RE: [Mailman-Developers] Adding headers to mailman 
> generated mails
> 
> At 5:50 PM -0500 2004/01/22, Barry Warsaw wrote:
> 
> >  Nigel can provide details, but I think the same embedding 
> feature could
> >  be used to have the MTA do the final stitching of content 
> template and
> >  personalized data.  It would be A Project to hack 
> together, but I think
> >  it could be a neat idea to play with, although I'm not 
> sure how much it
> >  would help.
> 
> 	This is similar to what Eric Allman (at that time, before 
> Sendmail Inc. existed), Bryan Costales (at the time, working for 
> InfoBeat/Mercury Mail), and I (working at AOL) were discussing back 
> in 1996, in the creation of a Mail-Merge Transport Protocol (MMTP) 
> server, based on a modified version of sendmail along with a standard 
> language for transmitting that content.  With MMTP servers on both 
> ends, it would not matter how many thousands or millions of 
> recipients you might have, only one copy of the message body would be 
> transmitted, and all the rest would be filled in on the remote end.
> 
> 	We ultimately gave up on this idea because we realized that it 
> would make the spam problem much, much worse.  The same things that 
> help regular MTAs transmit millions of customized messages per hour 
> to their paying customers would probably allow spammers to transmit 
> billions of messages per hour to everyone in the universe.
> 
> 
> 	Certainly, before any serious discussion of creating something 
> like an MMTP server, and trying to make that a standard which you 
> would expect programs like sendmail, postfix, and Exim to implement, 
> I believe that the spam issue needs to be addressed.  You need to be 
> able to prove how this cannot be abused to generate spam instead.
> 
> >  If the MTA could do what Mailman does here -- not creating 
> a disk image
> >  for each instance of the message, but stitching it 
> together in member as
> >  it's going out on the wire -- I think you'd greatly improve disk
> >  contention.
> 
> 	I'm not sure that the MTA could safely do that in memory.  At 
> least, it would be difficult to ensure that the MTA gets this done 
> right.  This would be akin to handling the entire message queue in 
> memory for all messages, something which can't really be done safely 
> except under very strict circumstances.
> 
> 	The only MTA I know of that is capable of doing things 
> like this 
> is the latest release of version 8 sendmail, and even then it 
> defaults to handling messages in memory that are no larger than 4KB.
> 
> 
> 	Yes, filesystem I/O is the number one killer, specifically 
> synchronous meta-data updates.
> 
> 	But then people on this list have heard me harping on this 
> subject for a long time, and should know by now that I will refer 
> them to Nick Christenson's book _Sendmail Performance Tuning_ (see 
> ), or my own slides from 
> an invited talk entitled "Sendmail Performance Tuning for Large 
> Systems" at 
> .
> 
> >  I don't think Postfix has the same embedding capabilities, 
> although I
> >  haven't looked at what Postfix 2.1 may provide.
> 
> 	I'm not aware of anything like this, but I'd have to check.
> 
> >  In a sense, that's what we've talked about before.  If there were a
> >  standard language that the mail server and list manager 
> could agree on
> >  for both defining the template, and defining the per-recipient data
> >  source, we could have a more efficient mechanism, with 
> perhaps a hope of
> >  mta agnosticism.
> 
> 	That would be nice.  However, I fear that we have much 
> more basic 
> problems that are much more serious, and which need to be resolved 
> before we can expect to start worrying about such subjects as 
> increasing efficiency in the interfaces between MLMs and MTAs.
> 
> -- 
> Brad Knowles, 
> 
> "They that can give up essential liberty to obtain a little temporary
> safety deserve neither liberty nor safety."
>      -Benjamin Franklin, Historical Review of Pennsylvania.
> 
> GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ 
> !E-(---) W+++(--) N+
> !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) 
> X++(+++) R+(+++)
> tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- 
> r---(+++)* z(+++)
> 


From brad.knowles at skynet.be  Fri Jan 23 18:04:36 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 23 18:05:50 2004
Subject: [Mailman-Developers] Adding headers to mailman
 generated mails
In-Reply-To: <21042380613285@smtp.westbay.net>
References: <21042380613285@smtp.westbay.net>
Message-ID: 

At 1:04 PM -0800 2004/01/23, Somuchfun wrote:

>  I did some testing with a 50,000 members list and the time it takes to send
>  when using personalization is twice as long (8hrs instead of 4).

	Not surprising.  See 
.

>  Of course that is bad news. I also tested some other mailing list software
>  like Gordano's Communicator and Lyris Listmanager and they both seem to be
>  able to send in batches but still add or include personalized fields to
>  trace the messages.
>  So if they can do it my simple question is why cannot mailman do it but
>  still send in batches?

	Tell me how they do it, and I might be able to tell you what the 
issue is with Mailman doing the same.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From barry at python.org  Sat Jan 24 12:45:13 2004
From: barry at python.org (Barry Warsaw)
Date: Sat Jan 24 12:45:25 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: 
References: <00462660303848@smtp.westbay.net>
	 <1074811832.31285.125.camel@anthem>
	 <1074815784.31285.181.camel@anthem>
	
Message-ID: <1074966312.4654.19.camel@anthem>

On Thu, 2004-01-22 at 19:31, Brad Knowles wrote:
> At 6:56 PM -0500 2004/01/22, Barry Warsaw wrote:
> 
> >  I had been thinking along the lines of the language for specifying the
> >  data source as a db connection and a SQL command.
> 
> 	We were working on a method to send a specially MIME-formatted 
> message to identify the various potential bodyparts, a list of 
> recipients and who gets which bodyparts, then a language for 
> specifying the template which pulls in the appropriate bodyparts for 
> the appropriate recipients (and ways to insert bits of information 
> about the user themselves into the message).

Did you publish any of this?  I'd like to read it.

> 	I think there are other areas in which I would be inclined to 
> focus my attentions within Mailman, at least as far as efficiency is 
> concerned.  Instead of using pickles, try Berkeley db b-trees, and 
> use that as your "queue" to be processed.  The reason here is that 
> b-trees are designed for lightning-fast cursor access, and all you 
> need to do is make sure it's indexed on certain key fields.  Let 
> Berkeley db take care of the reliability issues (what happens if 
> there's a crash), efficiently caching information in memory for 
> maximum performance, etc....

Mailman 3 will definitely be database backed, via an interface that
allows different back-ends to be pluggable.  My prototypes use
BerkeleyDB through Python 2.3's standard bsddb module.

However, my experience with transactional BerkeleyDB's performance
doesn't make me confident about using it for the queue runner
subsystem.  We'll very likely stick with the file-based qrunner
architecture, although I've worked out a way to use only one file per
message.

> >  I'm not even touching the 3rd rail of putting the MTA /in/ Mailman any
> >  more :).
> 
> 	That solution is called L-Soft Listserv, with LSMTP.  In fact, 
> you're talking about some of the same sorts of things that they do, 
> only you don't want to own both pieces of code that are implementing 
> the desired standard, which will make solving the problem orders of 
> magnitude more difficult.

I know that rail is shiny, but I'm not touching it. :)

-Barry



From barry at python.org  Sat Jan 24 13:30:18 2004
From: barry at python.org (Barry Warsaw)
Date: Sat Jan 24 13:30:24 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: <21042380613285@smtp.westbay.net>
References: <21042380613285@smtp.westbay.net>
Message-ID: <1074969017.4654.71.camel@anthem>

On Fri, 2004-01-23 at 16:04, Somuchfun wrote:
> Brad,
> I did some testing with a 50,000 members list and the time it takes to send
> when using personalization is twice as long (8hrs instead of 4).

This is not surprising given Chuq's analysis.

> Of course that is bad news. I also tested some other mailing list software
> like Gordano's Communicator and Lyris Listmanager and they both seem to be
> able to send in batches but still add or include personalized fields to
> trace the messages.
> So if they can do it my simple question is why cannot mailman do it but
> still send in batches?

Maybe they do XVERP?  There's a patch of SF I haven't looked at yet that
adds XVERP support to Mailman.  But that will only help you with bounce
detection, and only then with compliant servers.  AFAIK, XVERP isn't yet
a standard.

-Barry



From barry at python.org  Sat Jan 24 13:41:33 2004
From: barry at python.org (Barry Warsaw)
Date: Sat Jan 24 13:41:40 2004
Subject: [Mailman-Developers] Adding headers to mailman generated mails
In-Reply-To: <87oesur4mj.fsf@athene.jamux.com>
References: <00462660303848@smtp.westbay.net>
	 <1074811832.31285.125.camel@anthem>
	<87oesur4mj.fsf@athene.jamux.com>
Message-ID: <1074969692.4654.81.camel@anthem>

On Fri, 2004-01-23 at 11:50, John A. Martin wrote:
> Hmm... dunno, but perhaps XFORWARD as in postfix->filter->postfix.
> Found in recent snapshots and promised with improved documentation for
> 2.1.

Interesting.  That's a possible approach, although Exim's ability to
embed an interpreter in the MTA is pretty powerful.

As an aside, looking through the Postfix 2.1 changelog, I noticed that
Errors-To support has been removed. ;)

-Barry



From barry at python.org  Sat Jan 24 13:43:04 2004
From: barry at python.org (Barry Warsaw)
Date: Sat Jan 24 13:43:09 2004
Subject: [Mailman-Developers] Mailman and CVS
In-Reply-To: <40105E79.5040201@isu.edu>
References: <40105E79.5040201@isu.edu>
Message-ID: <1074969784.4654.83.camel@anthem>

On Thu, 2004-01-22 at 18:36, Kory Wheatley wrote:
> This question is not really related to Mailman, but I'm sure with the 
> knowledge of the developers it can easily be answered about CVS.
> 
> I've been asked to setup a CVS Repository  on a dedicated machine just 
> for this purpose, of course users will need to be able to add their 
> updates over the network.
> 
> My question is where is good instructions  about using CVS (commands),  
> and  installing it on one center Repository, and setting up rights for 
> individuals.
> 
> Thank you in advance.

Google for "CVS SSH".

-Barry



From barry at python.org  Sat Jan 24 13:46:56 2004
From: barry at python.org (Barry Warsaw)
Date: Sat Jan 24 13:47:07 2004
Subject: [Mailman-Developers] embedded html problem with large lists
In-Reply-To: 
References: 
	
Message-ID: <1074970016.4654.87.camel@anthem>

On Fri, 2004-01-23 at 11:11, Brad Knowles wrote:

> 	In this case, don't interpret silence at ignoring you.  Instead, 
> consider silence to be an acknowledgement that we have no clue as to 
> what could have caused this problem, and we're not inclined to try to 
> guess.

Seconded. :)

-Barry



From barry at python.org  Sat Jan 24 13:48:36 2004
From: barry at python.org (Barry Warsaw)
Date: Sat Jan 24 13:48:41 2004
Subject: [Mailman-Developers] prevent qp as qp routine broken
In-Reply-To: <91PnKWtKpwB@my.freexp.de>
References: <91PnKWtKpwB@my.freexp.de>
Message-ID: <1074970115.4654.90.camel@anthem>

On Fri, 2004-01-23 at 07:54, Michael Heydekamp wrote:

> as Mailman's qp routine seems to be completely broken I'd like to know
> how to prevent Mailman from qp'ing messages (for instance when sending
> out subscribeack.txt) at all.
> 
> I don't see the benefit of qp'ing anyway but wouldn't mind if Mailman
> would do it correctly.  But that's not the case ...

Michael, can you please provide some detail?  Reproducible examples
would help.

-Barry



From my at freexp.de  Sat Jan 24 17:07:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Sat Jan 24 17:09:12 2004
Subject: [Mailman-Developers] prevent qp as qp routine broken
In-Reply-To: <1074970115.4654.90.camel@anthem>
References: <1074970115.4654.90.camel@anthem>
Message-ID: <91Tq41KKpwB@my.freexp.de>

Hi Barry,

Barry Warsaw  to Michael Heydekamp on 24.01.04:

> On Fri, 2004-01-23 at 07:54, Michael Heydekamp wrote:

>> as Mailman's qp routine seems to be completely broken I'd like to
>> know how to prevent Mailman from qp'ing messages (for instance when
>> sending out subscribeack.txt) at all.

>> I don't see the benefit of qp'ing anyway but wouldn't mind if
>> Mailman would do it correctly.  But that's not the case ...

> Michael, can you please provide some detail?

Sure, but what about an answer to my question how to prevent qp'ing? ;-)

> Reproducible examples would help.

Simple - say we have the following lines in subscribeack.txt (German
text follows):

----------8<----------
1) Was Mailinglisten sind und wie man sie am besten und komfortabelsten
   in XP einrichtet und handhabt, ist ausf?hrlich in der Online-Hilfe
   [...]
----------8<----------


When receiving the greeting message after subscribing to the list, the
users see the following in their MUAs instead:

----------8<----------
1) Was Mailinglisten sind und wie man sie am besten und
komfortabelsten
   in XP einrichtet und handhabt, ist ausf?hrlich in der Online-Hilfe
   [...]
----------8<----------


Ugly, right?


There are two reasons for this:

1. Unnecessarily, Mailman wraps lines that are not longer than 76
   characters (note that the line in question is just 71 characters long
   and therefore Mailman wouldn't even need to touch it).

2. When Mailman wraps, it simply "forgets" to use soft line breaks which
   would have to look like this:

----------8<----------
1) Was Mailinglisten sind und wie man sie am besten und =
komfortabelsten
   in XP einrichtet und handhabt, ist ausf=FChrlich in der Online-Hilfe
   [...]
----------8<----------

Please note the "=" at the end of the first line.  Only then every
qp-aware MUA would be able to restore the original line.

The second reason is the more important one.  Mailman may wrap even at
pos 40 if it wants to but it *has* to use soft line breaks then.  For
more details see 6.7. of RFC2045.

Furthermore I'm not sure if Mailman calculates the line length correctly
at all (would have to run another test to be sure).  My impression is
that characters are counted before being qp'd (because if they would be
counted after being qp'd, also the line starting with "in XP einrichtet"
should be wrapped, given that the same logic is applied to the line).

Anyway, I see no reason for qp'ing at all and to avoid the ugly output
above, I'd like to get rid of it.

Just for demonstration purposes, I'm writing this last paragraph of the message in one long line and send it out qp'd. Look at the raw format to see how qp should work.  You MUA should restore it to one long line (and afterwards internally wrap it again for display purposes).


        Michael

From pioppo at ferrara.linux.it  Sun Jan 25 08:43:19 2004
From: pioppo at ferrara.linux.it (Simone Piunno)
Date: Sun Jan 25 08:36:21 2004
Subject: [Mailman-Developers] prevent qp as qp routine broken
In-Reply-To: <91Tq41KKpwB@my.freexp.de>
References: <1074970115.4654.90.camel@anthem> <91Tq41KKpwB@my.freexp.de>
Message-ID: <200401251443.19357.pioppo@ferrara.linux.it>

On Saturday 24 January 2004 23:07, Michael Heydekamp wrote:

> Simple - say we have the following lines in subscribeack.txt (German
> text follows):
>
> ----------8<----------
> 1) Was Mailinglisten sind und wie man sie am besten und komfortabelsten
>    in XP einrichtet und handhabt, ist ausf?hrlich in der Online-Hilfe
>    [...]
> ----------8<----------

Michael, what Mailman are you referring to?
There's no such text in the german translation for 2.1...
neither in subscribeack.txt, nor anywhere else.


From my at freexp.de  Sun Jan 25 09:45:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Sun Jan 25 10:03:59 2004
Subject: [Mailman-Developers] prevent qp as qp routine broken
In-Reply-To: <200401251443.19357.pioppo@ferrara.linux.it>
References: <200401251443.19357.pioppo@ferrara.linux.it>
Message-ID: <91Xr69N4pwB@my.freexp.de>

Hi Simone,

Simone Piunno  to Michael Heydekamp on 25.01.04:

> On Saturday 24 January 2004 23:07, Michael Heydekamp wrote:

>> Simple - say we have the following lines in subscribeack.txt (German
>> text follows):

>> ----------8<----------
>> 1) Was Mailinglisten sind und wie man sie am besten und
>>    komfortabelsten in XP einrichtet und handhabt, ist ausf?hrlich in
>>    der Online-Hilfe [...]
>> ----------8<----------

> Michael, what Mailman are you referring to?

2.1.3

> There's no such text in the german translation for 2.1...
> neither in subscribeack.txt, nor anywhere else.

Of course not, this is a list specific subscribeack.txt that I created
for each of our lists.

But what the heck :) has the (arbitrary) text itself to do with the fact
that Mailman creates a technically wrong and therefore ugly qp'd output
upon sending such texts?  It would do so with the original
subscribeack.txt in exactly the same manner.

Hey - anybody around who wouldn't mind to answer my original question?
:)  I'm asking questions and what I'm getting are more questions...  I'm
happy to answer them but at some stage an answer to my question would
also be nice, so:

At this stage I just need to know how to prevent Mailman from qp'ing.

Simone, there is BTW still another open question in
<91HicwPppwB@my.freexp.de> (how to make fuzzy translations non-fuzzy).

Well, there are even more open questions that I raised here, but as
nobody came up with useful answers, I have almost solved these issues on
INN or MTA level in the meantime.


        Michael

From pioppo at ferrara.linux.it  Sun Jan 25 11:50:01 2004
From: pioppo at ferrara.linux.it (Simone Piunno)
Date: Sun Jan 25 11:43:04 2004
Subject: [Mailman-Developers] prevent qp as qp routine broken
In-Reply-To: <91Xr69N4pwB@my.freexp.de>
References: <200401251443.19357.pioppo@ferrara.linux.it>
	<91Xr69N4pwB@my.freexp.de>
Message-ID: <200401251750.01909.pioppo@ferrara.linux.it>

On Sunday 25 January 2004 15:45, Michael Heydekamp wrote:

> > There's no such text in the german translation for 2.1...
> > neither in subscribeack.txt, nor anywhere else.
>
> Of course not, this is a list specific subscribeack.txt that I created
> for each of our lists.

Ah ok, maybe we're getting closer to the solution...

> But what the heck :) has the (arbitrary) text itself to do with the fact
> that Mailman creates a technically wrong and therefore ugly qp'd output
> upon sending such texts?  It would do so with the original
> subscribeack.txt in exactly the same manner.

That text is not completely arbitrary: there's a little structure in those 
apparently flat files!
The rationale is that, after variable interpolation, the text could become 
more ugly than you reported, because we don't know in advance how many chars 
will be used for each interpolation placeholder.
For this reason, after interpolation in Utils.maketext() the resulting string 
is filtered by Utils.wrap(), which basically reflows all not-indented 
paragraphs.
Probably you wrote your templates without taking care of these rules, and get 
ugly results.

If you want to completely disable the re-flowing logic, you should add a 
"raw=True" parameter to Utils.maketext() calls (e.g. at line 58 of 
Deliverer.py), or just change the defaults (at line 384 for Utils.py), by I 
bet after some experimentation you'll prefer to rollback ;)

> Hey - anybody around who wouldn't mind to answer my original question?
> :)  I'm asking questions and what I'm getting are more questions...  I'm
> happy to answer them but at some stage an answer to my question would
> also be nice, so:
> At this stage I just need to know how to prevent Mailman from qp'ing.

I'm not responding to this because I don't know.
I've insisted on the thread above because I think that's your real problem.

> Simone, there is BTW still another open question in
> <91HicwPppwB@my.freexp.de> (how to make fuzzy translations non-fuzzy).

You simply fix that translation string (if needed) and remove the fuzzy line.
Translation tools (such as kbabel) will do this automatically as soon as you 
touch that string.



From my at freexp.de  Sun Jan 25 12:26:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Sun Jan 25 12:31:02 2004
Subject: [Mailman-Developers] prevent qp as qp routine broken
In-Reply-To: <200401251750.01909.pioppo@ferrara.linux.it>
References: <200401251750.01909.pioppo@ferrara.linux.it>
Message-ID: <91XrenJppwB@my.freexp.de>

Hi Simone,

Simone Piunno  to Michael Heydekamp on
25.01.04:

> On Sunday 25 January 2004 15:45, Michael Heydekamp wrote:

>>> There's no such text in the german translation for 2.1...
>>> neither in subscribeack.txt, nor anywhere else.

>> Of course not, this is a list specific subscribeack.txt that I
>> created for each of our lists.

> Ah ok, maybe we're getting closer to the solution...

Let's see...

>> But what the heck :) has the (arbitrary) text itself to do with the
>> fact that Mailman creates a technically wrong and therefore ugly
>> qp'd output upon sending such texts?  It would do so with the
>> original subscribeack.txt in exactly the same manner.

> That text is not completely arbitrary: there's a little structure in
> those apparently flat files!
> The rationale is that, after variable interpolation, the text could
> become more ugly than you reported, because we don't know in advance
> how many chars will be used for each interpolation placeholder.

The text that I quoted did not contain any variables, Simone.

> For this reason, after interpolation in Utils.maketext() the
> resulting string is filtered by Utils.wrap(), which basically reflows
> all not-indented paragraphs.

This *is* an indented paragraph (with a negative indention in the first
line).  Quite common thing...

And the wrapping is done at a fixed position of ...?

I also had cases where an empty line was inserted in the middle of a
paragraph for no reason (for instance in the result of a 'help'
request).

> Probably you wrote your templates without taking care of these rules,
> and get ugly results.

Are these rules documented somewhere?

Anyway, when writing these texts, I was of course bearing in mind
possible variable interpolation and made sure that lines with variables
cannot exceed a reasonable line length.

> If you want to completely disable the re-flowing logic, you should
> add a "raw=True" parameter to Utils.maketext() calls (e.g. at line 58
> of Deliverer.py), or just change the defaults (at line 384 for
> Utils.py),

Ah, I'll try that.  Thanks!

> by I bet after some experimentation you'll prefer to rollback ;)

Never ever. :)  I hate when software is messing up my carefully prepared
text files without asking me.

>> Simone, there is BTW still another open question in
>> <91HicwPppwB@my.freexp.de> (how to make fuzzy translations
>> non-fuzzy).

> You simply fix that translation string (if needed) and remove the
> fuzzy line.

But as I told you already in the message above:

----------8<----------
> But as I said: Even if I remove the tag and recompile the file, the
> translation is still not used.  What's wrong?
----------8<----------

I thought it might have to do with the fact that the English text
additionally appears in line 17 of cmd_confirm.py, but I'm not sure
what's this for?

I also don't know if the line numbers in the language file do have any
effect (apart from being informational), but we can discuss this in the
appropriate mailing list as soon as I have accomplished some Exim stuff
here...


        Michael

From brad.knowles at skynet.be  Sun Jan 25 15:27:16 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Sun Jan 25 16:57:15 2004
Subject: [Mailman-Developers] Adding headers to mailman
 generated mails
In-Reply-To: <1074966312.4654.19.camel@anthem>
References: <00462660303848@smtp.westbay.net>	
	
	<1074811832.31285.125.camel@anthem>	
	
	<1074815784.31285.181.camel@anthem>	
	
	<1074966312.4654.19.camel@anthem>
Message-ID: 

At 12:45 PM -0500 2004/01/24, Barry Warsaw wrote:

>  Did you publish any of this?  I'd like to read it.

	Nope.  All private conversations with Eric and Bryan. 
Unfortunately, I've lost all mailboxes I had while I was at AOL, so I 
can't pull those messages back up.

	Sorry, guy.  ;-(

>  Mailman 3 will definitely be database backed, via an interface that
>  allows different back-ends to be pluggable.  My prototypes use
>  BerkeleyDB through Python 2.3's standard bsddb module.

	Cool.

>  However, my experience with transactional BerkeleyDB's performance
>  doesn't make me confident about using it for the queue runner
>  subsystem.  We'll very likely stick with the file-based qrunner
>  architecture, although I've worked out a way to use only one file per
>  message.

	My experience with Berkeley DB has been that you store the actual 
content in files, and you put meta-data in the database (with a field 
that tells you the full path to the file).  The file is touched only 
once in creation, read one or more times (on message delivery), and 
then deleted when all copies have been delivered.  All other activity 
occurs within the database.

	Used that way, it's blindingly fast, unbreakable, and amazingly 
efficient with memory.  However, I'm not convinced that using a 
standard Python access module is the way to get the best out of it -- 
I don't know how reliable that module is, and it could be a 
significant drain on the capabilities of Berkeley DB itself.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From somuchfun at atlantismail.com  Tue Jan 27 07:39:24 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Tue Jan 27 07:41:35 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
Message-ID: <12394579527884@smtp.westbay.net>

I have a question in regards to mailman's recovery abilities.
Let's say mailman is running and sending out messages to a large list and
the machine crashes or is rebooted, does mailman pickup where it stopped? Or
is the run gone forever?


From les at 2pi.org  Tue Jan 27 13:26:26 2004
From: les at 2pi.org (Les Niles)
Date: Tue Jan 27 13:26:39 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <12394579527884@smtp.westbay.net> (somuchfun@atlantismail.com)
References: <12394579527884@smtp.westbay.net>
Message-ID: <200401271826.i0RIQQ1H004658@grumman.kjsl.com>

On Tue, 27 Jan 2004 04:39:24 -0800 "Somuchfun"  wrote:
>I have a question in regards to mailman's recovery abilities.
>Let's say mailman is running and sending out messages to a large list and
>the machine crashes or is rebooted, does mailman pickup where it stopped? Or
>is the run gone forever?

I've had way too much experience with this lately... :(

Mailman soldiers on just fine.  The problems we've run into are
with some of mailman's files getting corrupted because they weren't
synced to disk when the machine crashed -- the standard problem
with any crash.  I've had to rebuild a couple of list config
databases, and toss out a few corrupted pending pickles.  We plan
to try the synchronous-write option, do backups (!), and maybe even
replace the flakey hardware that's been causing the crashes.

  -les

From craig.elkins at verizon.net  Tue Jan 27 13:36:30 2004
From: craig.elkins at verizon.net (Craig Elkins)
Date: Tue Jan 27 13:37:14 2004
Subject: [Mailman-Developers] PLEASE HELP!!!!! need to customize subject
	line in new subscribers welcome message! in english (en?)
Message-ID: <003201c3e504$7bc45480$0affff0a@computer>

PLEASE HELP!!!!! need to customize subject line in new subscribers welcome message! in english (en?)

i can't find the mailman.po english (US) version - i would assume the directory would be called "en" and it would be in messages - but I don't see it.  please tell me what i am missing here - my client is about to shoot me!!!!!!!!  also, after i change the message what else to i need to do?

thanks!

craig

From jdennis at redhat.com  Tue Jan 27 13:40:04 2004
From: jdennis at redhat.com (John Dennis)
Date: Tue Jan 27 13:40:50 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <200401271826.i0RIQQ1H004658@grumman.kjsl.com>
References: <12394579527884@smtp.westbay.net>
	<200401271826.i0RIQQ1H004658@grumman.kjsl.com>
Message-ID: <1075228804.27966.34.camel@finch.boston.redhat.com>

On Tue, 2004-01-27 at 13:26, Les Niles wrote:
> On Tue, 27 Jan 2004 04:39:24 -0800 "Somuchfun"  wrote:
> >I have a question in regards to mailman's recovery abilities.
> >Let's say mailman is running and sending out messages to a large list and
> >the machine crashes or is rebooted, does mailman pickup where it stopped? Or
> >is the run gone forever?
> 
> I've had way too much experience with this lately... :(
> 
> Mailman soldiers on just fine.  The problems we've run into are
> with some of mailman's files getting corrupted because they weren't
> synced to disk when the machine crashed -- the standard problem
> with any crash.  I've had to rebuild a couple of list config
> databases, and toss out a few corrupted pending pickles.  We plan
> to try the synchronous-write option, do backups (!), and maybe even
> replace the flakey hardware that's been causing the crashes.
> 

Doesn't the new SYNC_AFTER_WRITE flag address this issue? Here is the doc
for it:

# This flag causes Mailman to fsync() its data files after writing and
# flushing its contents.  While this ensures the data is written to disk,
# avoiding data loss, it may be a performance killer.  Note that this flag
# affects both message pickles and MailList config.pck files.



From brad.knowles at skynet.be  Tue Jan 27 14:06:38 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Tue Jan 27 14:07:25 2004
Subject: [Mailman-Developers] PLEASE HELP!!!!! need to
	customize subject 	line in new subscribers welcome message! in
	english (en?)
In-Reply-To: <003201c3e504$7bc45480$0affff0a@computer>
References: <003201c3e504$7bc45480$0affff0a@computer>
Message-ID: 

At 1:36 PM -0500 2004/01/27, Craig Elkins wrote:

>  i can't find the mailman.po english (US) version - i would assume the
>  directory would be called "en" and it would be in messages - but I
>  don't see it.  please tell me what i am missing here - my client is
>  about to shoot me!!!!!!!!

	The templates are in ~mailman/templates, and there is an en/ 
subdirectory under that (among all the other languages known).  If 
you want to make changes to the templates used for a supported 
language for a particular list, you need to copy the appropriate 
templates to a subdirectory by the same name under 
~mailman/lists/listname (e.g., ~mailman/lists/listname/en), and then 
make changes from there.

	If you're looking for a particular file (e.g., mailman.po), you 
can always do a "find ~mailman -name mailman.po -print", which on my 
system prints the following:

% find . -name mailman.po -print
find: ./archives/private: Permission denied
./messages/ca/LC_MESSAGES/mailman.po
./messages/cs/LC_MESSAGES/mailman.po
./messages/da/LC_MESSAGES/mailman.po
./messages/de/LC_MESSAGES/mailman.po
./messages/es/LC_MESSAGES/mailman.po
./messages/et/LC_MESSAGES/mailman.po
./messages/eu/LC_MESSAGES/mailman.po
./messages/fi/LC_MESSAGES/mailman.po
./messages/fr/LC_MESSAGES/mailman.po
./messages/hr/LC_MESSAGES/mailman.po
./messages/hu/LC_MESSAGES/mailman.po
./messages/it/LC_MESSAGES/mailman.po
./messages/ja/LC_MESSAGES/mailman.po
./messages/ko/LC_MESSAGES/mailman.po
./messages/lt/LC_MESSAGES/mailman.po
./messages/nl/LC_MESSAGES/mailman.po
./messages/no/LC_MESSAGES/mailman.po
./messages/pl/LC_MESSAGES/mailman.po
./messages/pt/LC_MESSAGES/mailman.po
./messages/pt_BR/LC_MESSAGES/mailman.po
./messages/ro/LC_MESSAGES/mailman.po
./messages/ru/LC_MESSAGES/mailman.po
./messages/sl/LC_MESSAGES/mailman.po
./messages/sr/LC_MESSAGES/mailman.po
./messages/sv/LC_MESSAGES/mailman.po
./messages/uk/LC_MESSAGES/mailman.po


	Note that there is no .../en/.../mailman.po file, which is a 
compiled binary python executable (so far as I can tell).  This is 
not a file that should be edited.  Try editing the templates instead.

>                             also, after i change the message what
>  else to i need to do?

	I'm not sure, but you may need to stop and restart mailman.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Tue Jan 27 14:10:12 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Tue Jan 27 14:19:50 2004
Subject: [Mailman-Developers] What happens with mailman after
 a crash
In-Reply-To: <200401271826.i0RIQQ1H004658@grumman.kjsl.com>
References: <12394579527884@smtp.westbay.net>
	<200401271826.i0RIQQ1H004658@grumman.kjsl.com>
Message-ID: 

At 10:26 AM -0800 2004/01/27, Les Niles wrote:

>                   I've had to rebuild a couple of list config
>  databases, and toss out a few corrupted pending pickles.  We plan
>  to try the synchronous-write option, do backups (!), and maybe even
>  replace the flakey hardware that's been causing the crashes.

	You could also change the filesystem where the mailman queue 
directory is located, so as to remove any asynchronous or delayed 
writes that are used by the filesystem, above and beyond any mailman 
SYNC_AFTER_WRITE feature that may or may not be enabled.  You could 
alternatively run on a journaling filesystem, which should also help.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From jeffw at chebucto.ns.ca  Tue Jan 27 14:29:04 2004
From: jeffw at chebucto.ns.ca (Jeff Warnica)
Date: Tue Jan 27 14:26:31 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <1075228804.27966.34.camel@finch.boston.redhat.com>
References: <12394579527884@smtp.westbay.net>
	<200401271826.i0RIQQ1H004658@grumman.kjsl.com>
	<1075228804.27966.34.camel@finch.boston.redhat.com>
Message-ID: <20040127152904.z5lvok84wkk4os88@george.warnica>


I just happen to remember a note from the Cyrus-IMAP docs that may help you, if
you happen to be running linux with ext2. Quote: "LINUX SYSTEMS USING EXT2FS
ONLY: Set the user, quota, and partition directories to update synchronously."

http://asg.web.cmu.edu/cyrus/download/imapd/install-configure.html

Quoting John Dennis :

> On Tue, 2004-01-27 at 13:26, Les Niles wrote:
>> On Tue, 27 Jan 2004 04:39:24 -0800 "Somuchfun" 
>>  wrote:
>> >I have a question in regards to mailman's recovery abilities.
>> >Let's say mailman is running and sending out messages to a large list and
>> >the machine crashes or is rebooted, does mailman pickup where it 
>> stopped? Or
>> >is the run gone forever?
>>
>> I've had way too much experience with this lately... :(
>>
>> Mailman soldiers on just fine.  The problems we've run into are
>> with some of mailman's files getting corrupted because they weren't
>> synced to disk when the machine crashed -- the standard problem
>> with any crash.  I've had to rebuild a couple of list config
>> databases, and toss out a few corrupted pending pickles.  We plan
>> to try the synchronous-write option, do backups (!), and maybe even
>> replace the flakey hardware that's been causing the crashes.
>>
>
> Doesn't the new SYNC_AFTER_WRITE flag address this issue? Here is the doc
> for it:
>

From rmg at terc.edu  Tue Jan 27 15:49:03 2004
From: rmg at terc.edu (Robby Griffin)
Date: Tue Jan 27 15:49:08 2004
Subject: [Mailman-Developers] PLEASE HELP!!!!! need to customize subject
	line in new subscribers welcome message! in english (en?)
In-Reply-To: <003201c3e504$7bc45480$0affff0a@computer>
Message-ID: <3DBAA32C-510A-11D8-9783-000A95A0C58E@terc.edu>

On Tuesday, Jan 27, 2004, at 13:36 US/Eastern, Craig Elkins wrote:

> PLEASE HELP!!!!! need to customize subject line in new subscribers 
> welcome message! in english (en?)
>
> i can't find the mailman.po english (US) version - i would assume the 
> directory would be called "en" and it would be in messages - but I 
> don't see it.  please tell me what i am missing here - my client is 
> about to shoot me!!!!!!!!  also, after i change the message what else 
> to i need to do?

The .po files are translations; there don't exist English translations 
because the English is the original. You can find the welcome message 
subject in Deliverer.py to make a quick-and-dirty change.

I am not an i18n expert though, so I'm not exactly sure what happens to 
translations when you change one of the original strings. My intuition 
is that the new string would be "untranslatable" using existing 
translations, so you'd always get the English version. If your client 
wishes to have properly translated welcome message subjects in other 
languages, you would probably need to update messages/mailman.pot and 
each of the translations. Also check README-I18N.en and the 
mailman-i18n list.

	--Robby


From les at 2pi.org  Tue Jan 27 17:15:04 2004
From: les at 2pi.org (Les Niles)
Date: Tue Jan 27 17:15:30 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <1075228804.27966.34.camel@finch.boston.redhat.com> (message from
	John Dennis on 27 Jan 2004 13:40:04 -0500)
References: <12394579527884@smtp.westbay.net>
	<200401271826.i0RIQQ1H004658@grumman.kjsl.com>
	<1075228804.27966.34.camel@finch.boston.redhat.com>
Message-ID: <200401272215.i0RMF48m012588@grumman.kjsl.com>

On 27 Jan 2004 13:40:04 -0500 John Dennis  wrote:
>On Tue, 2004-01-27 at 13:26, Les Niles wrote:
>> Mailman soldiers on just fine.  The problems we've run into are
>> with some of mailman's files getting corrupted because they weren't
>> synced to disk when the machine crashed -- the standard problem
>> with any crash.  I've had to rebuild a couple of list config
>> databases, and toss out a few corrupted pending pickles.  We plan
>> to try the synchronous-write option, do backups (!), and maybe even
>> replace the flakey hardware that's been causing the crashes.
>> 
>
>Doesn't the new SYNC_AFTER_WRITE flag address this issue? Here is the doc
>for it:
>
># This flag causes Mailman to fsync() its data files after writing and
># flushing its contents.  While this ensures the data is written to disk,
># avoiding data loss, it may be a performance killer.  Note that this flag
># affects both message pickles and MailList config.pck files.

Yes, that's what I meant by "synchronous-write option."  We haven't
turned it on yet, mostly because of the warning about performance,
but partially out of laziness since we need to upgrade MM yet
again.

  -les

From somuchfun at atlantismail.com  Tue Jan 27 17:32:02 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Tue Jan 27 17:32:27 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: 
Message-ID: <22322451535767@smtp.westbay.net>

So what happens when the server is normally rebooted? Does mailman remember
where it stopped? Does it start all over again? Or is the mailing just gone?

> -----Original Message-----
> From: 
> mailman-developers-bounces+somuchfun=atlantismail.com@python.o
> rg 
> [mailto:mailman-developers-bounces+somuchfun=atlantismail.com@
> python.org] On Behalf Of Brad Knowles
> Sent: Tuesday, January 27, 2004 11:10 AM
> To: les@2pi.org
> Cc: mailman-developers@python.org
> Subject: Re: [Mailman-Developers] What happens with mailman 
> after a crash
> 
> At 10:26 AM -0800 2004/01/27, Les Niles wrote:
> 
> >                   I've had to rebuild a couple of list config
> >  databases, and toss out a few corrupted pending pickles.  We plan
> >  to try the synchronous-write option, do backups (!), and maybe even
> >  replace the flakey hardware that's been causing the crashes.
> 
> 	You could also change the filesystem where the mailman queue 
> directory is located, so as to remove any asynchronous or delayed 
> writes that are used by the filesystem, above and beyond any mailman 
> SYNC_AFTER_WRITE feature that may or may not be enabled.  You could 
> alternatively run on a journaling filesystem, which should also help.
> 
> -- 
> Brad Knowles, 
> 
> "They that can give up essential liberty to obtain a little temporary
> safety deserve neither liberty nor safety."
>      -Benjamin Franklin, Historical Review of Pennsylvania.
> 
> GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ 
> !E-(---) W+++(--) N+
> !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) 
> X++(+++) R+(+++)
> tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- 
> r---(+++)* z(+++)
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers@python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> 


From craig.elkins at verizon.net  Wed Jan 28 11:48:13 2004
From: craig.elkins at verizon.net (Craig Elkins)
Date: Wed Jan 28 12:00:46 2004
Subject: [Mailman-Developers] re: change password: automatic email
	confirmation
Message-ID: <00ab01c3e5be$85739f80$0affff0a@computer>

Hi,

Is it possible for me to configure mailman so when a member changes their password it is automatically emailed to them? currently i can change it but i don't get a confirmation email unless i click the forgot password button.

thanks,
craig
From somuchfun at atlantismail.com  Wed Jan 28 16:33:51 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Wed Jan 28 16:34:03 2004
Subject: [Mailman-Developers] Mailman unsubscribe mechanism unclear
Message-ID: <21335446147366@smtp.westbay.net>

Hello,
Can someone explain the exact mechanism mailman uses for unsubscriptions? I
see some of them are done without a confirmation and some members get an
email with a token asked to return it to confirm their unsubscription.
I have played around with mailman but are not able to figure out how it
determines when a confirmation is needed and when not.
Hostile unsubscriptions are a growing problem and I would need a way to make
sure all unsubscriptions need to be confirmed.

Thanks as always for the great help!


From tomc at future-i.com  Wed Jan 28 17:30:13 2004
From: tomc at future-i.com (Tom McAdam)
Date: Wed Jan 28 17:30:18 2004
Subject: [Mailman-Developers] Patch submitted: Information leak when issuing
	a subscribe request
Message-ID: 

Hi,

I have submitted a patch (id 885098) for an information leak of
subscription status for an address by sending a subscribe request via
email.

Thanks are also due to Ashok Argent-Katwala for his assistance in 
identifying this problem.

Tom McAdam



From jimmy-ml at nccom.com  Wed Jan 28 19:01:05 2004
From: jimmy-ml at nccom.com (Jim Gottlieb)
Date: Wed Jan 28 19:04:28 2004
Subject: [Mailman-Developers] one-click unsubscribe
Message-ID: <20040129000105.GX27962@nccom.com>

Is anyone working on extending VERP-like features to allow for true
one-click unsubscribe?

Right now, it's at least three-click, plus waiting for the confirmation
to arrive.  We'd really like to include a link in each message that
will allow them to very easily get off our list.

On a related note, if we were to send HTML email (I know, I know), will
variables be expanded in the text we submit so that we could create
links that will be different for each user?  So that, say with the
current capabilities, we could have something like

Click
here to manage your subscription


From barry at python.org  Thu Jan 29 00:03:33 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 29 00:03:40 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <12394579527884@smtp.westbay.net>
References: <12394579527884@smtp.westbay.net>
Message-ID: <1075352612.23368.23.camel@anthem>

On Tue, 2004-01-27 at 07:39, Somuchfun wrote:
> I have a question in regards to mailman's recovery abilities.
> Let's say mailman is running and sending out messages to a large list and
> the machine crashes or is rebooted, does mailman pickup where it stopped? Or
> is the run gone forever?

If Mailman is in the middle of delivering a message and is killed
uncleanly, e.g. Python crashes, or the machine hard panics, then the
current run is lost.  If Mailman is stopped cleanly via 'mailmanctl
stop', then it's current place is remembered and resumed on restart.

I'd like to do better than this, but I think it's infeasible with the
current qrunner architecture, since the Switchboard removes the files
when they are dequeued for processing.  It seems to me the alternatives
are to either risk duplicate deliveries for some subset of recipients,
or really clobber performance by writing status information out after
each successful recipient delivery.

Of course, you'd hope that the window of opportunity for message loss in
Mailman is small, if it can hand off all the recipient chunks to the MTA
quickly.  Then the mail server's guarantees take over.

-Barry



From barry at python.org  Thu Jan 29 00:11:31 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 29 00:11:39 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <1075228804.27966.34.camel@finch.boston.redhat.com>
References: <12394579527884@smtp.westbay.net>
	<200401271826.i0RIQQ1H004658@grumman.kjsl.com>
	<1075228804.27966.34.camel@finch.boston.redhat.com>
Message-ID: <1075353091.23368.33.camel@anthem>

On Tue, 2004-01-27 at 13:40, John Dennis wrote:

> Doesn't the new SYNC_AFTER_WRITE flag address this issue? Here is the doc
> for it:
> 
> # This flag causes Mailman to fsync() its data files after writing and
> # flushing its contents.  While this ensures the data is written to disk,
> # avoiding data loss, it may be a performance killer.  Note that this flag
> # affects both message pickles and MailList config.pck files.

Note that this warning /may/ be superstition.  When I did some tests a
long while ago, I saw something like a 95% hit in performance on
ext3/RH9.  Since then, I've been told that others have seen much less of
a performance hit, and I've also heard that RH9 is particularly prone to
performance problems when under heavy I/O.

It would be nice for folks out there to enable SYNC_AFTER_WRITE on heavy
traffic sites and report back on performance.  Maybe we should enable
this option by default.

Also, I now know how to cut the number of files created and unlinked by
Mailman in half.  Currently, the qrunners create a .msg and .db file for
every message in the queues.  I can collapse that to one file, and I
think I can do this while still maintaining the Python 2.1 compatibility
requirement.  I think the upgrade procedure will be fairly
straightforward, so I'm seriously considering implementing this for
Mailman 2.1.5.  It's an important change, but it's mostly internal and I
think it would be a big enough win to slip it into a bug fix release. 
There are other advantages, such as getting rid of those pesky "lost
data files for filebase" messages.

-Barry



From barry at python.org  Thu Jan 29 00:13:24 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 29 00:13:32 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <22322451535767@smtp.westbay.net>
References: <22322451535767@smtp.westbay.net>
Message-ID: <1075353204.23368.37.camel@anthem>

On Tue, 2004-01-27 at 17:32, Somuchfun wrote:
> So what happens when the server is normally rebooted? Does mailman remember
> where it stopped? Does it start all over again? Or is the mailing just gone?

If you run "mailmanctl stop" first, as should happen when you change run
levels if you've installed the mailman init script, the qrunners will
get a Python exception, which they'll catch.  That should cause them to
re-queue the files and restart where they left off.

-Barry



From bob at nleaudio.com  Thu Jan 29 00:15:07 2004
From: bob at nleaudio.com (Bob Puff@NLE)
Date: Thu Jan 29 00:21:45 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <1075353204.23368.37.camel@anthem>
References: <22322451535767@smtp.westbay.net>
	<1075353204.23368.37.camel@anthem>
Message-ID: <401896DB.2060807@nleaudio.com>

I don't know about your system, but at least on mine, Mailman/Python are only working for a few 
seconds.  The rest of the time the MTA is busy sending out all the mail, and -that- is where you 
hope no problems are when you reboot.  Generally, the current MTAs are pretty good at handling this.

Bob

Barry Warsaw wrote:
> On Tue, 2004-01-27 at 17:32, Somuchfun wrote:
> 
>>So what happens when the server is normally rebooted? Does mailman remember
>>where it stopped? Does it start all over again? Or is the mailing just gone?
> 
> 
> If you run "mailmanctl stop" first, as should happen when you change run
> levels if you've installed the mailman init script, the qrunners will
> get a Python exception, which they'll catch.  That should cause them to
> re-queue the files and restart where they left off.
> 
> -Barry
> 
> 
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers@python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> 
> .
> 


From barry at python.org  Thu Jan 29 00:23:05 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 29 00:23:13 2004
Subject: [Mailman-Developers] one-click unsubscribe
In-Reply-To: <20040129000105.GX27962@nccom.com>
References: <20040129000105.GX27962@nccom.com>
Message-ID: <1075353784.23368.47.camel@anthem>

On Wed, 2004-01-28 at 19:01, Jim Gottlieb wrote:
> Is anyone working on extending VERP-like features to allow for true
> one-click unsubscribe?

When you say "VERP-like" I think you mean more along the lines of a
mail-merge or content personalization feature.  I try to use the term
"VERP" to mean header, and particularly envelope sender techniques.

> Right now, it's at least three-click, plus waiting for the confirmation
> to arrive.  We'd really like to include a link in each message that
> will allow them to very easily get off our list.

You have to do this in an unguessable way, otherwise, attackers can
simply unsubscribe anybody they want from a mailing list.

> On a related note, if we were to send HTML email (I know, I know), will
> variables be expanded in the text we submit so that we could create
> links that will be different for each user?  So that, say with the
> current capabilities, we could have something like
> 
> Click
> here to manage your subscription

Not in Mailman 2.1.  I know how to do this, and have proof-of-concept
code working for Mailman 3.  It requires Python 2.3 features, and will
look something like this:

http://lists.example.com/mailman/unsub/${mm:unsub_token}

I believe I have a way to limit, based on list configuration, such
"$mm:" substitutions to the headers and footers, or to allow such
substitutions to occur anywhere in the message text (headers, body, and
MIME attachments included).  Such substitutions would run at about the
speed of Python's string interpolation operator (read: much faster than
what Mailman 2.1 does).

-Barry



From barry at python.org  Thu Jan 29 00:24:13 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Jan 29 00:24:19 2004
Subject: [Mailman-Developers] What happens with mailman after a crash
In-Reply-To: <401896DB.2060807@nleaudio.com>
References: <22322451535767@smtp.westbay.net>
	<1075353204.23368.37.camel@anthem>  <401896DB.2060807@nleaudio.com>
Message-ID: <1075353852.23368.49.camel@anthem>

On Thu, 2004-01-29 at 00:15, Bob Puff@NLE wrote:
> I don't know about your system, but at least on mine, Mailman/Python are only working for a few 
> seconds.  The rest of the time the MTA is busy sending out all the mail, and -that- is where you 
> hope no problems are when you reboot.  Generally, the current MTAs are pretty good at handling this.

Yep.

-Barry



From jimmy-ml at nccom.com  Thu Jan 29 04:31:46 2004
From: jimmy-ml at nccom.com (Jim Gottlieb)
Date: Thu Jan 29 04:33:45 2004
Subject: [Mailman-Developers] one-click unsubscribe
In-Reply-To: <1075353784.23368.47.camel@anthem>
References: <20040129000105.GX27962@nccom.com>
	<1075353784.23368.47.camel@anthem>
Message-ID: <20040129093146.GA28332@nccom.com>

On 2004-01-29 at 00:23, Barry Warsaw (barry@python.org) wrote:

> When you say "VERP-like" I think you mean more along the lines of a
> mail-merge or content personalization feature.

What I meant was using a string including a random number, like what is
done for confirmations.  It's not really related to VERP.  That was a
poor choice of words.


> You have to do this in an unguessable way, otherwise, attackers can
> simply unsubscribe anybody they want from a mailing list.

Right.  That's what I was talking about.


> I believe I have a way to limit, based on list configuration, such
> "$mm:" substitutions to the headers and footers, or to allow such
> substitutions to occur anywhere in the message text (headers, body, and
> MIME attachments included).

That sounds encouraging.


From somuchfun at atlantismail.com  Thu Jan 29 17:21:02 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Thu Jan 29 17:21:27 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
Message-ID: <22210899256223@smtp.westbay.net>

Ok, I just got off the phone with AOL's postmaster dept. and found out the
whole scoop why they have to strip out the To: address from complaints.
There was a lawsuit of breach of privacy so AOL can no longer share the To:
field in their feedback loop.
So here is the solution:
Mailman needs to create something like an x-client-id header that has the
recipient email address in it because this header will stay intact when a
complaint comes back.
This header needs to be created whether mailman runs in personalization mode
or not.
So the questions is not can mailman do it or not?


From brad.knowles at skynet.be  Thu Jan 29 17:44:27 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Thu Jan 29 17:46:10 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <22210899256223@smtp.westbay.net>
References: <22210899256223@smtp.westbay.net>
Message-ID: 

At 2:21 PM -0800 2004/01/29, Somuchfun wrote:

>  This header needs to be created whether mailman runs in personalization mode
>  or not.
>  So the questions is not can mailman do it or not?

	In personalization mode, this kind of information could 
theoretically persist in the headers (with suitable source-code 
modifications).  Otherwise, I don't think there's a mailing list 
manager on the planet that could make this happen.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From somuchfun at atlantismail.com  Thu Jan 29 20:45:06 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Thu Jan 29 20:46:15 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
Message-ID: <01451307560141@smtp.westbay.net>

Brad,
That is actually not true. I tested both Gordano's communicator and Lyris
Listmanager and both are able to handle this requirement without a problem.
I guess we found one of mailman's real weaknesses then.

> -----Original Message-----
> From: Brad Knowles [mailto:brad.knowles@skynet.be] 
> Sent: Thursday, January 29, 2004 2:44 PM
> To: Somuchfun
> Cc: mailman-developers@python.org
> Subject: Re: [Mailman-Developers] AOL's requirements for spam 
> complaints
> 
> At 2:21 PM -0800 2004/01/29, Somuchfun wrote:
> 
> >  This header needs to be created whether mailman runs in 
> personalization mode
> >  or not.
> >  So the questions is not can mailman do it or not?
> 
> 	In personalization mode, this kind of information could 
> theoretically persist in the headers (with suitable source-code 
> modifications).  Otherwise, I don't think there's a mailing list 
> manager on the planet that could make this happen.
> 
> -- 
> Brad Knowles, 
> 
> "They that can give up essential liberty to obtain a little temporary
> safety deserve neither liberty nor safety."
>      -Benjamin Franklin, Historical Review of Pennsylvania.
> 
> GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ 
> !E-(---) W+++(--) N+
> !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) 
> X++(+++) R+(+++)
> tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- 
> r---(+++)* z(+++)
> 


From moron at industrial.org  Thu Jan 29 21:03:03 2004
From: moron at industrial.org (moron@industrial.org)
Date: Thu Jan 29 21:03:08 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <01451307560141@smtp.westbay.net> from "Somuchfun" at Jan 29,
	4 05:45:06 pm
Message-ID: 

> That is actually not true. I tested both Gordano's communicator and
Lyris
> Listmanager and both are able to handle this requirement without a problem.
> I guess we found one of mailman's real weaknesses then.

To enable custom headers for each message at least partially destroys the
intent of a mailing list - efficient delivery of messages.  If each
message is customized that way then you have to actually send one distinct
message to every user instead of sending a single message with multiple
RCPT-TO lines.  Enabling this would serve to radically increase the server
overhead that Mailman causes (back to the Smartlist olden days).  Instead
of injecting one message to hundreds of recipients, you would be injecting
hundreds of unique messages. 

This is hardly a weakness of mailman and instead a weird bit of behaviour
on AOLs part.  

IMHO of course.

Cheers

-- 
|)   __,,_____________      moron :       (|
|)  <     ___________/  EEEI news :    (|
|)  /   /-'    musician community : http://ampfea.org           (|
|) /___/ industrial & DIY culture : http://industrial.org       (|
|)           deterrent industries : http://deterrent.net        (|

From chuqui at plaidworks.com  Fri Jan 30 01:40:11 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 01:40:33 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: <22210899256223@smtp.westbay.net>
	
Message-ID: <26FF59BB-52EF-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 29, 2004, at 2:44 PM, Brad Knowles wrote:

> At 2:21 PM -0800 2004/01/29, Somuchfun wrote:
>
>>  This header needs to be created whether mailman runs in 
>> personalization mode
>>  or not.
>>  So the questions is not can mailman do it or not?
>
> 	In personalization mode, this kind of information could theoretically 
> persist in the headers (with suitable source-code modifications).  
> Otherwise, I don't think there's a mailing list manager on the planet 
> that could make this happen.

If you don't personalize, you can't do it. Why? Because with 
personalization off, you're sending one email in a batch to 1-n users. 
Since  it's one copy to more than one user, then you can't 
individualize it (makes sense, when you think about it, right?)

So that header is what personalization mode is all about, really -- 
personalization. turn off personalization, but still personalize?

What I do is use this as a footer:

>> _______________________________________________
>> Do not post admin requests to the list. They will be ignored.
>> %(real_name)s mailing list      (%(real_name)s@%(host_name)s)
>> Help/Unsubscribe/Update your Subscription:
>> %(user_optionsurl)s
>>
>> This email sent to %(user_address)s

works great with AOL, as long as it's not a digest being reported. 
That's so rare I don't bother with it.


From chuqui at plaidworks.com  Fri Jan 30 01:41:26 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 01:42:37 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <01451307560141@smtp.westbay.net>
References: <01451307560141@smtp.westbay.net>
Message-ID: <53E1D274-52EF-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 29, 2004, at 5:45 PM, Somuchfun wrote:
> That is actually not true. I tested both Gordano's communicator and 
> Lyris
> Listmanager and both are able to handle this requirement without a 
> problem.
> I guess we found one of mailman's real weaknesses then.
>
>> 	In personalization mode, this kind of information could
>> theoretically persist in the headers (with suitable source-code
>> modifications).  Otherwise, I don't think there's a mailing list
>> manager on the planet that could make this happen.
>>

I've written a couple of servers that do this. I think every server 
should now, so all of mine do.


From chuqui at plaidworks.com  Fri Jan 30 01:44:07 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 01:44:33 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 
Message-ID: 


On Jan 29, 2004, at 6:03 PM, moron@industrial.org wrote:

> To enable custom headers for each message at least partially destroys 
> the
> intent of a mailing list - efficient delivery of messages.

Sorry, I don't buy this argument. If you have two choices: use more CPU 
time and network, or improve the end-user experience, choosing "less 
work for the computer" is almost always the wrong answer.

> Instead
> of injecting one message to hundreds of recipients, you would be 
> injecting
> hundreds of unique messages.
>

Yes, you are. And fortunately, it's not the 70's any more, and the 
resource limitations that caused those design decisions are gone. We 
aren't on 9600 baud dialups any more, for instance, or trying to run 
large mailing list on 286 class machines. well, I'm sure there are a 
few of those still out there, but that's no reason to hobble the rest 
of the universe with designs aimed at the last century.



From moron at industrial.org  Fri Jan 30 02:04:19 2004
From: moron at industrial.org (moron)
Date: Fri Jan 30 02:04:05 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 
	
Message-ID: <200401292304.19166.moron@industrial.org>

On January 29, 2004 10:44 pm, you wrote:
> Sorry, I don't buy this argument. If you have two choices: use more CPU
> time and network, or improve the end-user experience, choosing "less
> work for the computer" is almost always the wrong answer.

Howdy.  I do not understand why you would feel that adding a personalized 
header makes the list experience any better.   Would Usenet be any better 
with a customized header for every news article you read?  There is a big 
difference in my mind between a discussion mailing list and a marketing  
system with "Dear  (culture) http://industrial.org : (label)  http://deterrent.net
---> (community) http://ampfea.org : (hire me) http://codegrunt.com
---> (send EEEI news to) infosuck@industrial.org
---> Whomever dies with the most URLs wins!!!!!!!!!!!!!


From chuqui at plaidworks.com  Fri Jan 30 02:19:11 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 02:19:32 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <200401292304.19166.moron@industrial.org>
References: 
	
	<200401292304.19166.moron@industrial.org>
Message-ID: <99F0A182-52F4-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 29, 2004, at 11:04 PM, moron wrote:

>
> Howdy.  I do not understand why you would feel that adding a 
> personalized
> header makes the list experience any better.

Lots of research with end-users, studying their needs and researching 
the places that they struggle using these systems, and having designed 
and built a number of list servers over the years that are used by a 
wide range of users, not all of them geeks.

>   Would Usenet be any better
> with a customized header for every news article you read?

different argument. you don't need that user data to unsubscribe from a 
usenet group.You do to unsubscribe from a list server.

>  There is a big
> difference in my mind between a discussion mailing list

Not from the point of view we're talking about here, which is giving 
the user the info they need to operate the list properly.

> I am also not convinced about the CPU argument.  That's a lot of 
> extraneous
> message IDs to keep track of in databases, bounce detecting, etc.

With the exception of network traffic, it's actually pretty trivial 
stuff. No, I can't explain how I know, but I've been there, done that. 
The only huge cost is the network bandwidth change, which is at least 
2X, and can be 5X, depending on your old configurations.

> Multiply that by a busy list (some of the ones I look
> over are up to 150 - 200 a day sometimes) and it is still significant,
> especially if binaries are involved.

um, heh. Busy. (grin)

>  I also wonder what effect it will have
> on archiving

none.

> But if it works for you, hey go nuts.  But the argument to me sounds
> dangerously similar to the one Microsoft used to push using HTML in 
> email
> which we are all still feeling the unfortunate fallout from

Um, of course, the fact that users want html email is irrelevant. Lots 
of studies show they prefer the look of HTML to text, actually. Except 
in the more hard-core geek crews, but we aren't writing stuff here JUST 
for people who run mutt, right?

>  (zero cognitive
> benefit, plenty of headache).  Just because computers are faster now 
> does not
> mean that resources are suddenly free (as in beer).
>

you might be surprised. Lots of benefit, no headaches.

chuq (guess what I do for a living?)



From moron at industrial.org  Fri Jan 30 02:51:04 2004
From: moron at industrial.org (moron)
Date: Fri Jan 30 02:50:49 2004
Subject: [Mailman-Developers] mailman-developers@python.org
In-Reply-To: <99F0A182-52F4-11D8-8BF3-0003934516A8@plaidworks.com>
References: 
	<200401292304.19166.moron@industrial.org>
	<99F0A182-52F4-11D8-8BF3-0003934516A8@plaidworks.com>
Message-ID: <200401292351.04623.moron@industrial.org>

On January 29, 2004 11:19 pm, Chuq Von Rospach wrote:
> Lots of research with end-users, studying their needs and researching
> the places that they struggle using these systems, and having designed
> and built a number of list servers over the years that are used by a
> wide range of users, not all of them geeks.

Howdy.  Again, how does including an extra header help the end user 
experience?  The original complaint was due to AOL being bass ackwards and 
somehow feeling that an email address in an arbitrary header was more 
"private" than the To field (which of course it is not).   In this scenario, 
the "customization" was simply to add the sender address back into the 
message which is hardly making the end experience any better (it should 
already be in something like "envelope-to" anyway).

How about something more concrete as to why this is such a great feature for 
something beyond a spam list? (I am *not* suggesting you are a spammer, just 
that customization would seem to be only really important for commercial 
mailouts which generally fall under the spam-brella).  

> different argument. you don't need that user data to unsubscribe from a
> usenet group.You do to unsubscribe from a list server.

The problem is AOL though, not Mailman.  Solution?  Switch to a real provider 
that uses RFC compliant software.  And be vocal as to why you are leaving.

> Not from the point of view we're talking about here, which is giving
> the user the info they need to operate the list properly.

The information they need is that AOL is running a broken SMTP server, no?

> With the exception of network traffic, it's actually pretty trivial
> stuff. No, I can't explain how I know, but I've been there, done that.
> The only huge cost is the network bandwidth change, which is at least
> 2X, and can be 5X, depending on your old configurations.

Ok though I have not seen evidence of this using Exim.  But a 2 to 5 times 
increase in bandwidth use is a lot.  The majority of traffic in the community 
server I look after at the moment is due to mail and we would definitely feel 
that.

> > Multiply that by a busy list (some of the ones I look
> > over are up to 150 - 200 a day sometimes) and it is still significant,
> > especially if binaries are involved.
> um, heh. Busy. (grin)

Well, it depends on membership of course and the total number of lists.  It's 
big enough for me to look after.

=)

But a 60,000+ member list on Mailman would suck due to the administration 
interface anyway.  

> >  I also wonder what effect it will have
> > on archiving
>
> none.

Are you sure of that?  I thought that Hypermail based its threading on message 
IDs which would be different in this case leading to far larger arrays and 
such to keep track of what article was connected to what.  I could see this 
having an exponential effect on the length to regenerate archives and for 
building indexes.  That could be a LOT of  RAM usage.  Any Hypermail gurus 
want to comment?  Am I flapping in the wind on this one?  

> Um, of course, the fact that users want html email is irrelevant. Lots
> of studies show they prefer the look of HTML to text, actually. Except
> in the more hard-core geek crews, but we aren't writing stuff here JUST
> for people who run mutt, right?

Hmm.  I have yet to see a case where HTML has helped readability and folks 
that use it seem to solely do so because it is there not because they are 
trying to impart meaning.  Some people like using their cell phones in 
theatres but that doesn't make it a postive feature.  When using webmail 
interfaces for example no one misses it that I have ever noticed.  As to the 
effect of HTML, even ignoring the obvious security and privacy nightmare it 
results in you still having horrific rendering problems depending on the 
exact path the message takes from sending client to viewing client.  Nothing 
like a missing table tag to make your message unviewable.  

> chuq (guess what I do for a living?)

Debate with me?

=)

Cheers

-- 
---> (culture) http://industrial.org : (label)  http://deterrent.net
---> (community) http://ampfea.org : (hire me) http://codegrunt.com
---> (send EEEI news to) infosuck@industrial.org
---> Whomever dies with the most URLs wins!!!!!!!!!!!!!


From brad.knowles at skynet.be  Fri Jan 30 07:51:08 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 08:24:42 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <01451307560141@smtp.westbay.net>
References: <01451307560141@smtp.westbay.net>
Message-ID: 

At 5:45 PM -0800 2004/01/29, Somuchfun wrote:

>  That is actually not true. I tested both Gordano's communicator and Lyris
>  Listmanager and both are able to handle this requirement without a problem.

	I would like to see the evidence of this claim.

	I've been doing mail systems for over ten years, and I used to be 
the Sr. Internet Mail Administrator for AOL.  I can't imagine how 
they could be doing this sort of thing short of using their 
equivalent of personalization mode, which defeats the purpose of a 
mailing list.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 07:52:24 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 08:24:47 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <53E1D274-52EF-11D8-8BF3-0003934516A8@plaidworks.com>
References: <01451307560141@smtp.westbay.net>
	<53E1D274-52EF-11D8-8BF3-0003934516A8@plaidworks.com>
Message-ID: 

At 10:41 PM -0800 2004/01/29, Chuq Von Rospach wrote:

>  I've written a couple of servers that do this.

	That do what?

>                                                  I think every server
>  should now, so all of mine do.

	What do they do?

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 07:59:46 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 08:24:52 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: 
References: 
	
Message-ID: 

At 10:44 PM -0800 2004/01/29, Chuq Von Rospach wrote:

>  Sorry, I don't buy this argument. If you have two choices: use
>  more CPU time and network, or improve the end-user experience,
>  choosing "less work for the computer" is almost always the wrong
>  answer.

	You know damn good and well that this is not a CPU issue.  This 
is a disk I/O capacity issue (synchronous meta-data updates). 
Moreover, you also know full well that there are serious performance 
issues with enabling personalization mode on large mailing lists, 
such that for some lists, it would simply be impossible to do.

	The increased CPU utilization and network bandwidth can be 
problems for some sites, but that is not the gating factor in most 
cases.

>  Yes, you are. And fortunately, it's not the 70's any more, and
>  the resource limitations that caused those design decisions are
>  gone.

	This is not a valid criticism.  As network bandwidth has 
increased, the numbers of messages being sent and the size of the 
messages being sent has also increased, and the number of recipients 
has also increased.  What has *not* increased significantly is disk 
I/O latency, which is the gating factor for synchronous meta-data 
updates.

	You've got a significant increase in demand along three separate 
axes, without a corresponding increase in capacity.  Something has to 
give.  We have to be more intelligent about how we deliver those 
messages, or the entire system grinds to a halt.

>         We aren't on 9600 baud dialups any more, for instance, or
>  trying to run large mailing list on 286 class machines.

	See above.  The CPU being utilized is irrelevant.  What is not 
irrelevant is disk I/O latency, a fact that I know that you know as 
well as or better than most.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 08:04:14 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 08:24:58 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <99F0A182-52F4-11D8-8BF3-0003934516A8@plaidworks.com>
References: 
	
	<200401292304.19166.moron@industrial.org>
	<99F0A182-52F4-11D8-8BF3-0003934516A8@plaidworks.com>
Message-ID: 

At 11:19 PM -0800 2004/01/29, Chuq Von Rospach wrote:

>  With the exception of network traffic, it's actually pretty trivial
>  stuff.

	Uh, no.  Not just "no", but "Hell, no!"

	Increased network traffic is one cost, yes.  But there are plenty 
of other additional costs as well, some of which are considerably 
more important.

>>   (zero cognitive
>>  benefit, plenty of headache).  Just because computers are faster 
>>now does not
>>  mean that resources are suddenly free (as in beer).
>
>  you might be surprised. Lots of benefit, no headaches.

	And you might be surprised.  Lots of benefit for the users, yes. 
Lots of cost on the system, also.

>  chuq (guess what I do for a living?)

	I know what you do for a living, and I know that you know what I 
have done for a living.

	The issue is not as simple as you make it out to be.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 08:10:49 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 08:25:03 2004
Subject: [Mailman-Developers] mailman-developers@python.org
In-Reply-To: <200401292351.04623.moron@industrial.org>
References: 
	<200401292304.19166.moron@industrial.org>
	<99F0A182-52F4-11D8-8BF3-0003934516A8@plaidworks.com>
	<200401292351.04623.moron@industrial.org>
Message-ID: 

At 11:51 PM -0800 2004/01/29, moron wrote:

>  Howdy.  Again, how does including an extra header help the end user
>  experience?

	It doesn't.  Enabling personalization does.

>               The original complaint was due to AOL being bass ackwards and
>  somehow feeling that an email address in an arbitrary header was more
>  "private" than the To field (which of course it is not).   In this scenario,
>  the "customization" was simply to add the sender address back into the
>  message which is hardly making the end experience any better (it should
>  already be in something like "envelope-to" anyway).

	That is pretty bloody stupid, and is the real issue that we 
should be discussing.

>  The problem is AOL though, not Mailman.  Solution?  Switch to a real provider
>  that uses RFC compliant software.  And be vocal as to why you are leaving.

	Your recipients are where they are.  You can't really make them 
move.  You can refuse to accept any recipients on AOL, but that's 
about it.

>  The information they need is that AOL is running a broken SMTP server, no?

	Not a broken SMTP server, per se.  It's broken policies with 
regards to handling spam and mailing lists, and their stupid 
sanitization methods which they are asking you to work around so that 
the very information they sanitized is exposed elsewhere in the 
message.

	Nevertheless, I was working at AOL when they implemented their 
current SMTP server, and I can confirm that it is pretty badly broken 
in plenty of other ways.

>  Well, it depends on membership of course and the total number of lists.  It's
>  big enough for me to look after.

	I guess that if you're running all the mailing lists for 
apple.com, then you don't really care about increased bandwidth 
charges, or any of the other increased costs of running the mailing 
lists.

>>  >  I also wonder what effect it will have
>>  > on archiving
>>
>>  none.
>
>  Are you sure of that?  I thought that Hypermail based its threading 
>on message
>  IDs which would be different in this case leading to far larger arrays and
>  such to keep track of what article was connected to what.

	No, Chuq is right about this.  The archiving is done on message 
input, which is not changed as a result of personalization on message 
output.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From kmccann at bellanet.org  Fri Jan 30 08:52:05 2004
From: kmccann at bellanet.org (Kevin McCann)
Date: Fri Jan 30 08:52:09 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 	
	
Message-ID: <401A6185.2080003@bellanet.org>

Brad Knowles wrote:

> At 10:44 PM -0800 2004/01/29, Chuq Von Rospach wrote:
>
>>  Sorry, I don't buy this argument. If you have two choices: use
>>  more CPU time and network, or improve the end-user experience,
>>  choosing "less work for the computer" is almost always the wrong
>>  answer.
>
>
>     You know damn good and well that this is not a CPU issue.  This is 
> a disk I/O capacity issue (synchronous meta-data updates). Moreover, 
> you also know full well that there are serious performance issues with 
> enabling personalization mode on large mailing lists, such that for 
> some lists, it would simply be impossible to do.

Why is it, then, that Lyris can send personalized messages to lists with 
hundreds of thousands of members with no problem?  I don't personally 
have any lists that are nearly that big but I can tell you that my Lyris 
box sends messages to my lists with a few thousand members extremely 
quickly. Having personalization as a *choice* is the best thing. Then, 
those who worry about disk I/O or whatever can live with 
non-personalized delivery (at the expense of the users, of course), and 
those who want to move forward into the 21st century can do so with 
personalized delivery.

Mailing list communities want more now. Especially in Communities of 
Practice. Our most recent request was to tack on a person's professional 
profile (from another datasource) on the end of each message he or she 
sends. Feasible? Maybe, maybe not. But people do want this kind of 
thing. And I get paid to deliver what is needed. The fact is that Lyris 
does personalization just fine. So why continue to let Mailman lag behind?

Barry and others will be (or are) working on Mailman 3. I think that 
he/they should take a long hard look at the commercial MLM success story 
(Lyris) and take a few pages out of that book. They spent millions of 
dollars on R&D and made decisions base on it. Why not tap into that? 
Personalized delivery is just one thing. Don't get me started on SQL 
issues and the need for vastly improved logging for forensic purposes.

- Kevin





From kmccann at bellanet.org  Fri Jan 30 09:14:13 2004
From: kmccann at bellanet.org (Kevin McCann)
Date: Fri Jan 30 09:14:20 2004
Subject: [Mailman-Developers] Any SQL people going to Barry's sprint?
Message-ID: <401A66B5.2070101@bellanet.org>

A while  ago Barry posted a message about a Mailman 3 sprint which is to 
occur in March. I was wondering if any SQL (especially MySQL) people are 
planning to go. If you have strong Python and MySQL skills, please drop 
me a note. I'm trying to identify a candidate from among a few 
organizations I'm working with on a project that will use an OS MLM 
(most likely Mailman), but I'm willing to look elsewhere if there is 
nobody suitable from that pool. Bottom line, I'm willing to pay to make 
Mailman SQL-able (as a choice) for all three data sources (list config, 
members, message archives).

Barry, what's the word? Have any SQL folks contacted you?

- Kevin

From dietmar at maurer-it.com  Fri Jan 30 10:07:47 2004
From: dietmar at maurer-it.com (Dietmar Maurer)
Date: Fri Jan 30 10:07:53 2004
Subject: [Mailman-Developers] message IDs
Message-ID: <87844F3B84E88E4EA0CD476EA3B9F71D432B57@himalaya.maurer-it.com>

Hi all,

is it possible to tell mailman to change the message ID of a mail, so that the original mail and the mails delivered to the members have different IDs?

best regards,

Dietmar 

---------------------------------------------------
Dietmar Maurer         Maurer IT Systeml?sungen KEG
                                 Technischer Leiter

Kohlgasse 51/9              Tel: +43  1 545 449 712
A - 1050 WIEN               Fax: +43  1 545 449 722
                          Mobil: +43 699 105 88 032
dietmar@maurer-it.com
http://www.maurer-it.com
---------------------------------------------------

From chuqui at plaidworks.com  Fri Jan 30 12:04:56 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 12:05:45 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 
	
	
Message-ID: <6E3D695E-5346-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 30, 2004, at 4:59 AM, Brad Knowles wrote:

>>  Sorry, I don't buy this argument. If you have two choices: use
>>  more CPU time and network, or improve the end-user experience,
>>  choosing "less work for the computer" is almost always the wrong
>>  answer.
>
> 	You know damn good and well that this is not a CPU issue.  This is a 
> disk I/O capacity issue (synchronous meta-data updates).

fair cop. you're right, Brad. I was tired, didn't think it through. But 
I still think the user experience issues trump the Network/disk issues. 
we're here to make life easier for people, not computers.

> 	You've got a significant increase in demand along three separate 
> axes, without a corresponding increase in capacity.  Something has to 
> give.  We have to be more intelligent about how we deliver those 
> messages, or the entire system grinds to a halt.

yup. and trust me, some of us are working on that part, too...


From chuqui at plaidworks.com  Fri Jan 30 12:09:42 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 12:10:40 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <401A6185.2080003@bellanet.org>
References: 	
	 <401A6185.2080003@bellanet.org>
Message-ID: <18491AC8-5347-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 30, 2004, at 5:52 AM, Kevin McCann wrote:

>>     You know damn good and well that this is not a CPU issue.  This 
>> is a disk I/O capacity issue (synchronous meta-data updates). 
>> Moreover, you also know full well that there are serious performance 
>> issues with enabling personalization mode on large mailing lists, 
>> such that for some lists, it would simply be impossible to do.
>
> Why is it, then, that Lyris can send personalized messages to lists 
> with hundreds of thousands of members with no problem?

Lyris has made the choice it's worth it. So has mailman with 
personalization.

Brad is right that I trivialized some resource issues last night -- but 
that doesn't change my belief that for the user, it's worth using those 
resources to improve things for them. You don't want to waste 
resources; you also don't want to not use them when it's the right 
thing to do.

> Barry and others will be (or are) working on Mailman 3. I think that 
> he/they should take a long hard look at the commercial MLM success 
> story (Lyris) and take a few pages out of that book. They spent 
> millions of dollars on R&D and made decisions base on it. Why not tap 
> into that? Personalized delivery is just one thing. Don't get me 
> started on SQL issues and the need for vastly improved logging for 
> forensic purposes.
>

Better yet, look at the users. They aren't geeks any more. They're your 
mom and dad, off on a cable modem somewhere. A cable modem who has been 
through two or three acquisitions and domain name changes, and these 
folks aren't really sure what their email address is (their smart son 
configured their computer for them), much less what it was three 
changes ago when they signed up for the list.

and now they want to turn it off and go on vacation for a month, and 
the plane leaves in six hours.

If they can't -- it's your fault, as admin. And they're right.

that's why personalization matters.


From carson at taltos.org  Fri Jan 30 12:57:41 2004
From: carson at taltos.org (Carson Gaspar)
Date: Fri Jan 30 12:57:51 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <401A6185.2080003@bellanet.org>
References: 
	
	 <401A6185.2080003@bellanet.org>
Message-ID: <2564415408.1075485461@taltos.ny.ficc.gs.com>



--On Friday, January 30, 2004 08:52:05 -0500 Kevin McCann 
 wrote:

> Why is it, then, that Lyris can send personalized messages to lists with
> hundreds of thousands of members with no problem?  I don't personally

If you'd read your own thread, you'd know the answer already. Lyris is its 
own MTA - it speaks SMTP directly to the recipients' mail servers. This 
allows it to do on-the-fly customization at SMTP transmit time instead of 
having to queue each unique message.

This is _very_ _hard_ to get right, just to do SMTP properly. 
Personalization makes it even more difficult. Simple example: someones mail 
server is down. Do you:

- queue the personalized message
- queue the message template, and the list of undelivered recipients
- queue the message template, the list of undelivered recipients, and the 
substitution db version

Each choice has significant implications. None is obviously correct.

It would be great if MTAs included this functionality, but there are major 
political players who are terrified this will just be used by spammers. 
Personally, I think spammers could do it trivially already, as they don't 
care about queueing mail properly and handling all the edge cases for SMTP. 
But I'm not the maintainer of postfix/exim/sendmail/etc., so it's not my 
decision.

I'll make you a deal - you write the MTA, and I'll add support in mailman 
to offload the personalization.

-- 
Carson


 

From kmccann at bellanet.org  Fri Jan 30 13:23:28 2004
From: kmccann at bellanet.org (Kevin McCann)
Date: Fri Jan 30 13:23:53 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <2564415408.1075485461@taltos.ny.ficc.gs.com>
References: 		
	<401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
Message-ID: <401AA120.3040002@bellanet.org>

Carson Gaspar wrote:

>
> If you'd read your own thread, you'd know the answer already. Lyris is 
> its own MTA - it speaks SMTP directly to the recipients' mail servers. 
> This allows it to do on-the-fly customization at SMTP transmit time 
> instead of having to queue each unique message.

Fair dues.

> I'll make you a deal - you write the MTA, and I'll add support in 
> mailman to offload the personalization.

I do not personally have the skills to do this but I wouldn't rule out 
trying to get the funding to help make it happen. I wonder if there is 
there enough collective know-how among Mailman developers and other 
interested parties. Let me ask: if you don't see this as being a 
priority now, do you see it as being such in 2 years, 3 years, five?

More than anything I'd like to see an open source MLM that can keep up. 
One that can meet the ever growing list of challenges as well as 
expectations. So, looking down the road, where do others see things 
going? Should the OS MLM status quo remain, or ought there be an effort 
to plan for the future? Now that Mailman 3 is on the table, is a 
built-in MTA an issue for discussion, or is it completely unrealistic? 
Will it always be unrealistic?

- Kevin








From somuchfun at atlantismail.com  Fri Jan 30 13:49:29 2004
From: somuchfun at atlantismail.com (Somuchfun)
Date: Fri Jan 30 13:49:46 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <2564415408.1075485461@taltos.ny.ficc.gs.com>
Message-ID: <18493813070347@smtp.westbay.net>

Here is what I do not understand from the discussion:
Mailman in its current form is slow and if personalization is turned on
users cannot even get into the mailman site anymore because it takes up all
available resources.
We are running a list with about 50,000 subscribers.
As an admin I do not really care if some people think AOL does not have
their act together or not - if I want to have my emails reach them then I
have to play by their rules.
Like I said, I have tried other softwares on the market and used their
personalization feature. I even tried the same list on the same machine.
Mailman needed with personalization about 8.5 hrs. to send out one message
to all 50,000 people and Lyris Listmanager needed about 4.5 hours.
Is disk I/O a problem? Of course it is, but it is a problem for all list
managing software packages. My experience is that mailman is just very slow
when it comes to db access. Just try to add 10,000 users at once and most
likely you get a time out.
So perhaps mailman is better for smaller discussion list than for larger
email lists.
Some people here have suggested that anything besides email discussion lists
are spam, I find statements like this alarming.
We run a newsletter where people actively want to get the newsletter and we
do not consider ourselves spamming these people. In fact we try very hard to
comply with all rules, regulations and expectations - more so than some
ISPs.
All I want is a fast and cheap engine that can help me reach my goal - to
get the email to my customers quickly and to offer easy management
capabilities.
So far I like mailman's management capabilities. The performance has left me
being disappointed.

> -----Original Message-----
> From: 
> mailman-developers-bounces+somuchfun=atlantismail.com@python.o
rg [mailto:mailman-developers->
bounces+somuchfun=atlantismail.com@python.org] On Behalf Of 
> Carson Gaspar
> Sent: Friday, January 30, 2004 9:58 AM
> To: mailman-developers@python.org
> Subject: Re: [Mailman-Developers] AOL's requirements for spam 
> complaints
> 
> 
> 
> --On Friday, January 30, 2004 08:52:05 -0500 Kevin McCann 
>  wrote:
> 
> > Why is it, then, that Lyris can send personalized messages 
> to lists with
> > hundreds of thousands of members with no problem?  I don't 
> personally
> 
> If you'd read your own thread, you'd know the answer already. 
> Lyris is its 
> own MTA - it speaks SMTP directly to the recipients' mail 
> servers. This 
> allows it to do on-the-fly customization at SMTP transmit 
> time instead of 
> having to queue each unique message.
> 
> This is _very_ _hard_ to get right, just to do SMTP properly. 
> Personalization makes it even more difficult. Simple example: 
> someones mail 
> server is down. Do you:
> 
> - queue the personalized message
> - queue the message template, and the list of undelivered recipients
> - queue the message template, the list of undelivered 
> recipients, and the 
> substitution db version
> 
> Each choice has significant implications. None is obviously correct.
> 
> It would be great if MTAs included this functionality, but 
> there are major 
> political players who are terrified this will just be used by 
> spammers. 
> Personally, I think spammers could do it trivially already, 
> as they don't 
> care about queueing mail properly and handling all the edge 
> cases for SMTP. 
> But I'm not the maintainer of postfix/exim/sendmail/etc., so 
> it's not my 
> decision.
> 
> I'll make you a deal - you write the MTA, and I'll add 
> support in mailman 
> to offload the personalization.
> 
> -- 
> Carson
> 
> 
>  
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers@python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> 


From moron at industrial.org  Fri Jan 30 14:10:32 2004
From: moron at industrial.org (moron@industrial.org)
Date: Fri Jan 30 14:10:38 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <18493813070347@smtp.westbay.net> from "Somuchfun" at Jan 30,
	4 10:49:29 am
Message-ID: 

> All I want is a fast and cheap engine that can help me reach my goal - to
> get the email to my customers quickly and to offer easy management
> capabilities.
> So far I like mailman's management capabilities. The performance has left me
> being disappointed.

Howdy.  I would think that Mailman's job is not to provide free marketing
tools but to act as a list processor.  For what it offers it is the best
trade off of features, performance and price going for small to medium
sized lists. If you want Lyris you should pony up and pay for it IMHO. 

If all you want is a customized one way mailout then it doesn't sound like
you are looking for a mailing list processor as much as a mass mailer and 
there are other options for that kind of thing.

But that's just my opinion of course.

Cheers

-- 
|)   __,,_____________      moron :       (|
|)  <     ___________/  EEEI news :    (|
|)  /   /-'    musician community : http://ampfea.org           (|
|) /___/ industrial & DIY culture : http://industrial.org       (|
|)           deterrent industries : http://deterrent.net        (|

From brad.knowles at skynet.be  Fri Jan 30 14:10:29 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 14:25:01 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <401A6185.2080003@bellanet.org>
References: 
	
	 <401A6185.2080003@bellanet.org>
Message-ID: 

At 8:52 AM -0500 2004/01/30, Kevin McCann wrote:

>  Why is it, then, that Lyris can send personalized messages to lists
>  with hundreds of thousands of members with no problem?

	Maybe they have their own custom MTA that is tightly integrated 
into the mailing list manager.

>                                                          I don't
>  personally have any lists that are nearly that big but I can tell you
>  that my Lyris box sends messages to my lists with a few thousand
>  members extremely quickly.

	Sending messages to large mailing lists very quickly is not a 
problem.  Doing so with personalization turned on, is a problem.

>                              Having personalization as a *choice* is
>  the best thing. Then, those who worry about disk I/O or whatever can
>  live with non-personalized delivery (at the expense of the users, of
>  course), and those who want to move forward into the 21st century
>  can do so with personalized delivery.

	Personalization is a valid choice.  Probably 99% of of mailman 
lists are small enough that the additional performance cost caused by 
turning on personalization doesn't cause too many problems.

	At issue is that other 1% of the largest mailing lists where 
turning on personalization would not be feasible.

>   The fact is that Lyris does personalization just fine.

	I don't doubt that Lyris can handle personalization just fine. 
For that matter, so can Mailman.  At issue is what cost do you pay to 
turn on personalization?

	So far, I have seen nothing that leads me to believe that Lyris 
is capable of doing this without doing a single delivery per 
recipient, which is exactly the same thing that Mailman has to do in 
order to achieve the same goal.

>                                                           So why continue
>  to let Mailman lag behind?

	If it requires implementing a custom MTA, that's not going to 
happen.  Barry has already ruled that out.

	If you want that kind of thing, go with Listserv and LSMTP.

>  Barry and others will be (or are) working on Mailman 3. I think
>  that he/they should take a long hard look at the commercial MLM
>  success story (Lyris) and take a few pages out of that book.
>  They spent millions of dollars on R&D and made decisions base on
>  it. Why not tap into that?

	Hey, give Barry a few million dollars to fix up Mailman properly, 
and I'm sure that he could come up with a way to write a custom MTA 
(or do whatever else is necessary) to make it competitive with other 
MLMs out there.

	Short of that, try contributing some code yourself to solve these problems.

>                              Personalized delivery is just one
>  thing. Don't get me started on SQL issues and the need for vastly
>  improved logging for forensic purposes.

	Mailman already does personalization.  If that's what you want, 
then stop complaining now.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 14:11:23 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 14:25:29 2004
Subject: [Mailman-Developers] message IDs
In-Reply-To: <87844F3B84E88E4EA0CD476EA3B9F71D432B57@himalaya.maurer-it.com>
References: <87844F3B84E88E4EA0CD476EA3B9F71D432B57@himalaya.maurer-it.com>
Message-ID: 

At 4:07 PM +0100 2004/01/30, Dietmar Maurer wrote:

>  is it possible to tell mailman to change the message ID of a mail,
>  so that the original mail and the mails delivered to the members
>  have different IDs?

	Sure.  You can hack the source code to do whatever you want.

	Short of hacking the source code, this is not possible.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 14:14:59 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 14:25:31 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <6E3D695E-5346-11D8-8BF3-0003934516A8@plaidworks.com>
References: 
	
	
	<6E3D695E-5346-11D8-8BF3-0003934516A8@plaidworks.com>
Message-ID: 

At 9:04 AM -0800 2004/01/30, Chuq Von Rospach wrote:

>  fair cop. you're right, Brad. I was tired, didn't think it through.
>  But I still think the user experience issues trump the Network/disk
>  issues. we're here to make life easier for people, not computers.

	I agree.  We are here to make life easier for people, and we 
should do whatever we can towards that goal.

	However, there are some things that are beyond our capabilities. 
Some features require a significant amount of additional effort on 
the part of the server, and while that is desirable and feasible for 
most mailing lists, there may well be some mailing lists where that's 
just not possible.  We have to acknowledge that issue.

>  yup. and trust me, some of us are working on that part, too...

	I know.  Big Mac is the third most powerful supercomputer cluster 
in the world, and cost much, much less than the two clusters that are 
more powerful than it is, and probably much less than any other 
cluster in the Top 100.

	But there are still limits, especially in the areas of certain 
types of common technology, such as disk drives.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 14:20:16 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 14:25:35 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <401AA120.3040002@bellanet.org>
References: 
	
	 <401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
	<401AA120.3040002@bellanet.org>
Message-ID: 

At 1:23 PM -0500 2004/01/30, Kevin McCann wrote:

>  I do not personally have the skills to do this but I wouldn't rule
>  out trying to get the funding to help make it happen. I wonder if
>  there is there enough collective know-how among Mailman developers
>  and other interested parties. Let me ask: if you don't see this as
>  being a priority now, do you see it as being such in 2 years, 3
>  years, five?

	Do you have any concept as to how much work has gone into 
sendmail over the past twenty-plus years?  Or postfix, or Exim?  They 
tend to get most things right, but even now they have plenty of 
problems -- they just have fewer problems than most other MTAs.

	You might be talking about a change that would require 
essentially throwing out everything that has been done before, and 
starting over from scratch.


	Do you have the millions of dollars and human-centuries worth of 
productive coding that it would take to write yet another MTA 
properly?

>  More than anything I'd like to see an open source MLM that can
>  keep up.

	There are some things that Barry has already ruled out.  Writing 
a custom MTA for Mailman is one of those things.

	Don't even bother barking up this tree.


	Perhaps, for Mailman 3, Barry could talk to people like Eric 
Allman, Wietse Venema, and other solid MTA authors, to see if there 
is a way we could get a certain amount of message customization 
pulled into the MTA, without killing the performance of the machine.

	But that's a question that Barry would have to answer.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 14:24:32 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 14:25:38 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <18493813070347@smtp.westbay.net>
References: <18493813070347@smtp.westbay.net>
Message-ID: 

At 10:49 AM -0800 2004/01/30, Somuchfun wrote:

>  Like I said, I have tried other softwares on the market and used their
>  personalization feature. I even tried the same list on the same machine.
>  Mailman needed with personalization about 8.5 hrs. to send out one message
>  to all 50,000 people and Lyris Listmanager needed about 4.5 hours.

	Which may be because they have implemented their own custom MTA, 
something that very few other MLMs in the world have done or can do. 
Listserv with LSMTP being the only other example I can think of off 
the top of my head.

>  Is disk I/O a problem? Of course it is, but it is a problem for all list
>  managing software packages.

	It can be less of an issue for those MLMs that have implemented 
their own custom MTA.

>  So perhaps mailman is better for smaller discussion list than for larger
>  email lists.

	Yup.  If your list is too big for Mailman, maybe you need to find 
a different MLM.  Perhaps some day Mailman will have had the 
performance increased enough that it could handle lists that large, 
but maybe it can't handle them today.

	Keep in mind that this is not a problem for 99% of the lists out 
there that are handled today with Mailman, and there are even lists 
with over 200,000 recipients in operation, which are running just 
fine with Mailman.

>  All I want is a fast and cheap engine that can help me reach my goal - to
>  get the email to my customers quickly and to offer easy management
>  capabilities.

	Maybe Mailman is not able to handle that load on the machines you have.

>  So far I like mailman's management capabilities. The performance has left me
>  being disappointed.

	Perhaps it is the wrong software for your application.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From kmccann at bellanet.org  Fri Jan 30 14:42:22 2004
From: kmccann at bellanet.org (Kevin McCann)
Date: Fri Jan 30 14:42:35 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 
	
	 <401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
	<401AA120.3040002@bellanet.org> 
Message-ID: <401AB39E.1070204@bellanet.org>

Brad Knowles wrote:

>     There are some things that Barry has already ruled out.  Writing a 
> custom MTA for Mailman is one of those things.
>     Don't even bother barking up this tree.
>     Perhaps, for Mailman 3, Barry could talk to people like Eric 
> Allman, Wietse Venema, and other solid MTA authors, to see if there is 
> a way we could get a certain amount of message customization pulled 
> into the MTA, without killing the performance of the machine.

I'm simply thinking about MLM challenges, which are increasing every 
day, it seems, and thinking about what can be done. Personalization 
without critical slowdown is an issue. I am thinking out loud about what 
can be done. I don't have the answers, which is why I invite discussion. 
Pleasant discussion, if you can find it within you. I have not been on 
mm-dev forever, so I am not aware of every thread that has been 
discussed. Further, am I willing to contribute? Yes I am. Millions, no. 
But something, yes. Give a guy a break.

- Kevin


From barry at python.org  Fri Jan 30 14:45:21 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 14:45:28 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <22210899256223@smtp.westbay.net>
References: <22210899256223@smtp.westbay.net>
Message-ID: <1075491920.1075.135.camel@anthem>

On Thu, 2004-01-29 at 17:21, Somuchfun wrote:

> Mailman needs to create something like an x-client-id header that has the
> recipient email address in it because this header will stay intact when a
> complaint comes back.

> This header needs to be created whether mailman runs in personalization mode
> or not.

> So the questions is not can mailman do it or not?

It would not be hard to add this to Mailman, when full personalization
is turned on.  Two lines of code.  

The difference between 'regular' personalization and full
personalization is that the latter inserts a custom To header for each
recipient.  There's no reason why Mailman couldn't also add an
X-800-Pound-Gorillas-Can-Hurt header with whatever information makes
your life easier.

The question isn't whether it can be done, but whether we should add it
to Mailman for everyone.  I'm not interested in adding even more
configuration options to control this, whether in Defaults.py or on the
admin pages.

-Barry



From ricardo at rixhq.nu  Fri Jan 30 15:06:05 2004
From: ricardo at rixhq.nu (Ricardo)
Date: Fri Jan 30 15:06:11 2004
Subject: [Mailman-Developers] my 2cts on Personalization, AOL, performance
Message-ID: <401AB92D.5010405@rixhq.nu>


Hi,

A few things I wanted to respond to.
First of all, I agree that personalization is an important feature for 
certain lists. Especially if your target audience isn't the average 
computer geek. I need it for two reasons: easy unsubscription and 
bulletproof bounce detection (or better: 
reported-as-spam-by-AOL-user-detection). One could argue that users need 
to be educated better, but sometimes that's almost like striving for 
world peace.
I can agree that mailman might be too slow for huge personalized lists, 
but my lists aren't bigger than 3k users... I have been running mailman 
for many years now. In the beginning it was running on an ancient 
Pentium 75 machine, so performance was a big issue for me (I couldn't 
even approve more than 5 posts at a time or else the system could crash 
;) ) but now I have a 2Ghz machine with fast disks at my disposal so 
mailman won't have any trouble sending out those 3000 messages...

Speaking of personalization: what is the main reason it hasn't been 
implemented for digests yet? Simply a lack of time or are there some 
important design issues which make it difficult? If isn't too difficult 
I could try to invest some time it myself to try to get it to work (I'm 
not a python god, but I can manage)

about improving delivery speed: what about implementing LMTP?
(http://www.networksorcery.com/enp/rfc/rfc2033.txt)


Regards,

Ricardo.

From barry at python.org  Fri Jan 30 15:06:06 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 15:06:16 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <401A6185.2080003@bellanet.org>
References: 
	
	  <401A6185.2080003@bellanet.org>
Message-ID: <1075493165.1075.157.camel@anthem>

On Fri, 2004-01-30 at 08:52, Kevin McCann wrote:

> Why is it, then, that Lyris can send personalized messages to lists with 
> hundreds of thousands of members with no problem?  I don't personally 
> have any lists that are nearly that big but I can tell you that my Lyris 
> box sends messages to my lists with a few thousand members extremely 
> quickly. 

And I think we can make Mailman clear its queue of a message very
quickly, even with full personalization turned on.  How Mailman 2.1 does
personalization is not as efficient as it could be, for technical
reasons I won't go into right now.  I believe we can make Mailman more
efficient here.

We've made the decision to not assimilate the MTA into Mailman.  The big
advantage here is that writing MTAs is hard, takes huge amount of
resources we don't have, and we can leverage the many good open source
MTAs out there.  The disadvantage is that we're going to pay for
personalization in MTA disk i/o.  Mailman 2.1 won't get clobbered here
(see my previous messages on the subject, and remember that during the
betas, MM2.1 /did/ queue each personal message to disastrous results),
and Mailman 3 will be better.

I totally buy that personalization improves the user experience, even
for discussion lists.  I think it's basically a no brainer, all other
things being equal.  I believe we're making the right choice here
because we can support a wide range of system configurations.  Small
sites that can't afford even moderate increase in cpu or bandwidth (they
turn off personalization), or that can and doesn't worry about i/o
because their traffic is light.  Larger sites can afford fast disks, mta
smurf farms, and other measures to mitigate the i/o requirements of the
mta.  Huge sites can write their own special Python delivery module to
speak WPMP (Wizzy Personalized Mail Protocol) to their custom in-house
blindingly fast weave-it-on-the-wire mail server.

> Having personalization as a *choice* is the best thing. Then, 
> those who worry about disk I/O or whatever can live with 
> non-personalized delivery (at the expense of the users, of course), and 
> those who want to move forward into the 21st century can do so with 
> personalized delivery.

Exactly.

> Mailing list communities want more now. Especially in Communities of 
> Practice. Our most recent request was to tack on a person's professional 
> profile (from another datasource) on the end of each message he or she 
> sends. Feasible? 

Yes.

> Maybe, maybe not. But people do want this kind of 
> thing. And I get paid to deliver what is needed. The fact is that Lyris 
> does personalization just fine. So why continue to let Mailman lag behind?
> 
> Barry and others will be (or are) working on Mailman 3. I think that 
> he/they should take a long hard look at the commercial MLM success story 
> (Lyris) and take a few pages out of that book. They spent millions of 
> dollars on R&D and made decisions base on it. Why not tap into that? 
> Personalized delivery is just one thing. Don't get me started on SQL 
> issues and the need for vastly improved logging for forensic purposes.

So Kevin, you coming to PyconII?  I still don't have (m)any volunteers
joining me in a Mailman 3 sprint. :(

-Barry



From barry at python.org  Fri Jan 30 15:13:46 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 15:13:52 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <401AA120.3040002@bellanet.org>
References: 
	
	 <401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
	<401AA120.3040002@bellanet.org>
Message-ID: <1075493625.1075.166.camel@anthem>

On Fri, 2004-01-30 at 13:23, Kevin McCann wrote:

> > I'll make you a deal - you write the MTA, and I'll add support in 
> > mailman to offload the personalization.
> 
> I do not personally have the skills to do this but I wouldn't rule out 
> trying to get the funding to help make it happen. I wonder if there is 
> there enough collective know-how among Mailman developers and other 
> interested parties. Let me ask: if you don't see this as being a 
> priority now, do you see it as being such in 2 years, 3 years, five?

It's possible.  But then again who knows what the email landscape will
look like in 5 years?  I'm betting it'll look a /lot/ different than it
does today, unless it doesn't.

> More than anything I'd like to see an open source MLM that can keep up. 

When you say "keep up", remember that because of the design decisions I
outlined in my previous message, you're really talking about the MLM+MTA
combination.

> One that can meet the ever growing list of challenges as well as 
> expectations. So, looking down the road, where do others see things 
> going? Should the OS MLM status quo remain, or ought there be an effort 
> to plan for the future? Now that Mailman 3 is on the table, is a 
> built-in MTA an issue for discussion, or is it completely unrealistic? 
> Will it always be unrealistic?

For all practical purposes, it's unrealistic for Mailman 3.  Our biggest
challenge is going to be avoiding second system syndrome, and we'll have
no hope if we don't limit scope.

OTOH, I think it's important to keep this in mind as we design the
delivery subsystem.  I think we win if MM3's architecture makes it
technically possible, even if it remains insane for us to attempt it
given our current level of resources.

-Barry



From kmccann at bellanet.org  Fri Jan 30 15:15:21 2004
From: kmccann at bellanet.org (Kevin McCann)
Date: Fri Jan 30 15:15:30 2004
Subject: [Mailman-Developers] A bit of perspective ....
Message-ID: <401ABB59.6080101@bellanet.org>

Maybe I ought to explain what I'm up against. At work I'm running Lyris. 
I have hundreds of lists and many, many members.  I also have Mailman 
running a handful of small-ish lists (and at home I run a server with 
Mailman for a personal interest discussion list of nearly 1,000 
members). My organization (and in fact, my entire industry of 
international development) is embracing open source with a fervor and 
one of the things on the to-do list is to move all services that run on 
commercial software to open source software. NT, IIS, ColdFusion and 
Lyris, are, for all intents and purposes, history.  I now have online 
communities  that have been very much accustomed to Lyris features, 
performance, functionality, etc. Not to mention the web interfaces that 
marry Lyris and other online resources (profiles, documents, calendar, 
etc.).  I have the unenviable challenge of moving these people to a 
Mailman environment without making them feel like it is a step 
backward.  I would really like to believe that this is possible. I'm not 
ready to give up on this quite yet. I have looked at other open source 
MLM's, and for one reason or another, the others aren't at all usable. 
Mailman is the closest thing I see to a solution.

Anyway, that's my sob story.

I'll continue to look forward for solutions. Flame away if it'll make 
you feel like a better person, but I'm not giving up that easily.

- Kevin







From barry at python.org  Fri Jan 30 15:16:36 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 15:16:47 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 
	
	 <401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
	<401AA120.3040002@bellanet.org>  
Message-ID: <1075493796.1075.170.camel@anthem>

On Fri, 2004-01-30 at 14:20, Brad Knowles wrote:

> 	Perhaps, for Mailman 3, Barry could talk to people like Eric 
> Allman, Wietse Venema, and other solid MTA authors, to see if there 
> is a way we could get a certain amount of message customization 
> pulled into the MTA, without killing the performance of the machine.
> 
> 	But that's a question that Barry would have to answer.

I'd support such an effort.  I think the right way to go about this
would be to design a protocol (or perhaps an API) for MLM/MTA
communication.  I'd be less enthusiastic about a solution that was
unique to a particular MTA.

Hey Brad, maybe you can dust off those protocol specs you once did.  Or
maybe you can make some first passes at a new specification.  That would
be a good jumping off point for approaching the MTA communities.

-Barry



From bob at nleaudio.com  Fri Jan 30 15:22:55 2004
From: bob at nleaudio.com (Bob Puff@NLE)
Date: Fri Jan 30 15:29:28 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <1075491920.1075.135.camel@anthem>
References: <22210899256223@smtp.westbay.net>
	<1075491920.1075.135.camel@anthem>
Message-ID: <401ABD1F.4060207@nleaudio.com>

Yes, that would be VERY helpful.  I've had a couple instances where that would have helped me.  Turn 
it on for all.

BOb

Barry Warsaw wrote:
> On Thu, 2004-01-29 at 17:21, Somuchfun wrote:
> 
> 
>>Mailman needs to create something like an x-client-id header that has the
>>recipient email address in it because this header will stay intact when a
>>complaint comes back.
> 
> 
>>This header needs to be created whether mailman runs in personalization mode
>>or not.
> 
> 
>>So the questions is not can mailman do it or not?
> 
> 
> It would not be hard to add this to Mailman, when full personalization
> is turned on.  Two lines of code.  
> 
> The difference between 'regular' personalization and full
> personalization is that the latter inserts a custom To header for each
> recipient.  There's no reason why Mailman couldn't also add an
> X-800-Pound-Gorillas-Can-Hurt header with whatever information makes
> your life easier.
> 
> The question isn't whether it can be done, but whether we should add it
> to Mailman for everyone.  I'm not interested in adding even more
> configuration options to control this, whether in Defaults.py or on the
> admin pages.
> 
> -Barry
> 
> 
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers@python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> 
> .
> 


From barry at python.org  Fri Jan 30 15:34:04 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 15:34:11 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <18493813070347@smtp.westbay.net>
References: <18493813070347@smtp.westbay.net>
Message-ID: <1075494844.1075.190.camel@anthem>

On Fri, 2004-01-30 at 13:49, Somuchfun wrote:
> Here is what I do not understand from the discussion:
> Mailman in its current form is slow and if personalization is turned on
> users cannot even get into the mailman site anymore because it takes up all
> available resources.

I don't totally believe that.

Understand that IMO, MM2.1's biggest architectural flaw right now is the
list data storage arrangement.  Pickles and list locks simply do not
scale.  Fixing this is a priority for MM3, but of course there will be
costs.  Backing Mailman with a real database (be it BerkeleyDB, MySQL,
or whatever) increases the administrative costs.  No way around it.

That said, MM2.1 does not retain a list lock while it delivers messages
to its MTA, so it should not lock out other access to the site.

> We are running a list with about 50,000 subscribers.
> As an admin I do not really care if some people think AOL does not have
> their act together or not - if I want to have my emails reach them then I
> have to play by their rules.
> Like I said, I have tried other softwares on the market and used their
> personalization feature. I even tried the same list on the same machine.
> Mailman needed with personalization about 8.5 hrs. to send out one message
> to all 50,000 people and Lyris Listmanager needed about 4.5 hours.
> Is disk I/O a problem? Of course it is, but it is a problem for all list
> managing software packages. My experience is that mailman is just very slow
> when it comes to db access. Just try to add 10,000 users at once and most
> likely you get a time out.

Again, the issue is likely deeper than it first appears.  I will bet you
that Mailman's "db access" is about as fast as you can possibly get,
because the list data resides completely in memory.  Lookups are a
simple dictionary access, which is very fast.

Where I believe you're getting clobbered is in the specific code that
generates the unique recipient copies.  The technique I'm using is about
as good as you can do in Python 2.1, which is MM2.1's minimum
requirement.  I can do a lot better if we set Python 2.3 as a baseline
and make other incompatible changes.  That's why it's all pushed off to
Mailman 3.

And being twice as slow as Lyris is actually not bad, IMO.  Lyris is
probably written in C or C++.  For a pure Python application like
Mailman to only take twice as long is not bad.  

> So perhaps mailman is better for smaller discussion list than for larger
> email lists.

As Mailman gains in popularity, people will try to make it do things it
wasn't necessarily designed for, or that weren't conceivable 6 years ago
when many of the basic architectural decisions were made.

> Some people here have suggested that anything besides email discussion lists
> are spam, I find statements like this alarming.

Spam is anything the user doesn't want to get.

> We run a newsletter where people actively want to get the newsletter and we
> do not consider ourselves spamming these people. In fact we try very hard to
> comply with all rules, regulations and expectations - more so than some
> ISPs.

I have no problem with that.

> All I want is a fast and cheap engine that can help me reach my goal - to
> get the email to my customers quickly and to offer easy management
> capabilities.
> So far I like mailman's management capabilities. The performance has left me
> being disappointed.

So how much would you pay to improve Mailman's performance?  If we could
raise a quarter million dollars in development funds, I doubt you'd be
disappointed for long .

-Barry



From kmccann at bellanet.org  Fri Jan 30 15:38:15 2004
From: kmccann at bellanet.org (Kevin McCann)
Date: Fri Jan 30 15:38:19 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <1075493165.1075.157.camel@anthem>
References: 	
		
	 <401A6185.2080003@bellanet.org>
	<1075493165.1075.157.camel@anthem>
Message-ID: <401AC0B7.1090504@bellanet.org>

Barry Warsaw wrote:

> So Kevin, you coming to PyconII? I still don't have (m)any volunteers
>
>joining me in a Mailman 3 sprint. :(
>
>-Barry
>  
>
Hi Barry,

Thanks for you cordial and helpful response.  If I can get up-to-speed 
with Python in order to work on the MySQL side of things, or if you 
think I could contribute with just the MySQL know-how, I'll go. 
Otherwise, I'll be sending someone else. I'd ideally like to send 
someone from one of our partner organizations involved with the Dgroups 
project, but if we can't  find a suitable candidate, then maybe we can 
find someone from this list. And as I've mentioned before, we'll fund 
it. So, if anyone is interested in working with Barry on Mailman/SQL in 
March, let me know. 

- Kevin

From barry at python.org  Fri Jan 30 15:38:28 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 15:38:36 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 
	
	 <401A6185.2080003@bellanet.org>
	
Message-ID: <1075495107.1075.194.camel@anthem>

On Fri, 2004-01-30 at 14:10, Brad Knowles wrote:

> 	Hey, give Barry a few million dollars to fix up Mailman properly, 
> and I'm sure that he could come up with a way to write a custom MTA 
> (or do whatever else is necessary) to make it competitive with other 
> MLMs out there.

Man, I was thinking too small.  Okay, Brad's elected Chief Mailman
Fundraiser. :)

-Barry



From barry at python.org  Fri Jan 30 15:44:21 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 15:45:00 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <401ABD1F.4060207@nleaudio.com>
References: <22210899256223@smtp.westbay.net>
	<1075491920.1075.135.camel@anthem>  <401ABD1F.4060207@nleaudio.com>
Message-ID: <1075495461.1075.208.camel@anthem>

On Fri, 2004-01-30 at 15:22, Bob Puff@NLE wrote:
> Yes, that would be VERY helpful.  I've had a couple instances where that would have helped me.  Turn 
> it on for all.

Turn what on?  So far there aren't any specific proposals, i.e. "Add
this header and make it contain that information".

-Barry



From chuqui at plaidworks.com  Fri Jan 30 15:50:23 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 15:50:38 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <401AA120.3040002@bellanet.org>
References: 		
	<401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
	<401AA120.3040002@bellanet.org>
Message-ID: 

>> If you'd read your own thread, you'd know the answer already. Lyris 
>> is its own MTA - it speaks SMTP directly to the recipients' mail 
>> servers. This allows it to do on-the-fly customization at SMTP 
>> transmit time instead of having to queue each unique message.
>
> Fair dues.
>
>> I'll make you a deal - you write the MTA, and I'll add support in 
>> mailman to offload the personalization.
>
> I do not personally have the skills to do this but I wouldn't rule out 
> trying to get the funding to help make it happen. I wonder if there is 
> there enough collective know-how among Mailman developers and other 
> interested parties.

Definitely. There are probably a six on this list who could write an 
MTA -- or have. The problem is, that dozen or so folks all (and I hope 
I speak for those people appropriately as I speak for myself) have come 
to the realization that it's rarely if ever cost-effective or worth the 
effort.

A secondary issue is there are more and more mutterings and grumblings 
that it's time to get serious and replace SMTP. If you integrate an 
SMTP server into Mailman and we go off and replace SMTP, where are you? 
out on a limb with a chain saw.

While Lyris has a lot going for it, it's tightly coupled MLM/MTA is a 
feature that's a mixed blessing. Now, if SMTP is replaced properly and 
the warts any MTA have to deal with (Hellow, Lotus Notes. Hello, 
exchange. hello, you know who you are) can get scraped off and not 
replaced with new warts, intefacing at the MTA level might be more 
practical.

But I wouldn't recommend it, support it, or encourage it with Mailman. 
not now, not in a year, not in five. Not to SMTP.

Mailman has a lot of things to do to become an even better mailing list 
manager before we should even think about trying to re-implement what 
the MTA teams are already spending all of their time on.

And I think we can do within Mailman what you think you need to 
integrate an MTA to do, without all of that pain and suffering. Or at 
least enough of it to not warrant going through the swamp to get there.

And trust me, SMTP is a swamp, with lots of hungry alligators.


From barry at python.org  Fri Jan 30 15:43:20 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 16:00:41 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <26FF59BB-52EF-11D8-8BF3-0003934516A8@plaidworks.com>
References: <22210899256223@smtp.westbay.net>
	
	<26FF59BB-52EF-11D8-8BF3-0003934516A8@plaidworks.com>
Message-ID: <1075495399.1075.206.camel@anthem>

On Fri, 2004-01-30 at 01:40, Chuq Von Rospach wrote:

> works great with AOL, as long as it's not a digest being reported. 
> That's so rare I don't bother with it.

Most lists on python.org have converted to doing the same.  I just
noticed mailman-developers didn't include %(user_optionsurl)s in its
footer even though it was personalizing.

  

Look down and enjoy.
-Barry




From barry at python.org  Fri Jan 30 15:59:29 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 16:01:50 2004
Subject: [Mailman-Developers] my 2cts on Personalization, AOL, performance
In-Reply-To: <401AB92D.5010405@rixhq.nu>
References: <401AB92D.5010405@rixhq.nu>
Message-ID: <1075496368.1075.223.camel@anthem>

On Fri, 2004-01-30 at 15:06, Ricardo wrote:

> Speaking of personalization: what is the main reason it hasn't been 
> implemented for digests yet? Simply a lack of time or are there some 
> important design issues which make it difficult?

For stupid reasons, Mailman decorates digests when it's composing them,
so by the time SMTPDirect.py is sending out personalization, it's too
late.  Untangling all that is more work than I'd be comfortable adding
to Mailman 2.1.

> about improving delivery speed: what about implementing LMTP?
> (http://www.networksorcery.com/enp/rfc/rfc2033.txt)

I'm not sure LMTP is appropriate, though I haven't studied this protocol
in detail, so I could be mistaken.  Mailman doesn't manage a queue in
the sense that LMTP requires; it relies on the MTA for most delivery
guarantees.

-Barry



From barry at python.org  Fri Jan 30 16:38:17 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 16:38:25 2004
Subject: [Mailman-Developers] mailman-developers@python.org
In-Reply-To: <200401292351.04623.moron@industrial.org>
References: 
	<200401292304.19166.moron@industrial.org>
	<99F0A182-52F4-11D8-8BF3-0003934516A8@plaidworks.com>
	<200401292351.04623.moron@industrial.org>
Message-ID: <1075498697.1075.251.camel@anthem>

On Fri, 2004-01-30 at 02:51, moron wrote:

> Ok though I have not seen evidence of this using Exim.  But a 2 to 5 times 
> increase in bandwidth use is a lot.

C'mon, isn't legitimate mail of any kind now just noise in the
spam/virii storms?  You won't even notice it. :)

-Barry



From brad.knowles at skynet.be  Fri Jan 30 16:00:16 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 16:43:52 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <18493813070347@smtp.westbay.net>
References: <18493813070347@smtp.westbay.net>
Message-ID: 

At 10:49 AM -0800 2004/01/30, Somuchfun wrote:

>  Like I said, I have tried other softwares on the market and used their
>  personalization feature. I even tried the same list on the same machine.
>  Mailman needed with personalization about 8.5 hrs. to send out one message
>  to all 50,000 people and Lyris Listmanager needed about 4.5 hours.

	Of course, this doesn't address the issue of MTA performance 
tuning.  I've seen situations where proper tuning resulted in a 
factor of ten (or more) improvement in the delivery times.  See 
 
and 
 
for two papers discussing this issue.

	See also my slides at 
 and 
, and the book 
_Sendmail Performance Tuning_ by Nick Christensen (at 
).


	If you haven't done your job in tuning the performance of the 
MTA, you really don't have much reason to complain about the 
performance of a mailing list manager with a lot of recipients.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 16:28:11 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 16:44:09 2004
Subject: [Mailman-Developers] my 2cts on Personalization, AOL, performance
In-Reply-To: <401AB92D.5010405@rixhq.nu>
References: <401AB92D.5010405@rixhq.nu>
Message-ID: 

At 9:06 PM +0100 2004/01/30, Ricardo wrote:

>  about improving delivery speed: what about implementing LMTP?
>  (http://www.networksorcery.com/enp/rfc/rfc2033.txt)

	That's for local message delivery, from the MTA to the recipient. 
That's not where we're hurting.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From brad.knowles at skynet.be  Fri Jan 30 16:32:07 2004
From: brad.knowles at skynet.be (Brad Knowles)
Date: Fri Jan 30 16:44:12 2004
Subject: [Mailman-Developers] AOL's requirements for spam
 complaints
In-Reply-To: <1075493796.1075.170.camel@anthem>
References: 	
		
	 <401A6185.2080003@bellanet.org>	
	<2564415408.1075485461@taltos.ny.ficc.gs.com>	
	<401AA120.3040002@bellanet.org> 
	<1075493796.1075.170.camel@anthem>
Message-ID: 

At 3:16 PM -0500 2004/01/30, Barry Warsaw wrote:

>  I'd support such an effort.  I think the right way to go about this
>  would be to design a protocol (or perhaps an API) for MLM/MTA
>  communication.  I'd be less enthusiastic about a solution that was
>  unique to a particular MTA.

	Agreed.  I think that would be a poor choice.

>  Hey Brad, maybe you can dust off those protocol specs you once did.  Or
>  maybe you can make some first passes at a new specification.  That would
>  be a good jumping off point for approaching the MTA communities.

	Sorry, we never got to the specs stage.  We got to talking about 
things, then talking about the spam issue, and then the whole idea 
basically died right there.

-- 
Brad Knowles, 

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

From barry at python.org  Fri Jan 30 16:07:01 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 16:50:49 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <401ABB59.6080101@bellanet.org>
References: <401ABB59.6080101@bellanet.org>
Message-ID: <1075496820.1075.226.camel@anthem>

On Fri, 2004-01-30 at 15:15, Kevin McCann wrote:

> I'll continue to look forward for solutions. Flame away if it'll make 
> you feel like a better person, but I'm not giving up that easily.

I for one, hope you don't give up Kevin!

Mailman's come a long way since its humble beginnings, and I enjoy
seeing where people are pushing it as software, and as a project.

it-only-looks-like-a-long-way-down-ly y'rs,
-Barry



From barry at python.org  Fri Jan 30 16:40:04 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 16:52:21 2004
Subject: [Mailman-Developers] Any SQL people going to Barry's sprint?
In-Reply-To: <401A66B5.2070101@bellanet.org>
References: <401A66B5.2070101@bellanet.org>
Message-ID: <1075498803.1075.254.camel@anthem>

On Fri, 2004-01-30 at 09:14, Kevin McCann wrote:

> Barry, what's the word? Have any SQL folks contacted you?

/No one/ has contacted me about SQL or anything else.  My co-worker Fred
Drake's offered to hack on a MM3 sprint, but since he's in the same boat
as I am, we probably won't do it if it's just the two of us.

-Barry



From barry at python.org  Fri Jan 30 15:53:24 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 17:05:28 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <401AC0B7.1090504@bellanet.org>
References: 
	
	 <401A6185.2080003@bellanet.org>
	<1075493165.1075.157.camel@anthem>  <401AC0B7.1090504@bellanet.org>
Message-ID: <1075496003.1075.218.camel@anthem>

On Fri, 2004-01-30 at 15:38, Kevin McCann wrote:

> Thanks for you cordial and helpful response.  If I can get up-to-speed 
> with Python in order to work on the MySQL side of things, or if you 
> think I could contribute with just the MySQL know-how, I'll go. 
> Otherwise, I'll be sending someone else. I'd ideally like to send 
> someone from one of our partner organizations involved with the Dgroups 
> project, but if we can't  find a suitable candidate, then maybe we can 
> find someone from this list. And as I've mentioned before, we'll fund 
> it. So, if anyone is interested in working with Barry on Mailman/SQL in 
> March, let me know. 

At this stage, I'd be happy with just MySQL, or more generally, database
expertise.  I'm at the stage in my MM3 experimentation where we need to
solidify the interfaces.  Code comes later, but I'd be really happy if
we could come away from a MM3 sprint with solid APIs to the various data
storages, and a good architecture for handling transactions across
potentially disparate databases, etc.  I have no problem implementing a
back-end for BerkeleyDB and/or ZODB.  I could probably kludge my way
through a MySQL back-end (although I'm not really a huge fan of the
MySQLdb package).

To Kevin and anyone else who wants to participate: please don't wait
until the last minute to sign up for the sprint, or at least signal your
intent.  Space at Pycon will probably be limited, and I will have to
take vacation if I'm going to participate on Monday and Tuesday.  I'm
not going to do that to sit at a table by myself though.  I plan on
being there the Saturday and Sunday before the conference no matter
what.

Pycon sprint page: http://www.python.org/cgi-bin/moinmoin/SprintPlan2004

Mailman sprint page:
http://www.python.org/cgi-bin/moinmoin/Mailman3Sprint

Please add your name to the latter if you're coming.

-Barry



From barry at python.org  Fri Jan 30 22:09:31 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 22:09:39 2004
Subject: [Mailman-Developers] Mailman unsubscribe mechanism unclear
In-Reply-To: <21335446147366@smtp.westbay.net>
References: <21335446147366@smtp.westbay.net>
Message-ID: <1075518570.1075.393.camel@anthem>

On Wed, 2004-01-28 at 16:33, Somuchfun wrote:

> Can someone explain the exact mechanism mailman uses for unsubscriptions? I
> see some of them are done without a confirmation and some members get an
> email with a token asked to return it to confirm their unsubscription.
> I have played around with mailman but are not able to figure out how it
> determines when a confirmation is needed and when not.
> Hostile unsubscriptions are a growing problem and I would need a way to make
> sure all unsubscriptions need to be confirmed.

It's simple.

If the unsub request comes from a trusted source, e.g. a logged in user
or the list admin through the admin pages, no confirmation message is
sent.

If the request comes from an untrusted source, e.g. a non-logged in user
sitting at the options page login, or via an email request, then a
confirmation message is sent.

-Barry



From barry at python.org  Fri Jan 30 22:12:12 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Jan 30 22:12:25 2004
Subject: [Mailman-Developers] How to change that unsubscriptions
	alsorequire confirmation
In-Reply-To: <20283299111534@smtp.westbay.net>
References: <20283299111534@smtp.westbay.net>
Message-ID: <1075518731.1075.396.camel@anthem>

On Wed, 2004-01-07 at 15:28, Somuchfun wrote:

> So basically anyone can unsubscribe someone else.

Not true in Mailman 2.1, unless that someone is a logged in list
administrator.  Then that person can unsub anybody through the
Membership Management pages.

-Barry



From chuqui at plaidworks.com  Fri Jan 30 23:32:38 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 23:32:45 2004
Subject: [Mailman-Developers] I'm hiring...
Message-ID: <7FE624F0-53A6-11D8-8BF3-0003934516A8@plaidworks.com>


god, I just realized I forgot to post these here. Hope you don't mind, 
Barry.

Full-time postmaster position (primarily administrative, not technical):

http://www.plaidworks.com/chuqui/blog/001267.html

"List mom": (moderately technical, with internal evangelism and needs 
analysis, and helping us figure out the next generation of "all this 
stuff")

http://www.plaidworks.com/chuqui/blog/001254.html

And finally: (very technical, short-term contract)

http://www.plaidworks.com/chuqui/blog/001138.html

(note: the job itself is filled, the contract spot it mentions needs to 
be filled in february)

All are in silicon valley, all are being actively interviewed. All have 
to be willing to put up with, well, me hanging around and asking weird 
questions and stuff. Just so you're warned upfront...


From chuqui at plaidworks.com  Fri Jan 30 23:37:23 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 23:37:30 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <18493813070347@smtp.westbay.net>
References: <18493813070347@smtp.westbay.net>
Message-ID: <29B8EA62-53A7-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 30, 2004, at 10:49 AM, Somuchfun wrote:

> We are running a list with about 50,000 subscribers.

that's a fair sized list, yes. What's it running on? Are you asking too 
much of your hardware?

> As an admin I do not really care if some people think AOL does not have
> their act together or not - if I want to have my emails reach them 
> then I
> have to play by their rules.

The people who think you can just ignore AOL have a really unrealistic 
view of the real world where most of us live.

(on the other hand, if you look at the latest numbers out of the direct 
marketing associations, AOL shed 800,000 paying customers last QUARTER. 
Of those, 450K were converted to a non-revenue style "incentive" 
account (something similar to "N months free if you agree to stay a 
year", but another 390K cancelled anyway despite being given that 
incentive.

By my count, that's over 3% of their user base -- in a quarter. And 
Morgan Stanley's analysts are saying they're expecting that loss to top 
a million paying accounts this calendar year, so unless AOL can figure 
this out, we're talking serious death spiral numbers. If you lose 1 out 
of every 30 customers in a three month period, something's seriously 
ugly...)

> Mailman needed with personalization about 8.5 hrs. to send out one 
> message
> to all 50,000 people and Lyris Listmanager needed about 4.5 hours.

And mailman is free and volunteer based, and Lyris, well, very much 
isn't. And that definitely makes a difference. there is a TAANSTAFL 
aspect here...

> So perhaps mailman is better for smaller discussion list than for 
> larger
> email lists.

that is true of almost all MLM's. there are very few specifically 
optimized for large-scale operations, and 50K is fairly large (well, 
not for me, but for most of the world). and I admit upfront I don't run 
any of my large lists on Mailman. They all run on custom built systems 
optimized for those operations. (and we're hiring help to work on these 
things, I just posted pointers to more info separately)

> Some people here have suggested that anything besides email discussion 
> lists
> are spam, I find statements like this alarming.

they room with the folks who think you can tell AOL to go to hell... 
(grin)

> So far I like mailman's management capabilities. The performance has 
> left me
> being disappointed.

I'll tell you what: if you find a better free and open source MLM than 
Mailman for your needs, I'll buy you a nice dinner (because you won't). 
At some point, "off the shelf" solutions stop scaling, no matter what 
they are. And at some point, either you find a company like Lyris and 
pay for their expertise, or find a geek like me or JC and pay us for 
ours. Even though both of us also volunteeer time back to Mailman as 
well, as does Barry and the other key developers, and we have the 
knowledge to take Mailman and build a tool that'd blow Lyris off the 
map (and we do), this ain't our paying job, and what we don't have is 
the time to do it. Nor, for 95% of the people who use Mailman, do we 
need to...



From chuqui at plaidworks.com  Fri Jan 30 23:40:25 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 23:40:33 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 
Message-ID: <961C945A-53A7-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 30, 2004, at 11:10 AM, moron@industrial.org wrote:
> Howdy.  I would think that Mailman's job is not to provide free 
> marketing
> tools but to act as a list processor.

Mailman is a tool. Asking it to discern intent in its use is like 
asking a gun to only shoot bad people. The gun does what it's told. So 
does Mailman.

And your view of this stuff is very simplistic, IMHO. The real world is 
a lot more complex.

> But that's just my opinion of course.
>

ditto, of course.



From chuqui at plaidworks.com  Fri Jan 30 23:47:15 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 23:47:46 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <401AB39E.1070204@bellanet.org>
References: 
	
	 <401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
	<401AA120.3040002@bellanet.org> 
	<401AB39E.1070204@bellanet.org>
Message-ID: <8AC2CA21-53A8-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 30, 2004, at 11:42 AM, Kevin McCann wrote:

> I'm simply thinking about MLM challenges, which are increasing every 
> day, it seems,

I disagree. the MLM stuff is doing quite well. There are challenges at 
the e-mail level, but non-MLM-email suffers as badly as MLM-email. And 
I really think the spammers have moved into their own version of the 
"battle of the bulge", an increasingly difficult fight with ever 
reducing gains. It's ugly right now, but it seems to me that's at least 
in part due to an increasing sense of urgency by the spammers.

(more here if you care: 
http://www.plaidworks.com/chuqui/blog/001252.html)

> and thinking about what can be done. Personalization without critical 
> slowdown is an issue.

define "critical". personalization is inherently more resource 
intensive than not personalizing. Physics wins. it's more difficult to 
send 20 emails to 20 people than one email to 20 people; and the 
reality is, you can't personalize without sending those 20 emails.

Can things be improved to minimize that resource cost? Definitely. Is 
it a high priority? For the vast majority of mailman users, no.


From chuqui at plaidworks.com  Fri Jan 30 23:49:54 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 23:50:03 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: <1075493625.1075.166.camel@anthem>
References: 
	
	 <401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
	<401AA120.3040002@bellanet.org> <1075493625.1075.166.camel@anthem>
Message-ID: 


On Jan 30, 2004, at 12:13 PM, Barry Warsaw wrote:
> It's possible.  But then again who knows what the email landscape will
> look like in 5 years?  I'm betting it'll look a /lot/ different than it
> does today, unless it doesn't.

Raises hand.

Or more correctly, two visions. what I think it should be, and what I'm 
afraid it will be.

But more on that sometime later....


From chuqui at plaidworks.com  Fri Jan 30 23:54:58 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Fri Jan 30 23:55:05 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <401ABB59.6080101@bellanet.org>
References: <401ABB59.6080101@bellanet.org>
Message-ID: <9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 30, 2004, at 12:15 PM, Kevin McCann wrote:

>  I have the unenviable challenge of moving these people to a Mailman 
> environment without making them feel like it is a step backward.


A laudible goal. And sometimes, the only way to make it possible is 
through some front end investment. Open Source isn't free, it simply 
doesn't have a price tag. the cost is in sweat equity and hours 
invested by interested parties.

So to leverage open source, you have three choices:

1) find packages and projects that are sympatico with your needs and 
interests, or at least come close enough to take advantage of them.

2) start typing -- dig in and do some sweating to take a package in the 
directions you need it taken.

3) influence and support others to do the sweating to your needs: in 
other words, patronage.

> I'll continue to look forward for solutions.

there are always solutions. Not all of them are already packaged up and 
waiting to be found, or easily created through talk.





From kmccann at bellanet.org  Sat Jan 31 00:46:44 2004
From: kmccann at bellanet.org (Kevin McCann)
Date: Sat Jan 31 00:24:23 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>
References: <401ABB59.6080101@bellanet.org>
	<9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>
Message-ID: <401B4144.9020905@bellanet.org>

>
>
>> I'll continue to look forward for solutions.
>
>
> there are always solutions. Not all of them are already packaged up 
> and waiting to be found, or easily created through talk.

I understand this, Chuq. For me, the most pressing thing is the SQL 
activity. And as I have mentioned, we're ready to contribute resources 
to this end. Either I will get involved, if I can get up to speed 
technically, or we'll hire someone to get involved with Barry. I don't 
want to just talk about this stuff. I want to see things happen. I don't 
know precisely how much funding I can get for this, but I'll be doing 
what I can. But some discussion *is* necessary. I have been harping 
about SQL in Mailman for a year now but it is only recently that the 
mere thought of it being worked into MM in one way or another has being 
entertained. Talking isn't entirely useless. In fact it's  necessary in 
order to influence change. Just so long as it doesn't stop at talk.

- Kevin



From chuqui at plaidworks.com  Sat Jan 31 00:35:29 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Sat Jan 31 00:35:36 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <401B4144.9020905@bellanet.org>
References: <401ABB59.6080101@bellanet.org>
	<9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>
	<401B4144.9020905@bellanet.org>
Message-ID: <47D6FF17-53AF-11D8-8BF3-0003934516A8@plaidworks.com>


On Jan 30, 2004, at 9:46 PM, Kevin McCann wrote:
>> there are always solutions. Not all of them are already packaged up 
>> and waiting to be found, or easily created through talk.
>
> I understand this, Chuq. For me, the most pressing thing is the SQL 
> activity. And as I have mentioned, we're ready to contribute resources 
> to this end.

  Fair 'nuff. I thought you did, but it's stuff that has to be said once 
in a while, anyway.

> I have been harping about SQL in Mailman for a year now but it is only 
> recently that the mere thought of it being worked into MM in one way 
> or another has being entertained. Talking isn't entirely useless. In 
> fact it's  necessary in order to influence change. Just so long as it 
> doesn't stop at talk.
>

I've got a bit of MySQL background. My time is still limited, so I 
won't commit to coding I can't depend on myself to finish -- but if you 
want someone to help out on DB design and stuff, I'm in. I'll do what I 
can and maybe help avoid some of the potholes. I'd like to see this, 
too.





From kmccann at bellanet.org  Sat Jan 31 01:17:41 2004
From: kmccann at bellanet.org (Kevin McCann)
Date: Sat Jan 31 00:55:21 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <47D6FF17-53AF-11D8-8BF3-0003934516A8@plaidworks.com>
References: <401ABB59.6080101@bellanet.org>
	<9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>
	<401B4144.9020905@bellanet.org>
	<47D6FF17-53AF-11D8-8BF3-0003934516A8@plaidworks.com>
Message-ID: <401B4885.2090305@bellanet.org>

Chuq Von Rospach wrote:

> I've got a bit of MySQL background. My time is still limited, so I 
> won't commit to coding I can't depend on myself to finish -- but if 
> you want someone to help out on DB design and stuff, I'm in. I'll do 
> what I can and maybe help avoid some of the potholes. I'd like to see 
> this, too.

Perfect.Any help would be appreciated. I'm packing it in for tonight (it 
has been a long day), but I'll be in sometime during the weekend. Maybe 
Barry has some ideas as to how things ought to move along wrt his mm3 
work in progress. I'm wondering if some inroads can be made even before 
the sprint session in March. I can show you and Barry the DB stuff I had 
in mind and maybe we can jumpstart from there.

- Kevin


From barry at python.org  Sat Jan 31 09:44:40 2004
From: barry at python.org (Barry Warsaw)
Date: Sat Jan 31 09:44:48 2004
Subject: [Mailman-Developers] AOL's requirements for spam complaints
In-Reply-To: 
References: 
	
	 <401A6185.2080003@bellanet.org>
	<2564415408.1075485461@taltos.ny.ficc.gs.com>
	<401AA120.3040002@bellanet.org> <1075493625.1075.166.camel@anthem>
	
Message-ID: <1075560280.1075.477.camel@anthem>

On Fri, 2004-01-30 at 23:49, Chuq Von Rospach wrote:
> On Jan 30, 2004, at 12:13 PM, Barry Warsaw wrote:
> > It's possible.  But then again who knows what the email landscape will
> > look like in 5 years?  I'm betting it'll look a /lot/ different than it
> > does today, unless it doesn't.
> 
> Raises hand.
> 
> Or more correctly, two visions. what I think it should be, and what I'm 
> afraid it will be.



-Barry



From jeffw at chebucto.ns.ca  Sat Jan 31 13:53:57 2004
From: jeffw at chebucto.ns.ca (Jeff Warnica)
Date: Sat Jan 31 13:51:07 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <47D6FF17-53AF-11D8-8BF3-0003934516A8@plaidworks.com>
References: <401ABB59.6080101@bellanet.org>
	<9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>
	<401B4144.9020905@bellanet.org>
	<47D6FF17-53AF-11D8-8BF3-0003934516A8@plaidworks.com>
Message-ID: <20040131145357.rydfbskgkw48og80@george.warnica>


I feel bad about jumping in to this discussion, and not necessaraly being
prepared to offer any real help, but anyway.

LDAP may be a better data[base|sink] for configuration data then SQL. It may be
harder to work with (only because less people have experience with it), and it
may be less usefull to a lot of sites (everyone has MySQL installed, not so for
LDAP), but preference storage is a common use for LDAP. For sites who already
have LDAP deployed, you can leverage the existing authentication data. LDAP
stores "objects", SQL "rows"...

Sory if I sound like a marketdroid, but I'd feel bad if LDAP wasent at least
considered.

Quoting Chuq Von Rospach :

> On Jan 30, 2004, at 9:46 PM, Kevin McCann wrote:

>> I have been harping about SQL in Mailman for a year now but it is 
>> only recently that the mere thought of it being worked into MM in 
>> one way or another has being entertained. Talking isn't entirely 
>> useless. In fact it's  necessary in order to influence change. Just 
>> so long as it doesn't stop at talk.
>>
>
> I've got a bit of MySQL background. My time is still limited, so I
> won't commit to coding I can't depend on myself to finish -- but if you
> want someone to help out on DB design and stuff, I'm in. I'll do what I
> can and maybe help avoid some of the potholes. I'd like to see this,
> too.

From chuqui at plaidworks.com  Sat Jan 31 14:10:36 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Sat Jan 31 14:11:23 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <20040131145357.rydfbskgkw48og80@george.warnica>
References: <401ABB59.6080101@bellanet.org>
	<9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>
	<401B4144.9020905@bellanet.org>
	<47D6FF17-53AF-11D8-8BF3-0003934516A8@plaidworks.com>
	<20040131145357.rydfbskgkw48og80@george.warnica>
Message-ID: <269B3300-5421-11D8-B9A7-0003934516A8@plaidworks.com>


On Jan 31, 2004, at 10:53 AM, Jeff Warnica wrote:

>
> I feel bad about jumping in to this discussion, and not necessaraly 
> being
> prepared to offer any real help, but anyway.
>
> LDAP may be a better data[base|sink] for configuration data then SQL.

actually, Jeff has a point. I was muttering to myself about this last 
night.

Mailman <-> LDAP as an interface means that anything that can generate 
an LDAP interface can talk to it. so perhaps the best thing to do is 
come up with an LDAP interface, define how the LDAP data should look, 
and then create a set of MySQL schemas that'll support that. I know 
barry's wanted to avoid requiring too many "things" to be installed to 
use Mailman, but when someone chooses to move to MySQL, I don't think 
it's unfair to assume they have or can install LDAP also.

And that would allow someone to use some other LDAP backing store more 
easily, since we're hiding the interface in a fairly standard setup. 
And with MySQL as the proof of concept, having somenoe add support for 
Postgres SQL, Oracle, DB files or something else would be a lot easier.

That'd also make life really rather nice for larger operations, because 
it's not far from that to splitting off the web piece from the delivery 
piece from the data store piece, allowing it to exist in organizations 
where services are cloistered from each other.

I like the idea. It's not much more work (if it's more work at all), 
and much more flexible, and creates an interface others can connect to 
as well, not a single vendor solution. I'll buy in...


From Dale at Newfield.org  Sat Jan 31 14:28:05 2004
From: Dale at Newfield.org (Dale Newfield)
Date: Sat Jan 31 14:28:07 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <401B4885.2090305@bellanet.org>
References: <401ABB59.6080101@bellanet.org>
	<9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>
	<401B4144.9020905@bellanet.org>
	<47D6FF17-53AF-11D8-8BF3-0003934516A8@plaidworks.com>
	<401B4885.2090305@bellanet.org>
Message-ID: 

On Sat, 31 Jan 2004, Kevin McCann wrote:
> I'm wondering if some inroads can be made even before the sprint session
> in March. I can show you and Barry the DB stuff I had in mind and maybe
> we can jumpstart from there.

I likewise had an abortive attempt to make the api changes necessary to
fold in a more scalable sql backend, and would offer that as another
example if we want to start these discussions before March.  I'm dubious,
though, that we'll get as much progress for the time investment doing this
through email instead of in person...

---
Dale Newfield 

"They that can give up essential liberty to obtain a little safety deserve
neither liberty nor safety." - Benjamin Franklin, on the Statue of Liberty

From Dale at Newfield.org  Sat Jan 31 14:34:26 2004
From: Dale at Newfield.org (Dale Newfield)
Date: Sat Jan 31 14:34:28 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: 
References: <401ABB59.6080101@bellanet.org>
	<9EF71992-53A9-11D8-8BF3-0003934516A8@plaidworks.com>
	<401B4144.9020905@bellanet.org>
	<47D6FF17-53AF-11D8-8BF3-0003934516A8@plaidworks.com>
	<401B4885.2090305@bellanet.org>
	
Message-ID: 

On Sat, 31 Jan 2004, Dale Newfield wrote:
> if we want to start these discussions before March.

Oh--I forgot to mention--I'll be joining Barry and whoever in March for
Sunday and Monday of the sprint, and I've got a bunch of SQL experience...
...wasn't somebody asking if anybody fit that description?

-Dale

From moron at industrial.org  Sat Jan 31 14:35:20 2004
From: moron at industrial.org (moron)
Date: Sat Jan 31 14:35:58 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <269B3300-5421-11D8-B9A7-0003934516A8@plaidworks.com>
References: <401ABB59.6080101@bellanet.org>
	<20040131145357.rydfbskgkw48og80@george.warnica>
	<269B3300-5421-11D8-B9A7-0003934516A8@plaidworks.com>
Message-ID: <200401311135.20670.moron@industrial.org>

On January 31, 2004 11:10 am, Chuq Von Rospach wrote:
> Mailman <-> LDAP as an interface means that anything that can generate
> an LDAP interface can talk to it. so perhaps the best thing to do is
> come up with an LDAP interface, define how the LDAP data should look,
> and then create a set of MySQL schemas that'll support that. I know
> barry's wanted to avoid requiring too many "things" to be installed to
> use Mailman, but when someone chooses to move to MySQL, I don't think
> it's unfair to assume they have or can install LDAP also.

Isn't LDAP a bit of a security hassle?  I would think it is pretty common to 
have Mailman running on a machine along side MySQL, Apache and and MTA of 
some sort but wouldn't throwing in LDAP be more like requiring people install  
a CVS daemon to use Mailman?  I'm no LDAP guru but from what I have looked at 
previously it certainly seemed that way.

Cheers

-- 
---> (culture) http://industrial.org : (label)  http://deterrent.net
---> (community) http://ampfea.org : (hire me) http://codegrunt.com
---> (send EEEI news to) infosuck@industrial.org
---> Whomever dies with the most URLs wins!!!!!!!!!!!!!


From chuqui at plaidworks.com  Sat Jan 31 14:40:16 2004
From: chuqui at plaidworks.com (Chuq Von Rospach)
Date: Sat Jan 31 14:41:04 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <200401311135.20670.moron@industrial.org>
References: <401ABB59.6080101@bellanet.org>
	<20040131145357.rydfbskgkw48og80@george.warnica>
	<269B3300-5421-11D8-B9A7-0003934516A8@plaidworks.com>
	<200401311135.20670.moron@industrial.org>
Message-ID: <4B76411C-5425-11D8-B9A7-0003934516A8@plaidworks.com>


>
> Isn't LDAP a bit of a security hassle?  I would think it is pretty 
> common to
> have Mailman running on a machine along side MySQL, Apache and and MTA 
> of
> some sort but wouldn't throwing in LDAP be more like requiring people 
> install
> a CVS daemon to use Mailman?  I'm no LDAP guru but from what I have 
> looked at
> previously it certainly seemed that way.
>

it's secure enough that most major IS/IT organizations seem to be 
standardizing it for distributing information like this. We now have, 
in fact, an internal list server that does exactly this now, but I 
wasn't directly involved in that project (it was a rewrite of a system 
I wrote years ago by someone else). And in MacOS X 10.3, Apple started 
moving it's stuff from NetInfo to OpenLDAP.

I think if it's secure enough for companies to implement, we don't have 
many worries if we take care to use it properly.





From jeffw at chebucto.ns.ca  Sat Jan 31 15:56:58 2004
From: jeffw at chebucto.ns.ca (Jeff Warnica)
Date: Sat Jan 31 15:54:11 2004
Subject: [Mailman-Developers] A bit of perspective ....
In-Reply-To: <200401311135.20670.moron@industrial.org>
References: <401ABB59.6080101@bellanet.org>
	<20040131145357.rydfbskgkw48og80@george.warnica>
	<269B3300-5421-11D8-B9A7-0003934516A8@plaidworks.com>
	<200401311135.20670.moron@industrial.org>
Message-ID: <20040131165658.ayumtq4g84w0804k@george.warnica>


I suppose it can be, but it is a question of where you implement your security.
If mailman is to use SQL to store preferences then it is up to mm to deal with
what records a user can update. If the mm interface to LDAP goes through one
master LDAP account, then it is still mm's job... But if mm binds to LDAP as
the mm user, then security is the responsibility of the LDAP server. With
OpenLDAP, and NDS permissions can be extreemly fine grained, down to the
attribute level. Ive not so much as seen ADS running anywhere, but I can only
assume that it does too.

How secure an admin might want to make it is likely to be related to what else,
if anything, their LDAP directory is being used for. A hypothetical site with
10,000 users in NDS, and 100,000 other things (printers, queues....), which
they have been using for a decade, may be very restrictive. Another site
installing MM+LDAP for fun as much as anything else, might just give the MM
user unlimited rights.

Kinda like the Bugzilla install docs something to the effect of  ".... MySQL's
security is an evil beast, and some people actualy use it. If you do, just make
sure that 'bugz' has the right rights...."


Quoting moron :

> On January 31, 2004 11:10 am, Chuq Von Rospach wrote:
>> Mailman <-> LDAP as an interface means that anything that can generate
>> an LDAP interface can talk to it. so perhaps the best thing to do is
>> come up with an LDAP interface, define how the LDAP data should look,
>> and then create a set of MySQL schemas that'll support that. I know
>> barry's wanted to avoid requiring too many "things" to be installed to
>> use Mailman, but when someone chooses to move to MySQL, I don't think
>> it's unfair to assume they have or can install LDAP also.
>
> Isn't LDAP a bit of a security hassle?  I would think it is pretty common to
> have Mailman running on a machine along side MySQL, Apache and and MTA of
> some sort but wouldn't throwing in LDAP be more like requiring people install
> a CVS daemon to use Mailman?  I'm no LDAP guru but from what I have looked at
> previously it certainly seemed that way.

From my at freexp.de  Sat Jan 31 17:49:00 2004
From: my at freexp.de (Michael Heydekamp)
Date: Sat Jan 31 17:51:26 2004
Subject: [Mailman-Developers] Translation F'up2 => Reply-To
Message-ID: <91v3LySKpwB@my.freexp.de>

Hi,

anybody here who could deliver Python code for the following?

In news2mail direction, I'd like to "translate" the F'up2 header
(containing one or more newsgroups) to the Reply-To: header by
preserving any already existing Reply-To addresses.

One of four mailing lists is assigned to each of four newsgroups
respectively.  These mailing list addresses shall be concatenated to the
Reply-To: header.  Other arbitrary newsgroups in the F'up2 header should
simply be disregarded.

In mail2news direction I managed this by tweaking the INN filter, but
I'm far beyond from being that familiar with Python as I am with Perl.

Some example code to start with would already help.


        Michael

From andrew_lists at crashbox.com  Fri Jan 23 12:27:45 2004
From: andrew_lists at crashbox.com (Andrew Mellinger)
Date: Sun Feb  8 13:48:52 2004
Subject: [Mailman-Developers] Tracking down a permissions bug in attachments
Message-ID: 

Y'all,

  I think I may have found a bug with the attachments code and hoped that
you guys could give me some input.

  When trying to save an attachment my installation of mailman provides the
following error.

+----

Jan 23 07:05:52 2004 (24615) Traceback (most recent call last):
  File "/usr/pkg/lib/mailman/Mailman/Queue/Runner.py", line 110, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/pkg/lib/mailman/Mailman/Queue/Runner.py", line 160, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/pkg/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in
_dispose
    mlist.ArchiveMail(msg)
  File "/usr/pkg/lib/mailman/Mailman/Archiver/Archiver.py", line 208, in
ArchiveMail
    h.processUnixMailbox(f)
  File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 544, in
processUnixMailbox
    m = mbox.next()
  File "/usr/pkg/lib/python2.2/mailbox.py", line 34, in next
    return self.factory(_Subfile(self.fp, start, stop))
  File "/usr/pkg/lib/mailman/Mailman/Mailbox.py", line 89, in scrubber
    return mailbox.scrub(msg)
  File "/usr/pkg/lib/mailman/Mailman/Mailbox.py", line 109, in scrub
    return self._scrubber(self._mlist, msg)
  File "/usr/pkg/lib/mailman/Mailman/Handlers/Scrubber.py", line 219, in
process
    url = save_attachment(mlist, part, dir, filter_html=0)
  File "/usr/pkg/lib/mailman/Mailman/Handlers/Scrubber.py", line 344, in
save_attachment
    makedirs(fsdir)
  File "/usr/pkg/lib/mailman/Mailman/Handlers/Scrubber.py", line 336, in
makedirs
    os.path.walk(dir, twiddle, None)
  File "/usr/pkg/lib/python2.2/posixpath.py", line 279, in walk
    func(arg, top, names)
  File "/usr/pkg/lib/mailman/Mailman/Handlers/Scrubber.py", line 335, in
twiddle
    os.chmod(dirname, 02775)
OSError: [Errno 1] Operation not permitted:
'/var/db/mailman/archives/private/rq-rules/attachments/20040123/72aeb309'

+----------

  Now, it looks to me like Mailman is trying to set permissions on the newly
created directory to 02775.  I'm running on NetBSD which doesn't let anyone
but the superuser set the 2000 (setuid) bit.  I imagine this setting (02775)
was done for linux which overloads the setuid bit for as 'set group' on new
files.

  Does this sound like a good assessment?  If so, is there a generic way to
turn this sort of thing off in a config?  (I searched by couldn't find
anything.)  Isn't this something that should be handled at build time?

  Thanks for any feedback!

-Andrew


From goebel at emunix.emich.edu  Fri Jan 23 09:42:58 2004
From: goebel at emunix.emich.edu (Matt Goebel)
Date: Sun Feb  8 13:49:08 2004
Subject: [Mailman-Developers] Filtering certain characters from email
	addresses?
Message-ID: <200401231442.i0NEgvfG016673@emunix.emich.edu>


Hello,

 Searched the archives and didn't see any references to this.

 Would it be possible to add some character filtering for email
addresses before they are added to a list?  I've been having more 
and more problems with admins using the mass subscribe function 
where they are cutting and pasting from word and picking up
null characters, and other odd things.  Mailman will happily 
add the addresses but then can't do anything with them later on, 
forcing me to use a hex editor to remove the odd characters.

Thanks,
Mattias

-- 
Matthew Goebel : goebel@emunix.emich.edu : Unix Jockey @ EMU : Hail Eris
Neo-Student, Net Lurker, Donut consumer, and procrastinating Furry Fan.
 "Always with the negative waves, Moriarty" - Oddball
 "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer

From rrussell at commandprompt.com  Thu Jan 22 15:12:56 2004
From: rrussell at commandprompt.com (Robby Russell)
Date: Sun Feb  8 13:49:37 2004
Subject: [Mailman-Developers] Mailman/PostgreSQL
Message-ID: <40102EBC.4040504@commandprompt.com>


I am considering starting a new python program that would work very 
similar to mailman but involve a PostgreSQL back end. I haven't looked 
too far into the mailman code to determine just how much data it is 
currently storing, so I am not sure if an abstraction layer to pgsql is 
the best solution for what I am aiming to do.

My goal is to create an application that works very similar to mailman 
with a database back end and contains reporting. At the moment, I am not 
sure whether or not mailman has any reporting features that would 
generate pretty pages with graphs, stats, etc.

Is there a reporting feature for mailman that someone has built already? 
I want statistics of who posts the most on a list, how many people 
subscribed between dates, by month, year, how many people unsub'd during 
any specific period. (perhaps people unsub'd during a heated thread... 
i'd like to track that sort of information).

So, as Mailman developers, do you see an 'easier' route using the 
existing infrastructure to handle this sort of reporting with a db 
backend? or should I start from scratch and use some of mailmans 
technology in the process?

Anyone interested in assisting me in such a project? are there any 
projects that some of you may be aware that already exist that work like 
this?

Cheers,

Robby


-- 
#-------------------------------------------------------
# Robby Russell,  |  Sr. Administrator / Lead Programmer
# Command Prompt, Inc.   |  http://www.commandprompt.com
# rrussell@commandprompt.com | Telephone: (503) 667.4564
#-------------------------------------------------------

