From mat at cisco.com Thu Feb 1 17:54:25 2007 From: mat at cisco.com (Michael Thomas) Date: Thu, 01 Feb 2007 08:54:25 -0800 Subject: [Mailman-Developers] dkim-signature headers Message-ID: <45C21B41.6030905@cisco.com> Hi all, I'm one of the authors of the DKIM protocol and it recently came to my attention that you've recently changed mailman to remove DK and DKIM signature headers when you remail the message. This is incorrect behavior: in Section 4: Signers SHOULD NOT remove any DKIM-Signature header fields from messages they are signing, even if they know that the signatures cannot be verified. This actually applies to everybody. There are several reasons for this. First is that DKIM allows you to specify the length of a body so it is not the case a priori that mailman will destroy the signature. Second, other heuristics can be applied to make mailing list traversal even better such as using the z= tag to determine whether trivial subject modifications have been made. Third and probably most important is that removing the signature is actually harmful rather than helpful: a broken signature and a missing signature MUST be treated as equivalent to no signature at all (lest an attacker just add a fake DKIM-signature header to get preferential treatment), and as above the verifier loses the ability to recover the signature. Just as an FYI, we have deployed DKIM across all of Cisco and our successful mailing list traversal rate is about 99% -- a large percentage of which are through mailman lists. By making this change, you've taken the verify rate from 99% to 0% in one swell foop. Not good. Mike From joe at skyrush.com Thu Feb 1 19:13:42 2007 From: joe at skyrush.com (Joe Peterson) Date: Thu, 01 Feb 2007 11:13:42 -0700 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C21B41.6030905@cisco.com> References: <45C21B41.6030905@cisco.com> Message-ID: <45C22DD6.5030208@skyrush.com> Hi Michael, Thanks for writing about this. I suspect many are under the impression that passing messages through mail lists tended to break DomainKeys and DKIM (I know I was one, at least back when I was experimenting a lot with it). In fact, it always seemed to break on my Mailman lists, leaving what seemed to be a misleading result. It seemed that mail lists would need to "re-sign" messages as the new sender for it to work right, and did that not require the old key to be removed? Maybe things have changed recently, or maybe at least I misunderstood some aspects of it. If, in fact, there is a way for the two to play well together, I think that would be great! It would remove one of the big "downsides" of using DK/DKIM. Since Mailman is one of the (if not *the* most) popular mail list systems, perhaps you can help Mailman developers to make the two things better integrated. I would think/hope there would be a way to increase the handling/interaction to allow for 100% reliable flow-through of keys. That would really help DKIM's cause, I think, since so many people use mailing lists. Is this not theoretically possible if both Mailman and DKIM obey an understood set of rules? I believe that a solid way to "fix" this is a lot better than a stochastic one. As for body length, would that not have to be a "guess"? I mean, short messages would fail due to the footer, and long ones would not get full checking (after the set length), right? If so, that sounds like a less than optimal solution. I, for one, would love to see the email "accountability" problem solved. With spam becoming the one thing that could ruin the Internet, we need all the help we can get. I used DKIM for a while on my own server, but because of Mailman gotchas, etc. (and maybe these were more a problem of perception on my part), it seemed too easy to break. Also, with anything short of near global acceptance, it really seems like a tough battle, especially when so many are adopting things like SPF, which I never liked at all. Certainly, resolving the mailing list issue would be a huge step in the right direction, so consider me an enthusiastic supporter of your efforts. -Joe Michael Thomas wrote: > Hi all, > > I'm one of the authors of the DKIM protocol and it recently came to > my attention that you've recently changed mailman to remove DK and > DKIM signature headers when you remail the message. This is incorrect > behavior: > > in Section 4: > > Signers SHOULD NOT remove any DKIM-Signature header fields from > messages they are > signing, even if they know that the signatures cannot be verified. > > This actually applies to everybody. There are several reasons for this. > First > is that DKIM allows you to specify the length of a body so it is not the > case > a priori that mailman will destroy the signature. Second, other > heuristics can > be applied to make mailing list traversal even better such as using the > z= tag > to determine whether trivial subject modifications have been made. Third and > probably most important is that removing the signature is actually > harmful rather > than helpful: a broken signature and a missing signature MUST be treated as > equivalent to no signature at all (lest an attacker just add a fake > DKIM-signature > header to get preferential treatment), and as above the verifier loses > the ability > to recover the signature. > > Just as an FYI, we have deployed DKIM across all of Cisco and our successful > mailing list traversal rate is about 99% -- a large percentage of which > are through > mailman lists. By making this change, you've taken the verify rate from > 99% to > 0% in one swell foop. Not good. > > Mike > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/joe%40skyrush.com > > Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp > From mat at cisco.com Thu Feb 1 20:17:04 2007 From: mat at cisco.com (Michael Thomas) Date: Thu, 01 Feb 2007 11:17:04 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C22DD6.5030208@skyrush.com> References: <45C21B41.6030905@cisco.com> <45C22DD6.5030208@skyrush.com> Message-ID: <45C23CB0.1060805@cisco.com> Joe Peterson wrote: > Hi Michael, > > Thanks for writing about this. I suspect many are under the impression > that passing messages through mail lists tended to break DomainKeys and > DKIM (I know I was one, at least back when I was experimenting a lot > with it). In fact, it always seemed to break on my Mailman lists, > leaving what seemed to be a misleading result. It seemed that mail > lists would need to "re-sign" messages as the new sender for it to work > right, and did that not require the old key to be removed? Maybe things > have changed recently, or maybe at least I misunderstood some aspects of it. > I'm not sure whether Murray's dkim milter allows you to sign with the l= option or not, but setting the body length allows text to be appended to the end of a message -- like a mailing list trailer. This combined with some heuristics with subject line modification gets you to nearly 100% success rates. The only other problem that I've had with mailman is that it sometimes reorders the dkim-signature header field itself which causes the signature to break. I've been trying to figure out where in the code it does that so that we could either fix it or compensate for it when I'm creating the signature itself. As for resigning -- that would certainly be a Good Thing, though not a cure-all. It's always better to have more accountability rather than less, but a valid first party signature (ie, From) is best of all. But in its absence, it would be nice for a remailer to sign the message so that it's possible for the receiver to whitelist them as a legit third party signer. It's not entirely clear to me whether implementing it in the MTA or mailman is better, though I suppose that there would be no harm having it in both places as an option. It would probably be a pretty modest matter to implement a signer -- dealing with keys is really the biggest PITA. > If, in fact, there is a way for the two to play well together, I think > that would be great! It would remove one of the big "downsides" of > using DK/DKIM. Since Mailman is one of the (if not *the* most) popular > mail list systems, perhaps you can help Mailman developers to make the > two things better integrated. > Yes, that would be most excellent -- I've been meaning to subscribe for a while about these issues. > I would think/hope there would be a way to increase the > handling/interaction to allow for 100% reliable flow-through of keys. > That would really help DKIM's cause, I think, since so many people use > mailing lists. Is this not theoretically possible if both Mailman and > DKIM obey an understood set of rules? I believe that a solid way to > "fix" this is a lot better than a stochastic one. > I've for quite a while thought that part of an ultimate DKIM BCP would give some guidance on what a "well behaved mailing list" would be. It would certainly be good if mailman were an example of that because at least at Cisco it accounts for the bulk of external mailing list traffic we see. > As for body length, would that not have to be a "guess"? I mean, short > messages would fail due to the footer, and long ones would not get full > checking (after the set length), right? If so, that sounds like a less > than optimal solution. > No, the body length just counts the number of canonical bytes in the body when it's computing the hash, so that if there are bytes that are appended, it doesn't affect the outcome. The lack of the l= tag implies that the entire body is hashed which is most likely what the sendmail dkim milter does (at least by default). The main issue is that there is a security/robustness tradeoff with the use of l=. That is, bad guys could append content too. On the other hand, *if* that comes to pass, receivers are completely at liberty to scan the covered and uncovered parts of the body differently, delete the appended text, etc, etc. > I, for one, would love to see the email "accountability" problem solved. > With spam becoming the one thing that could ruin the Internet, we need > all the help we can get. I used DKIM for a while on my own server, but > because of Mailman gotchas, etc. (and maybe these were more a problem of > perception on my part), it seemed too easy to break. Also, with > anything short of near global acceptance, it really seems like a tough > battle, especially when so many are adopting things like SPF, which I > never liked at all. Certainly, resolving the mailing list issue would > be a huge step in the right direction, so consider me an enthusiastic > supporter of your efforts. > Thanks, I appreciate that :) The one thing to be said is that you don't actually need to global acceptance to do useful things with dkim (though I'd certainly be happy if there were). The reason that Cisco deployed DKIM internally is to counter spear-phishing attacks. That is, forged mail that is a targeted social engineering attack. For this, we just need to determine whether the mail is or isn't from Cisco's servers and warn people when the message doesn't verify correctly. To pull this off, we need to keep the false positive rate fairly low which means that the legions of geeks using mailing list should mostly work. Which is why I was so alarmed to find out the newest versions of mailman was now doing the wrong thing. In any case, backing out this change would go a *long* way to helping DKIM. Mike From msapiro at value.net Thu Feb 1 21:00:29 2007 From: msapiro at value.net (Mark Sapiro) Date: Thu, 1 Feb 2007 12:00:29 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C23CB0.1060805@cisco.com> Message-ID: Michael Thomas wrote: > >I'm not sure whether Murray's dkim milter allows you to sign with the l= >option or not, but setting the body length allows text to be appended to the >end of a message -- like a mailing list trailer. This combined with some >heuristics >with subject line modification gets you to nearly 100% success rates. Consider that while Mailman doesn't do all of these things to every message, it can do any of the following: - Add text to the beginning of the message body (msg_header) - Add text to the end of the message body (msg_footer) - Remove text from the beginning of the message body (Approved: line) - Add additional MIME parts to a multipart message (msg_header, msg_footer) - Convert a single part message to multipart in order to add msg_header/msg_footer - Remove parts from a multipart message (content filtering) - Convert an HTML part to plain text (content filtering) - Decode a base64 or quoted-printable encoded part and perhaps re-encode it with a different encoding. - Change or delete various headers including Subject:, To:, From: - Replace some MIME parts with URLs of where they were stored and flatten the entire message into a single plain text message (scrubber). - Probably other things I'm overlooking. >The only >other problem that I've had with mailman is that it sometimes reorders the >dkim-signature header field itself which causes the signature to break. I've >been trying to figure out where in the code it does that so that we could >either fix it or compensate for it when I'm creating the signature itself. Other than the code which removes these headers, Mailman doesn't touch them. If they get modified in Mailman processing, it would have to be in the Python email library in the process of parsing the raw message into an email message object and then ultimately converting the message object back into raw text to be sent, but I don't think the email library does anything to headers that aren't explicitly processed in some way. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From mat at cisco.com Thu Feb 1 22:34:08 2007 From: mat at cisco.com (Michael Thomas) Date: Thu, 01 Feb 2007 13:34:08 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: Message-ID: <45C25CD0.1010301@cisco.com> Mark Sapiro wrote: > Michael Thomas wrote: > >> I'm not sure whether Murray's dkim milter allows you to sign with the l= >> option or not, but setting the body length allows text to be appended to the >> end of a message -- like a mailing list trailer. This combined with some >> heuristics >> with subject line modification gets you to nearly 100% success rates. >> > > > Consider that while Mailman doesn't do all of these things to every > message, it can do any of the following: > > - Add text to the beginning of the message body (msg_header) > - Add text to the end of the message body (msg_footer) > - Remove text from the beginning of the message body (Approved: line) > - Add additional MIME parts to a multipart message (msg_header, > msg_footer) > - Convert a single part message to multipart in order to add > msg_header/msg_footer > - Remove parts from a multipart message (content filtering) > - Convert an HTML part to plain text (content filtering) > - Decode a base64 or quoted-printable encoded part and perhaps > re-encode it with a different encoding. > - Change or delete various headers including Subject:, To:, From: > - Replace some MIME parts with URLs of where they were stored and > flatten the entire message into a single plain text message (scrubber). > - Probably other things I'm overlooking. > Yes, there's no question that mailman as well as lots of other software can destroy signatures. In practice as people seem to actually use them, it is more theoretical than real. We've been running DKIM signers/verifiers for going on a year now and the 99% I quoted is across a 25000 user population which probably uses mailing lists far more than most similarly sized companies. [note some of the things you mention can be worked around too] > > >> The only >> other problem that I've had with mailman is that it sometimes reorders the >> dkim-signature header field itself which causes the signature to break. I've >> been trying to figure out where in the code it does that so that we could >> either fix it or compensate for it when I'm creating the signature itself. >> > > > Other than the code which removes these headers, Mailman doesn't touch > them. If they get modified in Mailman processing, it would have to be > in the Python email library in the process of parsing the raw message > into an email message object and then ultimately converting the > message object back into raw text to be sent, but I don't think the > email library does anything to headers that aren't explicitly > processed in some way. > > That's something to go on. I haven't been able to figure out what exactly triggers this behavior because it doesn't appear to be very consistent. Mike From msapiro at value.net Fri Feb 2 00:06:25 2007 From: msapiro at value.net (Mark Sapiro) Date: Thu, 1 Feb 2007 15:06:25 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C25CD0.1010301@cisco.com> Message-ID: Michael Thomas wrote: > >Yes, there's no question that mailman as well as lots of other software >can destroy signatures. In practice as people seem to actually use them, >it is more theoretical than real. We've been running DKIM signers/verifiers >for going on a year now and the 99% I quoted is across a 25000 user >population which probably uses mailing lists far more than most similarly >sized companies. I'm sure your statistics are valid for your environment, but I'm not sure that they are universally applicable. Consider what I think is a fairly typical situation exemplified by mailman-users at python.org. I don't know what fraction of incoming posts to this list are multipart/alternative with text/plain and text/html alternative parts, but I see many just from people who Cc: me directly. It would be a fairly simple matter to go through the .mbox archive for any list that has one and count the number of X-Content-Filtered-By: Mailman/MimeDel and compare that to the number of messages. in fact, I just did that for a cycling club discussion list I managed, and just over 20% of the messages had content removed. Since the most common result of this is to throw away a text/html part and collapse the message to a single part, I submit that this will break a significant number of signatures. No if the only result of this were that the recipient's MTA/MUAs considered these messages to be unsigned, that would be OK, but my understanding is that in some cases at least, these messages are either discarded or flagged as having invalid signatures. Either of these alternatives is not good. The former discards wanted messages, and the latter trains recipients to ignore the fact that signatures are invalid. That said, it would be a simple matter to make the removal of these signature headers a site option (or even a list option, but I think a site option is more appropriate). It would be better still to be able to make Mailman play better with DKIM so that we wouldn't have to break or remove signatures. I note that Joe is one of the people who first identified the need to remove these headers. Perhaps together, we can find a better way. See for some discussion. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From joe at skyrush.com Fri Feb 2 01:06:24 2007 From: joe at skyrush.com (Joe Peterson) Date: Thu, 01 Feb 2007 17:06:24 -0700 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: Message-ID: <45C28080.6000403@skyrush.com> Yep, for a time I was doing some testing of DKIM on my server (using the sendmail milter). )I was using sendmail at the time, and I have since switched to Postfix.) I did stop using DKIM after a while, and one reason was the mailing list stumbling block. Since passing messages through Mailman appeared to always render a signature invalid, it certainly seemed correct to remove the sig from the header, thus allowing the outgoing MTA the chance to re-sign the message as the mail list server (not as good as an original sig, but at least it would pass the DKIM test at the receiving end). I know that one is supposed to treat an invalid signature the same as no signature, but knowingly letting email go out (and it is Mailman that is sending it) with a bad signature certainly felt "irresponsible" and potentially misleading. Now, if it is indeed possible to come up with a way to make Mailman successfully pass the original signature through validly, that would be a big win - a real challenge I suspect. But I guess the big question now is what to do in the short term. I'm happy to back out the change, but perhaps we should let a few others weigh in on this topic first. -Joe Mark Sapiro wrote: > I note that Joe is one of the people who first identified the need to > remove these headers. Perhaps together, we can find a better way. From rick at wzoeterwoude.net Fri Feb 2 02:03:40 2007 From: rick at wzoeterwoude.net (Rick van der Zwet) Date: Fri, 02 Feb 2007 02:03:40 +0100 Subject: [Mailman-Developers] [patch] mailman nl i18n fix Message-ID: <45C28DEC.5010201@wzoeterwoude.net> Hi All, % in template/nl/private.html are not escaped, which cause the other templates like realame not to be applied. Attached a small fix. Best regards, /Rick van der Zwet -- http://rickvanderzwet.nl -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: templates-nl-private.html.patch Url: http://mail.python.org/pipermail/mailman-developers/attachments/20070202/04886a4c/attachment.asc From bob at nleaudio.com Fri Feb 2 02:46:41 2007 From: bob at nleaudio.com (Bob Puff) Date: Thu, 1 Feb 2007 20:46:41 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: <45C25CD0.1010301@cisco.com> Message-ID: <20070202014359.M56219@nleaudio.com> I have demime in front of most of my larger lists, and I can tell you from casual peeks at the incoming copy that I keep, there are far too many people who send html email. My lists nuke all the html, so I'd say probably 75% of the incoming messages are modified. I also would concur that deleting the DKIM sig would be the proper thing to do, as an invalid sig is sure to count towards spamminess. Its a challenge at times to get these emails delivered (I've pretty much given up on AOL, they are in such a mess), so anything like this that will trigger more unsuccessful deliveries is a real problem. Bob ---------- Original Message ----------- From: Mark Sapiro To: Michael Thomas Cc: mailman-developers at python.org Sent: Thu, 1 Feb 2007 15:06:25 -0800 Subject: Re: [Mailman-Developers] dkim-signature headers > Michael Thomas wrote: > > > >Yes, there's no question that mailman as well as lots of other software > >can destroy signatures. In practice as people seem to actually use them, > >it is more theoretical than real. We've been running DKIM signers/verifiers > >for going on a year now and the 99% I quoted is across a 25000 user > >population which probably uses mailing lists far more than most similarly > >sized companies. > > I'm sure your statistics are valid for your environment, but I'm not > sure that they are universally applicable. Consider what I think is a > fairly typical situation exemplified by mailman-users at python.org. I > don't know what fraction of incoming posts to this list are > multipart/alternative with text/plain and text/html alternative > parts, but I see many just from people who Cc: me directly. > > It would be a fairly simple matter to go through the .mbox archive > for any list that has one and count the number of X-Content-Filtered- > By: Mailman/MimeDel and compare that to the number of messages. in > fact, I just did that for a cycling club discussion list I managed, > and just over 20% of the messages had content removed. Since the > most common result of this is to throw away a text/html part and > collapse the message to a single part, I submit that this will break > a significant number of signatures. > > No if the only result of this were that the recipient's MTA/MUAs > considered these messages to be unsigned, that would be OK, but my > understanding is that in some cases at least, these messages are > either discarded or flagged as having invalid signatures. Either of > these alternatives is not good. The former discards wanted messages, > and the latter trains recipients to ignore the fact that signatures > are invalid. > > That said, it would be a simple matter to make the removal of these > signature headers a site option (or even a list option, but I think a > site option is more appropriate). > > It would be better still to be able to make Mailman play better with > DKIM so that we wouldn't have to break or remove signatures. > > I note that Joe is one of the people who first identified the need to > remove these headers. Perhaps together, we can find a better way. > > See > > for some discussion. > > -- > Mark Sapiro The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/bob%40nleaudio.com > > Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp ------- End of Original Message ------- From mat at cisco.com Fri Feb 2 03:03:06 2007 From: mat at cisco.com (Michael Thomas) Date: Thu, 01 Feb 2007 18:03:06 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <20070202014359.M56219@nleaudio.com> References: <45C25CD0.1010301@cisco.com> <20070202014359.M56219@nleaudio.com> Message-ID: <45C29BDA.6030806@cisco.com> We've been running with dkim signatures over a large population for nearly a year and have had no indication whatsoever that broken signatures do anything of the sort. Leaving the signatures in allows smarter receivers to have a chance to verify them, not to mention that deleting them destroys the forensic value of the signature. I'm not sure that you all appreciate that resigning a message is no panacea. It's helpful, but only to a degree: a third party signature from a remailer is not a substitute for a first party signature from the From domain. If you have just a valid third party signature, the signing domain would need to be _specifically_ whitelisted by the receiver to be considered an acceptable signature lest you be subject to attacks by rogue third party signers. So to the degree that first party signatures can be validated the better. Mike Bob Puff wrote: > I have demime in front of most of my larger lists, and I can tell you from > casual peeks at the incoming copy that I keep, there are far too many people > who send html email. My lists nuke all the html, so I'd say probably 75% of > the incoming messages are modified. > > I also would concur that deleting the DKIM sig would be the proper thing to > do, as an invalid sig is sure to count towards spamminess. Its a challenge at > times to get these emails delivered (I've pretty much given up on AOL, they > are in such a mess), so anything like this that will trigger more unsuccessful > deliveries is a real problem. > > Bob > > ---------- Original Message ----------- > From: Mark Sapiro > To: Michael Thomas > Cc: mailman-developers at python.org > Sent: Thu, 1 Feb 2007 15:06:25 -0800 > Subject: Re: [Mailman-Developers] dkim-signature headers > > >> Michael Thomas wrote: >> >>> Yes, there's no question that mailman as well as lots of other software >>> can destroy signatures. In practice as people seem to actually use them, >>> it is more theoretical than real. We've been running DKIM signers/verifiers >>> for going on a year now and the 99% I quoted is across a 25000 user >>> population which probably uses mailing lists far more than most similarly >>> sized companies. >>> >> I'm sure your statistics are valid for your environment, but I'm not >> sure that they are universally applicable. Consider what I think is a >> fairly typical situation exemplified by mailman-users at python.org. I >> don't know what fraction of incoming posts to this list are >> multipart/alternative with text/plain and text/html alternative >> parts, but I see many just from people who Cc: me directly. >> >> It would be a fairly simple matter to go through the .mbox archive >> for any list that has one and count the number of X-Content-Filtered- >> By: Mailman/MimeDel and compare that to the number of messages. in >> fact, I just did that for a cycling club discussion list I managed, >> and just over 20% of the messages had content removed. Since the >> most common result of this is to throw away a text/html part and >> collapse the message to a single part, I submit that this will break >> a significant number of signatures. >> >> No if the only result of this were that the recipient's MTA/MUAs >> considered these messages to be unsigned, that would be OK, but my >> understanding is that in some cases at least, these messages are >> either discarded or flagged as having invalid signatures. Either of >> these alternatives is not good. The former discards wanted messages, >> and the latter trains recipients to ignore the fact that signatures >> are invalid. >> >> That said, it would be a simple matter to make the removal of these >> signature headers a site option (or even a list option, but I think a >> site option is more appropriate). >> >> It would be better still to be able to make Mailman play better with >> DKIM so that we wouldn't have to break or remove signatures. >> >> I note that Joe is one of the people who first identified the need to >> remove these headers. Perhaps together, we can find a better way. >> >> See >> >> > > >> for some discussion. >> >> -- >> Mark Sapiro The highway is for gamblers, >> San Francisco Bay Area, California better use your sense - B. Dylan >> >> _______________________________________________ >> Mailman-Developers mailing list >> Mailman-Developers at python.org >> http://mail.python.org/mailman/listinfo/mailman-developers >> Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py >> Searchable Archives: >> > http://www.mail-archive.com/mailman-developers%40python.org/ > >> Unsubscribe: >> > http://mail.python.org/mailman/options/mailman-developers/bob%40nleaudio.com > >> Security Policy: >> > http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp > ------- End of Original Message ------- > From mat at cisco.com Fri Feb 2 03:22:28 2007 From: mat at cisco.com (Michael Thomas) Date: Thu, 01 Feb 2007 18:22:28 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: Message-ID: <45C2A064.3090809@cisco.com> Mark Sapiro wrote: > Michael Thomas wrote: > >> Yes, there's no question that mailman as well as lots of other software >> can destroy signatures. In practice as people seem to actually use them, >> it is more theoretical than real. We've been running DKIM signers/verifiers >> for going on a year now and the 99% I quoted is across a 25000 user >> population which probably uses mailing lists far more than most similarly >> sized companies. >> > > > I'm sure your statistics are valid for your environment, but I'm not > sure that they are universally applicable. Consider what I think is a > fairly typical situation exemplified by mailman-users at python.org. I > don't know what fraction of incoming posts to this list are > multipart/alternative with text/plain and text/html alternative parts, > but I see many just from people who Cc: me directly. > Just to be clear, there are hacks that we do with the length such that mailing lists that insert trailers into mime structured posts still verify. This is done by not signing the trailing -- and/or . This works pretty well. Obviously any wholesale conversions like 8->7bit or other suchlike are going to be a problem, but at least from our vantagepoint of being a pretty big company with a lot of mailing list use, it seems to be pretty rare. > It would be a fairly simple matter to go through the .mbox archive for > any list that has one and count the number of X-Content-Filtered-By: > Mailman/MimeDel and compare that to the number of messages. in fact, I > just did that for a cycling club discussion list I managed, and just > over 20% of the messages had content removed. Since the most common > result of this is to throw away a text/html part and collapse the > message to a single part, I submit that this will break a significant > number of signatures. > Yes, that certainly would. > No if the only result of this were that the recipient's MTA/MUAs > considered these messages to be unsigned, that would be OK, but my > understanding is that in some cases at least, these messages are > either discarded or flagged as having invalid signatures. Either of > these alternatives is not good. The former discards wanted messages, > and the latter trains recipients to ignore the fact that signatures > are invalid. > We haven't seen anybody doing that, and I'd certainly hope that people don't use DKIM as the _sole_ indication of whether the message should be rejected; the false positive rate would be far too high. In any case, the follow on work from DKIM is the sender signing practices (SSP) work which in a nutshell allows a signer to publish whether they sign all of their mail or not. From the SSP standpoint, it makes no difference at all whether you strip the broken signature or not as both count as "no valid signature". I get the sense that people here think no signature would be better than a broken one, but I doubt that's how things will play out, especially with SSP. > That said, it would be a simple matter to make the removal of these > signature headers a site option (or even a list option, but I think a > site option is more appropriate). > I could live with that, especially if the default is to leave the signatures alone. > It would be better still to be able to make Mailman play better with > DKIM so that we wouldn't have to break or remove signatures. > Right... what would be good here I think is to define a "dkim friendly" profile for the signing domain, mailing list configuration, as well as the receiver and where this kind of profile is appropriate. It would be nice if the default thing that mailman did was dkim friendly (which it mostly was until this change). Mike > I note that Joe is one of the people who first identified the need to > remove these headers. Perhaps together, we can find a better way. > > See > > for some discussion. > > From barry at python.org Fri Feb 2 05:03:29 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Feb 2007 23:03:29 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C21B41.6030905@cisco.com> References: <45C21B41.6030905@cisco.com> Message-ID: <36BFFB9F-FA57-4463-9BF7-AFB39D4A42AA@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 1, 2007, at 11:54 AM, Michael Thomas wrote: > in Section 4: > > Signers SHOULD NOT remove any DKIM-Signature header fields from > messages they are > signing, even if they know that the signatures cannot be verified. > > This actually applies to everybody. There are several reasons for > this. > First > is that DKIM allows you to specify the length of a body so it is > not the > case > a priori that mailman will destroy the signature. Second, other > heuristics can > be applied to make mailing list traversal even better such as using > the > z= tag > to determine whether trivial subject modifications have been made. > Third and > probably most important is that removing the signature is actually > harmful rather > than helpful: a broken signature and a missing signature MUST be > treated as > equivalent to no signature at all (lest an attacker just add a fake > DKIM-signature > header to get preferential treatment), and as above the verifier loses > the ability > to recover the signature. I haven't had time to read the formal spec, but does it specifically handle the mailing list use case, where you cannot vouch for what comes out the other end when you send a message to it? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iEYEARECAAYFAkXCuBIACgkQ2YZpQepbvXFurQCgknmNA6pJC/RTdSCFNb3Eoygw u/EAoJYm/SEPQ+KG5NhACNWzt5zjbwp0 =ri/f -----END PGP SIGNATURE----- From barry at python.org Fri Feb 2 05:12:17 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Feb 2007 23:12:17 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C23CB0.1060805@cisco.com> References: <45C21B41.6030905@cisco.com> <45C22DD6.5030208@skyrush.com> <45C23CB0.1060805@cisco.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 1, 2007, at 2:17 PM, Michael Thomas wrote: > I've for quite a while thought that part of an ultimate DKIM BCP would > give some guidance on what a "well behaved mailing list" would be. It > would certainly be good if mailman were an example of that because at > least at Cisco it accounts for the bulk of external mailing list > traffic > we see. I agree with both statements. Note that there are many email related RFCs that are ambiguous in the mailing list use case. We make choices based on our best interpretation but it's never fully satisfactory. If there's a possibility to have DKIM specify what a properly behaving mailing should do (with of course, consensus from this community and other listserver vendors), then I'm all for it. > (at least by default). The main issue is that there is a security/ > robustness > tradeoff with the use of l=. That is, bad guys could append content > too. > On the other hand, *if* that comes to pass, receivers are > completely at > liberty to scan the covered and uncovered parts of the body > differently, > delete the appended text, etc, etc. Isn't it possible that from the point of view of the original sender, the mailing list /is/ the bad guy? (Note too that of course it's trivial to disable DKIM header cleansing in Cisco's own copy of Mailman.) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iEYEARECAAYFAkXCuiIACgkQ2YZpQepbvXF5OwCcCe2sET+qPrlQBMhwL9Aty9CL 6GEAn17BAMu9UC4p+mmUmigliEVDitQE =0INK -----END PGP SIGNATURE----- From barry at python.org Fri Feb 2 05:16:56 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Feb 2007 23:16:56 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C29BDA.6030806@cisco.com> References: <45C25CD0.1010301@cisco.com> <20070202014359.M56219@nleaudio.com> <45C29BDA.6030806@cisco.com> Message-ID: <266D7762-4895-4B40-8E20-ACCD38312712@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 1, 2007, at 9:03 PM, Michael Thomas wrote: > We've been running with dkim signatures over a large population for > nearly > a year and have had no indication whatsoever that broken signatures do > anything > of the sort. Leaving the signatures in allows smarter receivers to > have > a chance > to verify them, not to mention that deleting them destroys the > forensic > value of > the signature. Mailman could also copy the original DKIM headers to something like X- Original-DKIM-Signature. That would preserve it (albeit in a non- standard way) for the forensic value. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iEYEARECAAYFAkXCuzkACgkQ2YZpQepbvXGbZACfQ28xB5znVIWVcBTrjfHU2jbx Z18An22cSHA9s+Zh0/Fi8/2fV4BqtijH =w5cO -----END PGP SIGNATURE----- From stephen at xemacs.org Fri Feb 2 05:30:40 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 02 Feb 2007 13:30:40 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C2A064.3090809@cisco.com> References: <45C2A064.3090809@cisco.com> Message-ID: <87ps8tuqof.fsf@uwakimon.sk.tsukuba.ac.jp> Michael Thomas writes: > Just to be clear, there are hacks that we do with the length such > that mailing lists that insert trailers into mime structured posts > still verify. This is done by not signing the trailing -- and/or > . This works pretty well. Obviously any wholesale > conversions like 8->7bit or other suchlike are going to be a > problem, but at least from our vantagepoint of being a pretty big > company with a lot of mailing list use, it seems to be pretty rare. You don't work much with Japanese-language messages, obviously. *Most* of my Japanese mail gets translated in one direction or the other, and multiple translations are reasonably frequent for mailing lists. I suspect that you'll find that in mostly non-ASCII environments DKIM will have a lot more trouble. Internationalization is definitely related to header-munging in Mailman lists. What apparently happens with I18N is that adjacent MIME encoded words get coalesced, and in particular linear white space gets dropped. Obviously leading spaces may get translated to tabs, etc, when doing RFC 822 wrapping on the way out. Also, there is no canonical way to do MIME encoding; it is perfectly legal to simply MIME encode the whole header even though there's only one Chinese character in it, or just that single character. I would guess that DKIM headers are often fairly long. IIRC, the email module does the canonical thing (ie, just dropping the CRLF pair and leaving other linear white space alone) for wrapped headers. This will *probably* give you the identical headers when they're regenerated, but that's not guaranteed (eg, if there was trailing whitespace on some physical line in the original). Anyway, that's where I'd look first, if I had time to look. > > That said, it would be a simple matter to make the removal of these > > signature headers a site option (or even a list option, but I think a > > site option is more appropriate). If, as I suspect, I18N makes a difference, I would *definitely* want a list option. Most of my lists are English-only, but a few are for other languages and one is multilingual (sometimes in a single post! :-) From barry at python.org Fri Feb 2 05:22:31 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Feb 2007 23:22:31 -0500 Subject: [Mailman-Developers] dkim-signature headers Message-ID: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm not sure what the right answer is just yet, but I'll offer some of my thoughts FWIW. I think the fundamental question is whether the mailing list is the originator of the messages its members receive or whether the original author is. This question has come up in other contexts before, and I don't think it's ever been answered satisfactorily. A quick search through DKIM archives seems to indicate that this question has come up there too, and I think answering it will help us understand what we should be doing here. On Feb 1, 2007, at 3:00 PM, Mark Sapiro wrote: > Consider that while Mailman doesn't do all of these things to every > message, it can do any of the following: [munge the original message] From the DKIM FAQ: - What is the purpose of DKIM? DKIM lets an organization take responsibility for a message. The organization taking responsibility is a handler of the message, either as its originator or as an intermediary. Their reputation is the basis for evaluating whether to trust the message for delivery. I think you can make a legitimate case that Mailman is the originator of messages its members receive. The message is certainly different than the one the original poster sent to the list, and Mailman is clearly an intermediary. Perhaps the message has only been munged in very trivial ways, but it's also possible to munge it in ways that could potentially be viewed as spammy. For example, what if a site decides to put some advertisements in the footer? If you take this view then it seems reasonable to say that it is the mailing list's system that "take[s] responsibility for a message." Sure, the mailing list system could verify the DKIM headers on the message it receives, but ultimate, it is up to the mailing list system to decide whether that message (or some derivative of that message) gets transmitted to its recipients. Or looked at another way, if I send a message through a mailing list, I wouldn't want to vouch for whatever comes out the other end because I don't know what they're going to do to my original content. Maybe then, it's correct for the DKIM signature on the copied message to be broken because what recipients got was /not/ the message I sent, and I don't know how it was munged. But that view implies that I am the originator of the recipient's message. I am, sort of, but also sort of not. I'm not convinced that DKIM is really designed to handle the mailing list use case. It seems to me that it was designed to handle point- to-point messages, not messages that flow through an intermediary, because it's not an enveloping system. Contrast that with S/MIME or OpenPGP. I can sign the message I send from my mailer and that could be preserved through the transformations that Mailman performs, with Mailman wrapping my original in its own signature if it wanted to. Practically speaking, if we can't come up with a consensus on the interpretation of which "organization [should] take responsibility for" the actual message that recipients receive, then what would be the right thing to do? (Note that this answer is different depending on whether we're talking about Mailman 2.1 or some future version.) When this came up before I statement my preference not to make a "strip DKIM headers" selectable by the list owner. I still prefer this for Mailman 2.1 because doing so would clearly be a new feature. Maybe a future version could treat the DKIM header the way it treats the RFC 2369 headers, with a separate selector for List- Post. Ideally, we'd have a more general way to decide which headers get cleansed and which new ones get added. But that's for the future. One elaboration you /might/ be able to get away with in Mailman 2.1 occurs to me as I look at Mark's list: > - Add text to the beginning of the message body (msg_header) > - Add text to the end of the message body (msg_footer) > - Remove text from the beginning of the message body (Approved: line) > - Add additional MIME parts to a multipart message (msg_header, > msg_footer) > - Convert a single part message to multipart in order to add > msg_header/msg_footer > - Remove parts from a multipart message (content filtering) > - Convert an HTML part to plain text (content filtering) > - Decode a base64 or quoted-printable encoded part and perhaps > re-encode it with a different encoding. > - Change or delete various headers including Subject:, To:, From: > - Replace some MIME parts with URLs of where they were stored and > flatten the entire message into a single plain text message > (scrubber). > - Probably other things I'm overlooking. If you could identify the message transformations that break the signature, then you could remove the signature. If the signature of the outgoing message were still valid because Mailman didn't touch any part of the message affecting the signature, then you could keep it. The implementation of this would be fairly simple; the hard part is writing the code to verify a DKIM signature and parse the selectors (IIUC the specification) to figure out which of the above transformations would break the signature. That might be enough to not do it in Mailman 2.1. I'm not sure how much I like that anyway, so comments are definitely welcome. After mulling over this post for an hour ;) I'm starting to believe that it's the mailing list system that needs to vouch for the messages its recipients receive. Of course, it could be Mailman doing the DKIM signing, or it could be Mailman's outgoing MTA, etc. But, ISTM Mailman is ultimately deciding what goes into the list copy, so it is responsible for it. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iEYEARECAAYFAkXCvIgACgkQ2YZpQepbvXFnVgCfeUkfQ0+h/bBAKiwznDTdrHJ6 7V0An2O9TcUYBJYlFhYFpLtYUUGalabq =yCgT -----END PGP SIGNATURE----- From mat at cisco.com Fri Feb 2 16:44:35 2007 From: mat at cisco.com (Michael Thomas) Date: Fri, 02 Feb 2007 07:44:35 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <3A7592A9-0957-4E86-B04E-4B334930CD93@wooz.org> References: <3A7592A9-0957-4E86-B04E-4B334930CD93@wooz.org> Message-ID: <45C35C63.3020409@cisco.com> The very basic test I use is what's in the From: address. That's the thing that's pretty universally displayed and one that users are most likely to grok. Anything beyond From, and you've probably lost at least half of the user population at least. So mailing lists preserve the original From: and perhaps add a Sender:. If I had to chose a signature which was most relevant for _display_ purposes (note that display purposes is but _one_ reason for signatures), I'd like to be able to say something good about the From address that most users grok. It's unclear that I'd want to say something nice about the Sender or any other third party signature because it may confuse users that the mere presence of a signature imputes some trust, where it actually needs to be considered on a case by case basis. I for one would only render the sender: as "verified" both if it verified correctly as well as it being on some known whitelist of domains that I trust. So the bottom line is that a valid third party signature from, say, a mailing list is not safe a priori. It requires special handling by the ultimate receiver in the form of a trust relationship with that domain which needs be done out of band. The same is not of a first party signature because you're only vouching for yourself which is already as good you trust that domain (or not). Mike, is this making any sense? Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I'm not sure what the right answer is just yet, but I'll offer some of > my thoughts FWIW. > > I think the fundamental question is whether the mailing list is the > originator of the messages its members receive or whether the original > author is. This question has come up in other contexts before, and I > don't think it's ever been answered satisfactorily. A quick search > through DKIM archives seems to indicate that this question has come up > there too, and I think answering it will help us understand what we > should be doing here. > > On Feb 1, 2007, at 3:00 PM, Mark Sapiro wrote: > >> Consider that while Mailman doesn't do all of these things to every >> message, it can do any of the following: > [munge the original message] > > From the DKIM FAQ: > > - What is the purpose of DKIM? > > DKIM lets an organization take responsibility for a message. The > organization taking responsibility is a handler of the > message, either as its originator or as an intermediary. Their > reputation is the basis for evaluating whether to trust the > message for delivery. > > I think you can make a legitimate case that Mailman is the originator > of messages its members receive. The message is certainly different > than the one the original poster sent to the list, and Mailman is > clearly an intermediary. Perhaps the message has only been munged in > very trivial ways, but it's also possible to munge it in ways that > could potentially be viewed as spammy. For example, what if a site > decides to put some advertisements in the footer? > > If you take this view then it seems reasonable to say that it is the > mailing list's system that "take[s] responsibility for a message." > Sure, the mailing list system could verify the DKIM headers on the > message it receives, but ultimate, it is up to the mailing list system > to decide whether that message (or some derivative of that message) > gets transmitted to its recipients. > > Or looked at another way, if I send a message through a mailing list, > I wouldn't want to vouch for whatever comes out the other end because > I don't know what they're going to do to my original content. Maybe > then, it's correct for the DKIM signature on the copied message to be > broken because what recipients got was /not/ the message I sent, and I > don't know how it was munged. But that view implies that I am the > originator of the recipient's message. I am, sort of, but also sort > of not. > > I'm not convinced that DKIM is really designed to handle the mailing > list use case. It seems to me that it was designed to handle > point-to-point messages, not messages that flow through an > intermediary, because it's not an enveloping system. Contrast that > with S/MIME or OpenPGP. I can sign the message I send from my mailer > and that could be preserved through the transformations that Mailman > performs, with Mailman wrapping my original in its own signature if it > wanted to. > > Practically speaking, if we can't come up with a consensus on the > interpretation of which "organization [should] take responsibility > for" the actual message that recipients receive, then what would be > the right thing to do? (Note that this answer is different depending > on whether we're talking about Mailman 2.1 or some future version.) > > When this came up before I statement my preference not to make a > "strip DKIM headers" selectable by the list owner. I still prefer > this for Mailman 2.1 because doing so would clearly be a new feature. > Maybe a future version could treat the DKIM header the way it treats > the RFC 2369 headers, with a separate selector for List-Post. > Ideally, we'd have a more general way to decide which headers get > cleansed and which new ones get added. But that's for the future. > > One elaboration you /might/ be able to get away with in Mailman 2.1 > occurs to me as I look at Mark's list: > >> - Add text to the beginning of the message body (msg_header) >> - Add text to the end of the message body (msg_footer) >> - Remove text from the beginning of the message body (Approved: line) >> - Add additional MIME parts to a multipart message (msg_header, >> msg_footer) >> - Convert a single part message to multipart in order to add >> msg_header/msg_footer >> - Remove parts from a multipart message (content filtering) >> - Convert an HTML part to plain text (content filtering) >> - Decode a base64 or quoted-printable encoded part and perhaps >> re-encode it with a different encoding. >> - Change or delete various headers including Subject:, To:, From: >> - Replace some MIME parts with URLs of where they were stored and >> flatten the entire message into a single plain text message (scrubber). >> - Probably other things I'm overlooking. > > If you could identify the message transformations that break the > signature, then you could remove the signature. If the signature of > the outgoing message were still valid because Mailman didn't touch any > part of the message affecting the signature, then you could keep it. > The implementation of this would be fairly simple; the hard part is > writing the code to verify a DKIM signature and parse the selectors > (IIUC the specification) to figure out which of the above > transformations would break the signature. That might be enough to > not do it in Mailman 2.1. > > I'm not sure how much I like that anyway, so comments are definitely > welcome. After mulling over this post for an hour ;) I'm starting to > believe that it's the mailing list system that needs to vouch for the > messages its recipients receive. Of course, it could be Mailman doing > the DKIM signing, or it could be Mailman's outgoing MTA, etc. But, > ISTM Mailman is ultimately deciding what goes into the list copy, so > it is responsible for it. > > - -Barry > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (Darwin) > > iEYEARECAAYFAkXCt04ACgkQ2YZpQepbvXHsAACfVOu25u2Ps2MrC0qQE7i/W5sx > ZdYAoL0z9q0+zjt7gpI3JrFy62m4DkAq > =+a49 > -----END PGP SIGNATURE----- From mat at cisco.com Fri Feb 2 16:57:21 2007 From: mat at cisco.com (Michael Thomas) Date: Fri, 02 Feb 2007 07:57:21 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: <45C21B41.6030905@cisco.com> <45C22DD6.5030208@skyrush.com> <45C23CB0.1060805@cisco.com> Message-ID: <45C35F61.9000606@cisco.com> Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Feb 1, 2007, at 2:17 PM, Michael Thomas wrote: > >> I've for quite a while thought that part of an ultimate DKIM BCP would >> give some guidance on what a "well behaved mailing list" would be. It >> would certainly be good if mailman were an example of that because at >> least at Cisco it accounts for the bulk of external mailing list traffic >> we see. > > I agree with both statements. Note that there are many email related > RFCs that are ambiguous in the mailing list use case. We make choices > based on our best interpretation but it's never fully satisfactory. > If there's a possibility to have DKIM specify what a properly behaving > mailing should do (with of course, consensus from this community and > other listserver vendors), then I'm all for it. > >> (at least by default). The main issue is that there is a >> security/robustness >> tradeoff with the use of l=. That is, bad guys could append content too. >> On the other hand, *if* that comes to pass, receivers are completely at >> liberty to scan the covered and uncovered parts of the body differently, >> delete the appended text, etc, etc. > > Isn't it possible that from the point of view of the original sender, > the mailing list /is/ the bad guy? Absolutely! Consider: From: barry at python.org Sender: fooledyou at badguy.com DKIM-Signature: d=badguy.com [...] So you'd have a perfectly legitimate signature that corresponded with the Sender: badguy.com. I know who barry at python.org is but haven't a clue about badguy.com (or better, something just plain obscure). Would I want to bring it to the attention of the receiver (say in the MUA) "barry" is more well known now? Certainly not, because you're not. The only time you might consider that ok is if and only if you had some warm fuzzies about the third party signing. Which scales about as well as whitelists scale in general, which is to say that I would expect that we'll need a *lot* more experience to see how this actually will play out. Note that this is not to discourage resigning by lists: it's cheap enough these days that if they're there for a reasonable % of lists, it will at least sow the ground for making whitelisting-like approaches more friendly (either local, aggregated...). (Note too that of course it's trivial to disable DKIM header cleansing in Cisco's own copy of Mailman.) Right -- we found this on some non-Cisco run lists though. Mike > - From jon at latchkey.com Fri Feb 2 17:04:56 2007 From: jon at latchkey.com (Jon Scott Stevens) Date: Fri, 2 Feb 2007 08:04:56 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> Message-ID: <15A834D9-CB7E-4B29-824B-D35B1367239F@latchkey.com> A nice alternative to DKIM signatures... On Feb 1, 2007, at 8:22 PM, Barry Warsaw wrote: > -----BEGIN PGP SIGNATURE----- I really don't see what the point of DKIM signatures are on an internal Cisco mailing list. You either work for the company or you don't. If you can't verify a message came from someone you work with, then there must be something wrong with your network. I surely hope that isn't the case at Cisco(TM). Using StartTLS (RFC 2487) is a much less intrusive way to wrap and transport messages securely. This all smells like Cisco(TM) trying to force their spec's on Mailman. I don't see an other mailing list products on this page... http://www.dkim.org/deploy/index.htm There is no guarantee that a mailing list isn't going to modify the content of a message. +1 to strip DKIM headers. jon http://subetha.tigris.org/ From mat at cisco.com Fri Feb 2 17:45:19 2007 From: mat at cisco.com (Michael Thomas) Date: Fri, 02 Feb 2007 08:45:19 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <15A834D9-CB7E-4B29-824B-D35B1367239F@latchkey.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <15A834D9-CB7E-4B29-824B-D35B1367239F@latchkey.com> Message-ID: <45C36A9F.3060102@cisco.com> Um, Cisco participates extensively in external mailing list for standards bodies. And we're not trying to "force" anything; I speak for myself as one of the authors of the spec. Mike Jon Scott Stevens wrote: > A nice alternative to DKIM signatures... > > On Feb 1, 2007, at 8:22 PM, Barry Warsaw wrote: > >> -----BEGIN PGP SIGNATURE----- >> > > > I really don't see what the point of DKIM signatures are on an > internal Cisco mailing list. You either work for the company or you > don't. If you can't verify a message came from someone you work with, > then there must be something wrong with your network. I surely hope > that isn't the case at Cisco(TM). > > Using StartTLS (RFC 2487) is a much less intrusive way to wrap and > transport messages securely. > > This all smells like Cisco(TM) trying to force their spec's on > Mailman. I don't see an other mailing list products on this page... > http://www.dkim.org/deploy/index.htm > > There is no guarantee that a mailing list isn't going to modify the > content of a message. > > +1 to strip DKIM headers. > > jon > http://subetha.tigris.org/ > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/mat%40cisco.com > > Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp > From mat at cisco.com Fri Feb 2 18:05:10 2007 From: mat at cisco.com (Michael Thomas) Date: Fri, 02 Feb 2007 09:05:10 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> Message-ID: <45C36F46.3080704@cisco.com> Barry Warsaw wrote: > I'm not sure how much I like that anyway, so comments are definitely > welcome. After mulling over this post for an hour ;) I'm starting to > believe that it's the mailing list system that needs to vouch for the > messages its recipients receive. Of course, it could be Mailman doing > the DKIM signing, or it could be Mailman's outgoing MTA, etc. But, > ISTM Mailman is ultimately deciding what goes into the list copy, so > it is responsible for it. The big problem with the way that mailing lists interact is that in some scenarios they're not terribly different from a .forward file, and in other cases they for all intents and purposes completely originating a completely new message, cf translators, digestors, etc, etc. The _problems_ arise when the mailing list keeps the 822.From address intact from the original submitter. For things like digests it pretty much does the right thing: it sets the 2822.From to be something related to the list. For normal list traffic it keeps the original 2822.From. What it seems to me is that maybe we should look close at that behavior of when a list ought to take From: responsibility for a message ala digests. When a list completely mangles a message, is it really reasonable for it to keep acting as if it came from the original From: address? There's probably not a bright line here, but maybe we should force the issue with DKIM in that something that mangles a message in a way that it's impossible to have the original signature survive to... set the From: address to something list related so that it's the lists reputation that's considered rather than getting caught up in no-man's land of "it looks like a forgery because we/they sign all mail, but..." Mike From joe at skyrush.com Fri Feb 2 19:56:34 2007 From: joe at skyrush.com (Joe Peterson) Date: Fri, 02 Feb 2007 11:56:34 -0700 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C36F46.3080704@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> Message-ID: <45C38962.8010507@skyrush.com> I really do not think that a From address should be changed. This is where "Sender" comes in (and Sender is more "behind the scenes" and less important to the end user). So what Mailman does not, I believe, is correct: keep From set to the person who sent the email and set Sender to reflect that the mail list is the sender. I also do agree with Barry that the responsible party for an email from a mailing list should be the mail list and not the original person who sent it. If you are subscribed to a mailing list, you need to be able to verify that the message indeed came from the list itself (and you can do this if the mail list host adds its own DKIM sig). Ideally, the mail list host would do the initial DKIM check from the original person sending the message and mark the header to reflect that it passed, and then the receiver's host can check the DKIM sig from the mail list. This "chain" of verification would be sufficient to verify the message itself. Now, what about the original DKIM sig...? If left in the header, the end receiver might try to check it and get a bad match, even if the newer sig added by the mail list would verify. Yeah, I know that a "bad sig is like no sig", but this just seems ugly. Then there's the additional problem of the milter not re-signing if there exists a sig already (these were the two motivations for cleansing the sigs from the mail in Mailman). Bottom line is that if we let the system do two separate DKIM checks (orig sender -> mail list, and then mail list -> receiver), it avoids all the issues of changes made to the message. Somehow this seems more clean to me. -Joe Michael Thomas wrote: > Barry Warsaw wrote: >> I'm not sure how much I like that anyway, so comments are definitely >> welcome. After mulling over this post for an hour ;) I'm starting to >> believe that it's the mailing list system that needs to vouch for the >> messages its recipients receive. Of course, it could be Mailman doing >> the DKIM signing, or it could be Mailman's outgoing MTA, etc. But, >> ISTM Mailman is ultimately deciding what goes into the list copy, so >> it is responsible for it. > > The big problem with the way that mailing lists interact is that in some > scenarios they're not terribly different from a .forward file, and in other > cases they for all intents and purposes completely originating a completely > new message, cf translators, digestors, etc, etc. The _problems_ arise when > the mailing list keeps the 822.From address intact from the original > submitter. > For things like digests it pretty much does the right thing: it sets the > 2822.From > to be something related to the list. For normal list traffic it keeps > the original > 2822.From. > > What it seems to me is that maybe we should look close at that behavior of > when a list ought to take From: responsibility for a message ala > digests. When > a list completely mangles a message, is it really reasonable for it to > keep acting > as if it came from the original From: address? There's probably not a > bright line > here, but maybe we should force the issue with DKIM in that something that > mangles a message in a way that it's impossible to have the original > signature > survive to... set the From: address to something list related so that > it's the lists > reputation that's considered rather than getting caught up in no-man's > land of > "it looks like a forgery because we/they sign all mail, but..." > > Mike > From joe at skyrush.com Fri Feb 2 19:59:20 2007 From: joe at skyrush.com (Joe Peterson) Date: Fri, 02 Feb 2007 11:59:20 -0700 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C38962.8010507@skyrush.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <45C38962.8010507@skyrush.com> Message-ID: <45C38A08.3080507@skyrush.com> Joe Peterson wrote: > I really do not think that a From address should be changed. This is > where "Sender" comes in (and Sender is more "behind the scenes" and less > important to the end user). So what Mailman does not, I believe, is > correct... ^^^ I meant to say "does now" :) From jon at latchkey.com Fri Feb 2 21:27:58 2007 From: jon at latchkey.com (Jon Scott Stevens) Date: Fri, 2 Feb 2007 12:27:58 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C36A9F.3060102@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <15A834D9-CB7E-4B29-824B-D35B1367239F@latchkey.com> <45C36A9F.3060102@cisco.com> Message-ID: <791B04A3-03FC-4419-9B90-BCD8B062D6C3@latchkey.com> On Feb 2, 2007, at 8:45 AM, Michael Thomas wrote: > Um, Cisco participates extensively in external mailing list for > standards > bodies. And we're not trying to "force" anything; I speak for > myself as > one of the authors of the spec. > > Mike Um, then PGP sign your messages and we will call it all good. =) Your interest in DKIM and Cisco's interest in DKIM are the same to me cause you are posting with a Cisco.com email address and because Cisco is one of the companies pushing the spec. Sorry, I just don't see DKIM as a good solution for the problem. But that's just my personal opinion. jon From stephen at xemacs.org Sat Feb 3 06:43:55 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 03 Feb 2007 14:43:55 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C36F46.3080704@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> Message-ID: <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> Michael Thomas writes: > What it seems to me is that maybe we should look close at that > behavior of when a list ought to take From: responsibility for a > message ala digests. When a list completely mangles a message, is > it really reasonable for it to keep acting as if it came from the > original From: address? There's probably not a bright line here, > but maybe we should force the issue with DKIM "Force"?! You've got the tail wagging the dog here. >From is an *author* header, not property of the MTA. It indicates the *author*, not the editor or delivery agency. If the author approves the message munging (and this is implicit in Internet custom or even the mailing lists terms of use in most cases), DKIM MUST live with it. So there's your bright red line. I think this points in the exact opposite direction from what you claim: users already understand what From means in the mailing list context, so if From and the DKIM signature are in conflict, the latter SHOULD be adjusted. Since the ML is not in possession of the relevant key, removal is the only option. > The very basic test I use is what's in the From: address. That's > the thing that's pretty universally displayed and one that users > are most likely to grok. Anything beyond From, and you've probably > lost at least half of the user population at least. If the users don't know the difference between "From" and "Sender", they have little chance of using DKIM as intended *except in the case where From == Sender* (or they are in the same DKIM domain). Doesn't this amount to saying "we don't know what DKIM BCP for mailing lists is---that will be determined by the ability of average users to grok RFC 2822 (and we believe that is quite low)". > So the bottom line is that a valid third party signature from, say, > a mailing list is not safe a priori. It requires special handling > by the ultimate receiver in the form of a trust relationship with > that domain which needs be done out of band. The same is not of a > first party signature because you're only vouching for yourself > which is already as good you trust that domain (or not). That's specious IMO. Technically, when the mailing list signs a message, it's vouching for that message as the mailing list re-sent it. This is exactly the same as for the originating domain. From the UI standpoint, if a DKIM-enabled MUA doesn't present information (including their roles as originators or resenders) about *all* of the "DKIM senders" to the user, it is hopelessly broken for use in a resending environment. DKIM can help here. Eg, if DKIM-enabled clients are told that they SHOULD recognize RFC 2369 headers and try to match the list identity to the DKIM signature, and DKIM is extended to provide for resenders (including lists that move domains---the List-ID will have a different domain then). Again, your argument bites the other way around. I think it's a reasonable presumption that the user has *already* established the relevant trust relationship with the mailing list, because either the list is run by the user's employer (or similar), or the user opted in. Certainly you can presume that of any mailing list willing to cooperate with DKIM! I think that all of this really points to issues that need to be resolved in the DKIM specification, not with mailing lists or any other existing infrastructure. It's DKIM that needs to establish BCPs for user interface and user education, not other parts of the infrastructure that need to cover for DKIM's weaknesses at this point. Once DKIM has established its own BCPs, and technical analysis shows that it *requires* cooperation with other protocols to resolve important ambiguities, *then* it's time to ask MLMs etc to help with those issues. From jwblist3 at olympus.net Sat Feb 3 06:22:12 2007 From: jwblist3 at olympus.net (John W. Baxter) Date: Fri, 02 Feb 2007 21:22:12 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <20070202014359.M56219@nleaudio.com> Message-ID: On 2/1/07 5:46 PM, "Bob Puff" wrote: > I have demime in front of most of my larger lists, and I can tell you from > casual peeks at the incoming copy that I keep, there are far too many people > who send html email. Anyone using Windows machine, or a Mac starting with Tiger who hasn't made a change to the defaults out of the box sends HTML mail. Before Tiger, Mac users were sending text/rich. And there is little incentive to change the defaults unless one is an Internet veteran. Plus AOL. There is some self-selection away from HTML in the mailing list population, since many folks using lists know what's wrong with HMTL mail, and others get chided for sending it. --John From barry at python.org Sat Feb 3 23:57:46 2007 From: barry at python.org (Barry Warsaw) Date: Sat, 3 Feb 2007 17:57:46 -0500 Subject: [Mailman-Developers] My new job Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello everyone, I just wanted to drop a quick note to let you know that I have started working for Canonical, the folks who distribute Ubuntu Linux and develop the Launchpad service for distributed open source development. I'm quite excited to be working for Canonical. They are a Python and Zope shop, very open source friendly, and based on Linux, so I'm working in an environment that's both familiar and fun. They're also a distributed company, without traditional offices, but with many folks working all over the world. They're also hiring . :) While this is good for Barry (and hopefully good for Canonical :), I know you're thinking, "what does this mean for Mailman?" Well, I'm glad you asked! Part of my official duties will be to integrate Mailman with Launchpad so as to enable more powerful communication patterns among members of a distributed development team. While I won't be spending /all/ of my time on Mailman, it will be getting some official love. I hope that this will help move the current development branch closer to a release some time later this year. Other than that, not much will really change. Mailman is and will always be released under the GPL, and it will continue to be a GNU project. If you have any questions or concerns, please send them directly to me. I will attempt to answer them as best I can, and if there are common themes, I'll post updates here. Thanks for your support and keep on delivering with Mailman! - -Barry Here are some related links: http://www.canonical.com http://www.ubuntu.com http://launchpad.net P.S. I'll be at PyCon this year so if you're coming too, please say hi! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcUTb3EjvBPtnXfVAQIQKgP/XFVeeDBn6QVXkE9oK1YJxyrLZZET5GxT TAvTJgfndkcWPuUpbC5D6hcpDNa6sfIgJnR3enoW7MgKpOAtIXTuXqPpNiFMBVT2 qUhlDHhwYzdJWWyzI/oXGRvt0lMqqA69Iu7A6DAKrgksBB128V/mYxTfv8BPmF4W uASb/9MkmAQ= =h+IQ -----END PGP SIGNATURE----- From tkikuchi at is.kochi-u.ac.jp Sun Feb 4 02:49:10 2007 From: tkikuchi at is.kochi-u.ac.jp (Tokio Kikuchi) Date: Sun, 04 Feb 2007 10:49:10 +0900 Subject: [Mailman-Developers] My new job In-Reply-To: References: Message-ID: <45C53B96.4010907@is.kochi-u.ac.jp> Congratulations, Barry! I'm very happy to hear that you can spend more time on Mailman. I also learned by quick search that Canonical is founded by this nice guy. :) http://en.wikipedia.org/wiki/Mark_Shuttleworth Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hello everyone, > > I just wanted to drop a quick note to let you know that I have > started working for Canonical, the folks who distribute Ubuntu Linux > and develop the Launchpad service for distributed open source > development. > > I'm quite excited to be working for Canonical. They are a Python and > Zope shop, very open source friendly, and based on Linux, so I'm > working in an environment that's both familiar and fun. They're also > a distributed company, without traditional offices, but with many > folks working all over the world. They're also hiring . :) > > While this is good for Barry (and hopefully good for Canonical :), I > know you're thinking, "what does this mean for Mailman?" Well, I'm > glad you asked! > > Part of my official duties will be to integrate Mailman with > Launchpad so as to enable more powerful communication patterns among > members of a distributed development team. While I won't be > spending /all/ of my time on Mailman, it will be getting some > official love. I hope that this will help move the current > development branch closer to a release some time later this year. > Other than that, not much will really change. Mailman is and will > always be released under the GPL, and it will continue to be a GNU > project. > > If you have any questions or concerns, please send them directly to > me. I will attempt to answer them as best I can, and if there are > common themes, I'll post updates here. > > Thanks for your support and keep on delivering with Mailman! > > - -Barry > > Here are some related links: > > http://www.canonical.com > http://www.ubuntu.com > http://launchpad.net > > P.S. I'll be at PyCon this year so if you're coming too, please say hi! > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (Darwin) > > iQCVAwUBRcUTb3EjvBPtnXfVAQIQKgP/XFVeeDBn6QVXkE9oK1YJxyrLZZET5GxT > TAvTJgfndkcWPuUpbC5D6hcpDNa6sfIgJnR3enoW7MgKpOAtIXTuXqPpNiFMBVT2 > qUhlDHhwYzdJWWyzI/oXGRvt0lMqqA69Iu7A6DAKrgksBB128V/mYxTfv8BPmF4W > uASb/9MkmAQ= > =h+IQ > -----END PGP SIGNATURE----- -- Tokio Kikuchi, tkikuchi at is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/ From mat at cisco.com Mon Feb 5 16:56:45 2007 From: mat at cisco.com (Michael Thomas) Date: Mon, 05 Feb 2007 07:56:45 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <45C753BD.4010004@cisco.com> Stephen J. Turnbull wrote: > Michael Thomas writes: > > > What it seems to me is that maybe we should look close at that > > behavior of when a list ought to take From: responsibility for a > > message ala digests. When a list completely mangles a message, is > > it really reasonable for it to keep acting as if it came from the > > original From: address? There's probably not a bright line here, > > but maybe we should force the issue with DKIM > > "Force"?! You've got the tail wagging the dog here. > > From is an *author* header, not property of the MTA. It indicates the > *author*, not the editor or delivery agency. If the author approves > the message munging (and this is implicit in Internet custom or even > the mailing lists terms of use in most cases), DKIM MUST live with it. > So there's your bright red line. > Which MTA? The submission MTA? The submission MTA can often does have the final word on who the "author" is. In fact, the domain holder ultimately decides who and who it won't delegate local parts too. This is all really fuzzy though: barring S/MIME there's no guarantees about "authorship" per se. Once it's in the hands of your MSA, it owns it. What DKIM tries to do is at least give some guarantees at the administrative/delegation points (ie, the domain) to other administrative/delegation points. It can only do this if the intermediate agents in the delivery path don't destroy the signature. If something destroys the signature, it's then taking responsibility for the downgrade in the assurance that the signature provided. Some receivers may accept that downgrade, some may weight it negatively, some may reject it out of hand. But ultimately it's the responsibility of the signature mangler to make that tradeoff. > I think this points in the exact opposite direction from what you > claim: users already understand what From means in the mailing list > context, so if From and the DKIM signature are in conflict, the latter > SHOULD be adjusted. Since the ML is not in possession of the relevant > key, removal is the only option. > Which makes the message look unsigned which does no good at all and in fact manifestly causes harm. And I really have no clue what "adjustment" you might be thinking of. Removing things that are signed so as to make the signature useless? Something people need to grok is that transformations of messages by mailing lists need to be viewed as no different than transformations made by an attacker. How does a piece of delivery machinery know whether transformations are benign or malicious? Not very well, as it turns out. > > The very basic test I use is what's in the From: address. That's > > the thing that's pretty universally displayed and one that users > > are most likely to grok. Anything beyond From, and you've probably > > lost at least half of the user population at least. > > If the users don't know the difference between "From" and "Sender", > they have little chance of using DKIM as intended *except in the case > where From == Sender* (or they are in the same DKIM domain). Doesn't > this amount to saying "we don't know what DKIM BCP for mailing lists > is---that will be determined by the ability of average users to grok > RFC 2822 (and we believe that is quite low)". > What we've been going on is that people sort understand From and that's about it. Machine learning may be more clever in time. For mailing lists to destroy the From signature is a problem -- compounded immensely by arbitrarily deleting them from whence nothing can recover the signature. > > So the bottom line is that a valid third party signature from, say, > > a mailing list is not safe a priori. It requires special handling > > by the ultimate receiver in the form of a trust relationship with > > that domain which needs be done out of band. The same is not of a > > first party signature because you're only vouching for yourself > > which is already as good you trust that domain (or not). > > That's specious IMO. Technically, when the mailing list signs a > message, it's vouching for that message as the mailing list re-sent > it. This is exactly the same as for the originating domain. From the > UI standpoint, if a DKIM-enabled MUA doesn't present information > (including their roles as originators or resenders) about *all* of the > "DKIM senders" to the user, it is hopelessly broken for use in a > resending environment. > Spammers and bad guys can resign too. As I said, people grok From and that's about it. If adding a third party signature causes people to give more credence to the From address, then you've opened yourself up to malicious third parties. The draft specifically states that a third party signature needs to be an "acceptable" third party signature. That is, a third party you trust not to screw you. How you determine that trust is not specified, but it is still required. > DKIM can help here. Eg, if DKIM-enabled clients are told that they > SHOULD recognize RFC 2369 headers and try to match the list identity > to the DKIM signature, and DKIM is extended to provide for resenders > (including lists that move domains---the List-ID will have a different > domain then). > > Again, your argument bites the other way around. I think it's a > reasonable presumption that the user has *already* established the > relevant trust relationship with the mailing list, because either the > list is run by the user's employer (or similar), or the user opted in. > Certainly you can presume that of any mailing list willing to > cooperate with DKIM! > Lots of MUA's don't even show Sender or Listid. And I can pretty much guarantee that there's about zero infrastructure for whitelisting trusted third party signatures for mailing lists right now. It's not clear that it will be coming any time soon either. > I think that all of this really points to issues that need to be > resolved in the DKIM specification, not with mailing lists or any > other existing infrastructure. It's DKIM that needs to establish BCPs > for user interface and user education, not other parts of the > infrastructure that need to cover for DKIM's weaknesses at this point. > Once DKIM has established its own BCPs, and technical analysis shows > that it *requires* cooperation with other protocols to resolve > important ambiguities, *then* it's time to ask MLMs etc to help with > those issues. > I'm afraid that intransigence from the mailing list community is likely to really backfire. Mailing list traffic is an extremely small percentage of traffic, and most admins are likely to just ignore the collateral damage if it's too much a nuisance. Don't get me wrong: I spend far too much of my day on mailing lists and would really like things to work out. But hard line positions in the face of thorny engineering tradeoffs doesn't help. Mike From bob at nleaudio.com Mon Feb 5 19:02:24 2007 From: bob at nleaudio.com (Bob Puff@NLE) Date: Mon, 05 Feb 2007 13:02:24 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C753BD.4010004@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <45C753BD.4010004@cisco.com> Message-ID: <45C77130.7010306@nleaudio.com> > I'm afraid that intransigence from the mailing list community is likely to > really backfire. Mailing list traffic is an extremely small percentage > of traffic, > and most admins are likely to just ignore the collateral damage if it's too > much a nuisance. Don't get me wrong: I spend far too much of my day on > mailing lists and would really like things to work out. But hard line > positions > in the face of thorny engineering tradeoffs doesn't help. > > Mike Hmm, not sure where that statistic comes from, but I'd say 25% of the mail that enters my couple hundred domains I admin originates from mailing lists. They may not necessarily be discussion lists like this (although I'd say perhaps 5-10% is), but also the "targeted marketing" types are also using mailing lists. Bob From mat at cisco.com Mon Feb 5 20:17:01 2007 From: mat at cisco.com (Michael Thomas) Date: Mon, 05 Feb 2007 11:17:01 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C77130.7010306@nleaudio.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <45C753BD.4010004@cisco.com> <45C77130.7010306@nleaudio.com> Message-ID: <45C782AD.6080409@cisco.com> Bob Puff at NLE wrote: >> I'm afraid that intransigence from the mailing list community is likely to >> really backfire. Mailing list traffic is an extremely small percentage >> of traffic, >> and most admins are likely to just ignore the collateral damage if it's too >> much a nuisance. Don't get me wrong: I spend far too much of my day on >> mailing lists and would really like things to work out. But hard line >> positions >> in the face of thorny engineering tradeoffs doesn't help. >> >> Mike >> > > Hmm, not sure where that statistic comes from, but I'd say 25% of the mail that enters my couple > hundred domains I admin originates from mailing lists. They may not necessarily be discussion lists > like this (although I'd say perhaps 5-10% is), but also the "targeted marketing" types are also > using mailing lists. > We checked with two very large email providers who put it at less than 1% and just spot checking here it seems to be 2-3% and I expect that we're probably on the high side of your average megacorp. Note that discussion lists are probably the only real problem -- write-once lists which are essentially originated from the list owner shouldn't pose a problem for dkim. Mike From stephen at xemacs.org Mon Feb 5 21:15:27 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 06 Feb 2007 05:15:27 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C753BD.4010004@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <45C753BD.4010004@cisco.com> Message-ID: <87abzsico0.fsf@uwakimon.sk.tsukuba.ac.jp> Michael Thomas writes: > I'm afraid that intransigence from the mailing list community is > likely to really backfire. Mailing list traffic is an extremely > small percentage of traffic, and most admins are likely to just > ignore the collateral damage if it's too much a nuisance. We know. Mailing lists have always been vulnerable to such collateral damage by their very nature as bulk transports. > Don't get me wrong: I spend far too much of my day on mailing lists > and would really like things to work out. But hard line positions > in the face of thorny engineering tradeoffs doesn't help. First, I don't speak for the Mailman community. I fully expect that the Mailman developers will recognize that they have no choice but to provide DKIM-friendliness options. Second, you can call me hardline if you like, but you're advocating that we abandon RFC 2822's From-is-not-Sender semantics. You're advocating that the legitimate editorial role currently performed by many mailing lists (analogous to the Editor reflowing lines and fixing typos in letters submitted for the Op-Ed page) be reduced to mere relaying. This is pretty radical stuff, if you ask me. The fact that you talk about the serious damage being done by mailing lists removing signatures leads me to wonder what was happening to unsigned posts on those mailing lists before they upgraded to a recent Mailman. And the fact that you completely ignore the existing trust relationship that mailing lists have with their members in discussing third party signatures makes me wonder how carefully you and your colleagues have really thought about mailing lists. (Especially since that trust relationship has been explicitly pointed out!) So I just don't see an existing best practice here. I see an attempt to develop an extension to an existing draft based purely on theory, and theory not really grounded in current practice, to boot. I worry that an attempt to make Mailman conform to DKIM rather than write list-friendly wording into the standard will cause the collateral damage to be set in stone. From mat at cisco.com Mon Feb 5 23:10:10 2007 From: mat at cisco.com (Michael Thomas) Date: Mon, 05 Feb 2007 14:10:10 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87abzsico0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <45C753BD.4010004@cisco.com> <87abzsico0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <45C7AB42.7080906@cisco.com> Stephen J. Turnbull wrote: > Michael Thomas writes: > > > I'm afraid that intransigence from the mailing list community is > > likely to really backfire. Mailing list traffic is an extremely > > small percentage of traffic, and most admins are likely to just > > ignore the collateral damage if it's too much a nuisance. > > We know. Mailing lists have always been vulnerable to such collateral > damage by their very nature as bulk transports. > > > Don't get me wrong: I spend far too much of my day on mailing lists > > and would really like things to work out. But hard line positions > > in the face of thorny engineering tradeoffs doesn't help. > > First, I don't speak for the Mailman community. I fully expect that > the Mailman developers will recognize that they have no choice but to > provide DKIM-friendliness options. > > Second, you can call me hardline if you like, I'm not calling anybody hardline; I'm just saying that there is a rather unpleasant set of engineering tradeoffs for security and functionality that need to be sorted out. Some of these tradeoffs are not especially new considering that many apply to PGP and S/MIME too. What is different here is that DKIM is likely to scale up -- already Y!, Gmail sign billions of piece of email a day, and this is likely to really ramp up once dkim has a bright and shiny rfc # (it's just waiting to get through the IESG right now). So all of those things that are a theoretical problem with S/MIME and PGP will likely be a real problem with DKIM. > but you're advocating > that we abandon RFC 2822's From-is-not-Sender semantics. You're > advocating that the legitimate editorial role currently performed by > many mailing lists (analogous to the Editor reflowing lines and fixing > typos in letters submitted for the Op-Ed page) be reduced to mere > relaying. This is pretty radical stuff, if you ask me. > Let's be clear that I'm advocating a dialog here, not any particular solution and I'm hoping that we can come up with some finesse. The problem is that what you consider a legitimate editorial role presupposes a trust relationship with the editor. But that's a rather hard problem given the scale of email -- and spew -- that MTA's and MUA's are having to deal with these days. I used to know what senders I trusted 10 years ago; today absent cryptographic proof of identity, it's just about anybody's guess. And my poor MTA's are even more clueless still. > The fact that you talk about the serious damage being done by mailing > lists removing signatures leads me to wonder what was happening to > unsigned posts on those mailing lists before they upgraded to a recent > Mailman. We weren't expecting there to be signature before, so there wasn't any damage. Since we sign just about everything now, we do have an expectation that there are signatures there and are planning to take advantage of that to alert users when a signature is missing/broken. Note from where I sit, it seems that mailman upgrade cycles seem to be pretty slow in the field -- it's a small minority of the list traffic that seems to be using the new dkim-stripping upgrade. > And the fact that you completely ignore the existing trust > relationship that mailing lists have with their members in discussing > third party signatures makes me wonder how carefully you and your > colleagues have really thought about mailing lists. (Especially since > that trust relationship has been explicitly pointed out!) > Mailing list members are not automatons dealing with the vast open sewer of email. Those automatons currently have absolutely no clue who their users do and don't have trust relationships with, and it's no easy chore to come by it even if you are so inclined -- which I suspect most admins will not be. > So I just don't see an existing best practice here. I see an attempt > to develop an extension to an existing draft based purely on theory, > and theory not really grounded in current practice, to boot. I worry > that an attempt to make Mailman conform to DKIM rather than write > list-friendly wording into the standard will cause the collateral > damage to be set in stone. > In fact, the draft has been happily humming alone with current lists for quite some time. In any case, if you have some ideas about what list friendly wording is, I'd be happy to hear it. The matter of the DKIM BCP is still in its infancy. Mike From stephen at xemacs.org Tue Feb 6 16:33:14 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 07 Feb 2007 00:33:14 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C7AB42.7080906@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <45C753BD.4010004@cisco.com> <87abzsico0.fsf@uwakimon.sk.tsukuba.ac.jp> <45C7AB42.7080906@cisco.com> Message-ID: <874ppzi9mt.fsf@uwakimon.sk.tsukuba.ac.jp> Michael Thomas writes: > Let's be clear that I'm advocating a dialog here, In some sense, there's very little room for dialog, unless it involves substantial amendments to DKIM. This is inherent in the design: the whole message is signed. Preserve it nearly verbatim or break the signature. This need not be a problem, however, as long as users can be taught not to panic because one signature doesn't verify. See below. > I'm hoping that we can come up with some finesse. It's not obvious to me that a finesse is necessary or desirable. IMO, the right answer is for mailing lists to sign the posts that pass through them, and to publish a BCP that extols the manifold advantages to lazy admins of vetting a bunch of mailing lists and then trusting the signatures of the trustworthy ones. The transition period may be painful, but so is spam. > In any case, if you have some ideas about what list friendly > wording is, I'd be happy to hear it. After reading dkim-base-8, things are a lot clearer. Specifically, the updated Section 4 makes it clear that there's no reason why a mailing list is a second-class citizen: Of course, a message might also have multiple signatures because it passed through multiple signers. A common case is expected to be that of a signed message that passes through a mailing list that also signs all messages. Assuming both of those signatures verify, a recipient might choose to accept the message if either of those signatures were known to come from trusted sources. While I could wish for a stronger endorsement, I realize that is about as strong an endorsement as you'll find in an informative section in a standards-track RFC. I guess in the BCP I'd like to see that language (at least) repeated with encouragement to implementers to (eg) have verifiers look for a mailing list signature if RFC 2369 headers are present. The heuristic being the one I've been hammering on: if your users are subscribed to a mailing list, they evidently trust it to a greater or lesser degree. And of course users and ISPs should be encouraged to use MUAs and servers that employ verifiers with those features. By the way: *WARNING* Most of the links from the DKIM site point to version dkim-base-7b, while the current version is dkim-base-8. The latter has a much more satisfactory Section 4 (on multiple signatures). Most recent version (currently v8) is here: https://datatracker.ietf.org/public/idindex.cgi?command=id_detail&id=14210 From barry at python.org Tue Feb 6 22:31:43 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Feb 2007 16:31:43 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 3, 2007, at 12:43 AM, Stephen J. Turnbull wrote: > I think this points in the exact opposite direction from what you > claim: users already understand what From means in the mailing list > context, so if From and the DKIM signature are in conflict, the latter > SHOULD be adjusted. Since the ML is not in possession of the relevant > key, removal is the only option. ISTM that the DKIM spec is in agreement with you Stephen: http://www.dkim.org/specs/draft-ietf-dkim-overview-02.html#anchor61 I think we can say Mailman is in compliance with choice #3 in this list. I will also agree with the Note at the beginning of this section that this "may be controversial". Indeed. >> The very basic test I use is what's in the From: address. That's >> the thing that's pretty universally displayed and one that users >> are most likely to grok. Anything beyond From, and you've probably >> lost at least half of the user population at least. > > If the users don't know the difference between "From" and "Sender", > they have little chance of using DKIM as intended *except in the case > where From == Sender* (or they are in the same DKIM domain). Doesn't > this amount to saying "we don't know what DKIM BCP for mailing lists > is---that will be determined by the ability of average users to grok > RFC 2822 (and we believe that is quite low)". I'll just also comment that it is controversial that Mailman is adding a Sender header at all to outbound copies. We've had this discussion in the past (no time to find the archive threads), and IIRC we've decided that the relevant RFCs are ambiguous wrt mailing list best practices. No surprise there, but we also decided that adding Sender is a valid interpretation of the RFCs and the most reasonable thing for us to be doing. However, Sender isn't useful to people because they never see it. Every MUA I've ever used simply displays the From header as who the message was originally written by, and I don't think we would provide clarity to our users if we munged that, under normal circumstances. We /have/ to do so for digests or anonymized lists, but that's a different situation. > DKIM can help here. Eg, if DKIM-enabled clients are told that they > SHOULD recognize RFC 2369 headers and try to match the list identity > to the DKIM signature, and DKIM is extended to provide for resenders > (including lists that move domains---the List-ID will have a different > domain then). Agreed. > Again, your argument bites the other way around. I think it's a > reasonable presumption that the user has *already* established the > relevant trust relationship with the mailing list, because either the > list is run by the user's employer (or similar), or the user opted in. > Certainly you can presume that of any mailing list willing to > cooperate with DKIM! Agreed! How many of you have trust relationships with the other members of this list? What about your trust relationship with python.org, and this mailing list in particular? > I think that all of this really points to issues that need to be > resolved in the DKIM specification, not with mailing lists or any > other existing infrastructure. It's DKIM that needs to establish BCPs > for user interface and user education, not other parts of the > infrastructure that need to cover for DKIM's weaknesses at this point. > Once DKIM has established its own BCPs, and technical analysis shows > that it *requires* cooperation with other protocols to resolve > important ambiguities, *then* it's time to ask MLMs etc to help with > those issues. Agreed! - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcjzwHEjvBPtnXfVAQLPOgP/awMIcfI7J92Z9Lqvt7MYdm1mMFoHdpzU VjO8RBTf7d3468TtCVowwKskduRcQazbsXAHnvl4FfoIqVlVI/mnktD0Tr1N0/3h 56CIu5cVbrTZc1wISPvBhs2zzGZNRxy7BfkbK5YZxFZZ9VnaNtcue9C/WgHiQ4HA XosqMDkMm3E= =3MXm -----END PGP SIGNATURE----- From mat at cisco.com Tue Feb 6 22:40:24 2007 From: mat at cisco.com (Michael Thomas) Date: Tue, 06 Feb 2007 13:40:24 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> Message-ID: <45C8F5C8.3060601@cisco.com> Barry Warsaw wrote: > > ISTM that the DKIM spec is in agreement with you Stephen: > > http://www.dkim.org/specs/draft-ietf-dkim-overview-02.html#anchor61 This is not the spec -- and it's not been widely vetted. > > I think we can say Mailman is in compliance with choice #3 in this > list. I will also agree with the Note at the beginning of this > section that this "may be controversial". Indeed. The bottom line here is that you are removing signatures that are not broken. In fact, you don't even check to see if they're broken at all. That's bad all around. Mike From barry at python.org Tue Feb 6 22:52:46 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Feb 2007 16:52:46 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C753BD.4010004@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <45C753BD.4010004@cisco.com> Message-ID: <373657E5-445D-4E10-A6EB-E206C74C0A2B@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 5, 2007, at 10:56 AM, Michael Thomas wrote: > This is all > really fuzzy > though: barring S/MIME there's no guarantees about "authorship" per > se. That's reasonable and I don't think it contradicts anything else being said here. E.g. if I use OpenPGP signed mail because I want to assert that the words you read are mine. If some intermediary (some of which I know about and others I have no clue about) break my signature, well then you can't verify that what you're reading are my words as I typed them. That's exactly what should happen. > Once > it's in the hands of your MSA, it owns it. What DKIM tries to do is > at least > give some guarantees at the administrative/delegation points (ie, the > domain) > to other administrative/delegation points. It can only do this if the > intermediate > agents in the delivery path don't destroy the signature. If something > destroys > the signature, it's then taking responsibility for the downgrade in > the > assurance > that the signature provided. Some receivers may accept that > downgrade, some > may weight it negatively, some may reject it out of hand. But > ultimately > it's > the responsibility of the signature mangler to make that tradeoff. This aligns with the way I see DKIM being useful in a mailing list environment. The MLM mangles the message in ways appropriate to its use patterns, and it takes responsibility for the message by signing it with its own DKIM key. The fact that the From header still shows the original author is orthogonal. > And I really have no clue what "adjustment" you might be thinking of. > Removing things that are signed so as to make the signature useless? > Something people need to grok is that transformations of messages by > mailing lists need to be viewed as no different than > transformations made > by an attacker. How does a piece of delivery machinery know whether > transformations are benign or malicious? Not very well, as it turns > out. It's also no different than transformations that can happen to email for any number of possible reasons. I'm old enough to remember when UUCP delivery would do all kinds of transformations to emails. What about other delivery mechanisms such as NNTP gating or archiving? Maybe your imap server does something wacky to the messages. Those transformations can be perfectly legitimate, as I claim a mailing list's transformations are. You as the final recipient of this message can only verify that the MLM has vouched for the message and hope that whatever gateways between the MLM and you haven't broken the signature. > What we've been going on is that people sort understand From and > that's > about it. Machine learning may be more clever in time. For mailing > lists > to destroy the From signature is a problem -- compounded immensely by > arbitrarily deleting them from whence nothing can recover the > signature. I go back to section 9.3 of the DKIM overview. If you could implement the parsing of the DKIM headers and guarantee that any transformations made by Mailman would not break the signature, then Mailman could add its own signature and everything would be golden. Mailman currently cannot make such a guarantee and cannot sign outgoing message with its own DKIM signature, so ISTM that we're doing the right thing by removing the existing DKIM header. If you (or anyone else) would like to donate some code to allow Mailman to do something more intelligent with the DKIM headers, for example by allowing us to choose points 1 or 2 from section 9.3, such a contribution would definitely be welcome. > Spammers and bad guys can resign too. As I said, people grok From and > that's about it. If adding a third party signature causes people to > give > more > credence to the From address, then you've opened yourself up to > malicious > third parties. The draft specifically states that a third party > signature needs to > be an "acceptable" third party signature. That is, a third party you > trust not > to screw you. How you determine that trust is not specified, but it > is still > required. The right thing then would be for Mailman to sign the Sender header. Unfortunately, no MUA that I know of presents that information to a user, but it could still be used by a DKIM verifier. > I'm afraid that intransigence from the mailing list community is > likely to > really backfire. Mailing list traffic is an extremely small percentage > of traffic, > and most admins are likely to just ignore the collateral damage if > it's too > much a nuisance. Don't get me wrong: I spend far too much of my day on > mailing lists and would really like things to work out. But hard line > positions > in the face of thorny engineering tradeoffs doesn't help. Well, mailing lists and even email will probably die with us dinosaurs anyway. All the kids are using IM, IRC, and VOIP instead. But lets be honest, the DKIM specs themselves acknowledge that there are, at the very least, still open questions w.r.t. mailing list use cases. So, trying to explore what the right thing to do work better than threats or hard-line positions. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcj4rnEjvBPtnXfVAQKEogQAsz0wifOZ1ocEiGX8EoJVCAsDtta21+Cr zTX3+oihGJgiGiOnYLW7s/SgPBT3/KZugjQ0WWss4TkVss/YjAy/02mGh3i8uTkH OKlUpnIWWmhzLPjuexA1o2RS3IChjZODSJ80fJL99jfymaR3eoDA+a4BKQIpB8z/ Q9xvLNrJuZ0= =9teO -----END PGP SIGNATURE----- From barry at python.org Wed Feb 7 00:03:26 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Feb 2007 18:03:26 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C8F5C8.3060601@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 6, 2007, at 4:40 PM, Michael Thomas wrote: >> http://www.dkim.org/specs/draft-ietf-dkim-overview-02.html#anchor61 > > This is not the spec -- and it's not been widely vetted. Fair enough; it's also out of date as Stephen pointed out. Still, it does indicate that the DKIM authors acknowledge that there are compatibility issues with mailing lists. The updated section 4 that Stephen posted seems to be moving toward resolving those issues. I really want to see the spec address mailing list issues in a thorough way, with clear instructions on what such remailers must and should do. Then we can say "Mailman is broken wrt to the spec" or "Mailman complies with the spec" or "Can someone please contribute code to comply with the spec" or "the spec is broken, we don't agree with it, so we won't support it and everyone should abandon Mailman" :). >> I think we can say Mailman is in compliance with choice #3 in this >> list. I will also agree with the Note at the beginning of this >> section that this "may be controversial". Indeed. > > The bottom line here is that you are removing signatures that are not > broken. In fact, you don't even check to see if they're broken at all. > That's bad all around. We're removing signature that we know nothing about. As I said, IWBNI we had code that could check DKIM signatures and sign messages. So a question to ask, in the face of no available code to do the verifying or signing, is it better to possibly break signatures because of Mailman transformations or better to not have a signature at all. And why? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRckJPnEjvBPtnXfVAQL0KAP9Ejeu+dSdN838rCAlzpumrM4myazKsQ8D Ya7+yOQ5saKbmhFO/eZpeDn7YWRT2MUw2+P+BuFd0QKEOzmkAeowaLfqtz5r8mme NQDyXUsj22YGOGV5nK+i8egnmDVvspb4nJ1j9ahuqLmQ3RMtCoIYq+jRx8sCdWXo 5VQAsYjcNAE= =UnvC -----END PGP SIGNATURE----- From mat at cisco.com Wed Feb 7 00:55:47 2007 From: mat at cisco.com (Michael Thomas) Date: Tue, 06 Feb 2007 15:55:47 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> Message-ID: <45C91583.5020808@cisco.com> Barry Warsaw wrote: >> This is not the spec -- and it's not been widely vetted. > > Fair enough; it's also out of date as Stephen pointed out. Still, it > does indicate that the DKIM authors acknowledge that there are > compatibility issues with mailing lists. The updated section 4 that > Stephen posted seems to be moving toward resolving those issues. I'm afraid that there's not much consensus on how to deal with the mailing list issue; the people who say "resign" are guessing as there is little/no evidence that resigning is actually a viable strategy. In fact, up until I found that your new version removed signatures I had pretty much figured that this was a tempest in a teapot because the vast majority of first party DKIM signatures pass verification through lists in an actual large deployment. Let me float this though: how about a "signature friendly" knob that configures the list to not do things that are known to be harmful to signatures (including s/mime and pgp for that matter). I don't think this needs to be _especially_ complicated, but it would probably need to be fleshed out. > > I really want to see the spec address mailing list issues in a > thorough way, with clear instructions on what such remailers must and > should do. Then we can say "Mailman is broken wrt to the spec" or > "Mailman complies with the spec" or "Can someone please contribute > code to comply with the spec" or "the spec is broken, we don't agree > with it, so we won't support it and everyone should abandon Mailman" :). I think it would actually work a lot better the other way: with real life experience and real life running code we can make a much better case for what constitutes a best common practice. Part of the problem right now is that a lot of the speculation is just that. >> >> The bottom line here is that you are removing signatures that are not >> broken. In fact, you don't even check to see if they're broken at all. >> That's bad all around. > > We're removing signature that we know nothing about. As I said, IWBNI > we had code that could check DKIM signatures and sign messages. So a > question to ask, in the face of no available code to do the verifying > or signing, is it better to possibly break signatures because of > Mailman transformations or better to not have a signature at all. And > why? Do you remove PGP ascii armor just on the offhand chance that you might destroy a PGP signature with some configurations of mailman? By removing signatures you go from having about a 99% success rate to a 0% success rate. You don't have to have 100% success rate with filters to bias them to do the right thing, but removing signatures makes it stand out for the *lack* of its signature, especially given filters trained on signing all of their mail. As I mentioned, we are planning to annotate messages that don't verify from our domain. With this change there will be a lot more people getting useless false positives. Is that of no consequence? But let's turn this around: why do you think practice is helpful? I really don't understand the motivation at all. Destroying information -- especially when you're charged with forensic exercises like spam filters are -- is *rarely* the right thing to do. It seems to me that the burden of proof should be on why removing them is the right thing, not the other way around. Mike From joe at skyrush.com Wed Feb 7 02:07:10 2007 From: joe at skyrush.com (Joe Peterson) Date: Tue, 06 Feb 2007 18:07:10 -0700 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C91583.5020808@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> Message-ID: <45C9263E.9020306@skyrush.com> Michael Thomas wrote: > But let's turn this around: why do you think practice is helpful? I really > don't understand the motivation at all. Destroying information -- especially > when you're charged with forensic exercises like spam filters are -- is > *rarely* the right thing to do. It seems to me that the burden of proof > should be on why removing them is the right thing, not the other way > around. Hi Mike, The original rationale for removing the signature lines were the following (at least from my point of view): 1) It was believed that Mailman would almost certainly cause the signature to break, since it [often] adds info at the end of the message, among other possible changes. You have since told us that there are ways around this, but I wonder if there is a sure way for Mailman to know if the "modes" used in signing are compatible with its transformations (see my idea below for having Mailman re-check the sig after transformation and reporting this fact). 2) The outgoing MTA (sendmail) milter seemed to only want to sign emails that did *not* already have a signature. Therefore, Mailman enabled them to re-sign by removing the old (presumably invalid anyway) signature. At least this way *some* valid signature, even a Sender one, could be placed on the message. This "restriction" may not apply anymore or may be milter-specific... And there is one more thing that, at least in my mind, made leaving the "known bad" sig in there a bad thing: if it is destined to be invalid, especially if there is almost no way to keep it valid, it seemed best to remove the "doomed" signature. Even though DKIM states that bad sigs should be treated like "no sigs", it just seemed wrong to leave known bad sigs to potentially make the receiver think there is something awry. Instead, having the new message signed with a new good signature that makes the MLM responsible certainly seemed OK and better. Also, the results of the DKIM validation by the MTA (before it gets to Mailman) are *not* removed by Mailman, so there is a trace of the fact that (if the MLM host's MTA did DKIM checks) the original signature passed the test before the message was transformed, so there is a chain of trust (i.e. the reciever trusts the MLM, and the MLM host's MTA verifies the original sender and reports that in the header). Now, after saying all of this, it still may be best to leave the sigs in there. I am not convinced either way yet. I can see your point about the forensic value. Is there a way, somehow, that Mailman could verify the message before transformation (or the MTA could have done this) and then again after transformation - and then indicate in the header if the original signature has now been rendered invalid (through an as-yet undefined header line) - and then re-sign the message again (or maybe the MTA would re-sign it on the way out)? This could perhaps be a way for Mailman to indicate to the receiver that an invalid From signature is to be expected, and no alarm bells would then go off (the receiver could then rely on the chain of trust). But if the From signature proves to be still valid after transformation, the receiver would know this is expected and could check this again and be even more confident, relying on not only the chain of trust but also on the primary signature. This would probably require adding to the DKIM spec, but it might help to make the whole MLM situation more deterministic. Hope that was clear...? It's hard to tell from re-reading it, since this stuff is a bit mind-bending. -Joe From bob at nleaudio.com Wed Feb 7 02:51:03 2007 From: bob at nleaudio.com (Bob Puff) Date: Tue, 6 Feb 2007 20:51:03 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C9263E.9020306@skyrush.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> Message-ID: <20070207014225.M89512@nleaudio.com> I confess not having read up on Domain Keys.. I did get into SPF a little, but understand its flaws as well. If a bad DK isn't bad, then how is this supposed to help spam? I mean, if the mere presence of some signature in the headers will increase the likelihood of an email being delivered (or at least help it NOT be tagged as spam), surely the spammers will pick up on this, and the whole benefit lost. Example: Spammer takes a legit message from a DK sender, replaces it with his spam, and blasts it out with the original DK headers. The message has obviously been altered, and contains spam. Would it not be right to reject this message, since it fails the DK check? Now if the DK verification were done on the input side to Mailman (that is, in the MTA), I can see a benefit. But even in that scenerio, unless Mailman is signing, I'd think removal of the DK headers would be the right thing to do. Bob From joe at skyrush.com Wed Feb 7 04:12:39 2007 From: joe at skyrush.com (Joe Peterson) Date: Tue, 06 Feb 2007 20:12:39 -0700 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <20070207014225.M89512@nleaudio.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> <20070207014225.M89512@nleaudio.com> Message-ID: <45C943A7.3060207@skyrush.com> With DKIM, according to my understanding, you are supposed to treat a "bad" sig the same way you'd treat "no" sig. So it would neither help nor hurt to have a bad signature; it would be like having none (or a missing sig). Personally, I think DKIM would be a whole lot more effective and powerful if we *could* treat bad sigs as bad. Also, I think there is danger of people reacting to bad signatures negatively. Personally, I'd eye a failed sig with a more suspicious eye than no sig. -Joe Bob Puff wrote: > I confess not having read up on Domain Keys.. I did get into SPF a little, but > understand its flaws as well. > > If a bad DK isn't bad, then how is this supposed to help spam? I mean, if the > mere presence of some signature in the headers will increase the likelihood of > an email being delivered (or at least help it NOT be tagged as spam), surely > the spammers will pick up on this, and the whole benefit lost. > > Example: > > Spammer takes a legit message from a DK sender, replaces it with his spam, and > blasts it out with the original DK headers. The message has obviously been > altered, and contains spam. Would it not be right to reject this message, > since it fails the DK check? > > Now if the DK verification were done on the input side to Mailman (that is, in > the MTA), I can see a benefit. But even in that scenerio, unless Mailman is > signing, I'd think removal of the DK headers would be the right thing to do. > > Bob > From stephen at xemacs.org Wed Feb 7 07:39:18 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 07 Feb 2007 15:39:18 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C943A7.3060207@skyrush.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> <20070207014225.M89512@nleaudio.com> <45C943A7.3060207@skyrush.com> Message-ID: <87veieh3op.fsf@uwakimon.sk.tsukuba.ac.jp> Joe Peterson writes: > With DKIM, according to my understanding, you are supposed to treat a > "bad" sig the same way you'd treat "no" sig. I don't think the spec says that. It says: A verifier SHOULD NOT treat a message that has one or more bad signatures and no good signatures differently from a message with no signature at all; such treatment is a matter of local policy and is beyond the scope of this document. Ie, the *verifier* must treat those cases in the same way, but that clearly refers to passing the message on to the policy agent. Verifiers MAY note that a signature failed and that fact MAY be considered in subsequent processing (ie, by the policy agent): Verifiers SHOULD ignore any DKIM-Signature header fields where the signature does not validate. Verifiers that are prepared to validate multiple signature header fields SHOULD proceed to the next signature header field, should it exist. However, verifiers MAY make note of the fact that an invalid signature was present for consideration at a later step. and An MTA who has performed verification MAY communicate the result of that verification by adding a verification header field to incoming messages. This considerably simplifies things for the user, who can now use an existing mail user agent. Most MUAs have the ability to filter messages based on message header fields or content; these filters would be used to implement whatever policy the user wishes with respect to unsigned mail. A verifying MTA MAY implement a policy with respect to unverifiable mail, regardless of whether or not it applies the verification header field to signed messages. I think the intent here is that conceptually the MTA might wish to refuse to accept mail that is unverifiable, but that it may not make a distinction between no signature and a broken signature *at that stage*. (Of course the distinction between correct and broken signatures is ambiguous in the presence of multiple signatures. *sigh*) It is unfortunate that the draft conflates verifiers with policy agents. (Of course it makes sense that the same *application* might implement both verification and policy enforcement; conceptually they should be kept separate. This is especially important with respect to MTAs because milters are very frequently used to implement policy, and they should not be subject to these restrictions.) > Personally, I think DKIM would be a whole lot more effective and > powerful if we *could* treat bad sigs as bad. *You* (as a user or a site admin) can. This is explicitly left up to local policy. > Also, I think there is danger of people reacting to bad signatures > negatively. Personally, I'd eye a failed sig with a more > suspicious eye than no sig. Certainly. What we really want is policy agents that are smart enough to say to the user This message has a signature which verified successfully and one which failed. According to the Received trace and the List-Id header, and correlated with the SENDER_IS_MAILMAN_BOUNCE heuristic, the successful signature was added by the Mailman Users mailing list. The wooz.1april signature failed. In similar cases in the future for this mailing list, should I (o) Rely on the verified signature and silently accept the message ( ) Ask how to treat the message ( ) Silently discard the message [[Accept this message]] [Discard this message] (Of course if it's a milter it would have to be rule-based.) YMMV, but HTH Steve From stephen at xemacs.org Wed Feb 7 10:31:57 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 07 Feb 2007 18:31:57 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C91583.5020808@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> Message-ID: <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> Michael Thomas writes: > I'm afraid that there's not much consensus on how to deal with the > mailing list issue; the people who say "resign" are guessing as there > is little/no evidence that resigning is actually a viable strategy. >From the point of view of the mailing lists, resigning is *clearly* a viable strategy; the draft mandates that signature processing SHOULD continue until a success or all signatures are exhausted. (And it MAY continue after a success.) Assuming conformant verifiers (and no corruption in the pipeline after the mailing list, of course), resigning by the mailing list guarantees successful verification. Of course there's the issue that policy agents might choose to put 100% weight on the failure of the trusted domain's signature, and none on the success of the mailing list's signature. See my reply to Joe Petersen for a scenario of how this could be avoided. Whether it will work well in practice depends on whether policy agents (milters and MUAs, mostly, I should think) will offer such options. While clearly you can't make such handling a MUST, I would like to see the BCP specify that policy agents SHOULD provide such a feature to facilitate resigning by resenders. Finally, with regard to "loopback" verification, that's your local policy problem, really. It's true that munging by a few mailing lists is invalidating your outgoing signatures. Whether that consideration overrides the mailing lists' editorial policies is a question for negotiation among you, your users, and the list admins on a case by case basis. Remember, you always have the option of S/MIME on individual SignedData MIME body parts for this purpose, and your argument for resenders passing things through verbatim would be much stronger in that case. > Let me float this though: how about a "signature friendly" knob that > configures the list to not do things that are known to be harmful to > signatures (including s/mime and pgp for that matter). I don't think > this needs to be _especially_ complicated, but it would probably need > to be fleshed out. [ Actually, for Mailman 3 it would be nice to have a general "theme" engine that handles this kind of thing. ] I think as a practical matter it's only a matter of time before this becomes a FAQ in *both* directions (people who are having problems similar to Joe Petersen's as well as those similar to Cisco's). So I agree here. >>>>> "bw" == Barry Warsaw writes: bw> I really want to see the spec address mailing list issues in a bw> thorough way, with clear instructions on what such remailers must and bw> should do. > I think it would actually work a lot better the other way: with real life > experience and real life running code we can make a much better case > for what constitutes a best common practice. Part of the problem right > now is that a lot of the speculation is just that. I really really sympathize with Barry (especially since I don't really have the resources to help with implementation of a verifier or signer for Mailman), but I have to agree with you here. bw> We're removing signature that we know nothing about. As I said, IWBNI bw> we had code that could check DKIM signatures and sign messages. So a bw> question to ask, in the face of no available code to do the verifying bw> or signing, is it better to possibly break signatures because of bw> Mailman transformations or better to not have a signature at all. And bw> why? > > Do you remove PGP ascii armor just on the offhand chance that you might > destroy a PGP signature with some configurations of mailman? That's hardly fair. Joe Petersen writes (in a response to your message): 2) The outgoing MTA (sendmail) milter seemed to only want to sign emails that did *not* already have a signature. Therefore, Mailman enabled them to re-sign by removing the old (presumably invalid anyway) signature. First, note that the admin has explicitly stated that the existing signature is presumed invalid. He may be wrong, but this isn't an "offhand chance". Much more interestingly, note that the milter is broken with respect to mailing lists. IMO, Section 5.1 of dkim-base should get language that says a signer SHOULD NOT refuse to sign a message merely because there is an existing signature. Without that, Barry's question is very relevant. I would answer Barry that Mailman SHOULD preserve existing signatures because the spec will get language saying o signers SHOULD NOT refuse to re-sign o policy agents SHOULD report only successes by default o policy agents MAY provide an option to acquire information about failures In particular, in section 6.1, INFORMATIVE NOTE: The rationale of this requirement is to permit messages that have invalid signatures but also a valid signature to work. For example, a mailing list exploder might opt to leave the original submitter signature in place even though the exploder knows that it is modifying the message in some way that will break that signature, and the exploder inserts its own signature. In this case the message should succeed even in the presence of the known-broken signature. I'd like to see that last "should" in all uppercase. Then (sorry, Barry!) IMO the burden is clearly on Mailman and other list manager projects to either implement signing themselves, or publish their own BCPs strongly recommending use in combination with an MTA that will do the signing. > By removing signatures you go from having about a 99% success rate > to a 0% success rate. That's false. You go from a 99% success rate *on "loopback" messages* to 0%. On other messages the ex ante success rate will be much lower because the trust relationship is on average much weaker. However, the vast majority of *deliveries* are not "loopbacks", and those non-loopbacks are likely (see (1) below) to suffer from rejection on the basis that signatures fail. > With this change there will be a lot more people getting useless > false positives. Is that of no consequence? Of course it is of consequence. You keep ignoring the fact that Mailman has already experienced systematic useless false positives without this change. Is that of no consequence? > But let's turn this around: why do you think practice is helpful? I > really don't understand the motivation at all. What I really fear is that (1) everybody's intuition is that a failed verification is a positive indicator for spam (as compared to no signature, as well as compared to verified signature), and they want to apply policy filters based on that---we can expect that lots of people (and let's not forget AOL) will (ab)use the information that way, and (2) we've already seen list-hostile implementations in the wild--- we can expect that there will be more. Both of these are evidently conformant to the current spec. > Destroying information -- especially when you're charged with > forensic exercises like spam filters are -- is *rarely* the right > thing to do. True, but this is a Pythonic bunch. "Practicality beats purity." :-) From barry at python.org Wed Feb 7 16:18:00 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Feb 2007 10:18:00 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87veieh3op.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> <20070207014225.M89512@nleaudio.com> <45C943A7.3060207@skyrush.com> <87veieh3op.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <2E2B29CA-117D-4449-AC13-E78FA63030CC@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 7, 2007, at 1:39 AM, Stephen J. Turnbull wrote: > Certainly. What we really want is policy agents that are smart enough > to say to the user > > This message has a signature which verified successfully and one > which failed. According to the Received trace and the List-Id > header, and correlated with the SENDER_IS_MAILMAN_BOUNCE heuristic, > the successful signature was added by the Mailman Users mailing > list. The wooz.1april signature failed. > > In similar cases in the future for this mailing list, should I > > (o) Rely on the verified signature and silently accept the message > ( ) Ask how to treat the message > ( ) Silently discard the message > > [[Accept this message]] [Discard this message] Part of me agrees that this is what you'd like to see, but my gut tells me that this will never work in practice. First, no one but an email geek will even understand the question, let alone know how to answer it, and second, I fear that most u/i's and policy engines will boil this down to a very simple choice for the user: This message is unverified [Accept] [Discard] (o) Do the same for all similar messages - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcntrXEjvBPtnXfVAQKCHwP6A1hqINQZj+EFnC0Vr9i49/wdAx3lA3NW +E3LpOALR9rfhmTxr3IM7tK3niPz7BFl4s7aPZhTReHt2HqVuED4ZOZzV7z0s7hc x6UM/Cm05fiGAz0A3aTLtrJiq8zQfu0h8Vc4mBJxlUt4hOUB/In+gDsLAzVqyHOB N2qgM7Wll0w= =B1XG -----END PGP SIGNATURE----- From barry at python.org Wed Feb 7 16:32:26 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Feb 2007 10:32:26 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 7, 2007, at 4:31 AM, Stephen J. Turnbull wrote: >> Let me float this though: how about a "signature friendly" knob that >> configures the list to not do things that are known to be harmful to >> signatures (including s/mime and pgp for that matter). I don't think >> this needs to be _especially_ complicated, but it would probably need >> to be fleshed out. > > [ Actually, for Mailman 3 it would be nice to have a general "theme" > engine that handles this kind of thing. ] And I'm strongly in favor of having such a framework. > I think as a practical matter it's only a matter of time before this > becomes a FAQ in *both* directions (people who are having problems > similar to Joe Petersen's as well as those similar to Cisco's). So I > agree here. Yep. > I would answer Barry that Mailman SHOULD preserve existing signatures > because the spec will get language saying > > o signers SHOULD NOT refuse to re-sign > o policy agents SHOULD report only successes by default > o policy agents MAY provide an option to acquire information about > failures > > In particular, in section 6.1, > > INFORMATIVE NOTE: The rationale of this requirement is to > permit > messages that have invalid signatures but also a valid signature > to work. For example, a mailing list exploder might opt to > leave > the original submitter signature in place even though the > exploder > knows that it is modifying the message in some way that will > break > that signature, and the exploder inserts its own signature. In > this case the message should succeed even in the presence of the > known-broken signature. > > I'd like to see that last "should" in all uppercase. Then > (sorry, Barry!) IMO the burden is clearly on Mailman and other list > manager projects to either implement signing themselves, or publish > their own BCPs strongly recommending use in combination with an MTA > that will do the signing. I agree. > What I really fear is that > > (1) everybody's intuition is that a failed verification is a > positive indicator for spam (as compared to no signature, as > well as compared to verified signature), and they want to > apply policy filters based on that---we can expect that lots > of people (and let's not forget AOL) will (ab)use the > information that way, and > > (2) we've already seen list-hostile implementations in the wild--- > we can expect that there will be more. > > Both of these are evidently conformant to the current spec. I echo Steve's fear. I think without clarifying language, that's exactly what we'll see. We'll probably see it anyway, but at least when Mailman gets screwed, we can at least point to the spec and say "you're being unfair to us!". >> Destroying information -- especially when you're charged with >> forensic exercises like spam filters are -- is *rarely* the right >> thing to do. > > True, but this is a Pythonic bunch. "Practicality beats purity." :-) Indeed. Let me bring this back around to the question of what Mailman should do. Clearly we aren't going to implement DKIM verification and signatures in Mailman 2.1, although it would not be unreasonable to add such a handler modules as unsupported contributions. Is anybody motivated enough to implement them? I could see such handlers becoming officially supported in Mailman 2.2/3.0. What should MM2.1 do now? Here's a proposal for 2.1.10: Add an mm_cfg.py variable that controls whether DKIM headers are stripped or not. I think Mark suggested that this should be a site-wide variable, and I tend to agree. The reasoning being that all the outbound Mailman traffic will be ultimately delivered by an MTA under the site admin's control. Either they have a milter that refuses to resign or they have a working milter. If their milter doesn't resign, then less harm is done by stripping the header. If their milter does resign, then less harm is done by allowing it to resign, even if Mailman has broken the original signature. I don't think there's any feasible way for MM2.1 to determine whether its transformations break the original signature or not, but that infrastructure could be built into Mailman 2.2/3.0. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcnxCnEjvBPtnXfVAQKr0QQAmXahpxy7V6fRXJJb3rFaruS529hWHKeS QliPVV1CBdNFqCfsYGVkDKGMV/vdUvOdoOHgougIrWEjYZdlWp82LRzoWbjR4MS+ H0hMkUENuZA2oO/iBQ8K7hBlGoRLhHc2x8odyR0YhcMWBbDrq6I8ICSYrd3kMTu7 sFEOCGBTpFA= =7tCs -----END PGP SIGNATURE----- From mat at cisco.com Wed Feb 7 16:44:48 2007 From: mat at cisco.com (Michael Thomas) Date: Wed, 07 Feb 2007 07:44:48 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <20070207014225.M89512@nleaudio.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> <20070207014225.M89512@nleaudio.com> Message-ID: <45C9F3F0.7080404@cisco.com> Bob Puff wrote: > I confess not having read up on Domain Keys.. I did get into SPF a little, but > understand its flaws as well. > > If a bad DK isn't bad, then how is this supposed to help spam? I mean, if the > mere presence of some signature in the headers will increase the likelihood of > an email being delivered (or at least help it NOT be tagged as spam), surely > the spammers will pick up on this, and the whole benefit lost. > DKIM isn't about "solving" spam per se. It's about accountability. If you know about a source, you can treat it differently. DKIM allows you to know the source. That goes for both good and bad sources of mail. > Example: > > Spammer takes a legit message from a DK sender, replaces it with his spam, and > blasts it out with the original DK headers. The message has obviously been > altered, and contains spam. Would it not be right to reject this message, > since it fails the DK check? > It's no more right to reject based on a signature failure than any other single test; how strong a weighting you give a signature failures depends on a myriad of things -- if you want to prevent false positives. In fact, I'd say that one of the DKIM provides is a better way to prevent false positives rather than detecting spam per se. If you know and trust a source, mail talking about v**gr* is more likely to be legit. Mail without signatures or with broken signatures is just put through the normal unknown source spam filter, so it's just neutral rather than spammy. Mike From bob at nleaudio.com Wed Feb 7 16:48:12 2007 From: bob at nleaudio.com (Bob Puff) Date: Wed, 7 Feb 2007 10:48:12 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> Message-ID: <20070207154736.M9885@nleaudio.com> > What should MM2.1 do now? Here's a proposal for 2.1.10: Add an > mm_cfg.py variable that controls whether DKIM headers are stripped > or not. +1, with it defaulting to strip the DKIM. Bob From mat at cisco.com Wed Feb 7 16:53:27 2007 From: mat at cisco.com (Michael Thomas) Date: Wed, 07 Feb 2007 07:53:27 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C943A7.3060207@skyrush.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> <20070207014225.M89512@nleaudio.com> <45C943A7.3060207@skyrush.com> Message-ID: <45C9F5F7.30208@cisco.com> Joe Peterson wrote: > With DKIM, according to my understanding, you are supposed to treat a > "bad" sig the same way you'd treat "no" sig. So it would neither help > nor hurt to have a bad signature; it would be like having none (or a > missing sig). > > Personally, I think DKIM would be a whole lot more effective and > powerful if we *could* treat bad sigs as bad. Also, I think there is > danger of people reacting to bad signatures negatively. Personally, I'd > eye a failed sig with a more suspicious eye than no sig. > Until, of course, you rejected a piece of mail which had an x-million dollar deal in it... one thing we found out is that while people hate false negatives, mail admins *really* hate false positives. The truth of the matter is that shit happens in the mail system and overreacting based on single factors is a great recipe for generating lots of false positives. As an individual decision you can set your own tolerance level, but you quickly become a lot more conservative if you're doing it at a (large) group level. Mike From turnbull at sk.tsukuba.ac.jp Wed Feb 7 17:40:49 2007 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 08 Feb 2007 01:40:49 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <2E2B29CA-117D-4449-AC13-E78FA63030CC@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> <20070207014225.M89512@nleaudio.com> <45C943A7.3060207@skyrush.com> <87veieh3op.fsf@uwakimon.sk.tsukuba.ac.jp> <2E2B29CA-117D-4449-AC13-E78FA63030CC@python.org> Message-ID: <87hctxhqem.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > Part of me agrees that this is what you'd like to see, but my gut > tells me that this will never work in practice. First, no one but an > email geek will even understand the question, let alone know how to > answer it, Agreed. It's a stalking horse for the BCP; for anybody who's not an email geek, I have this simple advice regarding the BCP: Run away! Run away! > and second, I fear that most u/i's and policy engines will > boil this down to a very simple choice for the user: > > This message is unverified > > [Accept] [Discard] > (o) Do the same for all similar messages Make sure no spam gets through your double opt-in list, and you're golden, no? Similar is *not* going to reduce to "is unverified"; too many grandmothers will lose messages from the grandkids. From barry at python.org Wed Feb 7 17:38:00 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Feb 2007 11:38:00 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87hctxhqem.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> <20070207014225.M89512@nleaudio.com> <45C943A7.3060207@skyrush.com> <87veieh3op.fsf@uwakimon.sk.tsukuba.ac.jp> <2E2B29CA-117D-4449-AC13-E78FA63030CC@python.org> <87hctxhqem.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1CD902A7-66FA-4C5B-966D-4FA4F68A27FC@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 7, 2007, at 11:40 AM, Stephen J. Turnbull wrote: > Barry Warsaw writes: > >> Part of me agrees that this is what you'd like to see, but my gut >> tells me that this will never work in practice. First, no one but an >> email geek will even understand the question, let alone know how to >> answer it, > > Agreed. It's a stalking horse for the BCP; for anybody who's not an > email geek, I have this simple advice regarding the BCP: > > Run away! Run away! LOL. >> and second, I fear that most u/i's and policy engines will >> boil this down to a very simple choice for the user: >> >> This message is unverified >> >> [Accept] [Discard] >> (o) Do the same for all similar messages > > Make sure no spam gets through your double opt-in list, and you're > golden, no? Ideally yeah. But python.org does get reported occasionally since while I think we do a pretty good job of blocking most spam, some nasties gated from Usenet still get through. Sigh. (Today's rant: overly aggressive blackholes that block /all/ cable and DSL IPs that reverse to the ISP doman. For FSM's sake, I have a static IP address already!) > Similar is *not* going to reduce to "is unverified"; too many > grandmothers will lose messages from the grandkids. You might be right about that. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcoAaXEjvBPtnXfVAQIxrwP/XpboY8mzsbw5LG9L+8Un/M4ogwgP4FgO 6JYNwgaMUAl4cWb1uXBgVuICUEP9LJMwj24SnqENV5HSxDWL5ui2jLj4psPS0h3U 9JXVT/gOjrW2Mr6XeGpVhDO5Zwe8I6obJU4zy/Iey7cqkB+uRyVK+00R5DGmvvo+ cKtcxXYv5HI= =W+TD -----END PGP SIGNATURE----- From stephen at xemacs.org Wed Feb 7 17:49:49 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 08 Feb 2007 01:49:49 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> Message-ID: <87fy9hhpzm.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > What should MM2.1 do now? Here's a proposal for 2.1.10: Add an > mm_cfg.py variable that controls whether DKIM headers are stripped or > not. I think Mark suggested that this should be a site-wide > variable, and I tend to agree. I've expressed my reservations regarding list-specific issues like internationalization (especially random 7<->8 bit transformation), but on the one hand we don't have any practical experience showing it's needed, and on the other hand turning site-wide Mailman options into list-specific options is well-understood. Ie, it's a YAGNI until proven otherwise. What about the default? My strong inclination is don't strip, and word the docstring to discourage stripping. From mat at cisco.com Wed Feb 7 17:45:12 2007 From: mat at cisco.com (Michael Thomas) Date: Wed, 07 Feb 2007 08:45:12 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <45CA0218.7020000@cisco.com> Stephen J. Turnbull wrote: > Michael Thomas writes: > > > I'm afraid that there's not much consensus on how to deal with the > > mailing list issue; the people who say "resign" are guessing as there > > is little/no evidence that resigning is actually a viable strategy. > > From the point of view of the mailing lists, resigning is *clearly* a > viable strategy; the draft mandates that signature processing SHOULD > continue until a success or all signatures are exhausted. (And it MAY > continue after a success.) Assuming conformant verifiers (and no > corruption in the pipeline after the mailing list, of course), > resigning by the mailing list guarantees successful verification. > I'm not saying I think that resigning is a Bad Thing, I'm saying that it's speculative whether it's a Good Thing. You seem to keep ignoring the inherent attack involved with resigning: From: good at guy.org Sender: bad at fooledyou.com Dkim-Signature: d=fooledyou.com; [...] If all it takes to get preferential treatment (FSVO preferential) is adding any old third party signature, then we can pretty much guarantee that spammers will start doing that when it's in their interest. The draft specifically refers to "acceptable" in this case for that reason. This is the achillies heel of third party signatures: it presupposes that you -- or much worse your mail infrastructure -- knows what constitutes "acceptable". In practice I know that we as an organization have no clue who is acceptable, and I also know that even if we did, there's no infrastructure use/maintain it. I doubt that we're especially unique here. > > Do you remove PGP ascii armor just on the offhand chance that you might > > destroy a PGP signature with some configurations of mailman? > > That's hardly fair. Joe Petersen writes (in a response to your > message): > > 2) The outgoing MTA (sendmail) milter seemed to only want to sign > emails that did *not* already have a signature. Therefore, > Mailman enabled them to re-sign by removing the old (presumably > invalid anyway) signature. > That is just an implementation detail of the sendmail milter. Mine doesn't have that restriction and neither do others that I've seen. This is new code in a new area so getting all worked up about how a particular implementation behaves is not fair. > First, note that the admin has explicitly stated that the existing > signature is presumed invalid. He may be wrong, but this isn't an > "offhand chance". > I think the question is quite apt: there are lots of transforms that might damage PGP signatures as well: why aren't you stripping them en masse like you are dkim signatures? Could it be that it's because in reality they normally go through just fine? Well, the same is true of dkim signatures; that you've come to the opposite conclusion seems to be based off of experience with exactly one implementation that doesn't produce mailing list friendly signatures. But that implementation isn't inherent in the spec. Something to note is that a *lot* of the difference between DK and DKIM was our insistence that DKIM be far more robust through manglers than DK was. This was purposefully so that survival through mailing lists was much more likely -- something that a lot of people frankly didn't care about. > I'd like to see that last "should" in all uppercase. In RFC-speak, should and SHOULD are identical. > Then > (sorry, Barry!) IMO the burden is clearly on Mailman and other list > manager projects to either implement signing themselves, or publish > their own BCPs strongly recommending use in combination with an MTA > that will do the signing. > As I mentioned, part of the DKIM work is the SSP work as well. Being able to say that "I sign everything" is a pretty powerful indication that you should be more suspicious if you get a broken/unsigned piece of mail. So while I think it would be great if mailman pushed signing too, I think that there are other incentives for domains to bring mailing lists into the fold too. > > By removing signatures you go from having about a 99% success rate > > to a 0% success rate. > > That's false. You go from a 99% success rate *on "loopback" messages* > to 0%. On other messages the ex ante success rate will be much lower > because the trust relationship is on average much weaker. However, > the vast majority of *deliveries* are not "loopbacks", and those > non-loopbacks are likely (see (1) below) to suffer from rejection on > the basis that signatures fail. > Huh? There's no difference between "loopback" and anything else: either a signature verifies or it doesn't; we don't treat our own signatures any differently than any other signatures. When using the l= option, almost all signatures will verify. The only substantial difference between dkim and pgp signatures is when a mailing list annotates the subject line of a new thread, but this too can be worked around by intelligent use of z=. The only reason this isn't in the draft is because we didn't want to get bogged down describing heuristics in a standard (rightfully). All in all, I see no justification for doing anything different for DKIM that you wouldn't also do the corresponding thing to PGP. > > With this change there will be a lot more people getting useless > > false positives. Is that of no consequence? > > Of course it is of consequence. You keep ignoring the fact that > Mailman has already experienced systematic useless false positives > without this change. Is that of no consequence? > Which false positives? I don't remember seeing that, and I've certainly not seen anything in our deployment that suggests that bad signatures are causing FP's on other domains incoming mail. After a year of deployment signing millions of messages a day, I'd expect that I'd have heard at least *one* report of that. Heck Y! and Google signing billions of messages a day with DK which is a lot more fragile and I've never heard from either of them that they are having FP problems due to broken signatures. > > But let's turn this around: why do you think practice is helpful? I > > really don't understand the motivation at all. > > What I really fear is that > > (1) everybody's intuition is that a failed verification is a > positive indicator for spam (as compared to no signature, as > well as compared to verified signature), and they want to > apply policy filters based on that---we can expect that lots > of people (and let's not forget AOL) will (ab)use the > information that way, and > Yet all evidence in my experience is that that is not correct. Also: last I talked with the AOL guys they were planning to deploy DKIM as well, so they'd have a reasonable amount of incentive to understand the implications on FP's and signature breakage. In any case, I think it's unreasonable to be held to a standard that somebody, somewhere may have an overly aggressive filter; it's their problem honestly. > (2) we've already seen list-hostile implementations in the wild--- > we can expect that there will be more. > > Both of these are evidently conformant to the current spec. > There are legitimate reasons to be "list hostile". Statements at bigbank.com could probably care the least whether it survives a list, and in fact probably thinks that it's a feature not a bug. Note that there is a significant faction of DKIM proponents who are coming at it from that angle and from their perspective it's a correct stance. We, on the other hand, want DKIM to work in the more general user population case. So it shouldn't be surprising that different sites/implementations will have different signing policies. Mike > > Destroying information -- especially when you're charged with > > forensic exercises like spam filters are -- is *rarely* the right > > thing to do. > > True, but this is a Pythonic bunch. "Practicality beats purity." :-) > From barry at python.org Wed Feb 7 17:46:15 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Feb 2007 11:46:15 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87fy9hhpzm.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> <87fy9hhpzm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <81955F6C-F961-4802-8A1C-E173878C4542@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 7, 2007, at 11:49 AM, Stephen J. Turnbull wrote: > Barry Warsaw writes: > >> What should MM2.1 do now? Here's a proposal for 2.1.10: Add an >> mm_cfg.py variable that controls whether DKIM headers are stripped or >> not. I think Mark suggested that this should be a site-wide >> variable, and I tend to agree. > > I've expressed my reservations regarding list-specific issues like > internationalization (especially random 7<->8 bit transformation), but > on the one hand we don't have any practical experience showing it's > needed, and on the other hand turning site-wide Mailman options into > list-specific options is well-understood. Ie, it's a YAGNI until > proven otherwise. Agreed. I do think the effects of stripping vs. non-stripping may be felt on a per-list and per-user basis, but let's do the simplest thing possible now and see what happens. > What about the default? My strong inclination is don't strip, and > word the docstring to discourage stripping. Good question. Let's take an informal poll. Should we strip DKIM by default or not? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcoCV3EjvBPtnXfVAQIzyAP+MrDG7vgIr4fr/90Tz0xobxqvEJmqLAFB vt7bzZqhr1SRlg/k8Y6eED5TbVivqWxLV5Ea27Uy0OYdb4X2QnMOHVoBBF8NDKu9 61Rg33vH/taAWzlND20Tvp6S0QPi9eiS70NGe7RuXbISK2PSnvig5hxKx47uxJ/L hqpkyDPIgOQ= =/OAo -----END PGP SIGNATURE----- From joe at skyrush.com Wed Feb 7 17:49:46 2007 From: joe at skyrush.com (Joe Peterson) Date: Wed, 07 Feb 2007 09:49:46 -0700 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> Message-ID: <45CA032A.2060502@skyrush.com> Barry Warsaw wrote: > What should MM2.1 do now? Here's a proposal for 2.1.10: Add an > mm_cfg.py variable that controls whether DKIM headers are stripped or > not. I think Mark suggested that this should be a site-wide > variable, and I tend to agree. The reasoning being that all the > outbound Mailman traffic will be ultimately delivered by an MTA under > the site admin's control. Either they have a milter that refuses to > resign or they have a working milter. If their milter doesn't > resign, then less harm is done by stripping the header. If their > milter does resign, then less harm is done by allowing it to resign, > even if Mailman has broken the original signature. Yes! I think this is indeed the right solution. I also tend to agree with Bob that the default ought to be to strip DKIM headers, as DKIM-aware admins would know what this variable is about, but admins who know nothing about DKIM probably will not have re-signing implemented. -Joe From barry at python.org Wed Feb 7 18:19:13 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Feb 2007 12:19:13 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45CA0218.7020000@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <45CA0218.7020000@cisco.com> Message-ID: <86791B00-56A6-4633-8DBE-7F9EA6BDD75B@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 7, 2007, at 11:45 AM, Michael Thomas wrote: > I'm not saying I think that resigning is a Bad Thing, I'm saying > that it's > speculative whether it's a Good Thing. You seem to keep ignoring the > inherent attack involved with resigning: > > From: good at guy.org > Sender: bad at fooledyou.com > Dkim-Signature: d=fooledyou.com; [...] So wait, taken to its logical conclusion, doesn't this mean that really the only thing that DKIM cares about protecting is the sanctity of the From header? Tell me if I'm wrong but couldn't a spambot forge any header, create a valid signature for that header, and inject that message into the mail infrastructure? It could even forge the From header and have a perfectly valid signature for that. The reason From-forging may not be an effective strategy for the spambot though is because part of the point is to spoof the From header so that it looks like the spam is coming from someone you know. OTOH, how many people would smell something fishy if this message had this header: From: Barry Warsaw ? > If all it takes to get preferential treatment (FSVO preferential) > is adding > any old third party signature, then we can pretty much guarantee that > spammers will start doing that when it's in their interest. The draft > specifically refers to "acceptable" in this case for that reason. > This is > the achillies heel of third party signatures: it presupposes that > you -- > or much worse your mail infrastructure -- knows what constitutes > "acceptable". In practice I know that we as an organization have no > clue who is acceptable, and I also know that even if we did, there's > no infrastructure use/maintain it. I doubt that we're especially > unique > here. Then I think we're in a catch-22. We'd like the MLM to sign the message to verify that it is a valid resender. We'd like to keep the original signature for forensic reasons at the very least. Resigning is vulnerable to attack. There is no standard (de-facto or otherwise) for a valid resender to indicate "I got this message from so-and-so, munged it a bit, and resent it". We're stuck. You suggest that we just let the original signature through saying, well, that works most of the time and if it doesn't work for you, tough luck! But whatever choices we make will be deployed in the field for years and years and if the 800lb gorillas start to take a much harder line two years from now, our mailing lists will be screwed. I have a hard time seeing how, given the above scenario and inclination against resigning, that removing the DKIM signature and just letting our messages go through normal spam processing is worse than allowing a broken signature to sneak through. > I think the question is quite apt: there are lots of transforms > that might > damage PGP signatures as well: why aren't you stripping them en masse > like you are dkim signatures? Could it be that it's because in > reality they > normally go through just fine? No. It's because that would require a deletive transformation on the contents of the body (at least for OpenPGP) and Mailman doesn't do that[*]. Under a very controlled set of conditions, Mailman will / add/ text to the body of the message, and it may perform character set transformations, but in general Mailman isn't going to go searching through the body of a message to find some PGP delimiters to strip. Signatures that are embodied in MIME parts can and may very well be easily stripped. [*] the one minor exception is removing things like Accepted: headers from the body of the message, but this is tightly limited to the first 5 or so non-blank lines of the body. Even then it doesn't work in the face of multipart/alternatives. > Something to note is that a *lot* of the difference between DK and > DKIM > was our insistence that DKIM be far more robust through manglers than > DK was. This was purposefully so that survival through mailing > lists was > much more likely -- something that a lot of people frankly didn't > care about. Mailing list use cases are very often overlooked in email related RFCs. Funny because so many standards discussions are conducted on mailing lists. > Which false positives? I don't remember seeing that, and I've > certainly not > seen anything in our deployment that suggests that bad signatures > are causing > FP's on other domains incoming mail. After a year of deployment > signing > millions of messages a day, I'd expect that I'd have heard at least > *one* > report of that. Heck Y! and Google signing billions of messages a > day with > DK which is a lot more fragile and I've never heard from either of > them that > they are having FP problems due to broken signatures. What kind of reporting infrastructure is there for false positives? ISTM they'd be very hard for end users to spot and report. > Yet all evidence in my experience is that that is not correct. > Also: last I talked > with the AOL guys they were planning to deploy DKIM as well, so they'd > have a reasonable amount of incentive to understand the > implications on > FP's and signature breakage. In any case, I think it's unreasonable > to be held > to a standard that somebody, somewhere may have an overly aggressive > filter; it's their problem honestly. People complain all the time about never seeing messages posted to a mailing list. The only thing we can do is tell them whether their message made it to the list, and to check their junk folders. We can pass the buck to the users or their ISPs but the bottom line is that those kinds of aggressive tactics are employed all the time and there is often nothing the user can do about it. See me previous rant on blanket IP blackholeing. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcoKEXEjvBPtnXfVAQL0SAQAgETzDXQqrjlmSmqwf2fspGgM0idZGi2R piFWixLckRxEpqEqeKFWGS8DCqMprnZcfa4lQENRgOtBcQU0bH6QmpIgi0Y/XPI4 sRfhfHjNgd4HLdS0gKmDvJRwLdx/gcn8BGyLGAZbzAm0mabk3Z+TwexR7dYyhwOw HNtZ2g6wZG0= =H8ty -----END PGP SIGNATURE----- From barry at python.org Wed Feb 7 18:24:01 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Feb 2007 12:24:01 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <86791B00-56A6-4633-8DBE-7F9EA6BDD75B@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <45CA0218.7020000@cisco.com> <86791B00-56A6-4633-8DBE-7F9EA6BDD75B@python.org> Message-ID: <59D648E5-026F-4D7E-8E03-463C23292458@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 BTW, synchronicity is a weird thing. A friend of mine -- totally unaware of the current discussions -- just sent this to me: http://it.slashdot.org/comments.pl?sid=218726&cid=17752748 LOL. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcoLMXEjvBPtnXfVAQKQHwP+LHBd+K8kS0WP1D/YsZ9SCSzSqgXzqiY+ jjk1cPzvM5diDb9C5j1rkebZHKmHJtLlmgUHb5skxNr2OYz4uvVKFvm1eSWCGASW V8F9wATj0uO/Hiro+grBLoJsGuVc5+TBRjORqMXp3PPIhQy5rahobhTJSXu/Wya7 txI0BoqSQNY= =SCfD -----END PGP SIGNATURE----- From mat at cisco.com Wed Feb 7 21:08:51 2007 From: mat at cisco.com (Michael Thomas) Date: Wed, 07 Feb 2007 12:08:51 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45C9263E.9020306@skyrush.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> Message-ID: <45CA31D3.5000107@cisco.com> Joe Peterson wrote: > Michael Thomas wrote: > > 2) The outgoing MTA (sendmail) milter seemed to only want to sign emails > that did *not* already have a signature. Therefore, Mailman enabled > them to re-sign by removing the old (presumably invalid anyway) > signature. At least this way *some* valid signature, even a Sender one, > could be placed on the message. This "restriction" may not apply > anymore or may be milter-specific... > Yes, it is specific to sendmail's milter. My milter that we use doesn't do that. Actually, you should probably file a bug on that because the signer shouldn't be paying attention to signatures from other domains. My guess is that this was an attempt to prevent multiple signatures from the same domain, but I haven't looked at Murray's code to verify that. > And there is one more thing that, at least in my mind, made leaving the > "known bad" sig in there a bad thing: if it is destined to be invalid, > especially if there is almost no way to keep it valid, it seemed best to > remove the "doomed" signature. I don't see why that follows. As I've said, I've not seen anybody whose actually doing that in the field, and given gmail and Y! signing with DK, a pretty large percentage of the world's legitimate email is currently being signed. > Even though DKIM states that bad sigs > should be treated like "no sigs", it just seemed wrong to leave known > bad sigs to potentially make the receiver think there is something awry. > This is really the receiver's business, if you ask me. Bad receivers are going to do bad things regardless of dkim or anything else, but that's really their own problem. People make similar mistakes by rejecting via SPF instead of using it as a data point in a larger filtering decision. There's really nothing we can do about this. In fact, I'd say there is just as high a likelihood that receivers will find that mail from domains that normally sign their mail will look _more_ suspicious to filters if the signature is stripped. Consider what's happening to your Bayesian filters right now: they're being trained on DKIM-signature headers from Cisco as being ham (I hope!) rather than spam. The lack of the signature means less things for it lock on to... > Instead, having the new message signed with a new good signature that > makes the MLM responsible certainly seemed OK and better. Also, the > results of the DKIM validation by the MTA (before it gets to Mailman) > are *not* removed by Mailman, so there is a trace of the fact that (if > the MLM host's MTA did DKIM checks) the original signature passed the > test before the message was transformed, so there is a chain of trust > (i.e. the reciever trusts the MLM, and the MLM host's MTA verifies the > original sender and reports that in the header). > Actually, of all things that could be stripped, the verification result is actually what could/should be stripped since it's not cryptographically verifiable. I'm pretty simple minded: I either trust the domain that sent me the mail or I don't. It's mailman's problem, IMO, to determine whether the original sender should be passed through the list or not. > Now, after saying all of this, it still may be best to leave the sigs in > there. I am not convinced either way yet. I can see your point about > the forensic value. Is there a way, somehow, that Mailman could verify > the message before transformation (or the MTA could have done this) and > then again after transformation - and then indicate in the header if the > original signature has now been rendered invalid (through an as-yet > undefined header line) - and then re-sign the message again (or maybe > the MTA would re-sign it on the way out)? Yes, absolutely you could do this. In fact, with the milter or other similar pieces of software your MTA will already do those things for you. I don't really see the point of inventing a new header to do what an invalid DKIM-signature header will already do. Especially given that a lot of the things that you thought were invalid actually weren't. As for whether mailman itself should sign and/or verify... I suspect that letting the MTA do that is more expedient and will save a lot of work. I believe at this point many/most of the MTA vendors have DKIM or are working on them, as well as the open software community as well. On the other hand, _using_ dkim results within mailman as an anti spoofing mechanism might be interesting, though I really have no idea how much of a problem that is. > This could perhaps be a way > for Mailman to indicate to the receiver that an invalid From signature > is to be expected, and no alarm bells would then go off (the receiver > could then rely on the chain of trust). But if the From signature > proves to be still valid after transformation, the receiver would know > this is expected and could check this again and be even more confident, > relying on not only the chain of trust but also on the primary > signature. This would probably require adding to the DKIM spec, but it > might help to make the whole MLM situation more deterministic. > In general, I doubt it's going to help to convey your filtering results because the trust model wrong. (for the same reason we wouldn't trust the results of your AV engine... we'll check again, thankyouverymuch :) The same goes with the signatures: the incoming side is going to re-check them (or not!) and draw its own inferences based on that. Part of that is that those checks are going to have to deal with inevitable damage that happens -- which is a larger problem than just mailing lists; as ought to be pretty apparent by now, any one-dimensional test to determine spaminess is likely to have really terrible false positive rates, so fixating on just one particular feature of a piece of mail as being the make or break of passing isn't a very good strategy in this day and age. Mike > Hope that was clear...? It's hard to tell from re-reading it, since > this stuff is a bit mind-bending. > > -Joe > From mat at cisco.com Wed Feb 7 23:06:39 2007 From: mat at cisco.com (Michael Thomas) Date: Wed, 07 Feb 2007 14:06:39 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <86791B00-56A6-4633-8DBE-7F9EA6BDD75B@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <45CA0218.7020000@cisco.com> <86791B00-56A6-4633-8DBE-7F9EA6BDD75B@python.org> Message-ID: <45CA4D6F.3020502@cisco.com> Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Feb 7, 2007, at 11:45 AM, Michael Thomas wrote: > >> I'm not saying I think that resigning is a Bad Thing, I'm saying that >> it's >> speculative whether it's a Good Thing. You seem to keep ignoring the >> inherent attack involved with resigning: >> >> From: good at guy.org >> Sender: bad at fooledyou.com >> Dkim-Signature: d=fooledyou.com; [...] > > So wait, taken to its logical conclusion, doesn't this mean that > really the only thing that DKIM cares about protecting is the sanctity > of the From header? No, it doesn't. All it means is that you shouldn't blindly allow a third party to vouch for a first party (or any other party for that matter). This is just common sense: you need to have some trust in a third party before you trust what they have to say about another party, right? > Tell me if I'm wrong but couldn't a spambot forge any header, create a > valid signature for that header, and inject that message into the mail > infrastructure? It could even forge the From header and have a > perfectly valid signature for that. It can do all of those things _except_ create a valid signature for a domain that it does not control their DNS (ie, the repository for the public keys). That is, you can't forge cisco.com or python.org if you can't put your keys in our zones. This is really a feature if you consider it: making an environment where spammers have to sign their mail limits their degrees of freedom: they have to leave more tracks by registering domains, etc, etc. > > The reason From-forging may not be an effective strategy for the > spambot though is because part of the point is to spoof the From > header so that it looks like the spam is coming from someone you > know. OTOH, how many people would smell something fishy if this > message had this header: > > From: Barry Warsaw > > ? Er, I'm not following you. A lot of the motivation around DKIM was to limit the wholesale forgery that's possible now. If I could guess what you mean here is that you mean that it lacks a signature at all? That's where the SSP part of the work comes in: if you receive a piece of mail that's unsigned, you do a lookup to see what the originating domain's policy is with respect to signing mail. If they say "we sign everything", a receiver has a pretty good indication that something's not right and may bias the filtering and/or annotation accordingly. >> If all it takes to get preferential treatment (FSVO preferential) is >> adding >> any old third party signature, then we can pretty much guarantee that >> spammers will start doing that when it's in their interest. The draft >> specifically refers to "acceptable" in this case for that reason. >> This is >> the achillies heel of third party signatures: it presupposes that you -- >> or much worse your mail infrastructure -- knows what constitutes >> "acceptable". In practice I know that we as an organization have no >> clue who is acceptable, and I also know that even if we did, there's >> no infrastructure use/maintain it. I doubt that we're especially unique >> here. > > Then I think we're in a catch-22. We'd like the MLM to sign the > message to verify that it is a valid resender. We'd like to keep the > original signature for forensic reasons at the very least. Resigning > is vulnerable to attack. There is no standard (de-facto or otherwise) > for a valid resender to indicate "I got this message from so-and-so, > munged it a bit, and resent it". We're stuck. Yes it is. At this point I'm not pessimistic though; from my point of view it's more that we're just not sure how it will pan out. From that standpoint, we need to keep _all_ of the options open which means that it's prudent to _both_ try to get mailing lists to be dkim friendly for first party signatures _and_ resign things so that reputation engines, etc, have something to latch on to. > > You suggest that we just let the original signature through saying, > well, that works most of the time and if it doesn't work for you, > tough luck! But whatever choices we make will be deployed in the > field for years and years and if the 800lb gorillas start to take a > much harder line two years from now, our mailing lists will be > screwed. I have a hard time seeing how, given the above scenario and > inclination against resigning, that removing the DKIM signature and > just letting our messages go through normal spam processing is worse > than allowing a broken signature to sneak through. Frankly I think you'll be screwed even if you remove them too; removing them will not allow you to fly below the radar. Consider if Y! and Gmail had a bilateral agreement that they expect each other's mail to be signed and to put it in the bit bucket if it wasn't. It makes no difference whether you removed it or not: it lacks a valid signature in both cases. In that case, the only thing you could do is not destroy and/or remove the signature. > >> I think the question is quite apt: there are lots of transforms that >> might >> damage PGP signatures as well: why aren't you stripping them en masse >> like you are dkim signatures? Could it be that it's because in >> reality they >> normally go through just fine? > > No. It's because that would require a deletive transformation on the > contents of the body (at least for OpenPGP) and Mailman doesn't do > that[*]. Under a very controlled set of conditions, Mailman will > /add/ text to the body of the message, and it may perform character > set transformations, but in general Mailman isn't going to go > searching through the body of a message to find some PGP delimiters to > strip. Or base64 to QP or... My point is that you're holding DKIM to a standard that you don't hold PGP to. > Signatures that are embodied in MIME parts can and may very well be > easily stripped. Right, but at least in my experience they seem to get through... but this is why a "signature friendly" theme may well be useful. Probably a lot of list owners aren't even cognizant that there is a tradeoff that is made when they do their demimulating, etc. > Mailing list use cases are very often overlooked in email related > RFCs. Funny because so many standards discussions are conducted on > mailing lists. Tell me about it :| > >> Which false positives? I don't remember seeing that, and I've >> certainly not >> seen anything in our deployment that suggests that bad signatures are >> causing >> FP's on other domains incoming mail. After a year of deployment signing >> millions of messages a day, I'd expect that I'd have heard at least >> *one* >> report of that. Heck Y! and Google signing billions of messages a day >> with >> DK which is a lot more fragile and I've never heard from either of >> them that >> they are having FP problems due to broken signatures. > > What kind of reporting infrastructure is there for false positives? > ISTM they'd be very hard for end users to spot and report. Right -- they are, but it's not impossible, and with list traffic I think it's actually easier to spot. Given how much a lot of the upper level geeks here live and die by external mailing list traffic, I'm pretty sure that I'd have heard about it by now if it were a real problem (and heard, and heard, and heard...). > >> Yet all evidence in my experience is that that is not correct. Also: >> last I talked >> with the AOL guys they were planning to deploy DKIM as well, so they'd >> have a reasonable amount of incentive to understand the implications on >> FP's and signature breakage. In any case, I think it's unreasonable >> to be held >> to a standard that somebody, somewhere may have an overly aggressive >> filter; it's their problem honestly. > > People complain all the time about never seeing messages posted to a > mailing list. The only thing we can do is tell them whether their > message made it to the list, and to check their junk folders. We can > pass the buck to the users or their ISPs but the bottom line is that > those kinds of aggressive tactics are employed all the time and there > is often nothing the user can do about it. See me previous rant on > blanket IP blackholeing. Correct (and I'm in the same boat with your IP blackhole rant), but until there's a real life problem it's really sort of pointless to try to guess, IMO. As it turns out, the law of unintended consequences reared its head yet again, and voila here I am! :) Mike From barry at python.org Wed Feb 7 23:41:40 2007 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Feb 2007 17:41:40 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45CA4D6F.3020502@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <45CA0218.7020000@cisco.com> <86791B00-56A6-4633-8DBE-7F9EA6BDD75B@python.org> <45CA4D6F.3020502@cisco.com> Message-ID: <990F69D5-A571-4734-BB07-6F93BF90CC99@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 7, 2007, at 5:06 PM, Michael Thomas wrote: >>> I'm not saying I think that resigning is a Bad Thing, I'm saying >>> that it's >>> speculative whether it's a Good Thing. You seem to keep ignoring the >>> inherent attack involved with resigning: >>> >>> From: good at guy.org >>> Sender: bad at fooledyou.com >>> Dkim-Signature: d=fooledyou.com; [...] >> >> So wait, taken to its logical conclusion, doesn't this mean that >> really the only thing that DKIM cares about protecting is the >> sanctity of the From header? > No, it doesn't. All it means is that you shouldn't blindly allow a > third party > to vouch for a first party (or any other party for that matter). > This is just > common sense: you need to have some trust in a third party before you > trust what they have to say about another party, right? Sure. I guess my point was, that in your example above, what's being signed is the Sender header, and for that header, fooledyou.com /is/ the first party. So fooledyou.com is making no assertions about the From header. Is there a requirement in DKIM that the Sender domain is the same as the From domain? For a non-anonymized non-digest message, where Mailman isn't going to change the From header, it obviously cannot sign the From header. It will set its own Sender header, and is able to sign that. In that scenario there's no third party signing going on. Maybe the confusion is in the term "resigning". I'm not actually proposing Mailman (or its downstream MTA) resign anything; I'm proposing that we add another signature for the headers that Mailman does control. Like Sender. If we leave the original DKIM-Signature header alone, but simply add ours to match our Sender header, then we'll have at least one valid DKIM-Signature header, right? The one for the From header may indeed be broken. Maybe Mailman broke it or maybe some other system component broke it. Is that what you thought I meant? Is the scenario I just outlined inherently unsafe? So now in fact, this leads to a concrete proposal that is simple, MLM- friendly and requires no changes to existing standards: a mailing list BCP is to DKIM sign the List-Id header. You policy engine than can add weight for a message with a valid DKIM signature of the List- Id, even if other signatures, say by the original author are broken. I think for now I'll cut this response short, because I'd like to know what you think about that. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcpVpHEjvBPtnXfVAQKEagP/WbWH5+rQuAofi5QrWgabibU8RRXZ8yqs 3nY1sZlYB616N6vuJoY4aqVN6Ud4AiXIS4gZPOsX5IEXiihK2XLYEL+JPtHMINHZ al4aa/6sRxrizDGHDQH8db19umD0R9vYceBAoyjRwrE1b1XbBDh8+ALavXZ0Lum6 sD4/KOQC4+w= =IUsC -----END PGP SIGNATURE----- From msapiro at value.net Thu Feb 8 01:25:06 2007 From: msapiro at value.net (Mark Sapiro) Date: Wed, 7 Feb 2007 16:25:06 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45CA4D6F.3020502@cisco.com> Message-ID: Michael Thomas wrote: >Barry Warsaw wrote: >> >> The reason From-forging may not be an effective strategy for the >> spambot though is because part of the point is to spoof the From >> header so that it looks like the spam is coming from someone you >> know. OTOH, how many people would smell something fishy if this >> message had this header: >> >> From: Barry Warsaw >> >> ? > >Er, I'm not following you. So the bad guy fooled you! -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From jwblist3 at olympus.net Thu Feb 8 01:43:40 2007 From: jwblist3 at olympus.net (John W. Baxter) Date: Wed, 07 Feb 2007 16:43:40 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <20070207014225.M89512@nleaudio.com> Message-ID: On 2/6/07 5:51 PM, "Bob Puff" wrote: > If a bad DK isn't bad, then how is this supposed to help spam? I mean, if the > mere presence of some signature in the headers will increase the likelihood of > an email being delivered (or at least help it NOT be tagged as spam), surely > the spammers will pick up on this, and the whole benefit lost. They have. Much of the spam I see from the "new breed legitimate spammers"* contains DKIM signatures. (I have to assume those verify OK, else why put them in.) * "new breed legitimate spammers": That is, in the US, those who carefully conform to CAN-SPAM as far as one can tell--one can't really tell whether they "share" unsubscribed addresses with other spammers as likely new victims without probing. --John From jwblist3 at olympus.net Thu Feb 8 02:01:07 2007 From: jwblist3 at olympus.net (John W. Baxter) Date: Wed, 07 Feb 2007 17:01:07 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <03D6058E-A177-4751-AF96-2451B9EE44A9@python.org> Message-ID: On 2/7/07 7:32 AM, "Barry Warsaw" wrote: > Either they have a milter that refuses to > resign or they have a working milter. If their milter doesn't > resign, then less harm is done by stripping the header. If their > milter does resign, then less harm is done by allowing it to resign, > even if Mailman has broken the original signature. Or they have an MTA that doesn't care at all about DKIM, in which case less harm is done by not stripping. --John From msapiro at value.net Thu Feb 8 02:08:37 2007 From: msapiro at value.net (Mark Sapiro) Date: Wed, 7 Feb 2007 17:08:37 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45CA4D6F.3020502@cisco.com> Message-ID: Michael Thomas wrote: > >Frankly I think you'll be screwed even if you remove them too; removing >them will not allow you to fly below the radar. Consider if Y! and Gmail >had a bilateral agreement that they expect each other's mail to be signed >and to put it in the bit bucket if it wasn't. It makes no difference whether >you removed it or not: it lacks a valid signature in both cases. In that >case, >the only thing you could do is not destroy and/or remove the signature. Consider the headers and structure as follows (From: munged) of a message I just received from a Y! user. DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=Cz7v0Jd51rc72THmOlqA/rac152bUZdiFm2T2IAFflFuxVHHKJmabS0rzD2tU5zVSofVoc0rVg0g7t0NaDGRMt7JxXK8reox7TzMephYOxI0zcr5iWGejG57Fn/gcQtqng8uG0vAJLw1mfXHMaCcz726cj4iYQOYzbCb6UxXH4g=; X-YMail-OSG: 9QE.IHIVM1k8MHil45oNbkt10TvkD0DVytKmI1Ki4W.WDhIT4Qq6HnLM6dCWNcikXlMu.1lftQrfhq1fgEKml97AoKamDnsG4bZNT_FRLyHVTcU_cuUp7W04PgOjiNd9HJK6MSNeJsfDUfVqrnegItcDfJ1Kjs5tGYyMqDp084T22mRvpc3swag- Received: from [69.232.227.173] by web82813.mail.mud.yahoo.com via HTTP; Wed, 07 Feb 2007 15:00:43 PST Date: Wed, 7 Feb 2007 15:00:43 -0800 (PST) From: xxxxx Subject: Re: feb. 25th To: Mark Sapiro In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-1589639226-1170889243=:32077" Content-Transfer-Encoding: 8bit Message-ID: <276441.32077.qm at web82813.mail.mud.yahoo.com> --0-1589639226-1170889243=:32077 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit --0-1589639226-1170889243=:32077 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: 8bit --0-1589639226-1170889243=:32077-- (My MUA will mung the DomainKey-Signature: by wrapping, but it looked OK as I received it) I submit that there is nothing that Mailman could do to this message in the way of filtering content or adding msg_footer that wouldn't break the signature. I also submit that this message structure is typical of the vast majority of mail that originates from Y! Thus, it seems that the choice is break the signature or make no changes whatsoever to the message other than adding more headers. Mike talks about the l= parameter allowing adding trailing content, but I don't see Y! and Gmail using it, and even if they did, how would we (could we) add a footer without breaking either the signature or the MIME structure of the message. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From jwblist3 at olympus.net Thu Feb 8 02:10:27 2007 From: jwblist3 at olympus.net (John W. Baxter) Date: Wed, 07 Feb 2007 17:10:27 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <81955F6C-F961-4802-8A1C-E173878C4542@python.org> Message-ID: On 2/7/07 8:46 AM, "Barry Warsaw" wrote: > Should we strip DKIM by default or not? Not strip by default. Even though that changes the default vs the most recent Mailman, it leaves the default alone for everyone who jumps to 2.1.10 from earlier versions. --John From jwblist3 at olympus.net Thu Feb 8 02:14:37 2007 From: jwblist3 at olympus.net (John W. Baxter) Date: Wed, 07 Feb 2007 17:14:37 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <86791B00-56A6-4633-8DBE-7F9EA6BDD75B@python.org> Message-ID: On 2/7/07 9:19 AM, "Barry Warsaw" wrote: > OTOH, how many people would smell something fishy if this > message had this header: > > From: Barry Warsaw With many MUAs (including the vast majority of different MUA programs and versions) that would pass totally unnoticed, as the user only sees From: Barry Warsaw without taking some action. --John From msapiro at value.net Thu Feb 8 02:20:34 2007 From: msapiro at value.net (Mark Sapiro) Date: Wed, 7 Feb 2007 17:20:34 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: Message-ID: John W. Baxter wrote: >On 2/7/07 8:46 AM, "Barry Warsaw" wrote: > >> Should we strip DKIM by default or not? > >Not strip by default. > >Even though that changes the default vs the most recent Mailman, it leaves >the default alone for everyone who jumps to 2.1.10 from earlier versions. I think I am swayed by the arguments in this thread to favor Not Strip as the default, and I agree with John WRT its not being a behavior change for many. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From msapiro at value.net Thu Feb 8 03:39:33 2007 From: msapiro at value.net (Mark Sapiro) Date: Wed, 7 Feb 2007 18:39:33 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: Message-ID: Michael Thomas wrote: > >On Wed, 7 Feb 2007, Mark Sapiro wrote: > >> Mike talks about the l= parameter allowing adding trailing content, but >> I don't see Y! and Gmail using it, and even if they did, how would we >> (could we) add a footer without breaking either the signature or the >> MIME structure of the message. > > l= is the number of canonical bytes added to the body hash. > If l=5, for example, anything past the 5th canonical byte will > not affect the verification of the signature. That's the reason > we get such high verify rates through mailing lists. My point is that for what I consider good reasons, Mailman will add the msg_footer to such a message by wrapping additional MIME structure around the original multipart/alternative message. I.e., the original multipart/alternative text/plain text/html message will be recast as multipart/mixed multipart/alternative text/plain text/html text/plain with the final text/plain part containing the footer. Given that the original content-type header is included in the signature, the signature is now broken. If we were to take a different approach with a signature containing l=, either the l= includes all the text/plain and at least part of the text/html, in which we can't add the footer to the text/plain alternative without breaking the signature, or the l= includes none of the text/html part in which case the signature is not very good at verifying the validity of the text/html part. This further assumes we even know how to add a footer to a text/html part. See for some discussion of why we do it the way we do. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From stephen at xemacs.org Thu Feb 8 04:19:42 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 08 Feb 2007 12:19:42 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <1CD902A7-66FA-4C5B-966D-4FA4F68A27FC@python.org> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <45C9263E.9020306@skyrush.com> <20070207014225.M89512@nleaudio.com> <45C943A7.3060207@skyrush.com> <87veieh3op.fsf@uwakimon.sk.tsukuba.ac.jp> <2E2B29CA-117D-4449-AC13-E78FA63030CC@python.org> <87hctxhqem.fsf@uwakimon.sk.tsukuba.ac.jp> <1CD902A7-66FA-4C5B-966D-4FA4F68A27FC@python.org> Message-ID: <87abzpgwtt.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > > Make sure no spam gets through your double opt-in list, and you're > > golden, no? > > Ideally yeah. But python.org does get reported occasionally since > while I think we do a pretty good job of blocking most spam, some > nasties gated from Usenet still get through. Sigh. *sigh* is right. Mailing lists will take *some* collateral damage. But DKIM is a very plausible approach to not just nibbling at spam, but biting off as much as we can chew. Especially since (unlike SPF and challenge-reponse) it's already widely implemented. The benefits have to be assessed as potentially much bigger than the small deviation from the ideal. From stephen at xemacs.org Thu Feb 8 06:41:57 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 08 Feb 2007 14:41:57 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45CA0218.7020000@cisco.com> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <45CA0218.7020000@cisco.com> Message-ID: <878xf9gq8q.fsf@uwakimon.sk.tsukuba.ac.jp> Michael Thomas writes: > I'm not saying I think that resigning is a Bad Thing, I'm saying that it's > speculative whether it's a Good Thing. You seem to keep ignoring the > inherent attack involved with resigning: Have you actually read my posts, or just enough to feel defensive? I have specifically referred many times to the user-list trust relationship implicit in a double-opt-in list. I have provided examples of how the receiving system can acquire information about such relationships, as well as acknowledging that getting such information is a nontrivial problem in the context of busy admins and slow implementation cycles. The standard specifies that whether to trust a signature is a local policy decision. It seems to me that Mailman's decisions about whether to put signing and verification capabilities into Mailman itself will depend on the developers' estimate of whether list signatures will be trusted in those local policies. It seems to me that Mailman's decisions about how to default the signature-stripping functionality depend on our estimate of whether broken signatures will be given negative weight compared to no signature. My position is that Mailman lists[1] SHOULD participate in the DKIM protocol, because I think that if Mailman lists are important to users, they will respond to DKIM-related obstruction of their traffic by demanding that their MTAs trust their Mailman lists. If that's a PITA for the admins, too bad---that's the flip side of the admin benefits that come with use of DKIM to filter spam. And of course admins and MUA developers *will* develop trust management tools. Maybe you have less confidence in your vendors, of course---I *know* mine will rise to the occasion. So far, you talk out of both sides of your mouth. On the one hand, you fault my proposal because of the attacks inherent in trusting any old re-signature; on the other, you claim that you don't have trust in anybody's signature, and you don't even treat your own specially! Elsewhere you have implied that for you verification == trust if "From == signing domain". Just generalize that to include "List-Id == signing domain" in the policy agent software! And "Sender == signing domain". Users need never know that the "sender" who was verified is not the >From header. Why not recommend that UIs SHOULD treat *any* of the three conditions above as "sender verified", until experience shows that's a bad idea? I suspect that naive users will happily accept it if policy agents tell them "sender was verified, so we present you with this mail for your consideration." They may misconceive that the >From header was verified, but why should that matter? If it *does* matter to them (eg, because they know that "Mom would never send me porn spam"), then here's the recommended UI for spam discards when the sender has been verified as a mailing list: Treat similar messages as spam? [[yes]] [no] Note: this message was verified to have been relayed by the Mailman Developers mailing list at python.org, which accepts responsibility for relaying it to you. ( ) I am not a member of this list. Treat messages from this list as spam in the future. (o) I am a member of this list. Carefully scan messages from this list in the future. Something similar could be recommended for other cases where the verified sender is in a different domain from "From" (as I am, for example). Isn't that the way it's supposed to work? Do you really think that users will care as long as you explain that they're not blacklisting their mothers, and the spammer does get blacklisted? Footnotes: [1] Whether Mailman itself should be a signer or verifier is another question; I'm agnostic on that. From barry at python.org Thu Feb 8 19:27:42 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 8 Feb 2007 13:27:42 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: Message-ID: <1D67B8FF-C5B7-4E98-BCAE-33EF7D721EAE@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 7, 2007, at 8:20 PM, Mark Sapiro wrote: > John W. Baxter wrote: > >> On 2/7/07 8:46 AM, "Barry Warsaw" wrote: >> >>> Should we strip DKIM by default or not? >> >> Not strip by default. >> >> Even though that changes the default vs the most recent Mailman, >> it leaves >> the default alone for everyone who jumps to 2.1.10 from earlier >> versions. > > I think I am swayed by the arguments in this thread to favor Not Strip > as the default, and I agree with John WRT its not being a behavior > change for many. Me too. Here's my discussion on the topic, including a concrete proposal for Mailman 2.1.10 and 2.2/3.0. Feel free to comment on the wiki on in this thread. http://wiki.list.org/x/OgM - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRctro3EjvBPtnXfVAQIHMAP/X4kZL4llpLMLf0rtePsf15092VsF8Old AMZmEvkJ/MtFT1mTm+cFjWg6i4/wUHfP2LIBr8AwNcO8MIUHUbjOB7fLCn41v93n FIKLIlFp6liFqjv3167Mz1SRRnb5r5KAReyCoyRww+ogo/AgVn8HmekoG74DOwGp v/SJuD1YcPQ= =CuhH -----END PGP SIGNATURE----- From barry at python.org Thu Feb 8 19:35:31 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 8 Feb 2007 13:35:31 -0500 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <878xf9gq8q.fsf@uwakimon.sk.tsukuba.ac.jp> References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <45CA0218.7020000@cisco.com> <878xf9gq8q.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Excellent post Steve, thanks. I think we're converging on a solution for Mailman both in the short term and in the long term. See my previously posted wiki link for my current thoughts on the matter. I just wanted to add one other thing... On Feb 8, 2007, at 12:41 AM, Stephen J. Turnbull wrote: > "From == signing domain". Just generalize that to include > "List-Id == signing domain" in the policy agent software! And > "Sender == signing domain". I definitely agree that "List-ID == signing domain" should be added for interoperability with mailing lists. I'm not sure about Sender, only because Mailman's addition of Sender itself is not without some controversy (mostly over interpretation of RFC 2822 language IIRC). But there's no doubt that well-behaved mailing lists should include List-ID, so that makes a natural header to sign. See my discussion in the wiki page for situations where we might /not/ want to sign List-ID though. Michael, since you're a DKIM spec insider, can you please relay this discussion to that community (if you agree with us of course!). We're making a good faith effort to do our part, and I'd like to see the DKIM specs acknowledge the mailing list use case more strongly. Thanks, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRcttc3EjvBPtnXfVAQJXAAP+P9Ddon+VPGcu9JefS9gxWOVPuWJDNsWI 8r0l0DIxJ8AZysCLSVzXAXEJqTapQjWB8l7fGZQZjznPFjea6/L1jR9yVwZVqRYI J5nbpq2m3OerNpKkBM6rUHpSXKVs8GrDwMyi+st626UwJW93muDeeNPU1DPoLxj7 6Kagg+VD5Ts= =Nnuw -----END PGP SIGNATURE----- From mat at cisco.com Thu Feb 8 21:17:29 2007 From: mat at cisco.com (Michael Thomas) Date: Thu, 08 Feb 2007 12:17:29 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: Message-ID: <45CB8559.5090508@cisco.com> Mark Sapiro wrote: > Michael Thomas wrote: > My point is that for what I consider good reasons, Mailman will add the > msg_footer to such a message by wrapping additional MIME structure > around the original multipart/alternative message. > > I.e., the original > > multipart/alternative > text/plain > text/html > > message will be recast as > > multipart/mixed > multipart/alternative > text/plain > text/html > text/plain > Ah yes, multipart/alternative vs. lists is definitely unhappy for DKIM, though simpler mime layouts make it through on average -- at least given our sample. I agree that if you have to add a footer at all, that the way your doing it in this case seems perfectly reasonable. > with the final text/plain part containing the footer. Given that the > original content-type header is included in the signature, the > signature is now broken. > > If we were to take a different approach with a signature containing l=, > either the l= includes all the text/plain and at least part of the > text/html, in which we can't add the footer to the text/plain > alternative without breaking the signature, or the l= includes none of > the text/html part in which case the signature is not very good at > verifying the validity of the text/html part. This further assumes we > even know how to add a footer to a text/html part. > > Are you still speaking of multipart/alernative? Right now what we do for, say, text/html is not sign the trailing and final --. This allows lists to insert their trailers as they normally do in the mime/html body. Similar for text/plain too. For us at least (and it may be that we're just have a lot of html hating geeks), this seems to do the trick pretty well. I see some breakage from multipart/ alternative, but not _that_ much. Mike From mat at cisco.com Thu Feb 8 22:26:59 2007 From: mat at cisco.com (Michael Thomas) Date: Thu, 08 Feb 2007 13:26:59 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: <420E8D51-B986-4201-BAE2-736B58722AEE@python.org> <45C36F46.3080704@cisco.com> <87hcu3vlr8.fsf@uwakimon.sk.tsukuba.ac.jp> <42187335-3A8B-4F22-BFE8-06929C2B25FD@python.org> <45C8F5C8.3060601@cisco.com> <45C91583.5020808@cisco.com> <87tzxygvoy.fsf@uwakimon.sk.tsukuba.ac.jp> <45CA0218.7020000@cisco.com> <878xf9gq8q.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <45CB95A3.3070201@cisco.com> Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Excellent post Steve, thanks. > > I think we're converging on a solution for Mailman both in the short > term and in the long term. See my previously posted wiki link for my > current thoughts on the matter. I just wanted to add one other thing... > > On Feb 8, 2007, at 12:41 AM, Stephen J. Turnbull wrote: > >> "From == signing domain". Just generalize that to include >> "List-Id == signing domain" in the policy agent software! And >> "Sender == signing domain". > > I definitely agree that "List-ID == signing domain" should be added > for interoperability with mailing lists. I'm not sure about Sender, > only because Mailman's addition of Sender itself is not without some > controversy (mostly over interpretation of RFC 2822 language IIRC). > But there's no doubt that well-behaved mailing lists should include > List-ID, so that makes a natural header to sign. See my discussion in > the wiki page for situations where we might /not/ want to sign List-ID > though. I wouldn't get all hung up about what you're signing "for", per se. The right thing to do for a mailing list signature that, say, adds both ListId and Sender would be to: h=From:ListId:Sender:[all of the other headers like mime stuff, etc] and i=mailing at list.org Note that the i= is the way to assert which address if any you want to take responsibility, which in the mailing list case is the ListID or Sender. It is definitely not harmful to sign things like From too, and you definitely should do that (I believe it's a MUST anyway). The only trickiness is that you shouldn't sign things like Sender or ListID if they are empty and it's acceptable for them to be modified in flight (ie, by a mailing list)... that probably doesn't affect you unless there are signatures where you add ListID but don't add Sender or something like that. > > Michael, since you're a DKIM spec insider, can you please relay this > discussion to that community (if you agree with us of course!). We're > making a good faith effort to do our part, and I'd like to see the > DKIM specs acknowledge the mailing list use case more strongly. I'm not entirely sure what I'm being asked to do -- did you have anything in particular you want me to relay? I remember the part of wanting to have better guidance, but did I miss anything else? I will forward on your wiki entry though... Mike From jwblist3 at olympus.net Thu Feb 8 23:55:37 2007 From: jwblist3 at olympus.net (John W. Baxter) Date: Thu, 08 Feb 2007 14:55:37 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <1D67B8FF-C5B7-4E98-BCAE-33EF7D721EAE@python.org> Message-ID: On 2/8/07 10:27 AM, "Barry Warsaw" wrote: > Me too. Here's my discussion on the topic, including a concrete > proposal for Mailman 2.1.10 and 2.2/3.0. Feel free to comment on the > wiki on in this thread. > > http://wiki.list.org/x/OgM > Looks good to me. " IOW, a valid signed List-ID header should be indication enough that all other signatures were subject to breakage because of mailing list garbling" Probably should be "...all prior signatures..." not "...other...". --- Since the public key for a DKIM signature comes from DNS (as I understand it), an administrative detail is that a mailing list operator has to provide for the key's existence in order for Mailman's attempt to sign to be effective. --John From msapiro at value.net Fri Feb 9 02:13:52 2007 From: msapiro at value.net (Mark Sapiro) Date: Thu, 8 Feb 2007 17:13:52 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <45CB8559.5090508@cisco.com> Message-ID: Michael Thomas wrote: >Mark Sapiro wrote: >> >> If we were to take a different approach with a signature containing l=, >> either the l= includes all the text/plain and at least part of the >> text/html, in which we can't add the footer to the text/plain >> alternative without breaking the signature, or the l= includes none of >> the text/html part in which case the signature is not very good at >> verifying the validity of the text/html part. This further assumes we >> even know how to add a footer to a text/html part. >> >> >Are you still speaking of multipart/alernative? Yes, I am. >Right now what we do >for, say, >text/html is not sign the trailing and final --. This >allows lists >to insert their trailers as they normally do in the mime/html body. This assumes that something appended to an html body will render in a nice way. Maybe it will in most cases, but there's no guarantee. >Similar for >text/plain too. For us at least (and it may be that we're just have a >lot of html hating >geeks), this seems to do the trick pretty well. I see some breakage from >multipart/ >alternative, but not _that_ much. I'm not sure how typical your Mailman environment is. For regular, non-geek lists, I think that most posts from Y! will be multipart/alternative, and I think that will be a problem if the MLM touches the message body in almost any way. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From mat at cisco.com Fri Feb 9 03:43:12 2007 From: mat at cisco.com (Michael Thomas) Date: Thu, 08 Feb 2007 18:43:12 -0800 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: References: Message-ID: <45CBDFC0.5050905@cisco.com> Mark Sapiro wrote: > Michael Thomas wrote: > > >> Similar for >> text/plain too. For us at least (and it may be that we're just have a >> lot of html hating >> geeks), this seems to do the trick pretty well. I see some breakage from >> multipart/ >> alternative, but not _that_ much. >> > > > I'm not sure how typical your Mailman environment is. For regular, > non-geek lists, I think that most posts from Y! will be > multipart/alternative, and I think that will be a problem if the MLM > touches the message body in almost any way. > > You're possibly right. Is there any way to know though? Some stats around this would go a long way to helping inform this... Mike From stephen at xemacs.org Fri Feb 9 05:29:51 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 09 Feb 2007 13:29:51 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <1D67B8FF-C5B7-4E98-BCAE-33EF7D721EAE@python.org> References: <1D67B8FF-C5B7-4E98-BCAE-33EF7D721EAE@python.org> Message-ID: <87veickl6o.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > Me too. Here's my discussion on the topic, including a concrete > proposal for Mailman 2.1.10 and 2.2/3.0. Feel free to comment on the > wiki on in this thread. I'll try to post to the wiki later (I'm not a member yet and I'm suffering mail delays---I expect I'll need to do an email confirm), but assuming "on" should be "or" I'll reply here first. In "Problems with Mailman signing headers" you write that a Mailman signature could potentially "legitimize otherwise illegitimate message". Technically, this is confusing language. "Legitimate" is a local policy decision and will surely vary from site to site. I would prefer that this kind of idea be expressed as "result in recipients accepting spam and undesired messages that they otherwise would reject." You suggest as a solution not signing gated messages. This has its own problems, specifically that (1) the list is deliberately not signing some of its output, which will cost the host site the advantages of the claim that all of its mail is signed, and (2) many sites that know that this list signs, and will drop all unsigned posts. I think a plausible alternative strategy would be to sign Usenet-gated posts with a different key. I don't think this is generally going to be more than minor extra effort, as any decent signing agent will be prepared to deal with selectors. (I suppose this is mandated by DKIM, but I haven't checked.) In "Proposal", the language for 2.1.9 is excellent. For 2.2/3.0, I think that all RFC 2369 headers must be signed to prevent phishing attacks. Sender should be signed for Mailman's own potential benefit, I think, and of course DKIM itself mandates signing From. The discussion of the DKIM status header should note that DKIM does not even suggest a format for these headers, so they will be implementation-dependent. I think a call for Mailman advocates to participate in specification of a standard header, as well as to help with implementations' headers in the meantime, may be warranted. Regarding the call for verifiers to "look favorably on" verified list signatures, I'm of two minds. Technically, I think it's a bug in the DKIM spec that it doesn't thoroughly distinguish between verifiers and policy agents (not to mention that it fails to mention the role of policy agents for the signing decision at all!) So I don't like language that suggests that verifiers make policy decisions. There's planty of precedent for it in the latest draft, though. :-( I think that language to the effect that verifiers MAY make extra effort to find a valid list signature in the case where broken signatures are found is definitely appropriate, though. As Michael has repeatedly emphasized, very little is known about the pragmatics of re-signing. Nonetheless, I think that a better way to try to accomplish this is to have a Mailman implementation that wants to take responsibility for broken signatures is to sign those signatures. (Remember, it can also create another signature without signing them. Then the signature signature might fail due to an intermediary reordering DKIM signatures, but the signature of the mail itself would succeed.) Then the BCP for policy agents could say that the presence of a broken signature which is nonetheless verified by a valid signature SHOULD be considered an indication that the verified signer verified the broken signatures on the way in, and that the verified signer then broke them. (This doesn't mean that a policy agent should trust the broken signature. That depends on how much it trusts the verified signer.) I think your current language is a non-starter without signing the signatures. It's an obvious veector for reply attacks. From joe at skyrush.com Fri Feb 9 05:53:57 2007 From: joe at skyrush.com (Joe Peterson) Date: Thu, 08 Feb 2007 21:53:57 -0700 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <1D67B8FF-C5B7-4E98-BCAE-33EF7D721EAE@python.org> References: <1D67B8FF-C5B7-4E98-BCAE-33EF7D721EAE@python.org> Message-ID: <45CBFE65.9060204@skyrush.com> Barry, Nice document. I still feel like I do not know enough about the ramifications of stripping or not stripping the DKIM signature to be sure of the right default, and I still think we could use some more information and understanding of all of the factors. However, Your proposed default of not stripping the signatures seems reasonable, since at least it preserves the forensic information. At least Mailman sites will have the opportunity to adjust this should we find that one way or the other is clearly correct. Thanks for putting the effort into studying this. Clearly, these (DKIM-like) technologies are not yet mature and there is a lot to consider (email and all of its possible interactions are quite complex), and I hope that the interaction (what has happened and what is to come) between the Mailman developers and the DKIM developers can help to make it all really workable! To make DKIM, or something like it, widely accepted as a standard, it clearly has to be able to handle mailing lists, and ideally, with good integration and good specs, it will be able to do so elegantly and deterministically (and not just "99%"). -Joe Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Feb 7, 2007, at 8:20 PM, Mark Sapiro wrote: > >> John W. Baxter wrote: >> >>> On 2/7/07 8:46 AM, "Barry Warsaw" wrote: >>> >>>> Should we strip DKIM by default or not? >>> Not strip by default. >>> >>> Even though that changes the default vs the most recent Mailman, >>> it leaves >>> the default alone for everyone who jumps to 2.1.10 from earlier >>> versions. >> I think I am swayed by the arguments in this thread to favor Not Strip >> as the default, and I agree with John WRT its not being a behavior >> change for many. > > Me too. Here's my discussion on the topic, including a concrete > proposal for Mailman 2.1.10 and 2.2/3.0. Feel free to comment on the > wiki on in this thread. > > http://wiki.list.org/x/OgM > > - -Barry > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (Darwin) > > iQCVAwUBRctro3EjvBPtnXfVAQIHMAP/X4kZL4llpLMLf0rtePsf15092VsF8Old > AMZmEvkJ/MtFT1mTm+cFjWg6i4/wUHfP2LIBr8AwNcO8MIUHUbjOB7fLCn41v93n > FIKLIlFp6liFqjv3167Mz1SRRnb5r5KAReyCoyRww+ogo/AgVn8HmekoG74DOwGp > v/SJuD1YcPQ= > =CuhH > -----END PGP SIGNATURE----- > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/joe%40skyrush.com > > Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp > From stephen at xemacs.org Fri Feb 9 12:12:44 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 09 Feb 2007 20:12:44 +0900 Subject: [Mailman-Developers] dkim-signature headers In-Reply-To: <87veickl6o.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1D67B8FF-C5B7-4E98-BCAE-33EF7D721EAE@python.org> <87veickl6o.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87tzxvlh3n.fsf@uwakimon.sk.tsukuba.ac.jp> Stephen J. Turnbull writes: > Barry Warsaw writes: > > > Me too. Here's my discussion on the topic, including a concrete > > proposal for Mailman 2.1.10 and 2.2/3.0. Feel free to comment on the > > wiki on in this thread. > > I'll try to post to the wiki later (I'm not a member yet and I'm > suffering mail delays---I expect I'll need to do an email confirm), OK, I've done this. No email confirm needed, so I could have just done it right then. Ah well. The version on the wiki is lightly edited, specifically I fixed a typo of 2.1.9 for 2.1.10. From msapiro at value.net Sat Feb 10 18:06:32 2007 From: msapiro at value.net (Mark Sapiro) Date: Sat, 10 Feb 2007 09:06:32 -0800 Subject: [Mailman-Developers] Problems if shunting fails Message-ID: There is a bug report at . The basic issue here is there is an extremely large message in the 'in' queue. In followup emails with the submitter, he said "The file was nearly half a terabyte in size". Not surprisingly, IncomingRunner threw a MemoryError when SpamDetect tried to flatten() the message, but then Runner tried to shunt the message and Switchboard.enqueue() threw another MemoryError in the attempt to pickle the new shunt queue entry. The second MemoryError is uncaught in Runner._oneloop, so it causes the main loop in Runner.run to exit. Pre-Mailman 2.1.9, the master just restarts IncomingRunner - the message is lost, but everything else is OK. Because of the changes in 2.1.9 to prevent message loss in case of disaster, there is now a .bak file left in the 'in' queue. When IncomingRunner restarts, it recovers the .bak file and the whole scenario repeats until the master reaches MAX_RESTARTS on IncomingRunner and we are left with no IncomingRunner and the .bak file still in the 'in' queue. In order to fix this, I suggest we protect the shunt enqueue in a try. I have worked up a patch for this which is attached. This patch also adds a 'preserve' argument to Switchboard.finish such that if it is called with preserve=True, instead of removing the .bak file, it just renames it with a .psv extension. These changes ensure that IncomingRunner doesn't exit, and no .bak file is left to cause a subsequent problem while still preserving the original queue entry for further analysis if possible. I would like some feedback on whether or not this is the right approach. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.txt Url: http://mail.python.org/pipermail/mailman-developers/attachments/20070210/a5da4088/attachment.txt From cmpalmer at metalab.unc.edu Sun Feb 11 20:51:30 2007 From: cmpalmer at metalab.unc.edu (Cristóbal Palmer) Date: Sun, 11 Feb 2007 14:51:30 -0500 Subject: [Mailman-Developers] [ 1657458 ] feature request: batch deletion of subscription requests Message-ID: <20070211195130.GA32037@garp.metalab.unc.edu> Per http://www.list.org/bugs.html I am sending an email to this list about the feature request I reported at: https://sourceforge.net/tracker/index.php?func=detail&aid=1657458&group_id=103&atid=350103 It basically says that it would be nice to have another tickbox in the admindb page that lets you "delete all pending *subscription* *requests* marked /defer/" and not just a tickbox that does that for *messages* since we're getting a fair amount of spam in the form of subscription requests to lists that we manage. Thanks for your attention, -- Crist?bal M. Palmer ibiblio.org system administrator From peter at theneb.co.uk Fri Feb 9 15:37:43 2007 From: peter at theneb.co.uk (Peter Brooks) Date: Fri, 9 Feb 2007 14:37:43 +0000 Subject: [Mailman-Developers] Feature Suggestion: Digest Reply handling Message-ID: Anyone please correct me if this feature already exists, however I'm wondering if it's already been developed so that if a user reply's to a digest in their box then mailman will extract the relevant post they are replying to and also warn the user. Obviously there are a fair few metric with this, such as which post are they replying to. So is this already implemented somewhere or should I develop it? Regards From msapiro at value.net Mon Feb 12 06:09:13 2007 From: msapiro at value.net (Mark Sapiro) Date: Sun, 11 Feb 2007 21:09:13 -0800 Subject: [Mailman-Developers] Feature Suggestion: Digest Reply handling In-Reply-To: Message-ID: Peter Brooks wrote: >Anyone please correct me if this feature already exists, however I'm >wondering if it's already been developed so that if a user reply's to a >digest in their box then mailman will extract the relevant post they are >replying to and also warn the user. >Obviously there are a fair few metric with this, such as which post are they >replying to. I probably lack imagination, but if someone, as people do on lists I manage, replies to a digest, doesn't change the subject, top posts and quotes the entire digest, I don't see how you can figure out which individual message is being replied to. >So is this already implemented somewhere or should I develop it? I am unaware of any implementation of anything like this feature. I suggest, it might be easier to default everyone to the MIME digest and train them all to use MUAs that explode the digest into individual messages they can reply to individually. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From mobiledreamers at gmail.com Tue Feb 13 06:39:43 2007 From: mobiledreamers at gmail.com (mobiledreamers at gmail.com) Date: Mon, 12 Feb 2007 21:39:43 -0800 Subject: [Mailman-Developers] Reg mailman to yahoo groups kind of interface Message-ID: Hello I love and use python a lot. And I m trying to make mailman, into a yahoo groups kind of interface, In that I want to find out if any of you guys tried to do this and integrate it into another webservice? Does mailman uses coroutines to send email, because a coroutine based email gateway can support much more email messages perday than a thread based one Mark From lists at mschuette.name Wed Feb 14 18:04:13 2007 From: lists at mschuette.name (=?ISO-8859-15?Q?Martin_Sch=FCtte?=) Date: Wed, 14 Feb 2007 18:04:13 +0100 Subject: [Mailman-Developers] Show real names in roster? Message-ID: <45D3410D.8060904@mschuette.name> Hello, is there a reason why the roster page does not show full names? Obvious concern is the privacy of the list members. But on the other hand: - the roster has its own access control, - the mail address itself is usually more valuable and can be used to get the real name. - entering the real name on subscription is optional. Thus IMHO showing real names does not disclose significant information, but makes things more convenient for legitimate users. I have attached a three-line patch against Mailman/HTMLFormatter.py -- Martin Sch?tte -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: HTMLFormatter.py.diff Url: http://mail.python.org/pipermail/mailman-developers/attachments/20070214/012a1a22/attachment.diff From msapiro at value.net Fri Feb 16 03:00:58 2007 From: msapiro at value.net (Mark Sapiro) Date: Thu, 15 Feb 2007 18:00:58 -0800 Subject: [Mailman-Developers] Reg mailman to yahoo groups kind of interface In-Reply-To: Message-ID: mobiledreamers at gmail.com wrote: > >I love and use python a lot. And I m trying to make mailman, into a yahoo >groups kind of interface, > >In that I want to find out if any of you guys tried to do this and integrate >it into another webservice? I don't know the answer to this. >Does mailman uses coroutines to send email, because a coroutine based email >gateway can support much more email messages perday than a thread based one Mailman doesn't use coroutines per se, although I suppose you could think of Mailman's qrunner processes as generalized coroutines. Mailman also doesn't use threads. Mailman's processing consists of a set of queues and qrunner processes. Each qrunner process is a single thread. The basic flow, ignoring everything but posts for the moment, is the incoming MTA creates a pipe to a wrapper process which invokes the 'post' script to drop the message in the 'in' queue. IncomingRunner picks up the message from the 'in' queue processes it through a pipline of handler modules which check for holds, etc., build the recipient list and queue the message with recipient list in the 'out' queue. OutgoingRunner picks up the message from the 'out' queue and engages in an SMTP dialog with the outgoing MTA to send the message to the recipients. Depending on personalization and VERP options, there may be one SMTP transaction per recipient, or a few SMTP transactions with many recipients each. OutgoingRunner processes one message in its entirety before returning to the 'out' queue for the next message. Parallelism can be increased by 'slicing' any queue into 2^n slices based on a message hash and running 2^n runners for that queue, each processing one slice. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From msapiro at value.net Fri Feb 23 00:24:15 2007 From: msapiro at value.net (Mark Sapiro) Date: Thu, 22 Feb 2007 15:24:15 -0800 Subject: [Mailman-Developers] Problems if shunting fails In-Reply-To: Message-ID: Mark Sapiro wrote: > >Because of the changes in 2.1.9 to prevent message loss in case of >disaster, there is now a .bak file left in the 'in' queue. When >IncomingRunner restarts, it recovers the .bak file and the whole >scenario repeats until the master reaches MAX_RESTARTS on >IncomingRunner and we are left with no IncomingRunner and the .bak >file still in the 'in' queue. > >In order to fix this, I suggest we protect the shunt enqueue in a try. >I have worked up a patch for this which is attached. This patch also >adds a 'preserve' argument to Switchboard.finish such that if it is >called with preserve=True, instead of removing the .bak file, it just >renames it with a .psv extension. These changes ensure that >IncomingRunner doesn't exit, and no .bak file is left to cause a >subsequent problem while still preserving the original queue entry for >further analysis if possible. > >I would like some feedback on whether or not this is the right approach. Since posting the above a couple of weeks ago, Another situation has come up on mailman-users with a different scenario. The thread is at . In this case, the problem was an unparseable message, and part of the issue was due to the user not having email 2.5.8 installed in pythonlib. Correcting this avoided the problem in her case, but there is still an underlying issue here in that the only exception that we catch is email.Errors.MessageParseError, and at least in this case, email.message_from_string threw a ValueError. The other part of this, is we used to just give up on this message when the exception occurred as we didn't have anything else we could do. Starting with Mailman 2.1.9, we still have the .bak queue entry which can be 'preserved' as above. Thus, I have modified the previous suggested patch, and a new patch is attached which catches any exception from switchboard.dequeue() and logs and preserves the queue entry. Thus, with this patch, we have two different ways in which a queue entry can be 'preserved' for analysis. One is when dequeue() throws an exception and the other is when the attempt to enqueue() in the shunt queue throws an exception. The question I would like to discuss, is what is the best way to preserve the queue entry for analysis. The patch just renames the .bak file to .psv and leaves it in the original queue. This could potentially over time accumulate a lot of .psv files in the 'in' or other queues and impact processing. We can't shunt the entry in the normal way because in some cases at least, shunting has already thrown an exception. I can think of three things to do. 1) Just rename the entry .psv and leave it in the original queue. 2) Rename it .psv and attempt to move it to the shunt queue. 3) Rename it .psv and attempt to move it to the bad queue. I would like to get other people's thoughts on this. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From msapiro at value.net Fri Feb 23 00:35:28 2007 From: msapiro at value.net (Mark Sapiro) Date: Thu, 22 Feb 2007 15:35:28 -0800 Subject: [Mailman-Developers] Problems if shunting fails In-Reply-To: Message-ID: Mark Sapiro wrote: >Thus, I have modified the previous >suggested patch, and a new patch is attached which catches any >exception from switchboard.dequeue() and logs and preserves the queue >entry. Ooops! It's attached this time. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Queueing.patch.txt Url: http://mail.python.org/pipermail/mailman-developers/attachments/20070222/0165e9db/attachment.txt From barry at python.org Mon Feb 26 05:58:10 2007 From: barry at python.org (Barry Warsaw) Date: Sun, 25 Feb 2007 22:58:10 -0600 Subject: [Mailman-Developers] Problems if shunting fails In-Reply-To: References: Message-ID: <8BB934AD-B8FF-46E9-BF52-F20B3C9135C8@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Mark, I'm sorry I haven't been able to respond to this thread before now, but I've been traveling and PyCon'ing and either haven't had time or have had spotty access to email. On Feb 22, 2007, at 5:24 PM, Mark Sapiro wrote: > The question I would like to discuss, is what is the best way to > preserve the queue entry for analysis. The patch just renames the .bak > file to .psv and leaves it in the original queue. This could > potentially over time accumulate a lot of .psv files in the 'in' or > other queues and impact processing. > > We can't shunt the entry in the normal way because in some cases at > least, shunting has already thrown an exception. I can think of three > things to do. > > 1) Just rename the entry .psv and leave it in the original queue. > 2) Rename it .psv and attempt to move it to the shunt queue. > 3) Rename it .psv and attempt to move it to the bad queue. I'm pretty sure we don't want to leave the .psv file in the original queue. I agree that over time, we'll just end up filling up the queue directory with files we'll never process, wasting time listing them, or worse. A major reason for having the shunt queue in the first place was to not clutter up our processing queues with messages we couldn't do anything about. Of course, culling our shunt queue is another issues. :/ OTOH, if we're going to move the offending message to the shunt queue, then there's not much point in keeping the .psv extension. We pretty much always know that if the file is in shunt, it's bad, so maybe we just rename the offender to shunt/blah.txt (assuming it's an unparsed text file). The other thing to consider is adding a configuration variable that let's us limit the size of the files we'll handle. I can't imagine any scenario under which we'd want to (let alone be able to) handle a message of a half terabyte. Heck, you have to have a pretty misconfigured mail system to even allow such a message to get to Mailman, IMO. I think Postfix for example has a 10MB default size limit, and even cranking that up by a factor of 10 should allow most legitimate mail to go through. So, if we added a size limit configuration variable, we'd have to stat the file (os.path.getsize()) and just os.rename() the file to shunt (with a log message) when we see a file over that size limit. What do you think? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBReJo43EjvBPtnXfVAQKw8QP/Qm2twCuIkyQV4zXt1+Ei9PKoBpdBSiK9 G/LgwlUz8OCBRkonb+m4NTyMmZJTQUjqcLICvgFzZ9d+dxf5ekVA8KoXM1iJjjnE TQqS88CAgUYcWixkIfS5H1l0lVSKSGsUn8eUAXwmoVZVn3NaqCzhTo1wGV56KATc VesBGYhpGkQ= =juvu -----END PGP SIGNATURE----- From msapiro at value.net Mon Feb 26 22:40:51 2007 From: msapiro at value.net (Mark Sapiro) Date: Mon, 26 Feb 2007 13:40:51 -0800 Subject: [Mailman-Developers] Problems if shunting fails In-Reply-To: <8BB934AD-B8FF-46E9-BF52-F20B3C9135C8@python.org> Message-ID: Barry Warsaw wrote: > >I'm sorry I haven't been able to respond to this thread before now, >but I've been traveling and PyCon'ing and either haven't had time or >have had spotty access to email. That's OK. I almost went to PyCon myself this year, but didn't make it. Maybe next time. >I'm pretty sure we don't want to leave the .psv file in the original >queue. I agree that over time, we'll just end up filling up the >queue directory with files we'll never process, wasting time listing >them, or worse. A major reason for having the shunt queue in the >first place was to not clutter up our processing queues with messages >we couldn't do anything about. Of course, culling our shunt queue is >another issues. :/ I agree that it is not a good thing to leave these in the original queue. We should somehow move them elsewhere. The only reason for even thinking about leaving them in the original queue is that a rename, changing the extension only is almost guaranteed not to fail whereas a rename that changes the path could run into a permissions problem or a non-existant directory. I don't think these are significant though as the non-existant directory can be created, and permissions aren't likely to be an issue if Mailman is actually running. >OTOH, if we're going to move the offending message to the shunt >queue, then there's not much point in keeping the .psv extension. We >pretty much always know that if the file is in shunt, it's bad, so >maybe we just rename the offender to shunt/blah.txt (assuming it's an >unparsed text file). I'm not so sure about this one. The file may not be unparsed text, and we may not have the metadata to find out. In any case, we don't want to put a .bak file in the shunt queue because unshunt will 'recover' it. I don't think we want that. >The other thing to consider is adding a configuration variable that >let's us limit the size of the files we'll handle. I can't imagine >any scenario under which we'd want to (let alone be able to) handle a >message of a half terabyte. Heck, you have to have a pretty >misconfigured mail system to even allow such a message to get to >Mailman, IMO. I think Postfix for example has a 10MB default size >limit, and even cranking that up by a factor of 10 should allow most >legitimate mail to go through. > >So, if we added a size limit configuration variable, we'd have to >stat the file (os.path.getsize()) and just os.rename() the file to >shunt (with a log message) when we see a file over that size limit. I think the size limit is a good idea. I think it would have avoided the MemoryError that started me off on this in the first place. I do think, we also need to consider the possible exceptions from dequeue(). Now that we have a .bak file in the case of a MessageParseError exception, I think it is good to try to put it somewhere where it can be examined if desired. Also, we had a case of ValueError being thrown by message_from_string. Of course, the ValueError was the result of an email bug that's been fixed, but it may still be possible for message_from_string to throw exceptions other than MessageParseError, and I think we need to protect against that. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan