From barry at list.org Wed Oct 3 04:47:35 2007 From: barry at list.org (Barry Warsaw) Date: Tue, 2 Oct 2007 22:47:35 -0400 Subject: [Mailman-Developers] Improving the archives In-Reply-To: <46B94EDE.6070901@Newfield.org> References: <8BA8AA8B-2575-4794-AEB5-CF4CFAE99CE6@zone12.com> <877ioqys3k.fsf@athene.jamux.com> <2009EA3C-9E11-4B2A-BF57-A62C0EB11870@python.org> <46A84F41.4020003@Newfield.org> <46B94EDE.6070901@Newfield.org> Message-ID: <9E7B1418-2E72-4B34-B76F-31E87AF1F5C9@list.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Aug 8, 2007, at 1:04 AM, Dale Newfield wrote: > Jeff Breidenbach wrote: >> 5.85 million messages > >> That's 0.03% if you count all the messages. It is 0.008% if you >> discard the top three offenders, all of which I have contacted. > > I'd say that's a strong argument for just using the Message-ID and > simplifying this tremendously... > > ...Barry, do you disagree? No, I'm convinced. Apologies for taking so long to respond. The code in the Mailman 3.0 branch has been updated to use only the Message-ID. I still think the base32-encoded sha1 hash is a good user-friendlier option but of course and that archivers should accept either. One question: should the angle brackets on the Message-ID be part of the hash or not? I think they should, or IOW, the entire value of the Message-ID header is taken as the hash, though they should be stripped off if using the Message-ID in any kind of archive query. I'm open to suggestions though... comments? > (It can still be a base32 encoded SHA hash it to make it less user > hostile.) > http://wiki.list.org/display/DEV/Stable+URLs The wiki is down at the moment (I have a issue opened on the support tracker about that). When it comes up, I'll update the page. Thanks everyone for a very good thread, and especially for Jeff for doing the analysis on real data. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFHAwLI2YZpQepbvXERArC8AJ9xJAtqHQPwipUnZuMOvkQ2yxWa0QCbBf+D KnPkuOJEFTZD38BfupCLvk0= =/kr1 -----END PGP SIGNATURE----- From barry at list.org Wed Oct 3 05:07:37 2007 From: barry at list.org (Barry Warsaw) Date: Tue, 2 Oct 2007 23:07:37 -0400 Subject: [Mailman-Developers] The Approved: header in MM3 Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Those of you who have been watching the commit messages can see I've been making some good progress. I'm actually hoping to have a Mailman 3.0 alpha some time RSN which will almost allow you to run the system from the command line, but without a web u/i. So one of the things I'm looking at is the MM2.1 concept of an Approved header. If a message comes into a list with an Approved header (or an Approved line at the start of the message body), and that header has a password that matches the list admin or moderator password, the message is pre-approved and short-circuits the posting tests. The concept doesn't translate well in a Mailman 3 world where there is no shared admin or moderator password. Web access will be control via roles and protected by user authentication much like any modern web application. So the question is, what do we do about the Approved header? 1. We can drop the concept altogether. This means there'd be no way to post a message as coming from an approved source, with a bypass of the posting filters. Maybe because few people have MUAs that support adding custom headers, this feature just isn't used much in the real world these days. You'd still have the moderation bit for announce- only lists though. 2. Replace the concept with some other email authentication mechanism, e.g. something more secure like a signature check. The problem with this is that I still don't think message signing is common practice outside our small community of geeks. 3. Allow an owner or moderator to use their own password in the Approved header. I'm not crazy about this because it has to be sent in the clear and if (when?) it gets compromised, their account is compromised, and this includes their administration of the mailing list. 4. Add a new shared password just for this purpose. You'd still have to communicate it to all your moderators, probably via the web page, but at least this password wouldn't have any other purpose so if (when?) it gets compromised, the only asset it protects is approved postings. Bad yes, if a spammer gets it, but easily changed and hopefully fairly limited in the damage it can do. 5. Your suggestion. Comments? I think my preference would be for #1 with future support for #2 and just accepting the fact that message signatures are for power users. Maybe that set is pretty close to the set of people currently using Approved anyway. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFHAwd62YZpQepbvXERAmMiAKCm3EyxA1CWxWyz4zWkzNwIDpCNKQCbBSXz hGqwpKEGmUScNjov68TUdgs= =gUiT -----END PGP SIGNATURE----- From jeff at jab.org Wed Oct 3 08:48:59 2007 From: jeff at jab.org (Jeff Breidenbach) Date: Tue, 2 Oct 2007 23:48:59 -0700 Subject: [Mailman-Developers] Improving the archives In-Reply-To: <9E7B1418-2E72-4B34-B76F-31E87AF1F5C9@list.org> References: <8BA8AA8B-2575-4794-AEB5-CF4CFAE99CE6@zone12.com> <2009EA3C-9E11-4B2A-BF57-A62C0EB11870@python.org> <46A84F41.4020003@Newfield.org> <46B94EDE.6070901@Newfield.org> <9E7B1418-2E72-4B34-B76F-31E87AF1F5C9@list.org> Message-ID: Question: what about crossposted messages? Let's say a message gets sent to a list called mailman-developers with a CC to a list called pet-bunnies. Hypothetically, of course. Presumably, the person who got the message from pet-bunnies should probably end up at the pet-bunnies archive, where the message can be viewed in proper context; right before the processed carrots flamewar and after the manifesto on proper hopping technique. To make that work, I think we need some way to - at least optionally - allow one or more of the RFC 2369 headers to influence the archival URL. Reading the wiki, I guess that's where List-Archive comes into play? My other question is about the angle brackets. Barry, why are you inclined to include them in calculations? It's kind of arbitrary, but quoting RFC 2822, end of section 3.6.4: Semantically, the angle bracket characters are not part of the msg-id; the msg-id is what is contained between the two angle bracket characters. From iane at sussex.ac.uk Wed Oct 3 10:55:28 2007 From: iane at sussex.ac.uk (Ian Eiloart) Date: Wed, 03 Oct 2007 09:55:28 +0100 Subject: [Mailman-Developers] The Approved: header in MM3 In-Reply-To: References: Message-ID: --On 2 October 2007 23:07:37 -0400 Barry Warsaw wrote: > > 1. We can drop the concept altogether. This means there'd be no way > to post a message as coming from an approved source, with a bypass of > the posting filters. Maybe because few people have MUAs that support > adding custom headers, this feature just isn't used much in the real > world these days. You'd still have the moderation bit for announce- > only lists though. Sounds reasonable to me. I don't use this feature, and I don't think we've documented it for our users. I don't even recall being aware of it before. > 2. Replace the concept with some other email authentication > mechanism, e.g. something more secure like a signature check. The > problem with this is that I still don't think message signing is > common practice outside our small community of geeks. No, but it could be useful for some. I doubt that this is urgent though. > 3. Allow an owner or moderator to use their own password in the > Approved header. I'm not crazy about this because it has to be sent > in the clear and if (when?) it gets compromised, their account is > compromised, and this includes their administration of the mailing list. No, no, no. Or, at least let me disable it for my site. We're likely to want local people to authenticate with passwords that are shared with services other than Mailman. I think this proposal would be very dangerous in any corporate or educational site. > 4. Add a new shared password just for this purpose. You'd still have > to communicate it to all your moderators, probably via the web page, > but at least this password wouldn't have any other purpose so if > (when?) it gets compromised, the only asset it protects is approved > postings. Bad yes, if a spammer gets it, but easily changed and > hopefully fairly limited in the damage it can do. Erm, no thanks. We really are looking forward to being able to identify our Mailman admins! > 5. Your suggestion. > > Comments? I think my preference would be for #1 with future support > for #2 and just accepting the fact that message signatures are for > power users. Maybe that set is pretty close to the set of people > currently using Approved anyway. I agree. -- Ian Eiloart IT Services, University of Sussex x3148 From pabs at debian.org Wed Oct 3 11:51:46 2007 From: pabs at debian.org (Paul Wise) Date: Wed, 3 Oct 2007 19:21:46 +0930 Subject: [Mailman-Developers] The Approved: header in MM3 In-Reply-To: References: Message-ID: On 10/3/07, Ian Eiloart wrote: > > 1. We can drop the concept altogether. > > Sounds reasonable to me. I've used it before as a site admin to mail lists saying that the list will be closed for whatever reason (since it supports using the site password to approve stuff). Personally, I think a combination of 2, 3 & 4 - each user can set a GPG/etc key or a password they use for approving messages. Then MM would check the signature and or the Approved psuedo/header against the key/pass of the users who have high enough privileges (site admins, site staff, list admins, list moderators, etc). -- bye, pabs http://wiki.debian.org/PaulWise http://docs.indymedia.org/view/Main/PaulWise From iane at sussex.ac.uk Wed Oct 3 11:53:31 2007 From: iane at sussex.ac.uk (Ian Eiloart) Date: Wed, 03 Oct 2007 10:53:31 +0100 Subject: [Mailman-Developers] Improving the archives In-Reply-To: <9E7B1418-2E72-4B34-B76F-31E87AF1F5C9@list.org> References: <8BA8AA8B-2575-4794-AEB5-CF4CFAE99CE6@zone12.com> <877ioqys3k.fsf@athene.jamux.com> <2009EA3C-9E11-4B2A-BF57-A62C0EB11870@python.org> <46A84F41.4020003@Newfield.org> <46B94EDE.6070901@Newfield.org> <9E7B1418-2E72-4B34-B76F-31E87AF1F5C9@list.org> Message-ID: --On 2 October 2007 22:47:35 -0400 Barry Warsaw wrote: > One question: should the angle brackets on the Message-ID be part of > the hash or not? I think they should, or IOW, the entire value of > the Message-ID header is taken as the hash, though they should be > stripped off if using the Message-ID in any kind of archive query. > I'm open to suggestions though... comments? Mathematically, the two solutions are equivalent for valid headers, aren't they? OK, the hashes will be different, but only in a trivial sense. Technically, I imagine, it's going to be easier to handle bogus headers if you just hash the entire header. For example, what do you do if some piece of crapware gives you a message with a header missing the angle brackets? Or that adds something outside angle brackets? Or that includes a right-angle bracket in the message-id itself? You don't have to think about any of those situations if you either (A) reject the message or (B) encode the entire header. -- Ian Eiloart IT Services, University of Sussex x3148 From ml at ancalagon.inka.de Wed Oct 3 11:21:21 2007 From: ml at ancalagon.inka.de (Thomas Hochstein) Date: Wed, 03 Oct 2007 11:21:21 +0200 Subject: [Mailman-Developers] The Approved: header in MM3 References: Message-ID: Barry Warsaw schrieb: > 4. Add a new shared password just for this purpose. [...] I'd prefer that. > 1. We can drop the concept altogether. This means there'd be no way > to post a message as coming from an approved source, with a bypass of > the posting filters. That would be bad, I think. It's not uncommon to send mails from "on the road"; in that case you want to "send and forget" the mail without having to visit a website to approve it afterwards. And you may want to send pre-approved mails that are automatically generated. > Maybe because few people have MUAs that support > adding custom headers, this feature just isn't used much in the real > world these days. But you can add a Approved-line in the body, too. -thh From carson at taltos.org Thu Oct 4 15:14:52 2007 From: carson at taltos.org (Carson Gaspar) Date: Thu, 04 Oct 2007 09:14:52 -0400 Subject: [Mailman-Developers] The Approved: header in MM3 In-Reply-To: References: Message-ID: <4704E74C.3060703@taltos.org> Thomas Hochstein wrote: > Barry Warsaw schrieb: [2... signed email] >> 4. Add a new shared password just for this purpose. [...] > > I'd prefer that. +1 for ((2 || 4) || (2 && 4)) >> 1. We can drop the concept altogether. This means there'd be no way >> to post a message as coming from an approved source, with a bypass of >> the posting filters. > > That would be bad, I think. It's not uncommon to send mails from "on > the road"; in that case you want to "send and forget" the mail without > having to visit a website to approve it afterwards. And you may want > to send pre-approved mails that are automatically generated. Removing "Approved" makes mailman _require_ a functioning web UI. That's a bad thing. Right now I frequently approve blocked posts (crossposting, size, etc.) via email. If you remove this, I'd have to go to the web page to do so, which takes longer and requires that I be online (instead of working offline on an airplane, for example). I'd actually prefer to sign the message with PGP, but an approve-only password works for me (and is both less work to implement and more likely to be implemented correctly). -- Carson From lawrenqj at gmail.com Thu Oct 4 16:52:07 2007 From: lawrenqj at gmail.com (Lawren Quigley-Jones) Date: Thu, 4 Oct 2007 14:52:07 +0000 Subject: [Mailman-Developers] lowercase in mailman create function Message-ID: I'm sorry if this issue was already addressed in later versions of mailman. I'm running debian etch and the mailman deb: dpkg -l | grep mailman ii mailman 2.1.9-7 Powerful, web-based mailing list manager I wrote a majordomo like interface for mailman which allows users to perform some backwards compatible functions via email, and access some information, like subscribed lists and owned lists via a web interface. When I released the interface to users, one of the first things a user did was to use all caps in the list they were attempting to create. My script verifies that the list doesn't already exist: if Utils.list_exists(listName): which it didn't and then passes it into the mailman Create function. mlist.Create(listName, listOwner, listPasswordHash ) There were no errors in the create process. The problem was that the list was actually a duplicate of an existing list which obviously wasn't in all caps. Since the process by which mailman gets the list configuration seems to be case insensitive, both lists would access the old config, so functionally I had two references to the same list. The solution to the problem was to just .lower() the listName string at the top of the list create function, but it might make sense to lower the string in the Create function as mailman is not case sensitive after that point. Let me know if you'd like me to open a bug for this issue... Thanks, -Lawren Quigley-Jones lawrenqj at gmail.com From msapiro at value.net Fri Oct 5 17:38:13 2007 From: msapiro at value.net (Mark Sapiro) Date: Fri, 5 Oct 2007 08:38:13 -0700 Subject: [Mailman-Developers] lowercase in mailman create function In-Reply-To: Message-ID: Lawren Quigley-Jones wrote: > >I wrote a majordomo like interface for mailman which allows users to perform >some backwards compatible functions via email, and access some information, >like subscribed lists and owned lists via a web interface. > >When I released the interface to users, one of the first things a user did >was to use all caps in the list they were attempting to create. My script >verifies that the list doesn't already exist: >if Utils.list_exists(listName): So does mlist.Create() >which it didn't and then passes it into the mailman Create function. > >mlist.Create(listName, listOwner, listPasswordHash ) > >There were no errors in the create process. The problem was that the list >was actually a duplicate of an existing list which obviously wasn't in all >caps. Since the process by which mailman gets the list configuration seems >to be case insensitive, both lists would access the old config, so >functionally I had two references to the same list. I don't think this is quite correct in an OS with case sensitive path names. I think you should have ended up with the original lists/listname/ directory with it's original contents and a new lists/LISTNAME/ directory with the new list's config.pck and other files. The real problems arise because the web CGIs lower case the list name, and the MTA might not distinguish local parts that differ only in case. Thus, the LISTNAME list is effectively unusable. >The solution to the problem was to just .lower() the listName string at the >top of the list create function, but it might make sense to lower the string >in the Create function as mailman is not case sensitive after that point. Lower casing the listname before calling Create() is what bin/newlist and Mailman/Cgi/create.py do, so I could say that not doing that is a misuse of the Create() method, but I agree that since upper and mixed case names don't really work, the Create() method or possibly Utils.list_exists() should do this. >Let me know if you'd like me to open a bug for this issue... Thanks for the report. I don't think it's necessary to open a bug. I'll fix it without a bug in the tracker, but I would like some feedback if anyone has any opinion on the best place to do it, i.e., the Create() method, Utils.list_exists(), both or somewhere else. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From stephen at xemacs.org Fri Oct 5 20:25:37 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 06 Oct 2007 03:25:37 +0900 Subject: [Mailman-Developers] lowercase in mailman create function In-Reply-To: References: Message-ID: <87sl4ps9jy.fsf@uwakimon.sk.tsukuba.ac.jp> Mark Sapiro writes: > Thanks for the report. I don't think it's necessary to open a bug. I'll > fix it without a bug in the tracker, Up to you, but it is useful to people with lower versions to be able to see that the bug has been reported, and what progress is being made, in the tracker. > but I would like some feedback if > anyone has any opinion on the best place to do it, i.e., the Create() > method, Utils.list_exists(), both or somewhere else. IMO, fix it in Utils.list_exists() and call that from the Create() method. From msapiro at value.net Fri Oct 5 21:51:09 2007 From: msapiro at value.net (Mark Sapiro) Date: Fri, 5 Oct 2007 12:51:09 -0700 Subject: [Mailman-Developers] lowercase in mailman create function In-Reply-To: <87sl4ps9jy.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Stephen J. Turnbull wrote: >Mark Sapiro writes: > > > Thanks for the report. I don't think it's necessary to open a bug. I'll > > fix it without a bug in the tracker, > >Up to you, but it is useful to people with lower versions to be able >to see that the bug has been reported, and what progress is being made, >in the tracker. Good point. > > but I would like some feedback if > > anyone has any opinion on the best place to do it, i.e., the Create() > > method, Utils.list_exists(), both or somewhere else. > >IMO, fix it in Utils.list_exists() and call that from the Create() method. I realized that only fixes the problem of creating a list with an upper/mixed case name where the lower case name already exists. We still need to make sure that we only create lists with lower case names in the first place. The API which is the Create() method needs to enforce this. So, I'm inclined to lower-case the name provided to Create(), but also lower-case the name provided to Utils.list_exists(). -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at python.org Fri Oct 5 23:15:57 2007 From: barry at python.org (Barry Warsaw) Date: Fri, 5 Oct 2007 17:15:57 -0400 Subject: [Mailman-Developers] lowercase in mailman create function In-Reply-To: References: Message-ID: <556CE041-A686-4D6E-985B-455C5EF333EB@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 5, 2007, at 3:51 PM, Mark Sapiro wrote: > We still need to make sure that we only create lists with lower case > names in the first place. The API which is the Create() method needs > to enforce this. So, I'm inclined to lower-case the name provided to > Create(), but also lower-case the name provided to Utils.list_exists > (). I'm actually not convinced this is a bug, but a problem with the client of Create() and list_exists(). The problem is that once you lowercase the list name in those two methods, you might have to start lowercasing them in an API that might get called via script, web, or email. I really don't think we need viral .lower() calls all over the place. It's tempting to lowercase in Create() but I think it's better to lowercase the list names at the edges of the system, in much the same way that we should be converting strings to unicode at the edges of the system. This means that Lawren's email and web extensions should be doing the lowercasing there. Another problem of course is that the constraints for internal API functions like Create and list_exists() isn't really documented anywhere. I'm trying to improve that situation for Mailman 3. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRwapjnEjvBPtnXfVAQJZ4QP/RrFuMZ3FcwzCWFdDdq1xdBVvV6ZUcllB +J/nHS5vqY8Fw36JcbJ+BlmzDbwNcWk6FivcjQehktt2owkYJjEZPfvTXytpjgvr Um7UjX4Vbc+p9u2e3l2J3rQuwTt8HNZmU8yTgX7x/lzgLnvHOGujuqXbAkSv5ILv 3gpbHGaOqbM= =A+Ef -----END PGP SIGNATURE----- From msapiro at value.net Sat Oct 6 01:57:32 2007 From: msapiro at value.net (Mark Sapiro) Date: Fri, 05 Oct 2007 16:57:32 -0700 Subject: [Mailman-Developers] The Approved: header in MM3 In-Reply-To: References: Message-ID: <4706CF6C.2030805@value.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > >So the question is, what do we do about the Approved header? [...] >Comments? I think my preference would be for #1 with future support >for #2 and just accepting the fact that message signatures are for >power users. Maybe that set is pretty close to the set of people >currently using Approved anyway. I suspect it is close for people approving posts via email, but based on questions on the mailman-users list, there are some number of 'non-power users' using the Approved: header/body line for spoof-proof posting to announcement lists. I don't really know what the best mechanism is, but I think there needs to be some way for a user to post to an announcement list in a spoof-proof way without requiring a subsequent visit to the web interface to approve the post. I'm not sure that all the people who need to do this are comfortable with signing posts. - -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFHBs9sVVuXXpU7hpMRAqW3AJ0WsYWFOKu1pQM+CQv4MZBk9QSVuwCeJp0X +yOt0sOeUf70xRSbnq9cB6Y= =SHnn -----END PGP SIGNATURE----- From barry at python.org Sat Oct 6 21:22:23 2007 From: barry at python.org (Barry Warsaw) Date: Sat, 6 Oct 2007 15:22:23 -0400 Subject: [Mailman-Developers] The Approved: header in MM3 In-Reply-To: <4704E74C.3060703@taltos.org> References: <4704E74C.3060703@taltos.org> Message-ID: <170C34B1-1EAB-4EB1-904E-40D132872905@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 4, 2007, at 9:14 AM, Carson Gaspar wrote: > I'd actually prefer to sign the message with PGP, but an approve-only > password works for me (and is both less work to implement and more > likely to be implemented correctly). Thanks for the feedback everyone. I just pushed a change to implement the shared moderator password, which will only be used for that purpose. At some point it probably makes sense to add gpg signature support, but it sounds like this will work well enough for now. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRwfgb3EjvBPtnXfVAQJnJwQAgHBFopt6/5pW6LCLJZd6wjjN+UCumvOw 4L0rdh84RO56oR2eND+6ZqAPNs4IglwE2PWdpXeDH6u1HkKI2U5BfBmN98nf9f9v 7XxAIM6BVu21tSxC4tYOalMaAFGOx3ZByex9Cc15it7NeB4X8SQGkoB4X4W082hp V6hRnhI4gV4= =58u/ -----END PGP SIGNATURE----- From msapiro at value.net Mon Oct 8 00:17:02 2007 From: msapiro at value.net (Mark Sapiro) Date: Sun, 07 Oct 2007 15:17:02 -0700 Subject: [Mailman-Developers] lowercase in mailman create function In-Reply-To: <556CE041-A686-4D6E-985B-455C5EF333EB@python.org> References: <556CE041-A686-4D6E-985B-455C5EF333EB@python.org> Message-ID: <47095ADE.3040909@value.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw wrote: > > I'm actually not convinced this is a bug, but a problem with the client > of Create() and list_exists(). I did say in my original reply in this thread: | Lower casing the listname before calling Create() is what bin/newlist | and Mailman/Cgi/create.py do, so I could say that not doing that is a | misuse of the Create() method ... > The problem is that once you lowercase > the list name in those two methods, you might have to start lowercasing > them in an API that might get called via script, web, or email. I > really don't think we need viral .lower() calls all over the place. > > It's tempting to lowercase in Create() but I think it's better to > lowercase the list names at the edges of the system, in much the same > way that we should be converting strings to unicode at the edges of the > system. This means that Lawren's email and web extensions should be > doing the lowercasing there. I agree, but the problem remains that upper and mixed case internal names don't work because of the many user interfaces that lower case the name before accessing the list. So, what does Create() do when asked to create a list with a non-lower case name? 1) Go ahead and create it. 2) Lower case the name and create that. 3) Raise raise Errors.BadListNameError. Presently it does 1), but I think 2) is better and perhaps 3) is best. - -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFHCVreVVuXXpU7hpMRAjWcAJ0YYudg4Tl6zg6sPrOt9/Lc4/wdzQCeMbq8 ChiA4GBUrkacAIFaG909BN8= =Yw4C -----END PGP SIGNATURE----- From gmessmer at u.washington.edu Tue Oct 9 02:07:36 2007 From: gmessmer at u.washington.edu (Gordon Messmer) Date: Mon, 08 Oct 2007 17:07:36 -0700 Subject: [Mailman-Developers] mta integration Message-ID: <470AC648.80605@u.washington.edu> I'd like to write a filter for the Courier MTA which will run mailman's approval and spam tests before accepting messages to mailing lists. My filter framework is in python, so I think I can import the relevant bits of mailman (I'd like to start with MM2.1 compatibility). Can I get any pointers to the functions that check the sender for approval, and for the spam-specific posting filters? From barry at list.org Tue Oct 9 15:03:11 2007 From: barry at list.org (Barry Warsaw) Date: Tue, 9 Oct 2007 09:03:11 -0400 Subject: [Mailman-Developers] lowercase in mailman create function In-Reply-To: <47095ADE.3040909@value.net> References: <556CE041-A686-4D6E-985B-455C5EF333EB@python.org> <47095ADE.3040909@value.net> Message-ID: <478B7090-EBE4-4BB1-AC20-20D8055B22CF@list.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 7, 2007, at 6:17 PM, Mark Sapiro wrote: > I agree, but the problem remains that upper and mixed case internal > names don't work because of the many user interfaces that lower > case the > name before accessing the list. So, what does Create() do when > asked to > create a list with a non-lower case name? > > 1) Go ahead and create it. > 2) Lower case the name and create that. > 3) Raise raise Errors.BadListNameError. > > Presently it does 1), but I think 2) is better and perhaps 3) is best. Let's do #3, or alternatively, we could add an assert since this is a constraint of Create(). E.g. something like this before the Utils.list_exists() call: assert name == name.lower(), 'List name must be all lower case.' Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFHC3wP2YZpQepbvXERAtToAJ9t1NToyQCtohVl3ZV0PWv2r5OodwCbBn7G BRRVrLmbFfcGb72HlUxpkT8= =OKYE -----END PGP SIGNATURE----- From msapiro at value.net Tue Oct 9 18:55:21 2007 From: msapiro at value.net (Mark Sapiro) Date: Tue, 9 Oct 2007 09:55:21 -0700 Subject: [Mailman-Developers] mta integration In-Reply-To: <470AC648.80605@u.washington.edu> Message-ID: Gordon Messmer wrote: >I'd like to write a filter for the Courier MTA which will run mailman's >approval and spam tests before accepting messages to mailing lists. My >filter framework is in python, so I think I can import the relevant bits >of mailman (I'd like to start with MM2.1 compatibility). Can I get any >pointers to the functions that check the sender for approval, and for >the spam-specific posting filters? Everything is done by the handler modules in Mailman/Handlers. IncomingRunner processes the message by calling the process() function of each module in the GLOBAL_PIPELINE list in turn until the pipeline is exhausted or a handler raises an exception. See the definition of GLOBAL_PIPELINE in Defaults.py and the SpamDetect.py, Approve.py, Moderate.py and Hold.py handlers in particular. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From maximilian.mehnert at googlemail.com Tue Oct 9 12:58:52 2007 From: maximilian.mehnert at googlemail.com (Maximilian Mehnert) Date: Tue, 09 Oct 2007 12:58:52 +0200 Subject: [Mailman-Developers] running 2.1.9 completely from crontab Message-ID: <470B5EEC.10802@googlemail.com> I had a little problem because my provider does not allow running daemons. However, crontab is fine. Since setting up aliases does not work either and I did not want to change .procmailrc all the time, I'm using a catchall user and Maildir delivery. In order to make this work, I changed two things. diff -ur mailman-2.1.9/Mailman/Queue/MaildirRunner.py mailman-2.1.9-mod/Mailman/Queue/MaildirRunner.py --- mailman-2.1.9/Mailman/Queue/MaildirRunner.py 2005-08-27 01:40:17.000000000 +0000 +++ mailman-2.1.9-mod/Mailman/Queue/MaildirRunner.py 2007-10-06 16:08:29.000000000 +0000 @@ -123,7 +123,7 @@ # message was destined for. See verp_bounce() in # BounceRunner.py for why we do things this way. vals = [] - for header in ('delivered-to', 'envelope-to', 'apparently-to'): + for header in ('x-original-to','delivered-to', 'envelope-to', 'apparently-to'): vals.extend(msg.get_all(header, [])) for field in vals: to = parseaddr(field)[1] diff -ur mailman-2.1.9/bin/qrunner mailman-2.1.9-mod/bin/qrunner --- mailman-2.1.9/bin/qrunner 2006-01-19 01:07:40.000000000 +0000 +++ mailman-2.1.9-mod/bin/qrunner 2007-10-09 10:33:11.000000000 +0000 @@ -119,7 +119,8 @@ # Subclass to hack in the setting of the stop flag in _doperiodic() class Once(qrclass): def _doperiodic(self): - self.stop() + if self._oneloop()<1: + self.stop() qrunner = Once(slice, range) else: qrunner = qrclass(slice, range) First I added the x-original-to field since some mails got stuck. The second change in qrunner is the short hack to get all pending stuff done with one run from crontab: qrunner -v -r All -o Would that be something that could be included in future releases maybe as a separate option? Regards, Maximilian From gmessmer at u.washington.edu Thu Oct 18 00:21:08 2007 From: gmessmer at u.washington.edu (Gordon Messmer) Date: Wed, 17 Oct 2007 15:21:08 -0700 Subject: [Mailman-Developers] mta integration In-Reply-To: References: Message-ID: <47168AD4.6040909@u.washington.edu> Mark Sapiro wrote: > Everything is done by the handler modules in Mailman/Handlers. > IncomingRunner processes the message by calling the process() function > of each module in the GLOBAL_PIPELINE list in turn until the pipeline > is exhausted or a handler raises an exception. > > See the definition of GLOBAL_PIPELINE in Defaults.py and the > SpamDetect.py, Approve.py, Moderate.py and Hold.py handlers in > particular. > Thanks. I got a chance to look at this today. Since I only need an indication of whether or not the message will be rejected by mailman, I believe that I can use SpamDetect and Approve directly. However, I'd need to make sure that messages aren't held as a result of the examination. I'm not sure yet whether it'll be less ugly to make a work-alike function from Moderate.process(), leaving out the Hold bits, or perhaps import Moderate, and then replace Hold therein with one that does nothing. Any thoughts before I start? From sebastian at nanofortnight.org Tue Oct 23 03:23:39 2007 From: sebastian at nanofortnight.org (Sebastian Wieseler) Date: Tue, 23 Oct 2007 03:23:39 +0200 Subject: [Mailman-Developers] --with-user and qmail-to-mailman.py Message-ID: <20071023012339.GA6898@nanofortnight.forkbomb.ch> Hello. If I use "--with-user" I got some errors with qmail-to-mailman.py. That's why it has got on line 67: local = re.sub("^mailman-","",local) But "mailman" isn't correct, because of "--with-user". Can you please fix that with something like: local = re.sub("^" + os.environ['USER'] + "-","",local) or even better re.escape(os.environ["USER"]) Lots of greetings Sebastian 'kickino' Wieseler, one of the Savannah admins, -- ,= ,-_-. =. /"\ ((_/)o o(\_)) \ / ASCII Ribbon Campaign `-'(. .)`-' && X against HTML e-mail \_/ / \ From msapiro at value.net Wed Oct 24 06:01:01 2007 From: msapiro at value.net (Mark Sapiro) Date: Tue, 23 Oct 2007 21:01:01 -0700 Subject: [Mailman-Developers] --with-user and qmail-to-mailman.py In-Reply-To: <20071023012339.GA6898@nanofortnight.forkbomb.ch> Message-ID: Sebastian Wieseler wrote: > >But "mailman" isn't correct, because of "--with-user". Can you please >fix that with something like: [...] qmail-to-mailman.py is 'unofficial' contributed software. Please read contrib/README in the distribution. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From cloomis at astro.princeton.edu Tue Oct 30 23:26:28 2007 From: cloomis at astro.princeton.edu (Craig Loomis) Date: Tue, 30 Oct 2007 18:26:28 -0400 Subject: [Mailman-Developers] Improving the archives Message-ID: <0673D829-3875-4CC7-9175-D234F880D874@astro.princeton.edu> Or Re: [Mailman-Developers 10417] Improving the archives I would like to interject and highlight some use cases for stable and predictable IDs. For us, "message IDs" are directly used both by people and ignorant programs. Our mailing lists serve as a permanent and concise record of our discussions, decisions, and operations, and we find it invaluable to be able to refer to individual messages in a simple and memorable way: "message 1210 in the calibration list", say. Other people can then easily jot that info down or directly find the message. Some message IDs even become shorthands for a particular topic or decision. We have also added trac InterWiki templates pointing into our mail archives (as listname:number), which encourages desirable cross-referencing (PRs, wiki pages, and SVN change logs can refer to mail messages, just as wiki pages could always refer to changesets and PRs, etc, etc.) But trac InterWiki templates can only interpolate $1,$2,... arguments into strings, and could not possibly calculate anything based on the _content_ of the messages. Globally unique IDs, hashed IDs, etc., are very appealing from various CS-y and techie points of view, but are simply not memorable to humans or knowable by dumb external programs. I think as much, or more, effort should be put into delivering a straightforwardly useable naming scheme as goes into making an arbitrary message recoverable from anywhere. Basically, "friendly URLs" should be a primary requirement, not an optional afterthought for careless geeks like me to get wrong later.... We long ago added an extremely simple ID handoff between MM 2.1.8 and pipermail, and though imperfect it has served us well. Basically, we hijacked the .post_id member in mailman (otherwise basically unused, and mysteriously a floating point number); CookHeaders stuffed it into a X-Mailman-Sequence-ID header line, and AfterDelivery incremented it. In turn, pipermail uses the header to feed a sequence ID into make_article, and the message is squirreled away as $mailinglist/all/%d.html. There are a few other minor matters (e.g. post_id was added to Decorators, a couple of templates were changed, we lost having 'ls' sort chronologically [did we have to add .last and .prev to the HyperDatabase classes?]), but it really was a minor bit of work. And for stability, as long as the archive files aren't lost, pipermail rebuilds should yield the same URLs even if junk messages have been deleted. [Oh, we did also add a "never rotate" policy to our archives, but that is finesseable. ] As an aside on other discussions, can you get away without using Message-ID or Date? I.e., aren't those just more of those tokens which were standardized back before the Internet got tricky enough to invalidate the standards? Mailing lists serialize incoming messages, and so can generate their own unique and trustworthy IDs. "UUIDs" would work, but if you can trust yourself to generate them, consecutive integers provide minimal, order-preserving, perfect hashing, too! Anyhow, we have found that people will enthusiastically refer by name to individual messages within mail archives if they can. - craig