From lists at psycho.i21k.de Mon Aug 3 20:00:13 2009 From: lists at psycho.i21k.de (Siggy Brentrup) Date: Mon, 3 Aug 2009 20:00:13 +0200 Subject: [Mailman-Developers] Searchable archives for MM Message-ID: <20090803180013.GF15680@keuner.winnegan.fake> Hi list, please let me start by presenting myself. My name is Bernd "Siggy" Brentrup and I'm a longtime *nix user/programmer/admin, from '95 thru '04 I have been a DD, shortly before I vanished from Debian I was part of the 2 and 1/2 men show maintaining Debian's mailman package; during that time I wrote most of the debconf stuff and contributed postfix-to-mailman.py modelled after Bruce Perens's qmail-to-mailman.py. Even then, one of my biggest concerns with mailman was that the archives are not searchable, I just can't recall whether I published this in '04. Now trying to reenter the OSS community I observe that archives on mailman lists still aren't searchable. Barry told me pipermail didn't see much love during the last years and some revamp is required. In contrast to his suggestions (sth involving google search) I'm aiming at a solution akin to Debian's Xapian powered ML search. cf http://lists.debian.org/cgi-bin/search What do you think and is anybody else interested in working on this? Thanks for listening Siggy -- bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From jmhayes at speakeasy.net Fri Aug 7 07:19:30 2009 From: jmhayes at speakeasy.net (Jordan Hayes) Date: Thu, 6 Aug 2009 22:19:30 -0700 Subject: [Mailman-Developers] Searchable archives for MM References: <20090803180013.GF15680@keuner.winnegan.fake> Message-ID: Here's my solution: http://infothecary.org/jordan/mailman.html /jordan From skip at pobox.com Fri Aug 7 13:11:16 2009 From: skip at pobox.com (skip at pobox.com) Date: Fri, 7 Aug 2009 06:11:16 -0500 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <20090803180013.GF15680@keuner.winnegan.fake> References: <20090803180013.GF15680@keuner.winnegan.fake> Message-ID: <19068.3028.441462.232752@montanaro.dyndns.org> Siggy> In contrast to his suggestions (sth involving google search) I'm Siggy> aiming at a solution akin to Debian's Xapian powered ML search. I've never been all that pleased with Xapian search results. I'm one of those lazy people who feels that Google does a better job than I ever could. -- Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/ Getting old sucks, but it beats dying young From adam-mailman at amyl.org.uk Fri Aug 7 16:46:27 2009 From: adam-mailman at amyl.org.uk (Adam McGreggor) Date: Fri, 7 Aug 2009 15:46:27 +0100 Subject: [Mailman-Developers] [Mailman-Users] mailman user password In-Reply-To: References: <4A74A47C.6050402@libertytrek.org> <87eirud1dt.fsf@uwakimon.sk.tsukuba.ac.jp> <4A75F9C0.3020705@libertytrek.org> <87vdl09cwn.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20090807144626.GN30597@amyl.org.uk> *shimmied over to mailman-devs* (from -users) On Fri, Aug 07, 2009 at 10:33:06AM -0400, Barry Warsaw wrote: > It kind of sucks that there are so many other Mailman command line > scripts, which is one reason why I've always put them in a separate > Mailman specific bin directory. With MM3 though I intend to use a > 'subcommand' approach so that there's only one 'mailman' command. > Think things like 'mailman listmembers foo'. I'll probably keep > mailmanctl separate though I haven't decided about that yet. Would it too much to ask for the 'old' (read 'current') scripts/commands to be aliased: at least in the first few releases? e.g., for 'list_members' to link to 'mailman listmembers' Perhaps as an install/configure/compilation option? "Create old-style links?" --old-style-links (maybe that's more for people who package: ISTR a couple of packagers are on list, yes?) (wonders how many other people have scripts that would need re-writing otherwise...) From mark at msapiro.net Fri Aug 7 22:40:51 2009 From: mark at msapiro.net (Mark Sapiro) Date: Fri, 7 Aug 2009 13:40:51 -0700 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <20090803180013.GF15680@keuner.winnegan.fake> Message-ID: Siggy Brentrup wrote: > >In contrast to his suggestions (sth involving google search) I'm >aiming at a solution akin to Debian's Xapian powered ML search. > > cf http://lists.debian.org/cgi-bin/search > >What do you think and is anybody else interested in working on >this? There are patches at that integrate the ht://Dig search engine with Mailman's archives. They work well. The problem with integrating these or some of the other solutions into the source is they require installation of software over which we have no control. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at python.org Fri Aug 7 23:59:45 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 7 Aug 2009 17:59:45 -0400 Subject: [Mailman-Developers] [Mailman-Users] mailman user password In-Reply-To: <20090807144626.GN30597@amyl.org.uk> References: <4A74A47C.6050402@libertytrek.org> <87eirud1dt.fsf@uwakimon.sk.tsukuba.ac.jp> <4A75F9C0.3020705@libertytrek.org> <87vdl09cwn.fsf@uwakimon.sk.tsukuba.ac.jp> <20090807144626.GN30597@amyl.org.uk> Message-ID: On Aug 7, 2009, at 10:46 AM, Adam McGreggor wrote: > *shimmied over to mailman-devs* (from -users) Thanks! > On Fri, Aug 07, 2009 at 10:33:06AM -0400, Barry Warsaw wrote: >> It kind of sucks that there are so many other Mailman command line >> scripts, which is one reason why I've always put them in a separate >> Mailman specific bin directory. With MM3 though I intend to use a >> 'subcommand' approach so that there's only one 'mailman' command. >> Think things like 'mailman listmembers foo'. I'll probably keep >> mailmanctl separate though I haven't decided about that yet. > > Would it too much to ask for the 'old' (read 'current') scripts/ > commands > to be aliased: at least in the first few releases? > > e.g., for 'list_members' to link to 'mailman listmembers' > > Perhaps as an install/configure/compilation option? > "Create old-style links?" > --old-style-links > > (maybe that's more for people who package: ISTR a couple of packagers > are on list, yes?) > > (wonders how many other people have scripts that would need re-writing > otherwise...) There won't be a cmmi recipe for MM3 (no configure; make; make install) at least from upstream, but I'm happy to work with distros to figure out the right way to package Mailman 3. As for aliases, I think it won't be worth it. I also want to standardize and make consistent all the command line options so that the whole suite of commands makes more sense. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From lists at psycho.i21k.de Sat Aug 8 00:16:54 2009 From: lists at psycho.i21k.de (Bernd 'Siggy' Brentrup) Date: Sat, 8 Aug 2009 00:16:54 +0200 Subject: [Mailman-Developers] [Mailman-Users] mailman user password In-Reply-To: References: <4A74A47C.6050402@libertytrek.org> <87eirud1dt.fsf@uwakimon.sk.tsukuba.ac.jp> <4A75F9C0.3020705@libertytrek.org> <87vdl09cwn.fsf@uwakimon.sk.tsukuba.ac.jp> <20090807144626.GN30597@amyl.org.uk> Message-ID: <20090807221654.GA3536@keuner.winnegan.fake> On Fri, Aug 07, 2009 at 17:59 -0400, Barry Warsaw wrote: > There won't be a cmmi recipe for MM3 (no configure; make; make > install) at least from upstream, but I'm happy to work with distros > to figure out the right way to package Mailman 3. I'd suggest /usr/bin/mailman and /usr/sbin/mailmanctl Modules might take advantage of python-support or be installed under /usr/share/mailman/ resp. /usr/lib/mailman/ for arch dependent stuff, queues &c in /var/spool/mailman/, for archives I still have to consult LSB. > As for aliases, I think it won't be worth it. I also want to > standardize and make consistent all the command line options so that > the whole suite of commands makes more sense. Args, looks like I posted from my standard address bsb at p.i.d instead of from my subscription address lists at p.s.d, would you kindly approve those messages. Regards Siggy (going to add yet another hook to .muttrc) -- bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From barry at python.org Sat Aug 8 00:40:35 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 7 Aug 2009 18:40:35 -0400 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <19068.3028.441462.232752@montanaro.dyndns.org> References: <20090803180013.GF15680@keuner.winnegan.fake> <19068.3028.441462.232752@montanaro.dyndns.org> Message-ID: On Aug 7, 2009, at 7:11 AM, skip at pobox.com wrote: > Siggy> In contrast to his suggestions (sth involving google > search) I'm > Siggy> aiming at a solution akin to Debian's Xapian powered ML > search. > > I've never been all that pleased with Xapian search results. I'm > one of > those lazy people who feels that Google does a better job than I > ever could. Unfortunately, that doesn't work for private mailing lists. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Sat Aug 8 01:27:01 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 7 Aug 2009 19:27:01 -0400 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: References: Message-ID: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> On Aug 7, 2009, at 4:40 PM, Mark Sapiro wrote: > The problem with integrating these or some of the other solutions into > the source is they require installation of software over which we have > no control. Agreed, which has always been my argument for including Pipermail in the Mailman distribution. At least it gives people an archiver out of the box. OTOH, it really makes more sense to make Pipermail a separate project and let it live or die on its own. I'm not sure what the right thing to do is, but it sure would help if someone adopted Pipermail . -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Sat Aug 8 01:32:24 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 7 Aug 2009 19:32:24 -0400 Subject: [Mailman-Developers] [Mailman-Users] mailman user password In-Reply-To: <20090807221654.GA3536@keuner.winnegan.fake> References: <4A74A47C.6050402@libertytrek.org> <87eirud1dt.fsf@uwakimon.sk.tsukuba.ac.jp> <4A75F9C0.3020705@libertytrek.org> <87vdl09cwn.fsf@uwakimon.sk.tsukuba.ac.jp> <20090807144626.GN30597@amyl.org.uk> <20090807221654.GA3536@keuner.winnegan.fake> Message-ID: On Aug 7, 2009, at 6:16 PM, Bernd 'Siggy' Brentrup wrote: > On Fri, Aug 07, 2009 at 17:59 -0400, Barry Warsaw wrote: >> There won't be a cmmi recipe for MM3 (no configure; make; make >> install) at least from upstream, but I'm happy to work with distros >> to figure out the right way to package Mailman 3. > > I'd suggest /usr/bin/mailman and /usr/sbin/mailmanctl > Modules might take advantage of python-support or be installed > under /usr/share/mailman/ resp. /usr/lib/mailman/ for arch > dependent stuff, queues &c in /var/spool/mailman/, for archives > I still have to consult LSB. There should be little or no architecture dependent stuff in MM3. All those binaries for "security" are gone. I definitely want to support the LSB for MM3, but I haven't put too much thought into that yet. The command line scripts will be interesting, but less so in MM3 than in MM2. Already I have several REST APIs fleshed out and as soon as you can subscribe to mailing lists via REST I will release the next alpha (hopefully in the next week or two). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From lists at psycho.i21k.de Sat Aug 8 03:03:44 2009 From: lists at psycho.i21k.de (Bernd Siggy Brentrup) Date: Sat, 8 Aug 2009 03:03:44 +0200 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> References: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> Message-ID: <20090808010344.GA23395@keuner.winnegan.fake> On Fri, Aug 07, 2009 at 19:27 -0400, Barry Warsaw wrote: > On Aug 7, 2009, at 4:40 PM, Mark Sapiro wrote: > > >The problem with integrating these or some of the other solutions into > >the source is they require installation of software over which we have > >no control. > > Agreed, which has always been my argument for including Pipermail in > the Mailman distribution. At least it gives people an archiver out > of the box. OTOH, it really makes more sense to make Pipermail a > separate project and let it live or die on its own. I'm not sure > what the right thing to do is, but it sure would help if someone > adopted Pipermail . :) No need for this broad hint. I was thinking about this option too and as soon as the issues with the old account Ubuntu automatically created for me when importing my Debian packages are solved I will create a pipermail project on launchpad and announce it here. Siggy p.s. Please respect my .signature, I'd really like to reenable duplicate elimination. -- Please don't Cc: me when replying, I might not see either copy. bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From barry at python.org Sat Aug 8 04:02:56 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 7 Aug 2009 22:02:56 -0400 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <20090808010344.GA23395@keuner.winnegan.fake> References: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> <20090808010344.GA23395@keuner.winnegan.fake> Message-ID: <93B285C8-91A5-4FD2-8FBA-0BEAA0A1A003@python.org> On Aug 7, 2009, at 9:03 PM, Bernd Siggy Brentrup wrote: > :) No need for this broad hint. Tim Peters would be proud at the power of the ! > I was thinking about this option too and as soon as the issues > with the old account Ubuntu automatically created for me when > importing my Debian packages are solved I will create a pipermail > project on launchpad and announce it here. That would be cool. I wonder if we should make ~mailman-coders the owner of that project? If not, a different but parallel team structure might make sense. Note too that since Pipermail comes from Mailman, it's GNU so we need to get copyright assignments for all Pipermail hackers. I will volunteer to help separate the code, and to work out how best to integrate it for packaging. > p.s. Please respect my .signature, I'd really like to reenable > duplicate elimination. You can do this for Mailman lists, at least when you are explicitly CC'd with an address that's subscribed to the list. You'll get the direct copy and not the list copy though. That's the best Mailman can do (anything more takes a better MUA ;). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From lists at psycho.i21k.de Sat Aug 8 04:49:14 2009 From: lists at psycho.i21k.de (Bernd Siggy Brentrup) Date: Sat, 8 Aug 2009 04:49:14 +0200 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <93B285C8-91A5-4FD2-8FBA-0BEAA0A1A003@python.org> References: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> <20090808010344.GA23395@keuner.winnegan.fake> <93B285C8-91A5-4FD2-8FBA-0BEAA0A1A003@python.org> Message-ID: <20090808024914.GA29033@keuner.winnegan.fake> On Fri, Aug 07, 2009 at 22:02 -0400, Barry Warsaw wrote: > On Aug 7, 2009, at 9:03 PM, Bernd Siggy Brentrup wrote: > > > :) No need for this broad hint. > > Tim Peters would be proud at the power of the ! > > >I was thinking about this option too and as soon as the issues > >with the old account Ubuntu automatically created for me when > >importing my Debian packages are solved I will create a pipermail > >project on launchpad and announce it here. > > That would be cool. I wonder if we should make ~mailman-coders the > owner of that project? No objections from my side if you grant me admin rights. As a surplus we don't have to wait for lp ppl to merge my accounts; there are some restrictions that inhibit merging e.g. creating a PPA, after all I want to be known on lp as ~bsb and not as ~bsb-psycho :-P > If not, a different but parallel team > structure might make sense. Note too that since Pipermail comes > from Mailman, it's GNU so we need to get copyright assignments for > all Pipermail hackers. ack > I will volunteer to help separate the code, and to work out how best > to integrate it for packaging. Thanks for this, did you write down any issues that should be addressed when revamping pipermail? One thing that comes to mind is how to deal with mime types and attachments. I'd also like to see signatures validated. (On an aside, I hate these footers that make my messages not fully signed) > >p.s. Please respect my .signature, I'd really like to reenable > > duplicate elimination. > > You can do this for Mailman lists, at least when you are explicitly > CC'd with an address that's subscribed to the list. You'll get the > direct copy and not the list copy though. That's the best Mailman > can do (anything more takes a better MUA ;). I know that's because I mentioned it, I didn't recall the variable name though. In the first place I don't want to see 2 copies arriving, no MUA involved here. Siggy (birds are greeting the day, time to go to bed :) -- Please don't Cc: me when replying, I might not see either copy. bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From stephen at xemacs.org Sat Aug 8 08:43:17 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 08 Aug 2009 15:43:17 +0900 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <20090808024914.GA29033@keuner.winnegan.fake> References: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> <20090808010344.GA23395@keuner.winnegan.fake> <93B285C8-91A5-4FD2-8FBA-0BEAA0A1A003@python.org> <20090808024914.GA29033@keuner.winnegan.fake> Message-ID: <87skg2ajqy.fsf@uwakimon.sk.tsukuba.ac.jp> Bernd Siggy Brentrup writes: > [Barry Warsaw contributed the comment:] > > You can do this for Mailman lists, at least when you are explicitly > > CC'd with an address that's subscribed to the list. You'll get the > > direct copy and not the list copy though. That's the best Mailman > > can do (anything more takes a better MUA ;). They don't come *better* than mutt, though. At best, "different". > I know that's because I mentioned it, I didn't recall the variable > name though. In the first place I don't want to see 2 copies > arriving, no MUA involved here. I suggest "/etc/init.d/$MTA stop" is the only way to guarantee that two copies of something will never arrive. The next best is to set Reply-To, which might actually have the desired effect most of the time. The next best is to set Mail-Followup-To, which won't have any effect but you will get a lot of sympathy from Dan Bernstein and Emacs/Gnus users. P.S. You owe me $250/hr X 5 seconds / (3600 seconds/hour) = $0.35 for time & expense of fixing up the headers in this message. You pay the wire transfer and currency conversion costs. From lists at psycho.i21k.de Sat Aug 8 09:35:58 2009 From: lists at psycho.i21k.de (Bernd Siggy Brentrup) Date: Sat, 8 Aug 2009 09:35:58 +0200 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <87skg2ajqy.fsf@uwakimon.sk.tsukuba.ac.jp> References: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> <20090808010344.GA23395@keuner.winnegan.fake> <93B285C8-91A5-4FD2-8FBA-0BEAA0A1A003@python.org> <20090808024914.GA29033@keuner.winnegan.fake> <87skg2ajqy.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20090808073558.GD29033@keuner.winnegan.fake> On Sat, Aug 08, 2009 at 15:43 +0900, Stephen J. Turnbull wrote: > Bernd Siggy Brentrup writes: > > [Barry Warsaw contributed the comment:] > > > > You can do this for Mailman lists, at least when you are explicitly > > > CC'd with an address that's subscribed to the list. You'll get the > > > direct copy and not the list copy though. That's the best Mailman > > > can do (anything more takes a better MUA ;). > > They don't come *better* than mutt, though. At best, "different". Oh no, no yamuad (Yet Another MUA Discussion) please. > > I know that's because I mentioned it, I didn't recall the variable > > name though. In the first place I don't want to see 2 copies > > arriving, no MUA involved here. > > I suggest "/etc/init.d/$MTA stop" is the only way to guarantee that > two copies of something will never arrive. The next best is to set > Reply-To, which might actually have the desired effect most of the > time. The next best is to set Mail-Followup-To, which won't have any > effect but you will get a lot of sympathy from Dan Bernstein and > Emacs/Gnus users. I'm subscribed to some MLs that mangle with Reply-To to keep subscribers with broken MUAs happy, mutt does the Mail-Followup-To for me w/o my intervention. Do you know anybody who wants to see djb happy? > P.S. You owe me $250/hr X 5 seconds / (3600 seconds/hour) = $0.35 for > time & expense of fixing up the headers in this message. You pay the > wire transfer and currency conversion costs. You actually read those winnegan.fake (my version of localnet) headers? Nobody does this! As kind of a reward IOU a beer should we ever meet. Regards Siggy -- Please don't Cc: me when replying, I might not see either copy. bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From malveeka at gmail.com Sat Aug 8 10:25:58 2009 From: malveeka at gmail.com (Malveeka Tewari) Date: Sat, 8 Aug 2009 13:55:58 +0530 Subject: [Mailman-Developers] Storm based MemberAdaptor for Mailman Message-ID: <68779e050908080125h7c39da1co283a0c7e16ddd92f@mail.gmail.com> Hi all I am working on writing a storm based member adaptor for mailman so that the mailman membership data can be stored in a database instead of .pck files. The reason for choosing storm is that it provides an abstraction to use any underlying db language- either mysql, postgresql or sqllite. I have created a branch on launchpad for this adaptor and would be great if I can get a code review of the adaptor ,t he schema of the db that I am using for storing memberships data and an opinion of whether such a member adaptor is desired at all. Presently the storm adaptor is written for Postgresql but it will support mysql and sqllite with slight modifications. Please take a look at *Mailman/PgsqlMemberships.py* at * https://code.launchpad.net/~malveeka/+junk/StormMemberAdaptor* under the first revision. The work is under progress and for now, I using to storm to create a database with memberships data alongwith the pickle files. The Mailman/PgsqlMemberships.py subclasses OldStyleMemberships.py and still uses the accessor methods in OldStyleMemberships instead of accessing the data from the postgres db. The ideal case would be to use only a database and no pickle files for Memberships data but I have not reached there. I had tried to read and use the data from the database instead of pickle files and that had broken my Mailman which leaves me with few questions In OldStyleMemberships.py the lower cased email address is used as a key for accessing the membership properties. However in my schema, I am using the (listname, case preserved email address) as the PK. Is it possible that not storing and using LCE as a key might break something. I also want to make sure that in the database I am caturing all the Memberships data. Presently my database uses the following class as a storm abstraction for the database. Do I need to add/remove anything? class PgsqlMembers(object): __storm_table__ = "mailman_test" __storm_primary__ = "listname","address" listname = Unicode() address = Unicode() password = Unicode() lang = Unicode() name = Unicode() digest = Unicode() delivery_status = Int() user_options = Int() topics_userinterest = Unicode() bounce_info = Unicode() delivery_status_timestamp = Unicode() Looking forward to your reviews and finally getting the thing in place!! Thanks a lot all Malveeka From turnbull at sk.tsukuba.ac.jp Sat Aug 8 15:33:07 2009 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 08 Aug 2009 22:33:07 +0900 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <20090808073558.GD29033@keuner.winnegan.fake> References: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> <20090808010344.GA23395@keuner.winnegan.fake> <93B285C8-91A5-4FD2-8FBA-0BEAA0A1A003@python.org> <20090808024914.GA29033@keuner.winnegan.fake> <87skg2ajqy.fsf@uwakimon.sk.tsukuba.ac.jp> <20090808073558.GD29033@keuner.winnegan.fake> Message-ID: <87prb6a0rw.fsf@uwakimon.sk.tsukuba.ac.jp> Bernd Siggy Brentrup writes: > On Sat, Aug 08, 2009 at 15:43 +0900, Stephen J. Turnbull wrote: > > They don't come *better* than mutt, though. At best, "different". > > Oh no, no yamuad (Yet Another MUA Discussion) please. Oh, fer shure. My point was to remind Barry that there are some things you can't ask of any MUA, 'cause none of them do it (well). > I'm subscribed to some MLs that mangle with Reply-To to keep subscribers > with broken MUAs happy, mutt does the Mail-Followup-To for me w/o my > intervention. Indeed, that was quite my point. You just can't win on "no dupes, please," except by accepting them and handling them yourself. > You actually read those winnegan.fake (my version of localnet) > headers? Nobody does this! *chuckle* Dirty job, but somebody's gotta do it. From barry at python.org Sat Aug 8 23:41:22 2009 From: barry at python.org (Barry Warsaw) Date: Sat, 8 Aug 2009 17:41:22 -0400 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <87prb6a0rw.fsf@uwakimon.sk.tsukuba.ac.jp> References: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> <20090808010344.GA23395@keuner.winnegan.fake> <93B285C8-91A5-4FD2-8FBA-0BEAA0A1A003@python.org> <20090808024914.GA29033@keuner.winnegan.fake> <87skg2ajqy.fsf@uwakimon.sk.tsukuba.ac.jp> <20090808073558.GD29033@keuner.winnegan.fake> <87prb6a0rw.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Aug 8, 2009, at 9:33 AM, Stephen J. Turnbull wrote: > Oh, fer shure. My point was to remind Barry that there are some > things you can't ask of any MUA, 'cause none of them do it > (well). Writing the Best MUA In The World is on my list, right after I solve the health care issue. :) But yep, it's just a specification of Warsaw's Third Law. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From andy.grover at gmail.com Sun Aug 9 21:35:17 2009 From: andy.grover at gmail.com (Andrew Grover) Date: Sun, 9 Aug 2009 12:35:17 -0700 Subject: [Mailman-Developers] Getting code in? (was: Storm based MemberAdaptor for Mailman) Message-ID: On Sat, Aug 8, 2009 at 1:25 AM, Malveeka Tewari wrote: > I am working on writing a storm based member adaptor for mailman so that the > mailman membership data can be stored in a database instead of .pck files. > The reason for choosing storm is that it provides an abstraction to use any > underlying db language- either mysql, postgresql or sqllite. Hi, I'm one of the GSOC mentors for Systers, all of whose students are working on Mailman enhancements. Some of these (Like Malveeka's) would be suitable for merging upstream, but the review and acceptance process is not clear. * Do people on this list regularly provide review of proposed enhancement patches? * Is there a set of people who explicitly ACK or NAK all proposed patches? * Once reviewed, merging is via LP merge requests, right? Or does one do the merge request and *then* get reviewed? * More fundamentally, is Mailman currently accepting patches from external contributors? We have code to contribute. It's not perfect yet but we will make it better. We need some Official Mailman participation or our improvements will never make it out of our branch. Even a response of "we're not interested in X, thanks" would be much superior to no response at all. Thanks -- Regards -- Andy From bsb at psycho.i21k.de Fri Aug 7 13:34:46 2009 From: bsb at psycho.i21k.de (Siggy Brentrup) Date: Fri, 7 Aug 2009 13:34:46 +0200 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: References: Message-ID: <20090807113446.GO11633@keuner.winnegan.fake> Args, I knew I forgot sth, I should have selected MIME digest :) On Thu, Aug 06, 2009 at 22:19:30 -0700, Jordan Hayes wrote: > Here's my solution: > > http://infothecary.org/jordan/mailman.html Nice solution Jordan, but I think about a pythonic way to fully integrate searchable archives into MM. I was using the debian archive search only as an example for what can be achieved on big archives. Thanks Siggy -- bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bsb at psycho.i21k.de Fri Aug 7 17:26:21 2009 From: bsb at psycho.i21k.de (Bernd 'Siggy' Brentrup) Date: Fri, 7 Aug 2009 17:26:21 +0200 Subject: [Mailman-Developers] [Mailman-Users] mailman user password In-Reply-To: <20090807144626.GN30597@amyl.org.uk> References: <4A74A47C.6050402@libertytrek.org> <87eirud1dt.fsf@uwakimon.sk.tsukuba.ac.jp> <4A75F9C0.3020705@libertytrek.org> <87vdl09cwn.fsf@uwakimon.sk.tsukuba.ac.jp> <20090807144626.GN30597@amyl.org.uk> Message-ID: <20090807152621.GD4628@keuner.winnegan.fake> Sorry Adam for hijacking, I'm not subscribed to mm-users. On Fri, Aug 07, 2009 at 15:46 +0100, Adam McGreggor wrote: > *shimmied over to mailman-devs* (from -users) > > On Fri, Aug 07, 2009 at 10:33:06AM -0400, Barry Warsaw wrote: > > It kind of sucks that there are so many other Mailman command line > > scripts, which is one reason why I've always put them in a separate > > Mailman specific bin directory. With MM3 though I intend to use a > > 'subcommand' approach so that there's only one 'mailman' command. > > Think things like 'mailman listmembers foo'. I'll probably keep > > mailmanctl separate though I haven't decided about that yet. I applaud you on this, Barry. I seem to remember discussions where to place these commands for Debian, ATM they pollute the /usr/sbin namespace. May I suggest to put mailman in /usr/bin while mailmanctl stays in /usr/sbin. > Would it too much to ask for the 'old' (read 'current') scripts/commands > to be aliased: at least in the first few releases? > > e.g., for 'list_members' to link to 'mailman listmembers' > > Perhaps as an install/configure/compilation option? > "Create old-style links?" > --old-style-links You might contribute some shell scripts simulating old behavior, a Debian package would then ask an additional debconf question whether to install these compatibility scripts or not. Regards Siggy -- bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From bsb at psycho.i21k.de Fri Aug 7 23:55:00 2009 From: bsb at psycho.i21k.de (Bernd 'Siggy' Brentrup) Date: Fri, 7 Aug 2009 23:55:00 +0200 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: References: <20090803180013.GF15680@keuner.winnegan.fake> Message-ID: <20090807215500.GB31400@keuner.winnegan.fake> On Fri, Aug 07, 2009 at 13:40 -0700, Mark Sapiro wrote: > Siggy Brentrup wrote: > > > >In contrast to his suggestions (sth involving google search) I'm > >aiming at a solution akin to Debian's Xapian powered ML search. > > > > cf http://lists.debian.org/cgi-bin/search > > > >What do you think and is anybody else interested in working on > >this? > > > There are patches at that > integrate the ht://Dig search engine with Mailman's archives. They > work well. > > The problem with integrating these or some of the other solutions into > the source is they require installation of software over which we have > no control. From a Debian point of view this isn't much of a problem, I might create a patched mailman-searchable-archives package that depends on htdig, eventually factoring out common parts to yield: mailman mailman-searchable-archives \ / \ mailman-common htdig but it doesn't address concerns about pipermail. MM 3 seems to me to be an opportunity to address all these issues, we might learn from ht://Dig or Xapian (both written in perl and C iirc) to implement an enhanced pipermail. These are only first thoughts, before deciding which way to go a lot of research is still necessary and I'm open to any suggestions. Regards Siggy -- bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From hopkinsju at umsystem.edu Fri Aug 7 22:59:17 2009 From: hopkinsju at umsystem.edu (Hopkins, Justin) Date: Fri, 7 Aug 2009 15:59:17 -0500 Subject: [Mailman-Developers] Attachment icon in archives Message-ID: For me, having a paperclip or some other indication of an attached file on messages in the archive would go a long way towards helping me find the message I am looking for. Has anyone else thought this and found or know of a solution? Cheers, Justin From jmhayes at speakeasy.net Mon Aug 10 03:31:05 2009 From: jmhayes at speakeasy.net (Jordan Hayes) Date: Sun, 9 Aug 2009 18:31:05 -0700 Subject: [Mailman-Developers] Searchable archives for MM References: <20090807113446.GO11633@keuner.winnegan.fake> Message-ID: Siggy writes: >> Here's my solution: >> >> http://infothecary.org/jordan/mailman.html > > Nice solution Jordan, but I think about a pythonic way to > fully integrate searchable archives into MM. Um, you need Phython in this equation exactly why? The archives are web pages. Who cares what language was used to implement the search? You don't demand a Python web server to deliver your archives, do you? > I was using the debian archive search only as an example for > what can be achieved on big archives. For me, I think the people who Think Hard about searching text can work in whatever language they want; SWISH-E has done a good job of providing a platform for search, and it's easily integrated with Mailman. FWIW, I have lots of "big archives" ... /jordan From jeff at jab.org Mon Aug 10 03:36:07 2009 From: jeff at jab.org (Jeff Breidenbach) Date: Sun, 9 Aug 2009 18:36:07 -0700 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: References: <20090807113446.GO11633@keuner.winnegan.fake> Message-ID: > Nice solution Jordan, but I think about a pythonic way to > fully integrate searchable archives into MM. If you are interested in PyLucene, I would be very happy to share maintenance of the relevant Debian package. -Jeff From barry at python.org Mon Aug 10 04:19:22 2009 From: barry at python.org (Barry Warsaw) Date: Sun, 9 Aug 2009 22:19:22 -0400 Subject: [Mailman-Developers] [Mailman-Users] mailman user password In-Reply-To: <20090807152621.GD4628@keuner.winnegan.fake> References: <4A74A47C.6050402@libertytrek.org> <87eirud1dt.fsf@uwakimon.sk.tsukuba.ac.jp> <4A75F9C0.3020705@libertytrek.org> <87vdl09cwn.fsf@uwakimon.sk.tsukuba.ac.jp> <20090807144626.GN30597@amyl.org.uk> <20090807152621.GD4628@keuner.winnegan.fake> Message-ID: <46DB531B-1BF7-47F3-86D5-F3CDDBBC06B7@python.org> On Aug 7, 2009, at 11:26 AM, Bernd 'Siggy' Brentrup wrote: > I applaud you on this, Barry. I seem to remember discussions where to > place these commands for Debian, ATM they pollute the /usr/sbin > namespace. May I suggest to put mailman in /usr/bin while mailmanctl > stays in /usr/sbin. I'm not yet sure how the buildout based source build will translate into installation recipes, but I generally concur with this. I'm nearly certain that mailmanctl's functionality will not appear in the 'mailman' command (which I already have working and committed to the trunk!). We'll have to figure that out when the release is closer. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 10 04:31:59 2009 From: barry at python.org (Barry Warsaw) Date: Sun, 9 Aug 2009 22:31:59 -0400 Subject: [Mailman-Developers] Getting code in? (was: Storm based MemberAdaptor for Mailman) In-Reply-To: References: Message-ID: <9C658C12-BCCD-41DD-9BF1-5C783DB7596C@python.org> On Aug 9, 2009, at 3:35 PM, Andrew Grover wrote: > Hi, I'm one of the GSOC mentors for Systers, all of whose students are > working on Mailman enhancements. Some of these (Like Malveeka's) would > be suitable for merging upstream, but the review and acceptance > process is not clear. Very cool to hear about your work! Let's see if I can clarify things. Eventually this ought to make it into the wiki. > * Do people on this list regularly provide review of proposed > enhancement patches? > * Is there a set of people who explicitly ACK or NAK all proposed > patches? > * Once reviewed, merging is via LP merge requests, right? Or does one > do the merge request and *then* get reviewed? I think the process should generally work like this: * A developer has an idea for an enhancement. They describe it here and we can discuss general design issues, coding suggestions, decomposition of work into branches, etc. For higher bandwidth discussions we can use irc. * If the idea or approach isn't something the Mailman community thinks is worthwhile, we'll leave official participation at that. Of course, this being open source and bzr being a DVCS, you're free to do whatever you want. * If the idea is appropriate for Mailman, we must get copyright assignments from the developer. Let's do this early, since it can take some time to get all the paperwork to the FSF. * Branches should be as small as possible and still be self- contained. They must include documentation, a NEWS entry, and for Mailman 3 they must include tests. Ideally, there would be one bug for every branch you work on. The smaller the branch the easier it is to review. For Launchpad, we have a strict 800 line diff limit on branches. I'm not sure if that's appropriate or not for Mailman, but please understand that anything over 1000 lines of diff gets very difficult and time consuming to review. * When a branch is ready for review, create a merge proposal for it and email us here. * One of us will review the code. For Mailman 2.x, it will most likely be Mark. For Mailman 3 it will be me. * Once we've approved the branch, either Mark or I will merge it into the official trunks. Currently only a few of us have write permission to the official branches. I do hope more people become developers and reviewers! > * More fundamentally, is Mailman currently accepting patches from > external contributors? Yes. It's tough though, and I apologize in advance, but both Mark and I are very busy. I've tried to shed load by concentrating only on Mailman 3 and that seems to work for me (and hopefully MM3, which is making good progress). > We have code to contribute. It's not perfect yet but we will make it > better. We need some Official Mailman participation or our > improvements will never make it out of our branch. Even a response of > "we're not interested in X, thanks" would be much superior to no > response at all. Completely agreed! Let's talk about your ideas and see which might be appropriate for inclusion. I will really work at being more responsive. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 10 04:48:58 2009 From: barry at python.org (Barry Warsaw) Date: Sun, 9 Aug 2009 22:48:58 -0400 Subject: [Mailman-Developers] Storm based MemberAdaptor for Mailman In-Reply-To: <68779e050908080125h7c39da1co283a0c7e16ddd92f@mail.gmail.com> References: <68779e050908080125h7c39da1co283a0c7e16ddd92f@mail.gmail.com> Message-ID: <394F83AF-9A50-4A55-9118-D6E932987743@python.org> On Aug 8, 2009, at 4:25 AM, Malveeka Tewari wrote: > I am working on writing a storm based member adaptor for mailman so > that the > mailman membership data can be stored in a database instead of .pck > files. > The reason for choosing storm is that it provides an abstraction to > use any > underlying db language- either mysql, postgresql or sqllite. I'm a big fan of Storm. It's the ORM used in Mailman 3 and I've had a lot of good success with it. Please note that this new member adapter cannot go officially into Mailman 2.1, but I do think it might be useful for Mailman 2.2, probably as unofficial contrib, though Mark may want to weigh in on that. Because the schema is so different in Mailman 3 and we already use Storm, this won't be relevant for that branch. > The ideal case would be to use only a database and no pickle files for > Memberships data but I have not reached there. > I had tried to read and use the data from the database instead of > pickle > files and that had broken my Mailman which leaves me with few > questions Are you using Pickle() fields for non-scalar data types? (i.e. lists and dictionaries). Of course, you don't gain from relational data that way, but it's still useful. > In OldStyleMemberships.py the lower cased email address is used as a > key for > accessing the membership properties. > However in my schema, I am using the (listname, case preserved email > address) as the PK. > Is it possible that not storing and using LCE as a key might break > something. From OldStyleMembership's (very loose) contract, I think the answer is "yes". It doesn't matter so much how you store things in the database, but you will have to honor the contract that the MemberAdapter.py promises. > I also want to make sure that in the database I am caturing all the > Memberships data. Presently my database uses the following class as > a storm > abstraction for the database. Do I need to add/remove anything? > > class PgsqlMembers(object): > __storm_table__ = "mailman_test" > __storm_primary__ = "listname","address" > > listname = Unicode() > address = Unicode() > password = Unicode() > lang = Unicode() > name = Unicode() > digest = Unicode() > delivery_status = Int() > user_options = Int() > topics_userinterest = Unicode() > bounce_info = Unicode() > delivery_status_timestamp = Unicode() When I was creating the schemas for MM3, I basically had to inspect OldStyleMembership.py to see what fields it requires, then fix things as tests broke. MM2 has the great disadvantage that there isn't a usable test suite. This is fixed in MM3. So I think you're left to manual testing until it basically works for you. :( -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 10 04:56:21 2009 From: barry at python.org (Barry Warsaw) Date: Sun, 9 Aug 2009 22:56:21 -0400 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: References: <20090807113446.GO11633@keuner.winnegan.fake> Message-ID: <9073FDED-F6C3-4448-B103-EF8F72121BAC@python.org> On Aug 9, 2009, at 9:31 PM, Jordan Hayes wrote: > Um, you need Phython in this equation exactly why? > > The archives are web pages. Who cares what language was used to > implement the search? > > You don't demand a Python web server to deliver your archives, do you? Heh, well Mailman 3 will include a couple of Python web servers. There will definitely be a web server for the REST interface and the web ui will likely be vended from a Python web server. They're super easy to write! Note that we've adopted WSGI for all web protocols so the ultimate port 80 listener can be almost anything. This will make it easy to integrate with frameworks such as Zope, Django, Turbogears, etc. or even Apache through mod_wsgi. I'm going to need really strong arguments to bundle anything not written in Python. >> I was using the debian archive search only as an example for >> what can be achieved on big archives. > > For me, I think the people who Think Hard about searching text can > work in whatever language they want; SWISH-E has done a good job of > providing a platform for search, and it's easily integrated with > Mailman. FWIW, I have lots of "big archives" ... The way to think about this then is modularization and pluggability. The archiver itself is already pluggable (with more well defined interfaces in Mailman 3). The question is whether for Pipermail, the search itself should be pluggable. Think about how that would work. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 10 05:05:03 2009 From: barry at python.org (Barry Warsaw) Date: Sun, 9 Aug 2009 23:05:03 -0400 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <20090808024914.GA29033@keuner.winnegan.fake> References: <91B74427-D5C2-45D2-AA7D-78669F7AC427@python.org> <20090808010344.GA23395@keuner.winnegan.fake> <93B285C8-91A5-4FD2-8FBA-0BEAA0A1A003@python.org> <20090808024914.GA29033@keuner.winnegan.fake> Message-ID: On Aug 7, 2009, at 10:49 PM, Bernd Siggy Brentrup wrote: > On Fri, Aug 07, 2009 at 22:02 -0400, Barry Warsaw wrote: >> On Aug 7, 2009, at 9:03 PM, Bernd Siggy Brentrup wrote: >> >>> :) No need for this broad hint. >> >> Tim Peters would be proud at the power of the ! >> >>> I was thinking about this option too and as soon as the issues >>> with the old account Ubuntu automatically created for me when >>> importing my Debian packages are solved I will create a pipermail >>> project on launchpad and announce it here. >> >> That would be cool. I wonder if we should make ~mailman-coders the >> owner of that project? > > No objections from my side if you grant me admin rights. As a surplus > we don't have to wait for lp ppl to merge my accounts; there are some > restrictions that inhibit merging e.g. creating a PPA, after all I > want > to be known on lp as ~bsb and not as ~bsb-psycho :-P I think right now I don't want to open up admin rights on the mailman- coders team. If that means we should create a pipermail-coders team for the pipermail project, that would be cool with me. >> I will volunteer to help separate the code, and to work out how best >> to integrate it for packaging. > > Thanks for this, did you write down any issues that should be > addressed when revamping pipermail? One thing that comes to mind > is how to deal with mime types and attachments. I'd also like to > see signatures validated. (On an aside, I hate these footers that > make my messages not fully signed) Here are the top priorities for me: * Stable and predictable URLs. We must guarantee that if the HTML archive were blown away and regenerated from the raw messages, that the urls to those messages will not change. URLs must also be predictable given only the information in the original message. Mailman must be able to calculate a URL to the message in the archive without communicating to the archiver. See this wiki page for details (already implemented on the Mailman side in MM3) http://wiki.list.org/display/DEV/Stable+URLs * Move away from mbox format for raw messages. (maildir?) * Use a modern backend for persistence (pickles suck) * Use templates and make the archives easily customizable * Use modern CSS, but Javascript sparingly and then with progressive degradation. * IWBNI all content were vended dynamically, with caching, so that messages could be removed from the archive, or the style could be changed, without a hard regen. >>> p.s. Please respect my .signature, I'd really like to reenable >>> duplicate elimination. >> >> You can do this for Mailman lists, at least when you are explicitly >> CC'd with an address that's subscribed to the list. You'll get the >> direct copy and not the list copy though. That's the best Mailman >> can do (anything more takes a better MUA ;). > > I know that's because I mentioned it, I didn't recall the variable > name though. In the first place I don't want to see 2 copies > arriving, no MUA involved here. That's because I was nice the first time and removed your name from the recipients. As they say "the first one's free" :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From andy.grover at gmail.com Mon Aug 10 07:08:10 2009 From: andy.grover at gmail.com (Andrew Grover) Date: Sun, 9 Aug 2009 22:08:10 -0700 Subject: [Mailman-Developers] Getting code in? (was: Storm based MemberAdaptor for Mailman) In-Reply-To: <9C658C12-BCCD-41DD-9BF1-5C783DB7596C@python.org> References: <9C658C12-BCCD-41DD-9BF1-5C783DB7596C@python.org> Message-ID: On Sun, Aug 9, 2009 at 7:31 PM, Barry Warsaw wrote: >> Hi, I'm one of the GSOC mentors for Systers, all of whose students are >> working on Mailman enhancements. Some of these (Like Malveeka's) would >> be suitable for merging upstream, but the review and acceptance >> process is not clear. > > Very cool to hear about your work! ?Let's see if I can clarify things. > ?Eventually this ought to make it into the wiki. Hi Barry, thanks for your lengthy response. I've tried to be helpful by updating the wiki with the content from your email (http://wiki.list.org/display/DEV/Home). You really clarified a lot for us, and hopefully other potential contributors :) So the next question for Mark and everyone is: do you want a StormDB-backed MembershipAdaptor for Mailman 2.x? There are some further questions the Systers org needs to settle before proceeding, about the ownership status of GSOC-produced code, as well as if copyright assignment to FSF is acceptable, but it is moot if the feature is only interesting to us. Thanks -- Regards -- Andy From stephen at xemacs.org Mon Aug 10 08:09:40 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 10 Aug 2009 15:09:40 +0900 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <20090807215500.GB31400@keuner.winnegan.fake> References: <20090803180013.GF15680@keuner.winnegan.fake> <20090807215500.GB31400@keuner.winnegan.fake> Message-ID: <87d4749p3v.fsf@uwakimon.sk.tsukuba.ac.jp> Bernd 'Siggy' Brentrup writes: > learn from ht://Dig or Xapian (both written in perl and C iirc) > to implement an enhanced pipermail. Xapian is written in C++ (I'm pretty sure about that) and has Python bindings. Python bindings for sure: it's used in Roundup (though like much in Roundup I can't recommend it as exemplary implementation). From adam-mailman at amyl.org.uk Mon Aug 10 14:06:28 2009 From: adam-mailman at amyl.org.uk (Adam McGreggor) Date: Mon, 10 Aug 2009 13:06:28 +0100 Subject: [Mailman-Developers] [Mailman-Users] mailman user password In-Reply-To: <20090807152621.GD4628@keuner.winnegan.fake> References: <4A74A47C.6050402@libertytrek.org> <87eirud1dt.fsf@uwakimon.sk.tsukuba.ac.jp> <4A75F9C0.3020705@libertytrek.org> <87vdl09cwn.fsf@uwakimon.sk.tsukuba.ac.jp> <20090807144626.GN30597@amyl.org.uk> <20090807152621.GD4628@keuner.winnegan.fake> Message-ID: <20090810120628.GV30597@amyl.org.uk> On Fri, Aug 07, 2009 at 05:26:21PM +0200, Bernd 'Siggy' Brentrup wrote: > Sorry Adam for hijacking, I'm not subscribed to mm-users. no prob! > > Would it too much to ask for the 'old' (read 'current') scripts/commands > > to be aliased: at least in the first few releases? > > > > e.g., for 'list_members' to link to 'mailman listmembers' > > > > Perhaps as an install/configure/compilation option? > > "Create old-style links?" > > --old-style-links > > You might contribute some shell scripts simulating old behavior, > a Debian package would then ask an additional debconf question > whether to install these compatibility scripts or not. I'll see what comes out, before committing to write scripts for Debian &c. I've been known to whine about 'Debian policy': some Debianistic traits annoy me (having came to Debian via BSD). Saying that, a new version may be a good chance to migrate/re-write the scripts I've been using for some time: what may be quite useful is something to hunt various directory trees for references to 'old mailman' commands, and offer replacements for 'new mailman'. I'd probably write something that would do that, for my own installs, so it's not too bad releasing back to the community. (although I tend to license stuff under (Attribute Share-Alike) Creative Commons.) -- If tin whistles are made of tin, what are foghorns made of? From lists at psycho.i21k.de Mon Aug 10 14:21:13 2009 From: lists at psycho.i21k.de (Bernd Siggy Brentrup) Date: Mon, 10 Aug 2009 14:21:13 +0200 Subject: [Mailman-Developers] [Mailman-Users] mailman user password In-Reply-To: <20090810120628.GV30597@amyl.org.uk> References: <4A74A47C.6050402@libertytrek.org> <87eirud1dt.fsf@uwakimon.sk.tsukuba.ac.jp> <4A75F9C0.3020705@libertytrek.org> <87vdl09cwn.fsf@uwakimon.sk.tsukuba.ac.jp> <20090807144626.GN30597@amyl.org.uk> <20090807152621.GD4628@keuner.winnegan.fake> <20090810120628.GV30597@amyl.org.uk> Message-ID: <20090810122113.GB13301@puntila.winnegan.fake> On Mon, Aug 10, 2009 at 13:06 +0100, Adam McGreggor wrote: > On Fri, Aug 07, 2009 at 05:26:21PM +0200, Bernd 'Siggy' Brentrup wrote: > > You might contribute some shell scripts simulating old behavior, > > a Debian package would then ask an additional debconf question > > whether to install these compatibility scripts or not. > > I'll see what comes out, before committing to write scripts for > Debian &c. I've been known to whine about 'Debian policy': some > Debianistic traits annoy me (having came to Debian via BSD). I'm using Debian here only as a reference for a packaging system that I happen to be most acquainted with. You may read Ubuntu, RH or whatever you like. Don't ask me about 'Debian Policy' as long as I am waiting for DAM to reopen my account bsb at d.o, I might be forced to lie like a trooper before I have a vote again :) Regards Siggy -- O< ascii ribbon campaign - stop html mail - www.asciiribbon.org bsb-at-psycho-dot-informationsanarchistik-dot-de or: bsb-at-psycho-dot-i21k-dot-de ========> ceterum censeo javascriptum esse restrictam <======== -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From mark at msapiro.net Mon Aug 10 17:55:24 2009 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 10 Aug 2009 08:55:24 -0700 Subject: [Mailman-Developers] Storm based MemberAdaptor for Mailman In-Reply-To: <394F83AF-9A50-4A55-9118-D6E932987743@python.org> Message-ID: Barry Warsaw quoted: > >On Aug 8, 2009, at 4:25 AM, Malveeka Tewari wrote: > >> I also want to make sure that in the database I am caturing all the >> Memberships data. Presently my database uses the following class as >> a storm >> abstraction for the database. Do I need to add/remove anything? >> >> class PgsqlMembers(object): >> __storm_table__ = "mailman_test" >> __storm_primary__ = "listname","address" >> >> listname = Unicode() >> address = Unicode() >> password = Unicode() >> lang = Unicode() >> name = Unicode() >> digest = Unicode() >> delivery_status = Int() >> user_options = Int() >> topics_userinterest = Unicode() >> bounce_info = Unicode() This has caused problems with the MySQL MemberAdaptor in the past. bounce_info is a _BounceInfo class instance that is supposed to be opaque to the MemberAdaptor. The issue with the MySQL MemberAdaptor was that it 'knew' what the attributes of this instance were and stored them separately in the database, but then the class grew another attribute and the MemberAdaptor failed. The problem is that the methods setBounceInfo() and getBounceInfo() have to know something about the _BounceInfo class in order to be able to store something in the database that can later be returned as a class instance. The MySQL MemberAdaptor does this by hving setBounceInfo() store the attributes separately and getBounceInfo() instantiate the class with the retrieved attributes, but this fails if the number of arguments expected by _BounceInfo.__init__() changes. An alternative is to pickle the class instance into a string to be stored in the database, but this then loses the ability to see the various bounce attributes in the database. >> delivery_status_timestamp = Unicode() -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at python.org Mon Aug 10 18:11:34 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Aug 2009 12:11:34 -0400 Subject: [Mailman-Developers] Storm based MemberAdaptor for Mailman In-Reply-To: References: Message-ID: <681EF8FB-6C99-4B18-9E14-C578E072B58F@python.org> On Aug 10, 2009, at 11:55 AM, Mark Sapiro wrote: > Barry Warsaw quoted: >> >> On Aug 8, 2009, at 4:25 AM, Malveeka Tewari wrote: >> >>> I also want to make sure that in the database I am caturing all the >>> Memberships data. Presently my database uses the following class as >>> a storm >>> abstraction for the database. Do I need to add/remove anything? >>> >>> class PgsqlMembers(object): >>> __storm_table__ = "mailman_test" >>> __storm_primary__ = "listname","address" >>> >>> listname = Unicode() >>> address = Unicode() >>> password = Unicode() >>> lang = Unicode() >>> name = Unicode() >>> digest = Unicode() >>> delivery_status = Int() >>> user_options = Int() >>> topics_userinterest = Unicode() >>> bounce_info = Unicode() > > > This has caused problems with the MySQL MemberAdaptor in the past. > bounce_info is a _BounceInfo class instance that is supposed to be > opaque to the MemberAdaptor. The issue with the MySQL MemberAdaptor > was that it 'knew' what the attributes of this instance were and > stored them separately in the database, but then the class grew > another attribute and the MemberAdaptor failed. > > The problem is that the methods setBounceInfo() and getBounceInfo() > have to know something about the _BounceInfo class in order to be able > to store something in the database that can later be returned as a > class instance. > > The MySQL MemberAdaptor does this by hving setBounceInfo() store the > attributes separately and getBounceInfo() instantiate the class with > the retrieved attributes, but this fails if the number of arguments > expected by _BounceInfo.__init__() changes. This has been a PITA in MM3 too, and not something I've totally unfscked yet. > An alternative is to pickle the class instance into a string to be > stored in the database, but this then loses the ability to see the > various bounce attributes in the database. The nice thing is that Storm has a Pickle() type that makes it easy. But you're right, doing this is only a stopgap and makes the bounce information opaque to other proceses. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mark at msapiro.net Mon Aug 10 18:38:03 2009 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 10 Aug 2009 09:38:03 -0700 Subject: [Mailman-Developers] Getting code in? (was: Storm basedMemberAdaptor for Mailman) In-Reply-To: Message-ID: Andrew Grover wrote: > >So the next question for Mark and everyone is: do you want a >StormDB-backed MembershipAdaptor for Mailman 2.x? > >There are some further questions the Systers org needs to settle >before proceeding, about the ownership status of GSOC-produced code, >as well as if copyright assignment to FSF is acceptable, but it is >moot if the feature is only interesting to us. I'm interested. It is not a high priority for me because it is something that Barry is doing for MM 3 anyway. My concern is to not create additional 2.1 -> 2.2 -> 3 migration headaches. If this is something that could ease the migration to MM 3 by providing a 'stepwise' path, that would be great. However, if it is viewed as something that will be done for MM 2.2 and then totally redone for MM 3, I'm afraid it will just encourage people to skip MM 2.2 and wait for MM 3. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From jmhayes at speakeasy.net Mon Aug 10 19:09:33 2009 From: jmhayes at speakeasy.net (Jordan Hayes) Date: Mon, 10 Aug 2009 10:09:33 -0700 Subject: [Mailman-Developers] Searchable archives for MM References: <20090807113446.GO11633@keuner.winnegan.fake> <9073FDED-F6C3-4448-B103-EF8F72121BAC@python.org> Message-ID: <37B91FC25F384E728D66770AF3634FDB@pavepaws> Barry Warsaw writes: > Mailman 3 will include a couple of Python web servers. Sorry, my port 80 is already taken up by something else :-) > There will definitely be a web server for the REST interface and the > web ui will likely be vended from a Python web server. They're super > easy to write! I look forward to someone reimplementing in Python a search package as good as SWISH-E :-) /jordan From barry at python.org Mon Aug 10 19:36:08 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Aug 2009 13:36:08 -0400 Subject: [Mailman-Developers] Searchable archives for MM In-Reply-To: <37B91FC25F384E728D66770AF3634FDB@pavepaws> References: <20090807113446.GO11633@keuner.winnegan.fake> <9073FDED-F6C3-4448-B103-EF8F72121BAC@python.org> <37B91FC25F384E728D66770AF3634FDB@pavepaws> Message-ID: <10351C66-28EF-4B26-AFCC-D07BA263AA0A@python.org> On Aug 10, 2009, at 1:09 PM, Jordan Hayes wrote: > Barry Warsaw writes: > >> Mailman 3 will include a couple of Python web servers. > > Sorry, my port 80 is already taken up by something else :-) Web servers can bind to other ports. They can even be proxied :). >> There will definitely be a web server for the REST interface and the >> web ui will likely be vended from a Python web server. They're super >> easy to write! > > I look forward to someone reimplementing in Python a search package > as good as SWISH-E :-) If there are Python bindings to SWISH-E, then that might be good enough for inclusion in Mailman. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 10 19:57:52 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Aug 2009 13:57:52 -0400 Subject: [Mailman-Developers] Getting code in? (was: Storm basedMemberAdaptor for Mailman) In-Reply-To: References: Message-ID: <9AA21215-E4F3-45FB-A1FC-2CE904AADCBE@python.org> On Aug 10, 2009, at 12:38 PM, Mark Sapiro wrote: > My concern is to not create additional 2.1 -> 2.2 -> 3 migration > headaches. If this is something that could ease the migration to MM 3 > by providing a 'stepwise' path, that would be great. However, if it is > viewed as something that will be done for MM 2.2 and then totally > redone for MM 3, I'm afraid it will just encourage people to skip MM > 2.2 and wait for MM 3. The Storm based MemberAdapter is definitely a MM2-only thing. The schema is so different in MM3 I don't see it as being a possible stepping stone. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From andy.grover at gmail.com Mon Aug 10 21:07:09 2009 From: andy.grover at gmail.com (Andrew Grover) Date: Mon, 10 Aug 2009 12:07:09 -0700 Subject: [Mailman-Developers] Getting code in? (was: Storm basedMemberAdaptor for Mailman) In-Reply-To: <9AA21215-E4F3-45FB-A1FC-2CE904AADCBE@python.org> References: <9AA21215-E4F3-45FB-A1FC-2CE904AADCBE@python.org> Message-ID: On Mon, Aug 10, 2009 at 10:57 AM, Barry Warsaw wrote: > On Aug 10, 2009, at 12:38 PM, Mark Sapiro wrote: > >> My concern is to not create additional 2.1 -> 2.2 -> 3 migration >> headaches. If this is something that could ease the migration to MM 3 >> by providing a 'stepwise' path, that would be great. However, if it is >> viewed as something that will be done for MM 2.2 and then totally >> redone for MM 3, I'm afraid it will just encourage people to skip MM >> 2.2 and wait for MM 3. > > The Storm based MemberAdapter is definitely a MM2-only thing. ?The schema is > so different in MM3 I don't see it as being a possible stepping stone. One (bad) option would be to use a MM3-compatible schema for the new 2.2 code. This sounds bad for a few reasons: MM3 schema could change, and an incomplete implementation of the (large) schema in our 2.2 Adaptor could increase the complexity and fragility of the upgrade process. Have you given any thought yet to the upgrade process? I'm assuming there will be some utility to convert a MM2.2 installation to a MM3 one? One thought I had was that this utility would likely use the Membership adaptor classes to pull data (as opposed to reading .pck files directly) in which case the data source of the member data should be opaque[1]? Thanks -- Regards -- Andy [1] of course if database or table names overlap, it goes boom From barry at python.org Mon Aug 10 21:20:35 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Aug 2009 15:20:35 -0400 Subject: [Mailman-Developers] Getting code in? (was: Storm basedMemberAdaptor for Mailman) In-Reply-To: References: <9AA21215-E4F3-45FB-A1FC-2CE904AADCBE@python.org> Message-ID: <3E71A9E4-8920-4F05-B2EB-9D8D1F40585B@python.org> On Aug 10, 2009, at 3:07 PM, Andrew Grover wrote: > On Mon, Aug 10, 2009 at 10:57 AM, Barry Warsaw > wrote: >> On Aug 10, 2009, at 12:38 PM, Mark Sapiro wrote: >> >>> My concern is to not create additional 2.1 -> 2.2 -> 3 migration >>> headaches. If this is something that could ease the migration to >>> MM 3 >>> by providing a 'stepwise' path, that would be great. However, if >>> it is >>> viewed as something that will be done for MM 2.2 and then totally >>> redone for MM 3, I'm afraid it will just encourage people to skip MM >>> 2.2 and wait for MM 3. >> >> The Storm based MemberAdapter is definitely a MM2-only thing. The >> schema is >> so different in MM3 I don't see it as being a possible stepping >> stone. > > One (bad) option would be to use a MM3-compatible schema for the new > 2.2 code. This sounds bad for a few reasons: MM3 schema could change, > and an incomplete implementation of the (large) schema in our 2.2 > Adaptor could increase the complexity and fragility of the upgrade > process. I'm not in favor of this because one of the whole points of having a MM2.2 was to keep the data model unchanged from 2.1, but to fix and improve other aspects of the system, such as the web interface, some of the defaults, etc. Really fixing the data model should only happen in MM3. Of course, I welcome all participation in that effort :). > Have you given any thought yet to the upgrade process? I'm assuming > there will be some utility to convert a MM2.2 installation to a MM3 > one? One thought I had was that this utility would likely use the > Membership adaptor classes to pull data (as opposed to reading .pck > files directly) in which case the data source of the member data > should be opaque[1]? > > Thanks -- Regards -- Andy > > [1] of course if database or table names overlap, it goes boom I have thought about this, and done some preliminary work on this, but it could definitely use some more attention. It's absolutely critical that we have an easy migration mechanism before MM3 final can be released. My current thinking has been to use a flat file format somewhat like what bin/export produces and have an import script in MM3 that collates and updates its data. That's just a general direction though so lots of details need to be worked out. Another thing I have not been good at is tracking the MM2.2 changes and 2.1 bug fixes. At some point I'll have to do an audit of this, but it's too disruptive of my current flow to do that now. :( -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From john at postcompanies.com Tue Aug 18 03:08:45 2009 From: john at postcompanies.com (John Goodell) Date: Mon, 17 Aug 2009 21:08:45 -0400 Subject: [Mailman-Developers] Looking for a mailman developer to hire Message-ID: Am looking for an experienced mailman developer to hire for a few small installations/upgrade to our deployment, including: personalized header/footer fields (using qmail injection), attachment issue for HTML messages, and a few others. Thanks, John From barry at list.org Sat Aug 22 00:22:51 2009 From: barry at list.org (Barry Warsaw) Date: Fri, 21 Aug 2009 18:22:51 -0400 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) Message-ID: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> I am happy to announce the release of the third alpha version of Mailman 3, code named "Working Man". This is primarily a preview release so that developers and other interested people can download the code and participate in Mailman 3's further development. I believe we are on track for a final release by the end of the year, and your contributions of code, feedback, documentation, etc. will be welcome and appreciated! Please note that this is an alpha release and as such is not ready for production use. You can get the code from the Cheeseshop: http://pypi.python.org/pypi/mailman/3.0.0a3 Mailman 3 is buildout based and requires Python 2.6. To build it, run this after unpacking the tarball and cd'ing into it % python bootstrap.py % bin/buildout From there you can run the tests % bin/test and build the documentation % bin/docs Highlights in this release include the start of a REST admin server for integrating Mailman with external web sites, a combined bin/ mailman uber-command, configuration now done through ini-files using lazr.config, and better LMTP support. Enjoy, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mark at msapiro.net Sat Aug 22 00:50:16 2009 From: mark at msapiro.net (Mark Sapiro) Date: Fri, 21 Aug 2009 15:50:16 -0700 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> Message-ID: Barry Warsaw wrote: > >I am happy to announce the release of the third alpha version of >Mailman 3, code named "Working Man". Way to go Barry!! -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From stephen at xemacs.org Sat Aug 22 07:45:31 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 22 Aug 2009 14:45:31 +0900 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> Message-ID: <87ab1s4d1g.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > I am happy to announce the release of the third alpha version of > Mailman 3, code named "Working Man". Congratulations! From p at state-of-mind.de Sat Aug 22 07:40:22 2009 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Sat, 22 Aug 2009 07:40:22 +0200 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> Message-ID: <20090822054022.GB4625@x300.office.state-of-mind.de> * Barry Warsaw : > I am happy to announce the release of the third alpha version of > Mailman 3, code named "Working Man". "I guess that's what I am ..." Congratulations! p at rick -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 315 bytes Desc: Digital signature URL: From cite at incertum.net Sat Aug 22 10:23:37 2009 From: cite at incertum.net (Stefan =?utf-8?Q?F=C3=B6rster?=) Date: Sat, 22 Aug 2009 10:23:37 +0200 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> Message-ID: <20090822082336.GV17045@mail.incertum.net> * Barry Warsaw : > I am happy to announce the release of the third alpha version of Mailman > 3, code named "Working Man". Congratulations! > Highlights in this release include the start of a REST admin server for > integrating Mailman with external web sites, a combined bin/mailman > uber-command, configuration now done through ini-files using > lazr.config, and better LMTP support. I'm really looking forward not only to the REStful implementation of the admin server but to the design specs for the web UI I've seen so far, too. Keep up the good work Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From stephen at xemacs.org Sat Aug 22 20:25:52 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 23 Aug 2009 03:25:52 +0900 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> Message-ID: <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> Shouldn't these release messages for 3.0 alphas redirect to mailman-developers? Barry Warsaw writes: > Please note that this is an alpha release and as such is not ready for > production use. > > You can get the code from the Cheeseshop: And do what with it? Check it in to a git repo, right? I mean, you did say you were looking forward to content contributions! ;-) So, let's be modern and check it out from Launchpad. Problems: (1) Debian stable doesn't have Python2.6. *sigh* (Not your fault, but it's now 0237, and I'm in a bitchin' mood and even Pure Prairie League doesn't make me feel better.) (2) README.txt refers to a bunch of files in docs/readme/ that don't exist (including the directory itself). If you're planning to reorganize the docs directory, now might be as good a time as any to fix the README. If you plan to populate it instead, you could add a note that volunteers to write the as yet nonexistent docs would be welcome. (3) docs/ALPHA.txt says you need an unofficial branch of lazr.config ... which doesn't seem to exist: $ bzr branch lp:~barry/lazr.config/megamerge You have not informed bzr of your Launchpad ID, and you must do this to write to Launchpad or access private data. See "bzr help launchpad-login". http://bazaar.launchpad.net/~barry/lazr.config/megamerge/ is permanently redirected to https+urllib://bazaar.launchpad.net/~barry/lazr.config/megamerge/ bzr: ERROR: Not a branch: "http://bazaar.launchpad.net/~barry/lazr.config/megamerge/". Oh, OK, it's private data or something. OK, so we try telling bzr about our LP ID and try again: $ bzr branch lp:~barry/lazr.config/megamerge The authenticity of host 'bazaar.launchpad.net (91.189.90.11)' can't be established. RSA key fingerprint is 9d:38:3a:63:b1:d5:6f:c4:44:67:53:49:2e:ee:fc:89. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'bazaar.launchpad.net,91.189.90.11' (RSA) to the list of known hosts. Permission denied (publickey). bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist. *sigh* I guess I'll just be a good little boy and get the PyPI package, but ... nah, I'll just go to bed and when I wake up I'll discover it was all a bad dream, right? From lists at psycho.i21k.de Sat Aug 22 21:54:56 2009 From: lists at psycho.i21k.de (Bernd Siggy Brentrup) Date: Sat, 22 Aug 2009 21:54:56 +0200 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20090822195456.GB5091@puntila.winnegan.fake> On Sun, Aug 23, 2009 at 03:25 +0900, Stephen J. Turnbull wrote: > Shouldn't these release messages for 3.0 alphas redirect to > mailman-developers? > > Barry Warsaw writes: > > > Please note that this is an alpha release and as such is not ready for > > production use. > > > > You can get the code from the Cheeseshop: > > And do what with it? Check it in to a git repo, right? I mean, you > did say you were looking forward to content contributions! ;-) > > So, let's be modern and check it out from Launchpad. > > Problems: > > (1) Debian stable doesn't have Python2.6. *sigh* (Not your fault, > but it's now 0237, and I'm in a bitchin' mood and even Pure > Prairie League doesn't make me feel better.) Neither does Debian testing aka squeeze, as I noticed moments ago. Just another reason to run Ubuntu 9.04 or above. Regs Siggy -- O< ascii ribbon campaign - stop html mail - www.asciiribbon.org+ |55 days until|Open Source in Northern Germany: www.free-it.org| |www.Ubucon.de| tech contact: bsb at free-it.de| +-------> ceterum censeo javascriptum esse restrictam <--------+ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From stephen at xemacs.org Sun Aug 23 02:48:01 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 23 Aug 2009 09:48:01 +0900 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <20090822195456.GB5091@puntila.winnegan.fake> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> <20090822195456.GB5091@puntila.winnegan.fake> Message-ID: <87ljlb2w5a.fsf@uwakimon.sk.tsukuba.ac.jp> Bernd Siggy Brentrup writes: > > Problems: > > > > (1) Debian stable doesn't have Python2.6. *sigh* (Not your fault, > > but it's now 0237, and I'm in a bitchin' mood and even Pure > > Prairie League doesn't make me feel better.) > > Neither does Debian testing aka squeeze, as I noticed moments ago. > Just another reason to run Ubuntu 9.04 or above. It's taken me years to get Debian under control so that it installs what I need and only that. No, I don't think I'm going to switch to a distro whose goal is to eliminate the need for Windows. From lists at psycho.i21k.de Sun Aug 23 06:34:55 2009 From: lists at psycho.i21k.de (Bernd Siggy Brentrup) Date: Sun, 23 Aug 2009 06:34:55 +0200 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <87ljlb2w5a.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> <20090822195456.GB5091@puntila.winnegan.fake> <87ljlb2w5a.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20090823043455.GB14594@puntila.winnegan.fake> Hi Stephen, On Sun, Aug 23, 2009 at 09:48 +0900, you wrote: > Bernd Siggy Brentrup writes: > > > > Problems: > > > > > > (1) Debian stable doesn't have Python2.6. *sigh* (Not your fault, > > > but it's now 0237, and I'm in a bitchin' mood and even Pure > > > Prairie League doesn't make me feel better.) > > > > Neither does Debian testing aka squeeze, as I noticed moments ago. > > Just another reason to run Ubuntu 9.04 or above. > > It's taken me years to get Debian under control so that it installs > what I need and only that. No, I don't think I'm going to switch to a > distro whose goal is to eliminate the need for Windows. Sorry if I sounded like urging you to switch to Ubuntu, my statement was meant only as a pun towards certain people currently in charge at the Debian project I had my issues with. Looking at the python2.6[1] page in Debian's PTS you'll notice it is still in experimental so don't hold your breath waiting for it appearing in testing. Regards Siggy [1] http://packages.qa.debian.org/p/python2.6.html -- O< ascii ribbon campaign - stop html mail - www.asciiribbon.org+ |54 days until|Open Source in Northern Germany: www.free-it.org| |www.Ubucon.de| tech contact: bsb at free-it.de| +-------> ceterum censeo javascriptum esse restrictam <--------+ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From stephen at xemacs.org Sun Aug 23 10:34:36 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 23 Aug 2009 17:34:36 +0900 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <20090823043455.GB14594@puntila.winnegan.fake> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> <20090822195456.GB5091@puntila.winnegan.fake> <87ljlb2w5a.fsf@uwakimon.sk.tsukuba.ac.jp> <20090823043455.GB14594@puntila.winnegan.fake> Message-ID: <87iqgf2ajn.fsf@uwakimon.sk.tsukuba.ac.jp> Hi Bernd, Bernd Siggy Brentrup writes: > Looking at the python2.6[1] page in Debian's PTS you'll notice it is > still in experimental so don't hold your breath waiting for it > appearing in testing. I didn't plan to. I don't actually mind installing my own Python. It's just that I've been procrastinating about doing something active on MM3 for months, and on the spur of the moment decided NOW was the time. I wasn't expecting that I'd be spending an hour or so just installing Python (I needed to install a bunch of -dev packages, etc.) and was frustrated. :-? From barry at python.org Mon Aug 24 00:05:54 2009 From: barry at python.org (Barry Warsaw) Date: Sun, 23 Aug 2009 18:05:54 -0400 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <20090822054022.GB4625@x300.office.state-of-mind.de> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <20090822054022.GB4625@x300.office.state-of-mind.de> Message-ID: <1B492DB8-571A-4FC7-9AB7-8D076C97FAC6@python.org> On Aug 22, 2009, at 1:40 AM, Patrick Ben Koetter wrote: > * Barry Warsaw : >> I am happy to announce the release of the third alpha version of >> Mailman 3, code named "Working Man". > > "I guess that's what I am ..." > Thanks Patrick, and everyone! Patrick, I was going to award you the prize for first guessing of the naming scheme. But then I remembered you had insider information :). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 24 00:06:58 2009 From: barry at python.org (Barry Warsaw) Date: Sun, 23 Aug 2009 18:06:58 -0400 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <20090822082336.GV17045@mail.incertum.net> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <20090822082336.GV17045@mail.incertum.net> Message-ID: <76535B1E-A1DB-4964-AAAA-97D467D3EBFB@python.org> On Aug 22, 2009, at 4:23 AM, Stefan F?rster wrote: > I'm really looking forward not only to the REStful implementation of > the admin server but to the design specs for the web UI I've seen so > far, too. Yep! Patrick and his team have a nice start to this, and should be bringing it to the mailing list soon. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From rsk at gsp.org Mon Aug 24 16:37:31 2009 From: rsk at gsp.org (Rich Kulawiec) Date: Mon, 24 Aug 2009 10:37:31 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 Message-ID: <20090824143731.GA1949@gsp.org> Summary: Spammers now have so many ways of "harvesting" addresses from so many systems, and so many ways of exchanging those with each other, that any email address which is actually used WILL eventually be harvested. (Where what "eventually" means varies widely, of course, but can be expected to steadily decrease.) Pretending that address obfuscation in mailing list [or newsgroup] archives will have any meaningful effect on this process gives users a false sense of security and has zero anti-spam value. Summary of summary: It's pointless. Explanation: Spammers maintain extensive databases of email addresses. Some of those databases are merely lists of addresses; others are more sophisticated and include data such as "harvested-date", "havesting-method", "last-seen date", "last-seen-context", "last-known-valid date" and more. Some of these databases are private; others are available for sale/lease. Some are maintained by spammers themselves, others by spammer support services don't directly engage in spamming. The harvesting engines used to acquire email addresses are myriad, as are the methods by which spammers acquire the raw data to use as input to them. *Some* of those methods, and there are many more, include: - subscribing to mailing lists - acquiring Usenet news (NNTP) feeds - querying mail servers - acquiring corporate email directories - insecure LDAP servers - insecure AD servers - use of backscatter/outscatter - use of auto-responders - use of mailing list mechanisms - use of abusive "callback" mechanisms - dictionary attacks - construction of plausible addresses (e.g. "firstname.lastname") - purchase of addresses in bulk on the open market. - purchase of addresses from vendors, web sites, etc. - purchase of addresses from registrars, ISPs, web hosts, etc. - domain registration (some registrars ARE spammers) [1] - misplaced/lost/sold media (hard disk, tape, CD, DVD, USB stick, etc.) and perhaps most significantly: - harvesting of the mail, address books and any other files present on any of the hundreds of millions of compromised Windows systems [2] Consider for example: the first time a newly-created address is used by someone (who is sending a message to it), it's now present on their system: in their saved outbound mail, or perhaps in their address book (if they have one), or in some cache. Any sensible malware resident on their system will of course pick it up and eventually hand it over to a harvesting agent. (Competent malware will harvest it in real time *and* associate it with the sender's address.) And if that particular system happens to be clean? Doesn't help much, because the more times that address is used, the more systems it's present on. And the more systems it's present on, the greater the probability that one of them is already compromised or will be soon. Thus even if we eliminate the originating end-user system as a possible source, we still have to consider the outbound mail server used by that end-user system, which is also a candidate for compromise. And then the inbound mail server used by the recipient, and then the recipient end-user system. And if there's some filtering appliance or intermediate system in place at either end, then it's there too. If the message is forwarded to a third party, then another set of systems is in play. If mail server logs are rolled up and moved to some central location, then it's there too. If backups are made, then it's present there, and subsequently may be present on any system where the backups are read/restored. And finally, if the destination of a mail message isn't an individual user, but an entire mailing list, then we must multiply the number of possible harvesting points by at least the number of people on the mailing list plus a factor for mail servers/gateways/filters/etc. (modulo overlaps). This in turns means that messages to sent to lists of any appreciable size (say, 1000 members) will turn up on considerably more than 1000 systems -- and the chances that all 1000-plus are secure are microscopic. Please note that the previous paragraph's recitation only covered the last vector I enumerated in the [indented] list above: compromised systems. That laundry list of methods also affords many other opportunities for addresses to find their way into spammers' hands. As just one pointed example out of a great many more that could be cited: how do you know that the address user at example.com which has just subscribed to the list you run is a real person and not just the front-end for an address-harvester that will pick up every address used to send traffic to the list? And so on. There are far too many others to enumerate, all of which have discussed at great length in anti-spam forums for many years, and are depressingly familiar to experienced practitioners working in the field. The bottom line is that any email address which is actually used [3], *especially* any email address used to send traffic to a mailing list, is going to be harvested. It's only a matter of when, not if, and "when" is getting sooner all the time. Incidentally, everyone (including me) can produce anecdotal tales of addresses that have remained surprisingly under-targeted by spammers over long periods of time. But this is clearly not the way to bet: it is in spammers' interests to ferret out as many addresses as possible and to use them as soon and as often as possible. Note, however, that some addresses are *deliberately* un-/under-targeted, so lack of substantial spam traffic to a given address is NOT an indicator that the address hasn't been harvested. That's because along with target lists, spammers maintain "suppression" lists, which they use to avoid hitting the addresses of people they think are likely to cause issues for them. [4] And obviously, people with postmaster or mailing list roles would be good candidates for membership on those lists. I know that if I were in their shoes, I'd add everyone who's ever sent a message to the mailman-* mailing lists to mine: a quick check indicates that it's on the order of only 10K addresses. Skipping those would be inconsequential when sending spam to a few hundred million addresses, and I trust it's obvious why spammers would benefit from doing so. With all this in mind, it's clearly pointless to pretend that address obfuscation in archives provides any protection at all. [5] It would be better to remove the code entirely than to continue to maintain the facade that it actually has any anti-spam value. Everyone should simply presume that all email addresses are in the hands of spammers and prepare defenses accordingly -- because even if that's not quite true yet, it will be soon enough. Notes: [1] I deliberately didn't mention mass WHOIS queries. While some efforts in this direction were made by spammers years ago, they've found it far more efficient and cost-effective to simply buy WHOIS data in bulk. There's always someone who wants to sell, and a CD/DVD or USB stick will suffice. This is why attempts by registrars to rate-limit queries or restrict access are not only foolish, but disengenuous: spammers already *have* the data, and can acquire updates at will, and they are clearly doing so via processes that lead back to registrars themselves. [2] The exact number of such systems is not only unknown, but unknowable, since any compromised system which (a) doesn't make its presence known (b) to a suitable detector will remain undetected indefinitely. However, two things are clear: (1) any estimate under 100 million should be laughed out of the room, and (2) there is no reason to suspect that the number is decreasing, and there are numerous reasons to suspect that it's increasing. Note, incidentally, that some detectors have reported observing 200,000 new such systems in a single day; and further note that it's now quite routine for individual botnets with several million *known* members to turn up. [3] Addresses which aren't used may remain out of spammer view for considerable time, depending on the care with which they're selected and maintained. However, this obviously excludes addresses used for participation in mailing lists. [4] For the purpose of this discussion, I'm just talking about suppression lists which enumerate individual email addresses. It's well-known that spammers also maintain suppression lists of MX's, domains, network allocations, ASNs, etc., in an attempt to avoid hitting spamtraps and/or hitting the mailboxes of those who might be in a position to file complaints or take action against them. [5] The only people left who are impeded in the slightest by obfuscation code are NON-spammers: that is, people who are trying to contact someone who has previously sent a message to some mailing list. ---Rsk From lists at psycho.i21k.de Mon Aug 24 20:33:31 2009 From: lists at psycho.i21k.de (Bernd Siggy Brentrup) Date: Mon, 24 Aug 2009 20:33:31 +0200 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <20090824143731.GA1949@gsp.org> References: <20090824143731.GA1949@gsp.org> Message-ID: <20090824183331.GB24556@puntila.winnegan.fake> On Mon, Aug 24, 2009 at 10:37 -0400, Rich Kulawiec wrote: > Summary: Spammers now have so many ways of "harvesting" addresses from so > many systems, and so many ways of exchanging those with each other, that > any email address which is actually used WILL eventually be harvested. > (Where what "eventually" means varies widely, of course, but can be > expected to steadily decrease.) Pretending that address obfuscation > in mailing list [or newsgroup] archives will have any meaningful effect > on this process gives users a false sense of security and has zero > anti-spam value. Just a little sample: usually I obfuscated addresses in my .signature. Due to the same arguments you are elaborating I used an unobfuscated one for 3 days. Now this address is contaminated and I'm refraining to obfuscating. my 2? Siggy -- O< ascii ribbon campaign - stop html mail - www.asciiribbon.org+ |53 days until|Open Source in Northern Germany: www.free-it.org| |www.Ubucon.de| tech contact: bsb-at-free-dash-it-dot-de| +-------> ceterum censeo javascriptum esse restrictam <--------+ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From barry at list.org Mon Aug 24 23:16:03 2009 From: barry at list.org (Barry Warsaw) Date: Mon, 24 Aug 2009 17:16:03 -0400 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <2332816B-4DE1-4016-B2AA-3222DEBE8265@list.org> On Aug 22, 2009, at 2:25 PM, Stephen J. Turnbull wrote: > Shouldn't these release messages for 3.0 alphas redirect to > mailman-developers? They should, yes. The next ones will. > Barry Warsaw writes: > >> Please note that this is an alpha release and as such is not ready >> for >> production use. >> >> You can get the code from the Cheeseshop: > > And do what with it? Check it in to a git repo, right? I mean, you > did say you were looking forward to content contributions! ;-) :) > (2) README.txt refers to a bunch of files in docs/readme/ that don't > exist (including the directory itself). If you're planning to > reorganize the docs directory, now might be as good a time as any > to fix the README. If you plan to populate it instead, you could > add a note that volunteers to write the as yet nonexistent docs > would be welcome. I've done some reorganization of the documentation. The doc build process is a little wonky still, as is the documentation itself. For example, I think we need a lot more overview docs than we have now. Most of what we have now is doctest output. But in any case, I've uploaded the current set of documentation to the Cheeseshop: http://packages.python.org/mailman/docs/README.html Volunteers are definitely wanted to help smooth out the build process, theme the documentation to "Mailman style" and help fill out the content. > (3) docs/ALPHA.txt says you need an unofficial branch of lazr.config > ... which doesn't seem to exist: It's a lie. Mailman 3 now uses the standard lazr.config package. Documentation is updated. So bin/buildout will DTRT. > *sigh* I guess I'll just be a good little boy and get the PyPI > package, but ... nah, I'll just go to bed and when I wake up I'll > discover it was all a bad dream, right? > Yes, that's it! A bad dream. Check out http://packages.python.org/mailman/docs/ALPHA.html which should be much more helpful now. what-can-this-strange-device-be?-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 24 23:19:24 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 24 Aug 2009 17:19:24 -0400 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <20090823043455.GB14594@puntila.winnegan.fake> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> <20090822195456.GB5091@puntila.winnegan.fake> <87ljlb2w5a.fsf@uwakimon.sk.tsukuba.ac.jp> <20090823043455.GB14594@puntila.winnegan.fake> Message-ID: On Aug 23, 2009, at 12:34 AM, Bernd Siggy Brentrup wrote: > Looking at the python2.6[1] page in Debian's PTS you'll notice it is > still in experimental so don't hold your breath waiting for it > appearing in testing. Dang. Too bad Python is so hard to install from source. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 24 23:21:09 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 24 Aug 2009 17:21:09 -0400 Subject: [Mailman-Developers] RELEASED GNU Mailman 3.0 alpha 3 (Working Man) In-Reply-To: <87iqgf2ajn.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4ACF8ADB-583D-4E75-909A-D0A234009B76@list.org> <87ocq73du7.fsf@uwakimon.sk.tsukuba.ac.jp> <20090822195456.GB5091@puntila.winnegan.fake> <87ljlb2w5a.fsf@uwakimon.sk.tsukuba.ac.jp> <20090823043455.GB14594@puntila.winnegan.fake> <87iqgf2ajn.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <8D651358-C9E1-401A-88AE-9869011FC995@python.org> On Aug 23, 2009, at 4:34 AM, Stephen J. Turnbull wrote: >> Looking at the python2.6[1] page in Debian's PTS you'll notice it is >> still in experimental so don't hold your breath waiting for it >> appearing in testing. > > I didn't plan to. I don't actually mind installing my own Python. > It's just that I've been procrastinating about doing something active > on MM3 for months, and on the spur of the moment decided NOW was the > time. I wasn't expecting that I'd be spending an hour or so just > installing Python (I needed to install a bunch of -dev packages, etc.) > and was frustrated. :-? Steve, it will be awesome to get your input on MM3. I /think/ you should be able to do a straight up from-source Python 2.6 install, and then run the instructions on the ALPHA.txt page. If you hit any glitches, please do let me know. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From hopkinsju at umsystem.edu Mon Aug 24 20:15:03 2009 From: hopkinsju at umsystem.edu (Hopkins, Justin) Date: Mon, 24 Aug 2009 13:15:03 -0500 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code fromMailman 3 In-Reply-To: <20090824143731.GA1949@gsp.org> References: <20090824143731.GA1949@gsp.org> Message-ID: Thanks for such a detailed and compelling post..but I must disagree. I can't refute any of the arguments you made, they are all quite sound, but I do take issue with your conclusion. Obfuscating the email addresses is just a part of 'defense in depth' - same as patching your computer, using a firewall, etc. Each layer, no matter how thin, still adds something. Cheers, Justin From stephen at xemacs.org Tue Aug 25 07:35:49 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 25 Aug 2009 14:35:49 +0900 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <20090824143731.GA1949@gsp.org> References: <20090824143731.GA1949@gsp.org> Message-ID: <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> Rich Kulawiec writes: > Pretending that address obfuscation in mailing list [or newsgroup] > archives will have any meaningful effect on this process gives > users a false sense of security and has zero anti-spam value. You're missing the point. Our (often non-technical) users demand this feature. Even our technical audience (see Siggy's parallel post for example) perceives benefits from obfuscation, based on empirical tests. So you can explain why, in theory and in practice, obfuscation doesn't work. But the user base will (stubbornly, if you like) refuse to accept your logic. From stephen at xemacs.org Tue Aug 25 07:44:34 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 25 Aug 2009 14:44:34 +0900 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code fromMailman 3 In-Reply-To: References: <20090824143731.GA1949@gsp.org> Message-ID: <874orw30sd.fsf@uwakimon.sk.tsukuba.ac.jp> Justin Hopkins writes: > Obfuscating the email addresses is just a part of 'defense in > depth' - same as patching your computer, using a firewall, > etc. Each layer, no matter how thin, still adds something. That's true. Rich's argument is more subtle than a claim that obfuscation is worth nothing, though. It is that benefits to obfuscation are small, and the cost is significantly larger than the benefit. You have to address the issue of the cost (obfuscating the address obstructs legitimate third-party users) as well. Note that the other strategies you mention -- patches, firewalls, etc -- do not impose costs on third parties, only on you. Personally, I subscribe to Rich's argument. I do not obfuscate my own addresses, and I argue against it when I have input into policy for processes like archiving mailing list posts. But Mailman needs to serve people who have different cost/benefit tradeoffs than Rich and I do -- I agree with you and Bernd that Mailman should provide the facility (though I would advise against relying on it, and generally deprecate its use, myself). From barry at list.org Tue Aug 25 12:39:29 2009 From: barry at list.org (Barry Warsaw) Date: Tue, 25 Aug 2009 06:39:29 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Aug 25, 2009, at 1:35 AM, Stephen J. Turnbull wrote: > Rich Kulawiec writes: > >> Pretending that address obfuscation in mailing list [or newsgroup] >> archives will have any meaningful effect on this process gives >> users a false sense of security and has zero anti-spam value. > > You're missing the point. Our (often non-technical) users demand this > feature. Even our technical audience (see Siggy's parallel post for > example) perceives benefits from obfuscation, based on empirical > tests. > > So you can explain why, in theory and in practice, obfuscation doesn't > work. But the user base will (stubbornly, if you like) refuse to > accept your logic. As usual, Stephen hits the nail on the head. I can't disagree with much in Rich's post, and yet it's likely that we'll still obfuscate and/or conceal email addresses in the archives because users will demand it. You can and should educate them, but this is not a battle I wish to fight because I think we can't win it. The costs of obfuscation are 1) increased code complexity; 2) denying legitimate third party uses. 1) is not insignificant. Regexp filters are tricky/impossible to get 100% right, but not too bad to get maybe 90% right. They are low fidelity because scanning headers isn't enough; people embed email addresses in all kinds of weird places in the body and HTML filtering is brain hurty. Obfuscation techniques will be busted so only concealment is future proof. This is all pretty boring coding though. 2) is more interesting. What kinds of uses are we talking about? You see a message in an archive from three years ago and you want to contact the OP about it? Why not just follow up and contact the mailing list? IOW, if there was an easy way to inject yourself into an old thread, perhaps one that was created before you joined the list, wouldn't that cover a large part of the use case? Do you want to be contacted off-list for on-list topics? Well, things like an email forwarding service could solve that, although I think it's not worth the effort as much as the first use case. What other kinds of legitimate third party uses does obfuscation/concealment prevent? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From iane at sussex.ac.uk Tue Aug 25 12:36:07 2009 From: iane at sussex.ac.uk (Ian Eiloart) Date: Tue, 25 Aug 2009 11:36:07 +0100 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code fromMailman 3 In-Reply-To: References: <20090824143731.GA1949@gsp.org> Message-ID: <328E334F56C9671300C638FD@lewes.staff.uscs.susx.ac.uk> --On 24 August 2009 13:15:03 -0500 "Hopkins, Justin" wrote: > Thanks for such a detailed and compelling post..but I must disagree. I > can't refute any of the arguments you made, they are all quite sound, but > I do take issue with your conclusion. > > Obfuscating the email addresses is just a part of 'defense in depth' - > same as patching your computer, using a firewall, etc. Each layer, no > matter how thin, still adds something. > > Cheers, > Justin Quite right. Rich's argument is, essentially, that obfuscation isn't 100% effective so it shouldn't be used. Frankly, if it's 10% effective, then it's worth doing in my view. Further, Rich offers no evidence of significant harm done by obfuscation. Finally, there are other privacy concerns than spam harvesting that may also be mitigated by address obfuscation. -- Ian Eiloart IT Services, University of Sussex 01273-873148 x3148 For new support requests, see http://www.sussex.ac.uk/its/help/ From stephen at xemacs.org Tue Aug 25 14:30:29 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 25 Aug 2009 21:30:29 +0900 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > 2) is more interesting. What kinds of uses are we talking about? You > see a message in an archive from three years ago and you want to > contact the OP about it? Why not just follow up and contact the > mailing list? For all the reasons why Reply-To Munging Considered Harmful. > Do you want to be contacted off-list for on-list topics? Well, things > like an email forwarding service could solve that, although I think > it's not worth the effort as much as the first use case. What other > kinds of legitimate third party uses does obfuscation/concealment > prevent? Obfuscation is a minor annoyance, but concealment is problematic in cases where the email is the identity, eg, matching list posts to issue tracker IDs. For example, I signed up for and log in to Launchpad as "stephen at xemacs.org", but I have to tell bzr that my ID is "stephen-xemacs". Wow, that's transparent. But at least it's guessable. Getting from "Stephen J. Turnbull " to "stephen-xemacs" is not going to be easy if you don't already know me. From skip at pobox.com Tue Aug 25 13:42:12 2009 From: skip at pobox.com (skip at pobox.com) Date: Tue, 25 Aug 2009 06:42:12 -0500 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code fromMailman 3 In-Reply-To: <328E334F56C9671300C638FD@lewes.staff.uscs.susx.ac.uk> References: <20090824143731.GA1949@gsp.org> <328E334F56C9671300C638FD@lewes.staff.uscs.susx.ac.uk> Message-ID: <19091.52756.137869.981735@montanaro.dyndns.org> Ian> Quite right. Rich's argument is, essentially, that obfuscation Ian> isn't 100% effective so it shouldn't be used. Frankly, if it's 10% Ian> effective, then it's worth doing in my view. I would be quite surprised if address obfuscation is anywhere close to 10% effective. Maybe 0.01%. The problem I see with Barry's argument that users demand it so Mailman must provide it is that position just propagates misinformation about the ineffectiveness of the "feature". I would vote for tossing it out, or at the very least making it a per-list flag which admins could disable if they wanted. The other thing about Mailman's obfuscation is that I sorta think that by now the spammers have figured it out. I mean, "skip at pobox.com"? Come on. Even Barry stands a good chance of writing a regular expression that can locate something like that, his self-deprecation about his r.e. prowess notwithstanding. :-) If nothing else, all an enterprising spammer would have to do is steal Mailman's email address matcher and replace "@" with " at ". Oh, wait, it's open source. They wouldn't even have to steal the code. -- Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/ Getting old sucks, but it beats dying young From bob at nleaudio.com Tue Aug 25 18:48:56 2009 From: bob at nleaudio.com (Bob Puff) Date: Tue, 25 Aug 2009 12:48:56 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code fromMailman 3 In-Reply-To: <19091.52756.137869.981735@montanaro.dyndns.org> References: <20090824143731.GA1949@gsp.org> <328E334F56C9671300C638FD@lewes.staff.uscs.susx.ac.uk> <19091.52756.137869.981735@montanaro.dyndns.org> Message-ID: <20090825162910.M19348@nleaudio.com> You are presuming too much on spammers as a whole. I've dealt with a couple spammers, and they just used some tools they got online that search for username at domain.something. Everything else is ignored. I don't for a minute doubt that the advanced spammers will snag anything and everything no matter how strange it is obfusticated (sp?). But there are a LOT of low-tech spammers still out there, and there is enough "low hanging fruit" for them that this little bit we are discussing can be over their head. Bob ---------- Original Message ----------- From: skip at pobox.com To: Ian Eiloart Cc: mailman-developers at python.org, Rich Kulawiec Sent: Tue, 25 Aug 2009 06:42:12 -0500 Subject: Re: [Mailman-Developers] Proposed: remove address-obfuscation code fromMailman 3 > Ian> Quite right. Rich's argument is, essentially, that obfuscation > Ian> isn't 100% effective so it shouldn't be used. Frankly, if > it's 10% Ian> effective, then it's worth doing in my view. > > I would be quite surprised if address obfuscation is anywhere close > to 10% effective. Maybe 0.01%. > > The problem I see with Barry's argument that users demand it so > Mailman must provide it is that position just propagates > misinformation about the ineffectiveness of the "feature". I would > vote for tossing it out, or at the very least making it a per-list > flag which admins could disable if they wanted. > > The other thing about Mailman's obfuscation is that I sorta think > that by now the spammers have figured it out. I mean, "skip at > pobox.com"? Come on. Even Barry stands a good chance of writing a > regular expression that can locate something like that, his self- > deprecation about his r.e. prowess notwithstanding. :-) If nothing > else, all an enterprising spammer would have to do is steal > Mailman's email address matcher and replace "@" with " at ". Oh, > wait, it's open source. They wouldn't even have to steal the code. > > -- > Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/ > Getting old sucks, but it beats dying young > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/bob%40nleaudio.com > > Security Policy: http://wiki.list.org/x/QIA9 ------- End of Original Message ------- From julian at mehnle.net Tue Aug 25 23:02:01 2009 From: julian at mehnle.net (Julian Mehnle) Date: Tue, 25 Aug 2009 21:02:01 +0000 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <20090825162910.M19348@nleaudio.com> References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> Message-ID: <200908252102.05115.julian@mehnle.net> Bob Puff wrote: > You are presuming too much on spammers as a whole. I've dealt with a > couple spammers, and they just used some tools they got online that > search for username at domain.something. Everything else is ignored. > > I don't for a minute doubt that the advanced spammers will snag > anything and everything no matter how strange it is obfusticated (sp?). > But there are a LOT of low-tech spammers still out there, and there is > enough "low hanging fruit" for them that this little bit we are > discussing can be over their head. It's not. Spammers usually don't do address harvesting themselves nowadays, but outsource it to botnets (just like they outsource the spamming itself to botnets) that are running kind of "off the shelf" software tailored to the task. Today, as a spammer you go out and buy those services in online shops, paying by credit card. And parsing "localpart at domain" is among the most trivial things current harvester modules do. Any wanna-be spammers who still run their garage business with self written tools are pretty much meaningless in terms of magnitude. If anything, this kind of obfuscation is an inconvenience to legitimate users, but certainly not to spammers. -Julian -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From iane at sussex.ac.uk Wed Aug 26 11:57:06 2009 From: iane at sussex.ac.uk (Ian Eiloart) Date: Wed, 26 Aug 2009 10:57:06 +0100 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <200908252102.05115.julian@mehnle.net> References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> <200908252102.05115.julian@mehnle.net> Message-ID: <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> --On 25 August 2009 21:02:01 +0000 Julian Mehnle wrote: > Bob Puff wrote: > >> You are presuming too much on spammers as a whole. I've dealt with a >> couple spammers, and they just used some tools they got online that >> search for username at domain.something. Everything else is ignored. >> >> I don't for a minute doubt that the advanced spammers will snag >> anything and everything no matter how strange it is obfusticated (sp?). >> But there are a LOT of low-tech spammers still out there, and there is >> enough "low hanging fruit" for them that this little bit we are >> discussing can be over their head. > > It's not. Spammers usually don't do address harvesting themselves > nowadays, but outsource it to botnets (just like they outsource the > spamming itself to botnets) that are running kind of "off the shelf" > software tailored to the task. Today, as a spammer you go out and buy > those services in online shops, paying by credit card. And parsing > "localpart at domain" is among the most trivial things current harvester > modules do. > > Any wanna-be spammers who still run their garage business with self > written tools are pretty much meaningless in terms of magnitude. > > If anything, this kind of obfuscation is an inconvenience to legitimate > users, but certainly not to spammers. > > -Julian There's recently published research which suggests that simple obfuscation can be effective. Concealment, presumably, is more effective. At you can download "Spamology: A Study of Spam Origins" They say "Surprisingly, even simple email obfuscation approaches are still sufficient today to prevent spammers from harvesting emails." and "Commonly-used email obfuscation techniques are offering protection (for now). It is common practice to replace the conventional @ in email addresses by an AT in order to defeat email harvesting. We found that the spammers are still not parsing simple obfuscations as of now. However, one should not count on the protection offered by such simple obfuscation schemes, for they are trivial to defeat." Of course, list posts hang around for a long time, and may be mirrored (eg by Google caching). Therefore, concealment seems more sensible than obfuscation. Perhaps a captcha could be used to reveal sender addresses, for example. The paper might be more interesting for its discussion of techniques for detecting (eg with honeypots) and defeating harvesters. -- Ian Eiloart IT Services, University of Sussex 01273-873148 x3148 For new support requests, see http://www.sussex.ac.uk/its/help/ From barry at list.org Thu Aug 27 18:05:00 2009 From: barry at list.org (Barry Warsaw) Date: Thu, 27 Aug 2009 12:05:00 -0400 Subject: [Mailman-Developers] Sysadmin/Mailman gig Message-ID: <7EA5CDD4-91FB-4E86-A5B0-EAB0AB564901@list.org> A friend of mine forwarded this sysadmin/mailman gig announcement to me. Apparently the Mailman component is fairly high priority. If you're interested, please respond directly to them -- I have no connection to or vested interest in Dreamfish. http://network.dreamfish.com/wiki/Lead_Systems_Administrator_-_Dreamfish_Service_Job -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From rsk at gsp.org Thu Aug 27 15:58:17 2009 From: rsk at gsp.org (Rich Kulawiec) Date: Thu, 27 Aug 2009 09:58:17 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> <200908252102.05115.julian@mehnle.net> <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> Message-ID: <20090827135817.GA6651@gsp.org> On Wed, Aug 26, 2009 at 10:57:06AM +0100, Ian Eiloart wrote: > There's recently published research which suggests that simple > obfuscation can be effective. Concealment, presumably, is more effective. > At you can download "Spamology: A Study of Spam > Origins" I'm composing a combined reply to all of the comments here, but wish to reply to this single point separately. This paper seems well-intentioned, but has some very serious problems -- any one of which is sufficient to dismiss its conclusions entirely. Let me just enumerate a few of them; I'll spare you the entire list. 1. The authors presume that they can tell that an address has been harvested *and* added to at least one spammer database (or not) by observing spam sent to it. But that's wrong: we know that many addresses are harvested and never spammed, or not spammed for a very long time (as in "years"). Conversely, many addresses are spammed that have *never* been harvested. And some addresses that are harvested are spammed, but not because they were harvested. [1] And some addresses are picked up by routine/ordinary web crawlers, and then subsequently spammed, but not by the people running those crawlers. [2] This invalidates their measurement technique. 2. There's a major methodology error here: "We began by registering a dedicated domain for this project, which we hosted on servers in our department." We know that some spammers -- the competent ones, who are the ones that matter -- use suppression lists based not just on domains, but TLDs, IP addresses, network allocations, ASNs, NS records, MX records, etc. We further know that anything tracing to a .edu or a network allocation/ASN associated with a .edu is quite likely to appear on those suppression lists. (This is an "old tradition" among spammers. Not all of them follow it, but quite a few do.) This also invalidates their measurement technique. 3. Statistics from any single domain are often wildly skewed one way or another. For example: I happen to host three domains which have the same name, but in three different TLDs. Everything else about them is exactly the same: NS, MX, web content, valid email addresses, etc. The spam they receive varies over three orders of magnitude. 4. And then there's this: it doesn't cover use of the single largest current vector for address harvesting -- zombied systems. No discussion of contemporary address harvesting techniques can even be begun without considering this. It's like writing a paper on tides without factoring in the moon's gravitation. [3] (I checked to see if perhaps this paper's publication predated the rise of the zombies earlier this decade, but it's from 2009.) To put it another way: yes, there are still address harvesters using the techniques that these researchers were looking for. But these harvesters are outdated and unimportant; they're only used by spammers who don't have the expertise and resources to do better. And not only is that class of spammer is steadily shrinking, that's NOT the class of spammer we need to worry about, as it's quite easy to block just about all their traffic whether they have valid addresses or not. (C'mon, these are people who can't decode rskATgsp.org, do you really think they constitute a serious threat?) So like I said above, I'll spare you points 5-N, but they're similar. None of what I've said here is new or novel: it's common knowledge among experienced people working in the field. I think perhaps in the future that people trying to conduct this kind of research should spend a few years reading spam-l and other similar lists before diving in. The bottom line is that (a) the numbers they've produced have no meaning and (b) their conclusions are all wrong. Notes: [1] As an example: conside joe at example.com, and let's suppose that it's been deliberately exposed to one method of harvesting because it's published at http://www.example.com. If spam arrives, then it may be because the address was harvested by a web crawler and added to a spammer database -- or it may be because "joe" is very common LHS string and thus one that spammers are very likely to try in *any* domain. Note that while spammers' list of such likely LHS were quite limited years ago, they're not any more: spammers now have the resources to try all known and all plausible LHS strings if they wish. And they are: check your logs. You may be surprised at which LHS strings are being tried: what was computationally infeasible a decade ago is now routine. [2] It's not difficult to figure out who's running a web crawler: just setting up a web site, making sure it's linked to, waiting, and then analyzing logs will reveal a candidate list. It's somewhat more work to figure out which of those crawler operations can be broken into, but it has significant advantages: it allows one to mine all their data without the expense/hassle of collecting it, and it conceals the source/use of that data. There are a lot of crawler operations out there. It would be silly to think that they're *all* secure. [3] Harvesting addresses on zombies has quite a few advantages over other techniques: It uses the host's own resources. It's unlikely to be detected. It won't be stopped by firewalls or rate-limiting at the network level. It provides social graph information. It provides timestamp information. It provides MUA information. It may yield useful phishing information. It may yield useful identity theft information. It may yield useful blackmail information. And all of this can be bundled up by suitable extraction software and delivered as a package back to a C&C node. For example, from a single email message sitting on Fred's computer: Fred last received email from Barney at 2009-08-11 07:32:12 UTC, thus Barney's address is known-good as of that time, Fred will probably accept suitably-forged mail from Barney, and vice-versa. And of course since Fred's computer is now owned by spammers, no anti-forgery mechanism of any kind will detect the latter. And maybe an appropriate malware payload from Fred to Barney will yield another zombie, where "appropriate" may be partially inferred by checking the headers and seeing what MUA Barney is using. Maybe those headers will also identify what MTA and associated anti-malware software Barney's site is using, so that the payload can be appropriately chosen. Phishing bonus if Barney's address is barney at some-bank or similar. Blackmail bonus if Barney's address is on an "adult dating" or "escort" site. Identity theft bonus if regexp matching on message-body turns up NNN-NN-NNNN (US social security number) and the like. &etc. Now multiply this by a billion. At least -- because there are at least a hundred million zombies and estimating only 10 stored messages per zombie gets us to a billion. This is why the serious/"professional" address harvesting operations have shifted from some of the older and less efficient techniques to this one, and why defending against those methods is now pointless. ---Rsk From barry at list.org Sat Aug 29 00:03:29 2009 From: barry at list.org (Barry Warsaw) Date: Fri, 28 Aug 2009 18:03:29 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> On Aug 25, 2009, at 8:30 AM, Stephen J. Turnbull wrote: >> 2) is more interesting. What kinds of uses are we talking about? >> You >> see a message in an archive from three years ago and you want to >> contact the OP about it? Why not just follow up and contact the >> mailing list? > > For all the reasons why Reply-To Munging Considered Harmful. What I'm thinking is that there should be a "send me this message" link in the archive, which gets you a copy as it was originally sent to the list. That let's you jump into a conversation as if you'd been there originally. Something like this would be cool for another reason. Assuming you could trust the long term storage at the archive site (enough) it would eliminate the last reason why I locally archive any public mailing list messages. >> Do you want to be contacted off-list for on-list topics? Well, >> things >> like an email forwarding service could solve that, although I think >> it's not worth the effort as much as the first use case. What other >> kinds of legitimate third party uses does obfuscation/concealment >> prevent? > > Obfuscation is a minor annoyance, but concealment is problematic in > cases where the email is the identity, eg, matching list posts to > issue tracker IDs. > > For example, I signed up for and log in to Launchpad as > "stephen at xemacs.org", but I have to tell bzr that my ID is > "stephen-xemacs". Wow, that's transparent. But at least it's > guessable. Getting from "Stephen J. Turnbull " to > "stephen-xemacs" is not going to be easy if you don't already know me. True. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Sat Aug 29 00:08:05 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 28 Aug 2009 18:08:05 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code fromMailman 3 In-Reply-To: <19091.52756.137869.981735@montanaro.dyndns.org> References: <20090824143731.GA1949@gsp.org> <328E334F56C9671300C638FD@lewes.staff.uscs.susx.ac.uk> <19091.52756.137869.981735@montanaro.dyndns.org> Message-ID: <85EA12BE-C8F1-4780-97D3-E822403B2B41@python.org> On Aug 25, 2009, at 7:42 AM, skip at pobox.com wrote: > The other thing about Mailman's obfuscation is that I sorta think > that by > now the spammers have figured it out. I mean, "skip at pobox.com"? > Come > on. Even Barry stands a good chance of writing a regular expression > that > can locate something like that, his self-deprecation about his r.e. > prowess > notwithstanding. :-) If nothing else, all an enterprising spammer > would > have to do is steal Mailman's email address matcher and replace "@" > with " > at ". Oh, wait, it's open source. They wouldn't even have to steal > the > code. I've always wanted to re-architect the archives so that they would / always/ vend the messages from an active process. I wouldn't have any static files, except a cache for efficiency, and I would generate the HTML on demand. My guess is that 99% of all archived messages are never read by a human. The problem of course is spiders but I guess they'll just warm up your cache. ;/ This would allow: * easy redeployment of new obfuscation techniques * on demand take downs or sanitization * easy site regeneration for style changes. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Sat Aug 29 03:46:01 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 28 Aug 2009 21:46:01 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <20090827135817.GA6651@gsp.org> References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> <200908252102.05115.julian@mehnle.net> <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> <20090827135817.GA6651@gsp.org> Message-ID: <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> Something else that occurs to me. If we accept that obfuscation is worthless and stop doing it, then there's no reason we shouldn't make the raw mbox files available for anyone to download. Mailman used to do this, but we removed the feature due to user outcry. Now you can download the gzip monthly .txt files, but they are sanitized. If we stop obfuscating, is there any reason not to make the raw messages available for download? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From bob at nleaudio.com Sat Aug 29 04:20:22 2009 From: bob at nleaudio.com (Bob Puff) Date: Fri, 28 Aug 2009 22:20:22 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> <200908252102.05115.julian@mehnle.net> <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> <20090827135817.GA6651@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> Message-ID: <20090829020752.M10054@nleaudio.com> That's the logical progression of that argument, and is the good reason why obfuscation or even removal of parts is not only a good idea, its a necessity. Exposing raw email addresses in their normal form is real low-hanging fruit. Regardless of what I think, my clients will cry bloody murder if emails leak out. I had one person recently google their email address, and found a link to an archive file that should have been private. I had removed all links to the archives, but somehow Google found it, indexed it, and the guy threatened me with bloody murder if I didn't take it down. Sheesh. Bob ---------- Original Message ----------- From: Barry Warsaw To: Rich Kulawiec Cc: mailman-developers at python.org Sent: Fri, 28 Aug 2009 21:46:01 -0400 Subject: Re: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 > Something else that occurs to me. > > If we accept that obfuscation is worthless and stop doing it, then > there's no reason we shouldn't make the raw mbox files available for > anyone to download. Mailman used to do this, but we removed the > feature due to user outcry. Now you can download the gzip monthly > .txt files, but they are sanitized. If we stop obfuscating, is > there any reason not to make the raw messages available for download? > > -Barry ------- End of Original Message ------- From julian at mehnle.net Sat Aug 29 06:19:58 2009 From: julian at mehnle.net (Julian Mehnle) Date: Sat, 29 Aug 2009 04:19:58 +0000 Subject: [Mailman-Developers] =?utf-8?q?Proposed=3A_remove_address-obfusca?= =?utf-8?q?tion_code=09from_Mailman_3?= In-Reply-To: <20090829020752.M10054@nleaudio.com> References: <20090824143731.GA1949@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> Message-ID: <200908290420.04618.julian@mehnle.net> Bob Puff wrote: > That's the logical progression of that argument, and is the good reason > why obfuscation or even removal of parts is not only a good idea, its a > necessity. Exposing raw email addresses in their normal form is real > low-hanging fruit. > > Regardless of what I think, my clients will cry bloody murder if emails > leak out. I had one person recently google their email address, and > found a link to an archive file that should have been private. I had > removed all links to the archives, but somehow Google found it, indexed > it, and the guy threatened me with bloody murder if I didn't take it > down. Sheesh. There's robots.txt, you know? If this is just about user outcry, then robots.txt will fix it (since all legitimate search engines honor it). -Julian -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From jeff at jab.org Sat Aug 29 06:21:58 2009 From: jeff at jab.org (Jeff Breidenbach) Date: Fri, 28 Aug 2009 21:21:58 -0700 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <20090829020752.M10054@nleaudio.com> References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> <200908252102.05115.julian@mehnle.net> <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> <20090827135817.GA6651@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> Message-ID: > > the archives, but somehow Google found it, indexed it, and the guy > threatened > me with bloody murder if I didn't take it down. Yes. It is critical to keep user perception in mind. Specifically, if you don't keep email addresses off the global search engines, there will be a deluge of vocal complaints from users who neither care about nor understand the technical aspects. That can be as simple as robots.txt configuration, or as fancy as using a captcha based system to reveal addresses like the one offered by reCaptcha. But my main point is you need to cover the user perception angle almost independtly from the core technical aspects of anti-harvesting. For the record, I prefer keeping data as unadulterated as possible because it helps interoperability. But we also need to keep users happy. -Jeff From lists at psycho.i21k.de Sat Aug 29 07:10:38 2009 From: lists at psycho.i21k.de (Bernd Siggy Brentrup) Date: Sat, 29 Aug 2009 07:10:38 +0200 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> Message-ID: <20090829051038.GA6030@puntila.winnegan.fake> On Fri, Aug 28, 2009 at 18:03 -0400, Barry Warsaw wrote: > What I'm thinking is that there should be a "send me this message" > link in the archive, which gets you a copy as it was originally sent > to the list. That let's you jump into a conversation as if you'd > been there originally. Another use case comes up when coming back from temporarily disabled delivery where you want to participate in an ongoing discussion. I've always dreamed of a ml-request at listdomain function that retransmits any messages in References to me. It's clear that MM has to delegate this to the archiver. > Something like this would be cool for another reason. Assuming you > could trust the long term storage at the archive site (enough) it > would eliminate the last reason why I locally archive any public > mailing list messages. ... indicating your internet connection is by orders of magnitude better than mine :) To get on topic again: regarding address obfuscation in the archives, I noted: - obfuscate by default, - the archive admin may choose not to obfuscate but this fact will be stated clearly on every archive page ? la: Email addresses are visible per choice of mailto:archiv-owner. Regards Siggy -- O< ascii ribbon campaign - stop html mail - www.asciiribbon.org+ |48 days until|Open Source in Northern Germany: www.free-it.org| |www.Ubucon.de| tech contact: bsb-at-free-dash-it-dot-de| +-------> ceterum censeo javascriptum esse restrictam <--------+ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From stephen at xemacs.org Sat Aug 29 09:01:34 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 29 Aug 2009 16:01:34 +0900 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> Message-ID: <87praft86p.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > What I'm thinking is that there should be a "send me this message" > link in the archive, which gets you a copy as it was originally sent > to the list. That let's you jump into a conversation as if you'd been > there originally. I don't understand. Do you mean the raw message received by the list, or the processed message as distributed by the list? The former means you don't have RFC 2369 headers, etc. I'm not sure I understand what the efficacy of the latter is; does address-munging happen only in the archives? I find it hard to believe that could be at all effective, except for what I would think is an unusual case (a closed-subscription list with public archives). From CNulk at scu.edu Mon Aug 31 19:15:43 2009 From: CNulk at scu.edu (C Nulk) Date: Mon, 31 Aug 2009 10:15:43 -0700 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <200908290420.04618.julian@mehnle.net> References: <20090824143731.GA1949@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> <200908290420.04618.julian@mehnle.net> Message-ID: <4A9C053F.9080501@scu.edu> I am pretty sure allowing the raw email addresses to be available is going to go over like a lead balloon here. Anything (however minor) to help protect the users/clients email addresses is helpful despite what others think. It is fine if someone considers the obfuscation that Mailman uses is trivial, however, anything I can do to make it harder or more computationally time-invested to get the email address is better than giving it away. Sure bots are out there but if what I do helps slow down someones system to make them look at it (and hopefully get rid of the bot), then great. But at least give me the choice to be able to do it. I happened to like Barry's (?) earlier comment about the "send me this message" link. Or maybe "send my message to the original poster" link where you can click on the link, compose your message, and send it through Mailman all without the original sender's address. Mailman or whatever process can figure out the original sender and pass on the your message. Yes, I know it is more work that is why we have computers :) As for using robots.txt, hmm, it is not the legitimate search engines I care about, it is the search engines/crawlers that do not respect my robots.txt file that I care about. If I had an effective way to consistently identify those non-legitimate crawlers, I would add what I needed to drop them into my firewall as I recognized them. Chris Julian Mehnle wrote: > Bob Puff wrote: > > >> That's the logical progression of that argument, and is the good reason >> why obfuscation or even removal of parts is not only a good idea, its a >> necessity. Exposing raw email addresses in their normal form is real >> low-hanging fruit. >> >> Regardless of what I think, my clients will cry bloody murder if emails >> leak out. I had one person recently google their email address, and >> found a link to an archive file that should have been private. I had >> removed all links to the archives, but somehow Google found it, indexed >> it, and the guy threatened me with bloody murder if I didn't take it >> down. Sheesh. >> > > There's robots.txt, you know? If this is just about user outcry, then > robots.txt will fix it (since all legitimate search engines honor it). > > -Julian > > ------------------------------------------------------------------------ > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/cnulk%40scu.edu > > Security Policy: http://wiki.list.org/x/QIA9 From barry at python.org Mon Aug 31 20:22:52 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 31 Aug 2009 14:22:52 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> <200908252102.05115.julian@mehnle.net> <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> <20090827135817.GA6651@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> Message-ID: On Aug 29, 2009, at 12:21 AM, Jeff Breidenbach wrote: > Yes. It is critical to keep user perception in mind. Specifically, > if you > don't keep email addresses off the global search engines, there will > be a > deluge of vocal complaints from users who neither care about nor > understand > the technical aspects. That can be as simple as robots.txt > configuration, or > as fancy as using a captcha based system to reveal addresses like > the one > offered by reCaptcha. But my main point is you need to cover the user > perception angle almost independtly from the core technical aspects of > anti-harvesting. > > For the record, I prefer keeping data as unadulterated as possible > because > it helps interoperability. But we also need to keep users happy. Trust me, I'm keenly aware of this as I probably get 3x the nasty hate mail that most of you get. I try to be nice and patient and that usually calms people down. :) Mailman will always still collect the raw data for messages sent to the list. There are legitimate uses for allowing outsiders access to that data (say, the list is moving and you want to migrate the archives), so I think we always want to support this. The question is how much if any of the raw data does the general public get access to? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 31 20:25:19 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 31 Aug 2009 14:25:19 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <20090829051038.GA6030@puntila.winnegan.fake> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> <20090829051038.GA6030@puntila.winnegan.fake> Message-ID: On Aug 29, 2009, at 1:10 AM, Bernd Siggy Brentrup wrote: > On Fri, Aug 28, 2009 at 18:03 -0400, Barry Warsaw wrote: > >> What I'm thinking is that there should be a "send me this message" >> link in the archive, which gets you a copy as it was originally sent >> to the list. That let's you jump into a conversation as if you'd >> been there originally. > > Another use case comes up when coming back from temporarily disabled > delivery where you want to participate in an ongoing discussion. I've > always dreamed of a ml-request at listdomain function that retransmits > any messages in References to me. It's clear that MM has to delegate > this to the archiver. I dream of a 'vacation' setting where you could tell Mailman the start and end dates of your "delivery stop" and then those messages would just be forwarded to you (perhaps as a digest) upon your return. Almost exactly like what the US Post Office does IRL. >> Something like this would be cool for another reason. Assuming you >> could trust the long term storage at the archive site (enough) it >> would eliminate the last reason why I locally archive any public >> mailing list messages. > > ... indicating your internet connection is by orders of magnitude > better than mine :) And yet, it's never enough! :) > To get on topic again: regarding address obfuscation in the archives, > I noted: > > - obfuscate by default, > - the archive admin may choose not to obfuscate but this fact > will be stated clearly on every archive page ? la: > Email addresses are visible per choice of mailto:archiv-owner. Yep, something like that. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at list.org Mon Aug 31 20:29:23 2009 From: barry at list.org (Barry Warsaw) Date: Mon, 31 Aug 2009 14:29:23 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <87praft86p.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> <87praft86p.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Aug 29, 2009, at 3:01 AM, Stephen J. Turnbull wrote: > Barry Warsaw writes: > >> What I'm thinking is that there should be a "send me this message" >> link in the archive, which gets you a copy as it was originally sent >> to the list. That let's you jump into a conversation as if you'd >> been >> there originally. > > I don't understand. Do you mean the raw message received by the list, > or the processed message as distributed by the list? The former means > you don't have RFC 2369 headers, etc. I'm not sure I understand what > the efficacy of the latter is; does address-munging happen only in the > archives? I find it hard to believe that could be at all effective, > except for what I would think is an unusual case (a closed- > subscription > list with public archives). Yes, address munging only happens in the HTML archives and in the outgoing queue processor. Mailman keeps a copy of the raw received message which for MM2 is only in the mbox file, but for MM3 will be in a "message store". Let's say I just joined the XEmacs development mailing list after a long absence. I find a message in the archive from two years ago that is relevant to an issue I'm having. I'd like to follow up to that message using my normal mail toolchain, but I found the archive page through Google. I should be able to click on a link on that page, enter my email address (perhaps through some validation dance, or subject to a request governor) and then the message -- as it was originally copied to the list membership -- would show up in my inbox, exactly as if I were a list member at the time. Now I can hit 'reply' and inject myself seamlessly into that 2 year old thread. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 31 20:31:47 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 31 Aug 2009 14:31:47 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <4A9C053F.9080501@scu.edu> References: <20090824143731.GA1949@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> <200908290420.04618.julian@mehnle.net> <4A9C053F.9080501@scu.edu> Message-ID: <23492CEA-7577-4C01-8DE1-922321F21492@python.org> On Aug 31, 2009, at 1:15 PM, C Nulk wrote: > I am pretty sure allowing the raw email addresses to be available is > going to go over like a lead balloon here. Anything (however minor) > to > help protect the users/clients email addresses is helpful despite what > others think. It is fine if someone considers the obfuscation that > Mailman uses is trivial, however, anything I can do to make it > harder or > more computationally time-invested to get the email address is better > than giving it away. Sure bots are out there but if what I do helps > slow down someones system to make them look at it (and hopefully get > rid > of the bot), then great. But at least give me the choice to be > able to > do it. Agreed. > I happened to like Barry's (?) earlier comment about the "send me this > message" link. Or maybe "send my message to the original poster" link > where you can click on the link, compose your message, and send it > through Mailman all without the original sender's address. Mailman or > whatever process can figure out the original sender and pass on the > your > message. Yes, I know it is more work that is why we have computers :) The difficult part about the latter is that I hate web interfaces for reading/composing email (Gmail included). I want to use my mail reader for that! > As for using robots.txt, hmm, it is not the legitimate search > engines I > care about, it is the search engines/crawlers that do not respect my > robots.txt file that I care about. If I had an effective way to > consistently identify those non-legitimate crawlers, I would add > what I > needed to drop them into my firewall as I recognized them. Agreed. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mark at msapiro.net Mon Aug 31 21:00:01 2009 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 31 Aug 2009 12:00:01 -0700 Subject: [Mailman-Developers] Proposed: remove address-obfuscation codefrom Mailman 3 In-Reply-To: <23492CEA-7577-4C01-8DE1-922321F21492@python.org> Message-ID: Barry Warsaw wrote: > >On Aug 31, 2009, at 1:15 PM, C Nulk wrote: > >> As for using robots.txt, hmm, it is not the legitimate search >> engines I >> care about, it is the search engines/crawlers that do not respect my >> robots.txt file that I care about. If I had an effective way to >> consistently identify those non-legitimate crawlers, I would add >> what I >> needed to drop them into my firewall as I recognized them. > >Agreed. The point in the original post about robots.txt was that if you think obfuscation is undesirable and don't do it, but you get complaints from people who find their unobfuscated addresses on your pages via legitimate search engines, you can use robots.txt to keep the search engines out. However, robots.txt is not completely effective in this. You can use it to prevent Google from crawling your site or portions thereof, but it won't prevent Google from indexing your pages that it finds via external links. To prevent this, you need a tag on the pages themselves. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From dale at newfield.org Mon Aug 31 21:00:59 2009 From: dale at newfield.org (Dale Newfield) Date: Mon, 31 Aug 2009 15:00:59 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> <87praft86p.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4A9C1DEB.1090405@Newfield.org> Barry Warsaw wrote: > Let's say I just joined the XEmacs development mailing list after a long > absence. I find a message in the archive from two years ago that is > relevant to an issue I'm having. I'd like to follow up to that message > using my normal mail toolchain, but I found the archive page through > Google. I should be able to click on a link on that page, enter my > email address (perhaps through some validation dance, or subject to a > request governor) and then the message -- as it was originally copied to > the list membership -- would show up in my inbox, exactly as if I were a > list member at the time. > > Now I can hit 'reply' and inject myself seamlessly into that 2 year old > thread. As long as the mailing list name/address hasn't migrated/changed in the interim... ...perhaps the original message munged to ensure current accuracy of the to/cc/reply-to fields? -Dale From barry at python.org Mon Aug 31 21:27:07 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 31 Aug 2009 15:27:07 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <4A9C1DEB.1090405@Newfield.org> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> <87praft86p.fsf@uwakimon.sk.tsukuba.ac.jp> <4A9C1DEB.1090405@Newfield.org> Message-ID: <82453FB5-4153-47E5-A99A-5088163E84EA@python.org> On Aug 31, 2009, at 3:00 PM, Dale Newfield wrote: > Barry Warsaw wrote: >> Let's say I just joined the XEmacs development mailing list after a >> long absence. I find a message in the archive from two years ago >> that is relevant to an issue I'm having. I'd like to follow up to >> that message using my normal mail toolchain, but I found the >> archive page through Google. I should be able to click on a link >> on that page, enter my email address (perhaps through some >> validation dance, or subject to a request governor) and then the >> message -- as it was originally copied to the list membership -- >> would show up in my inbox, exactly as if I were a list member at >> the time. >> Now I can hit 'reply' and inject myself seamlessly into that 2 year >> old thread. > > As long as the mailing list name/address hasn't migrated/changed in > the interim... Good point. > ...perhaps the original message munged to ensure current accuracy of > the to/cc/reply-to fields? Not sure I understand; can you elaborate? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From CNulk at scu.edu Mon Aug 31 22:39:20 2009 From: CNulk at scu.edu (C Nulk) Date: Mon, 31 Aug 2009 13:39:20 -0700 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <23492CEA-7577-4C01-8DE1-922321F21492@python.org> References: <20090824143731.GA1949@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> <200908290420.04618.julian@mehnle.net> <4A9C053F.9080501@scu.edu> <23492CEA-7577-4C01-8DE1-922321F21492@python.org> Message-ID: <4A9C34F8.7040009@scu.edu> Barry Warsaw wrote: > On Aug 31, 2009, at 1:15 PM, C Nulk wrote: > >> I am pretty sure allowing the raw email addresses to be available is >> going to go over like a lead balloon here. Anything (however minor) to >> help protect the users/clients email addresses is helpful despite what >> others think. It is fine if someone considers the obfuscation that >> Mailman uses is trivial, however, anything I can do to make it harder or >> more computationally time-invested to get the email address is better >> than giving it away. Sure bots are out there but if what I do helps >> slow down someones system to make them look at it (and hopefully get rid >> of the bot), then great. But at least give me the choice to be able to >> do it. > > Agreed. > >> I happened to like Barry's (?) earlier comment about the "send me this >> message" link. Or maybe "send my message to the original poster" link >> where you can click on the link, compose your message, and send it >> through Mailman all without the original sender's address. Mailman or >> whatever process can figure out the original sender and pass on the your >> message. Yes, I know it is more work that is why we have computers :) > > The difficult part about the latter is that I hate web interfaces for > reading/composing email (Gmail included). I want to use my mail > reader for that! Actually, I had more of a mailto style link in mind that sends the message to the list (run by Mailman naturally) and as part of the body/subject include an encrypted form of the message id (providing it is unique). You would use your mail client to read/compose. Maybe something similar to a list's listname-bounces address but with the message id could be done. Don't know. Mailman would receive your message, decrypt the message id, look up the message, then forward your message to the original sender. I am not particularly fond of web interfaces for reading/composing email. Well, maybe when I travel overseas without a laptop, then it is minimally okay. > >> As for using robots.txt, hmm, it is not the legitimate search engines I >> care about, it is the search engines/crawlers that do not respect my >> robots.txt file that I care about. If I had an effective way to >> consistently identify those non-legitimate crawlers, I would add what I >> needed to drop them into my firewall as I recognized them. > > Agreed. > -Barry > Now, totally off-topic, anyone have a recommendation for a book on learning Python so I am no longer truly dangerous, just slightly. Thanks, Chris From dale at newfield.org Mon Aug 31 22:41:28 2009 From: dale at newfield.org (Dale Newfield) Date: Mon, 31 Aug 2009 16:41:28 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <82453FB5-4153-47E5-A99A-5088163E84EA@python.org> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> <87praft86p.fsf@uwakimon.sk.tsukuba.ac.jp> <4A9C1DEB.1090405@Newfield.org> <82453FB5-4153-47E5-A99A-5088163E84EA@python.org> Message-ID: <4A9C3578.80002@Newfield.org> Barry Warsaw wrote: >>> Now I can hit 'reply' and inject myself seamlessly into that 2 year >>> old thread. >> >> As long as the mailing list name/address hasn't migrated/changed in >> the interim... > > Good point. > >> ...perhaps the original message munged to ensure current accuracy of >> the to/cc/reply-to fields? > > Not sure I understand; can you elaborate? We can tell from a mailing list's configuration what the distribution address should be, but I guess we don't know what previous addresses it had, so it's not as simple as I was thinking to do this munging (I was thinking just a search/replace). Maybe the appropriate modifications from the original message would be to add as a "To" address the current list address iff it does not appear in the To or CC addresses in the archived message (and to re-set ReplyTo, if reply-to-munging is set). -Dale From barry at python.org Mon Aug 31 22:53:01 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 31 Aug 2009 16:53:01 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <4A9C34F8.7040009@scu.edu> References: <20090824143731.GA1949@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> <200908290420.04618.julian@mehnle.net> <4A9C053F.9080501@scu.edu> <23492CEA-7577-4C01-8DE1-922321F21492@python.org> <4A9C34F8.7040009@scu.edu> Message-ID: On Aug 31, 2009, at 4:39 PM, C Nulk wrote: > Now, totally off-topic, anyone have a recommendation for a book on > learning Python so I am no longer truly dangerous, just slightly. There are zillions of books available now for learning Python (I think there was only 1 when I first learned it 15 years ago :). http://wiki.python.org/moin/PythonBooks For various reasons, it's difficult for me to recommend one over the other. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From barry at python.org Mon Aug 31 22:53:56 2009 From: barry at python.org (Barry Warsaw) Date: Mon, 31 Aug 2009 16:53:56 -0400 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: <4A9C3578.80002@Newfield.org> References: <20090824143731.GA1949@gsp.org> <8763cc316y.fsf@uwakimon.sk.tsukuba.ac.jp> <87ws4s13fe.fsf@uwakimon.sk.tsukuba.ac.jp> <1166767B-B4CB-4329-9424-D9FA44E2A7F5@list.org> <87praft86p.fsf@uwakimon.sk.tsukuba.ac.jp> <4A9C1DEB.1090405@Newfield.org> <82453FB5-4153-47E5-A99A-5088163E84EA@python.org> <4A9C3578.80002@Newfield.org> Message-ID: <4FA82515-9F0C-41DF-90C5-085599017397@python.org> On Aug 31, 2009, at 4:41 PM, Dale Newfield wrote: > Maybe the appropriate modifications from the original message would > be to add as a "To" address the current list address iff it does not > appear in the To or CC addresses in the archived message (and to re- > set ReplyTo, if reply-to-munging is set). That seems reasonable. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From CNulk at scu.edu Mon Aug 31 22:54:06 2009 From: CNulk at scu.edu (C Nulk) Date: Mon, 31 Aug 2009 13:54:06 -0700 Subject: [Mailman-Developers] Proposed: remove address-obfuscation codefrom Mailman 3 In-Reply-To: References: Message-ID: <4A9C386E.2050502@scu.edu> Mark Sapiro wrote: > Barry Warsaw wrote: > >> On Aug 31, 2009, at 1:15 PM, C Nulk wrote: >> >> >>> As for using robots.txt, hmm, it is not the legitimate search >>> engines I >>> care about, it is the search engines/crawlers that do not respect my >>> robots.txt file that I care about. If I had an effective way to >>> consistently identify those non-legitimate crawlers, I would add >>> what I >>> needed to drop them into my firewall as I recognized them. >>> >> Agreed. >> > > > The point in the original post about robots.txt was that if you think > obfuscation is undesirable and don't do it, but you get complaints > from people who find their unobfuscated addresses on your pages via > legitimate search engines, you can use robots.txt to keep the search > engines out. > I understood the original post and I agree. > However, robots.txt is not completely effective in this. You can use it > to prevent Google from crawling your site or portions thereof, but it > won't prevent Google from indexing your pages that it finds via > external links. To prevent this, you need a content="noindex"> tag on the pages themselves. > I agree with you here. The robots.txt and the " References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> <200908252102.05115.julian@mehnle.net> <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> <20090827135817.GA6651@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> <20090831204801.GM1861@monkey.uchicago.edu> Message-ID: <46B5E105-9317-4531-9EDC-BDF359179139@python.org> On Aug 31, 2009, at 4:48 PM, David Champion wrote: > I'm going to embracing and extend something Barry suggested in > private mail. He suggested a list setting that permits signed-in > list subscribers to download raw archives if they have some > 'archive-approved' status. What if that is a three-way switch: > approved, unapproved, and blacklisted? New subscribers would always > be unapproved. An unapproved subscriber who successfully posted to > the list, clearing any approval mechanisms in place and subject to a > list configuration option, would get approved for raw archive access. > (Automatic posting-equals-approval would not be desirable for all > lists, but it would for many.) An approved user could be blacklisted > by moderator action or by an automated moderation filter. Coming off > blacklist status would require manual action by the moderator. > > And there could be a form in the application to request approval or > de-blacklisting, of course. Launchpad's mailing lists have a very similar concept, although it's not used for access to the archives. The concept there is called "standing" and currently has four levels: excellent, good, poor, and unknown. You start out with unknown standing, but after you prove yourself (in much the same way as you describe above), you get to be in good standing, which gives you other benefits, such as being able to email a list you're not on without moderation. You can't get to excellent standing on your own and there are currently no benefits of excellent over good standing. Poor standing is much like your blacklist idea. The way I look at it is that Launchpad prototyped this concept and I do think it could be useful in Mailman itself. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From dgc at uchicago.edu Mon Aug 31 22:48:01 2009 From: dgc at uchicago.edu (David Champion) Date: Mon, 31 Aug 2009 15:48:01 -0500 Subject: [Mailman-Developers] Proposed: remove address-obfuscation code from Mailman 3 In-Reply-To: References: <20090824143731.GA1949@gsp.org> <19091.52756.137869.981735@montanaro.dyndns.org> <20090825162910.M19348@nleaudio.com> <200908252102.05115.julian@mehnle.net> <28D6BEE0D626089CE1779FC5@lewes.staff.uscs.susx.ac.uk> <20090827135817.GA6651@gsp.org> <8BF2AC28-BFCD-4F0B-AB28-CA6D47B863C2@python.org> <20090829020752.M10054@nleaudio.com> Message-ID: <20090831204801.GM1861@monkey.uchicago.edu> * On 31 Aug 2009, Barry Warsaw wrote: > > Mailman will always still collect the raw data for messages sent to > the list. There are legitimate uses for allowing outsiders access > to that data (say, the list is moving and you want to migrate the > archives), so I think we always want to support this. The question > is how much if any of the raw data does the general public get > access to? It seems clear that there are legitimate use cases for raw archives, so I'll skip the justifications and just address how we can balance between transparency and security. I'm going to embracing and extend something Barry suggested in private mail. He suggested a list setting that permits signed-in list subscribers to download raw archives if they have some 'archive-approved' status. What if that is a three-way switch: approved, unapproved, and blacklisted? New subscribers would always be unapproved. An unapproved subscriber who successfully posted to the list, clearing any approval mechanisms in place and subject to a list configuration option, would get approved for raw archive access. (Automatic posting-equals-approval would not be desirable for all lists, but it would for many.) An approved user could be blacklisted by moderator action or by an automated moderation filter. Coming off blacklist status would require manual action by the moderator. And there could be a form in the application to request approval or de-blacklisting, of course. -- -D. dgc at uchicago.edu NSIT University of Chicago