From alexander at sulfrian.net Sun Apr 1 16:08:00 2012 From: alexander at sulfrian.net (Alexander Sulfrian) Date: Sun, 01 Apr 2012 16:08:00 +0200 Subject: [Mailman-Developers] GSoC 2012 - NNTP archive access In-Reply-To: <20120327165204.255d7527@resist.wooz.org> Message-ID: <87obrbilz3.wl%alexander@sulfrian.net> Hi, On Tue Mar 27 22:52:04 CEST 2012, Barry Warsaw wrote: > > On Mar 27, 2012, at 09:09 PM, Alexander Sulfrian wrote: > > > What are the next steps you would propose. I unfortunately not up > > to date with the development of mailman 3. But I am a little bit > > familiar with the mailman 2 source code. > > MM3 will be a better platform to build something like the NNTP > access on. The question in my mind is whether this should be done > as part of the various independent (but related) archiver projects, > or whether it should be done as a separate "archiver". there is a second question connected with that: Should the messages be kept in an additional storage for NNTP access or should the default archiver be responsible for storage and should be extended with methods for accessing specific messages? > In mm3, there's an API for feeding posted messages to an IArchiver, > but this is quite flexible. I could imagine that something on the > other end of this vended messages via NNTP instead of HTTP. This would be the scenario if implementing the NNTP access in a new archiver, separated from the other. > The one key difference is that you'd like to be able to post to the > mailing list through NNTP, with probably some additional posting > rules (e.g. if you're not a member, but we "know" you, or you've > been approved for posting a few times, your message wouldn't get > held for moderator approval). If it should be possible to post messages over the NNTP transport, that does not match the classic design of an archiver. I do not know, whether there is an API to post messages, but eventually it would be better to implement the NNTP archive as external module, that could maybe even run on a separate server. > If I was doing this, I'd probably looks seriously at Twisted as the > basis for implementing the NNTP side of things. I haven't looked in > quite a while, but at the time, it had great support for NNTP > server-side. Yes, twisted should be the right choice. There is a twisted module for implementing a NNTPServer[1], but it is not very well documented. But even if it is not working, it should not be hard to implement it. The NNTP commands described in RFC3977[2] do not look very complicated. Additional to that, there is also the question, whether it should be possible to sync a few mailman server over the NNTP protocol. That would be a possibility to do clustering for load balancing or something like that. > Cheers, > -Barry Thanks, Alex [1] http://twistedmatrix.com/documents/current/api/twisted.news.nntp.NNTPServer.html [2] http://tools.ietf.org/html/rfc3977 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From barry at list.org Sun Apr 1 21:05:15 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 1 Apr 2012 13:05:15 -0600 Subject: [Mailman-Developers] GSoC 2012 - NNTP archive access In-Reply-To: <87obrbilz3.wl%alexander@sulfrian.net> References: <20120327165204.255d7527@resist.wooz.org> <87obrbilz3.wl%alexander@sulfrian.net> Message-ID: <20120401130515.05305816@resist.wooz.org> BTW, the NNTP queue runner has now been ported to Mailman 3. You will need to re-run bin/buildout though, to pick up the new dependency on the mock library. On Apr 01, 2012, at 04:08 PM, Alexander Sulfrian wrote: >> MM3 will be a better platform to build something like the NNTP >> access on. The question in my mind is whether this should be done >> as part of the various independent (but related) archiver projects, >> or whether it should be done as a separate "archiver". > >there is a second question connected with that: Should the messages >be kept in an additional storage for NNTP access or should the default >archiver be responsible for storage and should be extended with methods >for accessing specific messages? This is a good, but larger question. I've always thought that Mailman will require a "message store" as defined in the IMessageStore interface. What might make sense is to have a single implementation that satisfies the IArchiver and IMessageStore (and possibly other interfaces), but with a single on-disk storage. This could in fact be the thing that backs the prototype archiver. >> In mm3, there's an API for feeding posted messages to an IArchiver, >> but this is quite flexible. I could imagine that something on the >> other end of this vended messages via NNTP instead of HTTP. > >This would be the scenario if implementing the NNTP access in a new >archiver, separated from the other. With the above, you probably wouldn't need this except as you say, if it is a separate archiver. >> The one key difference is that you'd like to be able to post to the >> mailing list through NNTP, with probably some additional posting >> rules (e.g. if you're not a member, but we "know" you, or you've >> been approved for posting a few times, your message wouldn't get >> held for moderator approval). > >If it should be possible to post messages over the NNTP transport, >that does not match the classic design of an archiver. I do not know, >whether there is an API to post messages, but eventually it would be >better to implement the NNTP archive as external module, that could >maybe even run on a separate server. Yes, now that the NNTPRunner is functional, it should be possible to set this up as posting to an NNTP service that a site could run, independent of Mailman. >> If I was doing this, I'd probably looks seriously at Twisted as the >> basis for implementing the NNTP side of things. I haven't looked in >> quite a while, but at the time, it had great support for NNTP >> server-side. > >Yes, twisted should be the right choice. There is a twisted module for >implementing a NNTPServer[1], but it is not very well documented. But >even if it is not working, it should not be hard to implement it. The >NNTP commands described in RFC3977[2] do not look very complicated. > >Additional to that, there is also the question, whether it should be >possible to sync a few mailman server over the NNTP protocol. That >would be a possibility to do clustering for load balancing or >something like that. That's a pretty cool idea, actually. Something fun to explore for 3.1 perhaps. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From terri at zone12.com Mon Apr 2 02:14:23 2012 From: terri at zone12.com (Terri Oda) Date: Sun, 01 Apr 2012 18:14:23 -0600 Subject: [Mailman-Developers] For prospective GSoC students Message-ID: <4F78EF5F.3040500@zone12.com> Some things you should know: 1. Mailman is working under the umbrella organization of the Python Software Foundation, so we get hundreds of applications to sort through not all of which are related to Mailman. Please make sure to put "Mailman" somewhere in the subject of your application so it doesn't get lost in the crowd! 2. On a related note, please make sure your application has a descriptive title. i.e. "GNU Mailman: improving archives by extending hyperkitty" or somesuch. Again, this makes it easier for us to sort through the applications in the system. 3. Applications are due April 6th. Google will not extend this deadline for any reason, including if the entire melange system goes down. (And this *has* happened at the last minute.) Please make sure to get your applications in early if you can! If you don't have an application in the system, there is no way we can accept you. 4. You can edit any application you submit, so what I recommend is that you all go and submit something right now with a note at the top saying that it is a draft. You can edit it after this is done, and when you're ready to finalize you can take the "this is a draft" note off the top. 5. Not all of us are set up as mentors in Melange yet (most PSF mentor accounts were only authorized today) so it may still be a couple of days before you get feedback. If you want feedback sooner, please feel free to post your proposal to this list! Terri From msk at cloudmark.com Mon Apr 2 23:58:55 2012 From: msk at cloudmark.com (Murray S. Kucherawy) Date: Mon, 2 Apr 2012 21:58:55 +0000 Subject: [Mailman-Developers] Presenting on anti-abuse developments Message-ID: <9452079D1A51524AA5749AD23E0039280C6A46@exch-mbx901.corp.cloudmark.com> Hi all, One of the hats I wear these days is technical committee co-chair for the Messaging Anti-Abuse Working Group (MAAWG). I'm looking to fill slots for our Berlin (June) and Baltimore (October) conferences. If someone on the mailman development team would like to come and speak about developments and features of Mailman (especially the new version) that try to deal with abuse mitigation issues, please contact me off-list. I have a request in to the executive director to find out what support we offer to speakers in terms of expenses, etc., so I'll pass that on once I have it to anyone that replies. Thanks, -MSK From terri at zone12.com Tue Apr 3 00:07:00 2012 From: terri at zone12.com (Terri Oda) Date: Mon, 02 Apr 2012 16:07:00 -0600 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: Message-ID: <4F7A2304.5060408@zone12.com> On 03/29/2012 02:27 PM, David Jeske wrote: > On Thu, Mar 29, 2012 at 9:16 AM, Stephen J. Turnbullwrote: >> I would say you should try to retain copyright, and have the Mailman >> project distribute it with the S-BSD license under the "mere >> aggregation" clause of the GPL. > This agrees with my view of the situation as well. Which leads to the > question, is the above approach interesting/viable for Mailman-team? > (assuming the code does something awesome that people want) If the question is just "would you like another archiver even if the licenses don't match?" then I believe the answer is yes. I think it would be really beneficial for us to have more than one archiver on the table sooner rather than later, and working with you to make sure all the plumbing is there to connect things would be really beneficial to us. The licensing issue might mean you're probably not guaranteed a blessing as the standard archiving utility for Mailman, but that never stopped other projects like MhonArc! But... since you arrived around the same time GSoC started, I should ask whether you were hoping to do this as a GSoC project? It'd be a worthwhile project to put out there, but it might be lower priority for us than more direct development, since one of the goals of GSoC is to get new developers who are going to stay around and do future work with the project. Terri From terri at zone12.com Tue Apr 3 00:37:04 2012 From: terri at zone12.com (Terri Oda) Date: Mon, 02 Apr 2012 16:37:04 -0600 Subject: [Mailman-Developers] Google Summer of Code: Integration of Search Code In-Reply-To: References: <20120326194107.GX11151@unaka.lan> <4F721331.60807@zone12.com> Message-ID: <4F7A2A10.6060202@zone12.com> On 03/29/2012 11:58 PM, Shayan Md wrote: > Okay then, can you please tell me how we can put this search code in best > use of mailman3? I have a proposal to write, I am getting unsure of things > day by day. Can you also tell me who is the mentor of this project? When it comes to writing your proposal, I'd be most impressed if you looked at search in terms of how it's going to be used. Take a look at some of the work generated by previous years students on search use cases: http://systers.org/systers-dev/doku.php/usecases-priya:start http://systers.org/systers-dev/doku.php/mailman_archives_ui_-_yian_shang These were generated from surveys of mailman users worldwide, so they probably show a reasonable picture of expected behaviour. Figure out what sort of data structures an indexes you need to support these use cases and work from there. Don't worry if it's not perfect; your best guess is fine for an initial application and we'll ask for clarification as necessary. Please also remember to give a reasonably detailed timeline for what you plan to do (e.g. weekly milestones) and how you will integrate code on a weekly or biweekly basis. That helps us a lot when evaluating your proposal! As for who will be mentoring search-related projects... we haven't decided. I was planning to just let the mentors fight for the best students once we have all the applications in. ;) More seriously, though, search touches on interests and expertise for pretty much all of our mentors, so the primary mentor for a search project will depend on what other applications we get. Terri From terri at zone12.com Tue Apr 3 00:46:31 2012 From: terri at zone12.com (Terri Oda) Date: Mon, 02 Apr 2012 16:46:31 -0600 Subject: [Mailman-Developers] [GSoC 2012] Candidate on 'Integration of (existing) search code into Mailman archives' In-Reply-To: References: Message-ID: <4F7A2C47.1070002@zone12.com> Hi George, Your MailmanStats project looks great and would totally fit with what we have in mind for stats, though I'm guessing the hyperkitty team has some much more extensive work in mind making use of post ratings, tags, etc. If you're putting together your proposal now, do feel free to mention both projects as sources of interest. Since you already have the stats code available, it might be possible to toss the integration in there after doing some other work. Normally I worry about students biting off more than they can chew, but given your prior experience with Mailman and the fact that you already have the basic code, you can make a case for being able to package up that code and contribute it in a week or two our of your summer if you're ready for a code review. Terri PS - For further advice regarding search projects, see my previous post to mailman-developers. On 03/26/2012 03:38 PM, George Chatzisofroniou wrote: > Hello Mailman Developers, > > My name is George Chatzisofroniou, i'm 20 years old and i'm an > undergraduate student in the Department of Informatics at the > University of Piraeus (Greece). > > ? have really good previous experience with Mailman. This is because i > use it for managing mailing lists for almost three years. > > I have also developed, with a friend of mine, MailmanStats [1], a > Python software that outputs statistics for a mailing list based on > Mailman. I think this implements the 'metric' idea in some way. I > would like to know your opinion about MailmanStats. > > I'm sending this mail to inform you about my will to be part of > Mailman Development team starting by Google Summer of Code 2012. The > idea that excites me more is the 'Integration of (existing) search > code into Mailman archives'. I think it is better to be developed on > Mailman v3 rather than v2. I realize the significance of a feature > like this. Many times before, i've got through the archives to search > for a specific thread, so an addition like this would be great! > > As another student mentioned this idea is kinda small for the whole > summer, so if there is time left i could integrate my MailmanStats [1] > software into Mailman and/or build CSS styles for the web UI. > > Please tell me what you think. I'm also on IRC by the name sophron. > > Thanks, > > [1]: http://mailmanstats.latthi.com/ > > > From davidj at gmail.com Tue Apr 3 05:04:23 2012 From: davidj at gmail.com (David Jeske) Date: Mon, 2 Apr 2012 20:04:23 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <4F7A2304.5060408@zone12.com> References: <4F7A2304.5060408@zone12.com> Message-ID: On Apr 2, 2012 3:07 PM, "Terri Oda" wrote: >> This agrees with my view of the situation as well. Which leads to the >> question, is the above approach interesting/viable for Mailman-team? >> (assuming the code does something awesome that people want) > > If the question is just "would you like another archiver even if the licenses don't match?" then I believe the answer is yes. The question i "would you BUNDLE another archiver even if the licenses don't match?" My archiver has been available for download (like many others) for ten years. All these sites are still running a limping pipermail archive, because it's bundled. I want to get Mailman a better bundled archive. > But... since you arrived around the same time GSoC started, I should ask whether you were hoping to do this as a GSoC project? Perhaps it would make things more clear if I expledin why I'm here... I'm not a student. I've been working in software for 15 years, programming for almost 30 (since I was 9). I wrote large portions of eGroups / Yahoo Groups / Google Groups. I'm a successful post-Google entrepreneur. Since leaving Google I've been angel investing mostly in tech stuff (see my Angel List).. I've been donating notable chunks of money and time to open source projects (with my blender donations working out the best so far). Given my history, and the fact that I keep wanting to tear my hear out reading mailing list archives in pipermail, I thought I'd give you folks an archiver that would be nice. HOWEVER, I personally will not write GPL code. I might submit a tiny patch or bugfix, but I'm simply opposed to restrictions on how someone uses something that I'm trying to donate to the software community. (i.e. you're never going to turn me into a mailman developer, the best you'd get is me writing my own mailman-ish and releassing it under S-BSD.. if you want that, let me know) From pingou at pingoured.fr Tue Apr 3 08:15:48 2012 From: pingou at pingoured.fr (Pierre-Yves Chibon) Date: Tue, 03 Apr 2012 08:15:48 +0200 Subject: [Mailman-Developers] Additional Mailman GSoC mentors In-Reply-To: <20120329145310.GE11151@unaka.lan> References: <4F7415CF.20304@zone12.com> <20120329145310.GE11151@unaka.lan> Message-ID: <1333433748.24909.11.camel@ambre.pingoured.fr> On Thu, 2012-03-29 at 07:53 -0700, Toshio Kuratomi wrote: > On Thu, Mar 29, 2012 at 01:57:03AM -0600, Terri Oda wrote: > > It's looking like we're going to have more student applicants than in > > previous years, so I think it'd be great if we could get a few more > > mentors to match. > > > > If you're a semi-active mailman developer (i.e. I'm going to > > recognize your name from your mailman-developers postings) and you > > think you might interested in mentoring for GSoC this summer or just > > want to know what's involved, please get in touch with me! > > > I'm willing to help mentor some work. I'd really like to mentor with > some other people -- especially at the application review stages -- I do > have more time for day-to-day mentoring if that's done on IRC (Interrupt > Driven Design, anyone ;-) But so far, my knowledge of the mailman codebase > is limited mainly to archiver stuff. > > I can answer questions about using bzr and some launchpad questions > (although I also have lots of launchpad questions of my own :-). I'm now > fully versed in Warsaw import style rules although I should probably > recertify at the next pycon :-) > > It does seem like there's a lot of interest in archivers this year (at > least, people have been pinging me about that. Since archivers for mailman3 > are somewhat in their infancy, it would be good to think of a "what do we > want the state of archivers to be after the summer and a year from now" so > that we can make sure that GSoC work fits into that. Hi, I actually would not mind give a hand to monitor someone, only point I do not want to supervise alone, I have no experience with the GSoC either as student or supervisor. Regards, Pierre From a.badger at gmail.com Tue Apr 3 20:58:22 2012 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 3 Apr 2012 11:58:22 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> Message-ID: <20120403185822.GI11151@unaka.lan> On Mon, Apr 02, 2012 at 08:04:23PM -0700, David Jeske wrote: > On Apr 2, 2012 3:07 PM, "Terri Oda" wrote: > >> This agrees with my view of the situation as well. Which leads to the > >> question, is the above approach interesting/viable for Mailman-team? > >> (assuming the code does something awesome that people want) > > > > If the question is just "would you like another archiver even if the > licenses don't match?" then I believe the answer is yes. > > The question i "would you BUNDLE another archiver even if the licenses > don't match?" > > My archiver has been available for download (like many others) for ten > years. All these sites are still running a limping pipermail archive, > because it's bundled. I want to get Mailman a better bundled archive. > From the talk about what it means to be a FSF project at the mailman sprint at pycon I don't think a non-FSF copyright assigned archiver would be bundled into mailman (Core). Distributed/pointed to by list.org along with mailman and postorius might be negotiable though :-) Would that be something you'd like to pursue? Also -- mailman3's builtin archiver is extremely minimal -- at the moment, it archives (stores) mail but it doesn't have a means to display that email on a web page or similar. Given that sort of bundled archiver, I have a feeling sites are going to want to run a third-party archiver of some sort instead of the default. > > HOWEVER, I personally will not write GPL code. I might submit a tiny patch > or bugfix, but I'm simply opposed to restrictions on how someone uses > something that I'm trying to donate to the software community. (i.e. you're > never going to turn me into a mailman developer, the best you'd get is me > writing my own mailman-ish and releassing it under S-BSD.. if you want > that, let me know) > General impression from talking to a few other developers at PyCon is we generally like copyleft licenses. Some version of copyleft is likely what a lot of us would choose to license our own code under. A few of us are unhappy when our code is used to make closed source applications. Mailman2 is an FSF project. mailman3 and postorius are both derivatives of mailman2 and so they are both FSF projects. FSF projects must do copyright assignment to the FSF and are licensed with one of the GNU licenses. Where could your archiver fit into that sequence of impressions? I'm not entirely sure. I think that it probably couldn't be bundled into the same tarball with mailman core due to mailman being an FSF project. But pointing to it from list.org or blessing it as the "standard archiver" for mailman3 is probably something that could be discussed by the core devs and yourself. I don't think you're going to find the will to make this sort of decision right at this instant because what we want the archiver ecosystem to look like for mailman3 is somewhat in the air. Do we really want an obviously less capable archiver to be the bundled archiver? Do we want to have a single blessed archiver (probably in a separate tarball as postorius, the admin web ui, is separate) as an eventual goal? Do we want (at least for a year or two) to let people go to town with their new ideas for archivers and then see if a best-of-breed archiver is raising its head? I don't believe any of this is decided inside of our minds yet, so, for now, people are defaulting to wait and see. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From adam-mailman at amyl.org.uk Wed Apr 4 00:48:08 2012 From: adam-mailman at amyl.org.uk (Adam McGreggor) Date: Tue, 3 Apr 2012 23:48:08 +0100 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> Message-ID: <20120403224808.GF4783@hendricks.amyl.org.uk> On Mon, Apr 02, 2012 at 08:04:23PM -0700, David Jeske wrote: > HOWEVER, I personally will not write GPL code. I might submit a tiny patch > or bugfix, but I'm simply opposed to restrictions on how someone uses > something that I'm trying to donate to the software community. +1. (as well as the bloody length of the GPL, and its age.) -- "Of course we are not patronising women. We are just going to explain to them, in words of one syllable, what it is all about." -- Olga Maitland From pabs3 at bonedaddy.net Wed Apr 4 03:32:55 2012 From: pabs3 at bonedaddy.net (Paul Wise) Date: Wed, 4 Apr 2012 09:32:55 +0800 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120403185822.GI11151@unaka.lan> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> Message-ID: On Wed, Apr 4, 2012 at 2:58 AM, Toshio Kuratomi wrote: > I don't think you're going to find the will to make this sort of decision > right at this instant because what we want the archiver ecosystem to look > like for mailman3 is somewhat in the air. ?Do we really want an obviously > less capable archiver to be the bundled archiver? ?Do we want to have > a single blessed archiver (probably in a separate tarball as postorius, the > admin web ui, is separate) as an eventual goal? ?Do we want (at least for > a year or two) to let people go to town with their new ideas for archivers > and then see if a best-of-breed archiver is raising its head? ?I don't > believe any of this is decided inside of our minds yet, so, for now, people > are defaulting to wait and see. I think it would be a mistake to bundle any archiver with mailman3. Listing the available archiver options and their features and shortcomings would be a better way to go. -- bye, pabs http://wiki.debian.org/PaulWise From bob at nleaudio.com Wed Apr 4 05:21:14 2012 From: bob at nleaudio.com (Bob Puff) Date: Tue, 3 Apr 2012 23:21:14 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> Message-ID: <20120404031831.M75170@nleaudio.com> > I think it would be a mistake to bundle any archiver with mailman3. > Listing the available archiver options and their features and > shortcomings would be a better way to go. -1 I think the majority of MM users will be simply using the RPM that comes with their distro, and there is a real benefit to stuff working right "out of the box". This includes the Archiving functions. Its great to have options, and giving a list of possible alternatives for users is excellent, but I think releasing MM 3 without -any- archiver is a down-grade from the current MM 2.x. Bob From pabs at debian.org Wed Apr 4 05:41:42 2012 From: pabs at debian.org (Paul Wise) Date: Wed, 4 Apr 2012 11:41:42 +0800 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120404031831.M75170@nleaudio.com> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <20120404031831.M75170@nleaudio.com> Message-ID: On Wed, Apr 4, 2012 at 11:21 AM, Bob Puff wrote: > I think the majority of MM users will be simply using the RPM that comes with > their distro, and there is a real benefit to stuff working right "out of the > box". ?This includes the Archiving functions. > > Its great to have options, and giving a list of possible alternatives for > users is excellent, but I think releasing MM 3 without -any- archiver is a > down-grade from the current MM 2.x. In the Debian world we would do something like this: Package: mailman3 Depends: mailman3-archiver-default | mailman3-archiver Package: mailman3-archiver-default Depends: mailman3-archiver-hyperkitty Package: mailman3-archiver-hyperkitty Provides: mailman3-archiver Is something like that not possible in the RPM world? I'm subscribed to the list, no need to CC me. -- bye, pabs http://bonedaddy.net/pabs3/ From davidj at gmail.com Wed Apr 4 06:16:28 2012 From: davidj at gmail.com (David Jeske) Date: Tue, 3 Apr 2012 21:16:28 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120404031831.M75170@nleaudio.com> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <20120404031831.M75170@nleaudio.com> Message-ID: On Apr 3, 2012 8:14 PM, "Bob Puff" wrote: > > I think it would be a mistake to bundle any archiver with mailman3. > > Listing the available archiver options and their features and > > shortcomings would be a better way to go. > > -1 > > I think the majority of MM users will be simply using the RPM that comes with > their distro, and there is a real benefit to stuff working right "out of the > box". This includes the Archiving functions. > > Its great to have options, and giving a list of possible alternatives for > users is excellent, but I think releasing MM 3 without -any- archiver is a > down-grade from the current MM 2.x. I agree. If MM2 and pipermail is any indication of how often admins just 'leave the defaults', then bunding no archive interface with MM3 would mean most mailing lists would have no archive. I'd personally like to see a better archiver rolled into an MM2 point release, as well as upcoming MM3 development. (I understand pipermail URL compat would be nice in that case). From davidj at gmail.com Wed Apr 4 06:33:33 2012 From: davidj at gmail.com (David Jeske) Date: Tue, 3 Apr 2012 21:33:33 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120403185822.GI11151@unaka.lan> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> Message-ID: On Apr 3, 2012 11:58 AM, "Toshio Kuratomi" wrote: > > The question is "would you BUNDLE another archiver even if the licenses > > don't match?" > Where could your archiver fit into that sequence of impressions? I'm not > entirely sure. I think that it probably couldn't be bundled into the same > tarball with mailman core due to mailman being an FSF project. I'm just going to charge down the path I was on and finish up something that's a great drop in for MM2/MM3. I'll even try to add some pipermail URL compatibility. It'll be S-BSD, so (if you like it) the MM devs and the FSF can wrestle with issues of whether you want to bundle it as is, put a rubber GPL stamp on it, or just point to it like you would any other archiver. I honestly expected to have an updated UI to show by now. I've been busy with some code-restructuring, and an unbelievable amount of life-stuff came across my bow in the past week. It shouldn't be too long now. > But pointing to it from list.org or blessing it as the "standard archiver" for mailman3 > is probably something that could be discussed by the core devs and yourself. I'm a bit scared of a world where MM3 does not include any archiver. If pipermail popularity is any indication of how often admins 'stick with the bundled defaults', we could have an unreasonable number of MM3 lists with no archives at all. Obviously the team is free to bless any archiver it wants, mine or others. Also, I'm certainly NOT trying to get anyone to agree to bless an archiver before they've even seen it working and kicking butt. I was just trying to understand the many issues as I'm cleaning up my code and trying to find it a home with a bit more utility. I think I have a great idea from all the disussions here.. THANKS! From a.badger at gmail.com Wed Apr 4 06:55:54 2012 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 3 Apr 2012 21:55:54 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <20120404031831.M75170@nleaudio.com> Message-ID: <20120404045554.GJ11151@unaka.lan> On Wed, Apr 04, 2012 at 11:41:42AM +0800, Paul Wise wrote: > On Wed, Apr 4, 2012 at 11:21 AM, Bob Puff wrote: > > > I think the majority of MM users will be simply using the RPM that comes with > > their distro, and there is a real benefit to stuff working right "out of the > > box". ?This includes the Archiving functions. > > > > Its great to have options, and giving a list of possible alternatives for > > users is excellent, but I think releasing MM 3 without -any- archiver is a > > down-grade from the current MM 2.x. > > In the Debian world we would do something like this: > > Package: mailman3 > Depends: mailman3-archiver-default | mailman3-archiver > > Package: mailman3-archiver-default > Depends: mailman3-archiver-hyperkitty > > Package: mailman3-archiver-hyperkitty > Provides: mailman3-archiver > > Is something like that not possible in the RPM world? > Not sure what the | syntax is so it may not be. But what we might do would be have a virtual provide. Package mailman3 Requires: mailman3-archiver Package: mailman3-archiver-hyperkitty Provides: mailman3-archiver Package: mailman3-archiver-pipermail-eeewwww Provides: mailman3-archiver -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From stephen at xemacs.org Wed Apr 4 07:03:00 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 4 Apr 2012 14:03:00 +0900 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <20120404031831.M75170@nleaudio.com> Message-ID: On Wed, Apr 4, 2012 at 1:16 PM, David Jeske wrote: > On Apr 3, 2012 8:14 PM, "Bob Puff" wrote: >> I think the majority of MM users will be simply using the RPM that comes with >> their distro, and there is a real benefit to stuff working right "out of the >> box". ?This includes the Archiving functions. I don't see why that precludes having the archiver in a separate recommended or required RPM, .deb, ebuild, or whatever dependency, and I imagine the distros can and will deal with that (as most of them use Mailman themselves, they'd have to do without dogfood). The problem as I see it is that many distros (I'm looking at you, Debian!) get woefully out of date, and their packaging often pays more attention to "fitting in to the distro" than to what we consider best practice. So users will often upgrade from our sources (and that is historically what we recommend). Also, many non-OS distros (*gag* *spit* Plesk *barf* cPanel) will roll their own derivatives (typically with little care for what we consider best practice). > I'd personally like to see a better archiver rolled into an MM2 point > release, as well as upcoming MM3 development. (I understand pipermail URL > compat would be nice in that case). That, and automatic storage conversion to whatever the new archive UI prefers. The caveats above notwithstanding, at this point I'm definitely with David and Bob on this issue -- +1 for including batteries. I'd like to hear from Mark, though (even more so than from Barry; Mark is the guy who's been guiding people through upgrades on a daily basis for the last decade or so). From stephen at xemacs.org Wed Apr 4 07:08:39 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 4 Apr 2012 14:08:39 +0900 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120403185822.GI11151@unaka.lan> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> Message-ID: On Wed, Apr 4, 2012 at 3:58 AM, Toshio Kuratomi wrote: > From the talk about what it means to be a FSF project at the mailman sprint > at pycon I don't think a non-FSF copyright assigned archiver would be > bundled into mailman (Core). AFAIK there are no "FSF projects", although the FSF does support "The" GNU Project and sometimes specific GNU projects. According to the criteria for being a GNU project (http://www.gnu.org/help/evaluation.html) For a program to be GNU software does not require transferring copyright to the FSF; that is a separate question. If you transfer the copyright to the FSF, the FSF will enforce the GPL for the program if someone violates it; if you keep the copyright, enforcement will be up to you. What *is* required is the GPL: A GNU program should use the latest version of the license that the GNU Project recommends?not just any free software license. For most packages, this means using the GNU GPL. So David's program can't be *part* of GNU Mailman without special permission, which I doubt the GNU Project (ie, RMS, AFAIK) will grant (and would require delicate negotations in extreme good humor on our part, based on past experience trying to negotiate licensing exceptions with RMS). It is not obvious that it can't be bundled with Mailman distributions, however. To my mind, bundling is a very strong recommendation, and the official standard for GNU projects merely says: A GNU program should not recommend use of any non-free program[...]. We could also redistribute verbatim, as part of Mailman under the GPL, with pointers to upstream (I would be happy personally host a mirror of a permissively licensed distribution). Perhaps with an ElementTree-like agreement that David makes the call on changes to the archiver he contributed. AIUI, that would make David happy (enough), as he doesn't believe you can really restrict redistribution of a simplified BSD-licensed program merely by incorporating it in a GPLed distribution. The main stinker there is the "David is the boss" agreement, if he wants it. I personally have been working with that kind of agreement for years in XEmacs, and it makes our package contributors happy, although it pisses off some of our core contributors. Similar to the ElementTree controversy in the Python stdlib, although none of the packages where issues have come up matters as much to us as ElementTree does to Python. So that would be mostly up to Barry (if David decides he wants that kind of power over the future of his archiver after contributing it to Mailman). > General impression from talking to a few other developers at PyCon is we > generally like copyleft licenses. ?Some version of copyleft is likely what > a lot of us would choose to license our own code under. ?A few of us are > unhappy when our code is used to make closed source applications. Sure, but this isn't our code yet, it's David's, and he proposes to do much of the work involved in adapting his code to Mailman 3. > Mailman2 is an FSF project. ?mailman3 and postorius are both derivatives of > mailman2 and so they are both FSF projects. That logic is inaccurate. There's no must about it; Mailman 3 could just as well be a fork. But since the FSF is the owner of most of our code, there are certain important conveniences to continuing that practice, and no real benefit to not doing so since we can't choose our own license because of the derivation from Mailman under the GPL. >?FSF projects must do copyright assignment to the FSF Not true, see above. > and are licensed with one of the GNU licenses. This is true, and I'm pretty sure it will be GPL v3, although given the functionality there is some chance the GNU Project would push for AGPL (but AFAIK RMS still considers the Affero clause optional, even for out-and-out Web 2.0 webapps). > Do we want to have a single blessed archiver (probably > in a separate tarball as postorius, the admin web ui, is > separate) as an eventual goal? I believe that we won't have a blessed archiver, in the sense that any archiver we distribute will have to use the same APIs that other archivers do. But having followed mailman-users for a decade now, I think it would be a bad idea to have a "batteries not included" distribution for Mailman 3.1. Which webmin, which archiver, is a different question. From terri at zone12.com Wed Apr 4 08:19:32 2012 From: terri at zone12.com (Terri Oda) Date: Wed, 04 Apr 2012 00:19:32 -0600 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> Message-ID: <4F7BE7F4.7070607@zone12.com> On 12-04-03 11:08 PM, Stephen J. Turnbull wrote: > So David's program can't be *part* of GNU Mailman without special > permission, which I doubt the GNU Project (ie, RMS, AFAIK) will grant > (and would require delicate negotations in extreme good humor on our > part, based on past experience trying to negotiate licensing > exceptions with RMS). It is not obvious that it can't be bundled with > Mailman distributions, however. It occurs to me that it's perfectly reasonable to assume that people who *package* mailman for different distributions may choose different recommended/required archive software, since they can (and with the license hassle likely should)) be separate packages. So what works for the FSF, what works for us as a dev team, and what works for the distributions may actually be different things. So no matter what, having David release his work is potentially going to lead to people getting it as a default, somewhere along the line, if he's got a great solution available. People get something better than pipermail *and* it doesn't result in me getting more angry emails from RMS? Sounds like a winner to me. BTW, I *will* argue that we should have a bundled archiver that does something more than make mbox files, and you can all expect to have a big argument with me about it later. ;) But I'm not in a hurry to make a decision about which one Right Now because I'm going to want to do a deeper usability analysis of Postorius + archive and I can't do that until we have them both on the table for user testing. Terri From davidj at gmail.com Wed Apr 4 08:56:30 2012 From: davidj at gmail.com (David Jeske) Date: Tue, 3 Apr 2012 23:56:30 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <4F7BE7F4.7070607@zone12.com> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <4F7BE7F4.7070607@zone12.com> Message-ID: This thread is slowing down my coding! :) (it's been really helpful though all, thanks for the many perspectives!) On Tue, Apr 3, 2012 at 11:19 PM, Terri Oda wrote: > It occurs to me that it's perfectly reasonable to assume that people who > *package* mailman for different distributions may choose different > recommended/required archive software, since they can (and with the license > hassle likely should)) be separate packages. So what works for the FSF, > what works for us as a dev team, and what works for the distributions may > actually be different things. > I agree I'm coming around to the sensibility of possibly not including an archiver with MM3, just so long as there actually ARE solid and working archivers that plug right in with nothing more than an apt-get (or equiv). It's just as fine if the distribution maintainers pick which one to include, and this gets around all this FSF/GPL/whatsit stuff... without bascally getting a pipermail default. I still think it's dangerous for people landing on Mailman's website and downloading source.. > So no matter what, having David release his work is potentially going to > lead to people getting it as a default, somewhere along the line, if he's > got a great solution available. > I know this thread is long and in pieces, but just to clarify, my code is already released and has been S-BSD for **ten** years. The UI is a little dated, so I'm cleaning up both the UI and the code right now, but I just want folks to know the code is already out there.. http://www.clearsilver.net/archive/ http://dj1.willowmail.com/csla/Mailman-Developers ...this discussion is all just about whether mailman wants to bundle (or reference) near-future updates to this stuff. I was hoping that rather than create my own separate OSS-y website and such for it, I could just hang out here and roll it into Mailman-land. You guys have done great work. If this GPL/S-BSD issue turns out to be a blocker, then I'll just make my own site and maintain (my version) there because I want to release my code S-BSD. Also, there will be *zero* ill-will if you folks want to wrap it up in a GPL license and stick it into mailman... i just won't be maintaining that, or assigning copyright, and any patches I make will be into my S-BSD tree. Perhaps not ideal, but still seems a better outcome than pipermail. From stephen at xemacs.org Wed Apr 4 09:58:13 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 4 Apr 2012 16:58:13 +0900 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <4F7BE7F4.7070607@zone12.com> Message-ID: On Wed, Apr 4, 2012 at 3:56 PM, David Jeske wrote: > ...this discussion is all just about whether mailman wants to bundle (or > reference) near-future updates to this stuff. I was hoping that rather than > create my own separate OSS-y website and such for it, I could just hang out > here and roll it into Mailman-land. You guys have done great work. I can't see any technical or legal reason why you would need to maintain a separate site. I would be more than happy to help maintain a Mailman-based site providing links to resources such as tarballs, VCS repos, and docs (this site would presumably be on the wiki) and a repo on Launchpad, which I think takes care of any social issues. It wouldn't be specific to the Clearsilver archiver, I'd do the same for any other archivers people care to recommend. As for a GPL-wrapped release bundled into Mailman, I'll do admin-level work to make that possible if the Clearsilver archiver is adopted and that's the way people want to go. I have experience with that kind of thing, and am happy to help lubricate that kind of friction. (I might be willing to do more hacking/maintainer-like work if people decide to make significant GPL-only enhancements, but I won't decide whether to do that until I've seen both the existing code and any such enhancements.) From benedict.stein at googlemail.com Wed Apr 4 18:07:43 2012 From: benedict.stein at googlemail.com (Benedict Stein) Date: Wed, 04 Apr 2012 18:07:43 +0200 Subject: [Mailman-Developers] Gsoc In-Reply-To: References: Message-ID: <4F7C71CF.2020109@gmail.com> Salut Fran?ois, GSOC Application are generally done through the Melange Web Interface which is on the Google Summer of Code Site. Regarding the matter of Mailman - Florian / The Mailinglist is probably the best way to contact the project people. You'll find both things in CC Also you'll find lots of addition information about mailman and GSOC opportunities on the MAilinglist. I've got only one question left - where did you get my Email ? On 04/04/2012 17:16, Fran?ois Rib?mont wrote: > Hello, > I am a 4th year student in software in engineering who would like to > apply for the project you have submitted on google summer of code. > This will be my first gsoc, and I don't know much about it. So my > first question is: is it the good way to contact you ? Is there any > kind of : "How to apply" on your project ? > > Regards > > -- > Ribemont Fran?ois > ribemont.francois at gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rkw at dataplex.net Thu Apr 5 05:57:59 2012 From: rkw at dataplex.net (Richard Wackerbarth) Date: Wed, 4 Apr 2012 22:57:59 -0500 Subject: [Mailman-Developers] [Merge] lp:~crodjer/postorius/postorius into lp:postorius In-Reply-To: References: Message-ID: <615831E3-3609-48CE-9692-8B1ACEC4E890@dataplex.net> On Apr 4, 2012, at 9:40 PM, Rohan Jain wrote: > Okay. I changed it to mutest.db because someone by the nick wacky asked > over the IRC First, that was a typo in my IRC message. Second, I did not request that you change the file name. I ASK why you changed the definition (from Florian's computed path). You never responded. From pingou at pingoured.fr Thu Apr 5 15:41:28 2012 From: pingou at pingoured.fr (Pierre-Yves Chibon) Date: Thu, 05 Apr 2012 15:41:28 +0200 Subject: [Mailman-Developers] From the creation of a ThreadID Message-ID: <1333633288.23207.26.camel@ambre.pingoured.fr> Hi, In HyperKitty to be able to easily retrieve from the database all the threads of a given month or just all the emails of a thread, I created a Field in the database called ThreadID. When I load the archives from mailman into mongo, I look for the absence of the headers 'References' or 'In-Reply-To' to define an email that starts a new thread. Then all emails which have the header 'References' or 'In-Reply-To' will look for the preceding email and extract the ThreadID from it. This seems to work fine. At the beginning I was using a simple integer as identifier but of course if you changed the order in which the archives are loaded or just if you miss like one month than the ThreadID is not consistent anymore. So I changed to use the Message-ID of the first email of the Thread as ThreadID. Problem is of course, if the admin removes the first email of a thread for x or y reasons, then when reloading the archives (for z or a reasons), we will loose the ThreadID and actually, the integrity of the Thread (each reply to the first email will be split into their own thread). Would anyone have an idea on how to generate a stable and delete/reload proof ThreadID? The other solution of course being that I regenerate the thread on the fly based on the first email (which is still easy to find), but that will be a lot of db querying. Thinking about it, generating the thread on the fly would also give the possibility to regenerate the thread view from anywhere (so you could generate a thread view for only a sub-thread). Do you have any suggestions/preferences ? Thanks, Pierre From stephen at xemacs.org Thu Apr 5 17:10:22 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 6 Apr 2012 00:10:22 +0900 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <1333633288.23207.26.camel@ambre.pingoured.fr> References: <1333633288.23207.26.camel@ambre.pingoured.fr> Message-ID: On Thu, Apr 5, 2012 at 10:41 PM, Pierre-Yves Chibon wrote: > In HyperKitty to be able to easily retrieve from the database all the > threads of a given month or just all the emails of a thread, I created a > Field in the database called ThreadID. > When I load the archives from mailman into mongo, I look for the absence > of the headers 'References' or 'In-Reply-To' to define an email that > starts a new thread. This fails when a thread crosses channels. Eg, To: Pierre From: Steve Message-Id: is followed by To: Steve From: Pierre Cc: SomeList References: Message-Id: > Would anyone have an idea on how to generate a stable and delete/reload > proof ThreadID? I don't see how this can be possible. Eg, in the above scenario you construct a thread based on your reply to me. Then I go, "oh, really I should have posted to mm-dev" and repost the thread. So the "Message-ID of root message" fails, and I don't see an alternative that can be predicted. So it may as well be arbitrary (eg, any message in the thread) and stored in the database with appropriate linkage from thread IDs to message IDs (one-to-many), and vice versa (many-to-one). > The other solution of course being that I regenerate the thread on the > fly based on the first email (which is still easy to find), but that > will be a lot of db querying. I haven't thought about it deeply, but I would say just give the thread an arbitrary ID in the database. Message-IDs are supposed to universally unique, so what's wrong with keeping the thread in the database as a tree of message IDs? Some Message-IDs will not have corresponding messages but that's always a problem with threading (see http://www.jwz.org/doc/threading.html, and RFC 5256). There are other problems with threading that need to be dealt with as well, such as References being inconsistent across messages in the same thread and people who continue a thread with a new message, etc. From richard at NFSNet.org Thu Apr 5 17:57:04 2012 From: richard at NFSNet.org (Richard Wackerbarth) Date: Thu, 5 Apr 2012 10:57:04 -0500 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> Message-ID: I agree with Steve. In general you cannot solve the problem with only the information contained in the message headers. You will need to maintain parallel meta-data for each message/thread. The header info would provide an initialization at the time of insertion. Presumedly this thread tree could be edited by an administrator to "correct" broken chains, etc. On Apr 5, 2012, at 10:10 AM, Stephen J. Turnbull wrote: > On Thu, Apr 5, 2012 at 10:41 PM, Pierre-Yves Chibon wrote: > >> In HyperKitty to be able to easily retrieve from the database all the >> threads of a given month or just all the emails of a thread, I created a >> Field in the database called ThreadID. >> When I load the archives from mailman into mongo, I look for the absence >> of the headers 'References' or 'In-Reply-To' to define an email that >> starts a new thread. > > This fails when a thread crosses channels. Eg, > > To: Pierre > From: Steve > Message-Id: > > is followed by > > To: Steve > From: Pierre > Cc: SomeList > References: > Message-Id: > I haven't thought about it deeply, but I would say just give the > thread an arbitrary ID in the database. Message-IDs are supposed to > universally unique, so what's wrong with keeping the thread in the > database as a tree of message IDs? Some Message-IDs will not have > corresponding messages but that's always a problem with threading (see > http://www.jwz.org/doc/threading.html, and RFC 5256). > > There are other problems with threading that need to be dealt with as > well, such as References being inconsistent across messages in the > same thread and people who continue a thread with a new message, etc. From sophron at latthi.com Thu Apr 5 20:38:58 2012 From: sophron at latthi.com (George Chatzisofroniou) Date: Thu, 5 Apr 2012 21:38:58 +0300 Subject: [Mailman-Developers] [GSoC 2012] Candidate on 'Integration of (existing) search code into Mailman archives' In-Reply-To: <4F7A2C47.1070002@zone12.com> References: <4F7A2C47.1070002@zone12.com> Message-ID: 2012/4/3 Terri Oda : > Hi George, > > Your MailmanStats project looks great and would totally fit with what we > have in mind for stats, though I'm guessing the hyperkitty team has some > much more extensive work in mind making use of post ratings, tags, etc. > > If you're putting together your proposal now, do feel free to mention both > projects as sources of interest. ?Since you already have the stats code > available, it might be possible to toss the integration in there after doing > some other work. ?Normally I worry about students biting off more than they > can chew, but given your prior experience with Mailman and the fact that you > already have the basic code, you can make a case for being able to package > up that code and contribute it in a week or two our of your summer if you're > ready for a code review. > > > ?Terri > > PS - ?For further advice regarding search projects, see my previous post to > mailman-developers. > > Thanks for your respond Terri, I thought about it quite a lot. Eventually, I think it is better to implement only the Metric idea (since i already have the base code) by integrating my software into Mailman. My previous experience with Mailman and the fact i have done some work already will make a more awesome result. I'll send my proposal in the next hours. I'll appreciate any feedback. Thanks, -- George Chatzisofroniou sophron.latthi.com From pingou at pingoured.fr Thu Apr 5 20:42:51 2012 From: pingou at pingoured.fr (Pierre-Yves Chibon) Date: Thu, 05 Apr 2012 20:42:51 +0200 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> Message-ID: <1333651371.6278.16.camel@ambre.pingoured.fr> On Fri, 2012-04-06 at 00:10 +0900, Stephen J. Turnbull wrote: > On Thu, Apr 5, 2012 at 10:41 PM, Pierre-Yves Chibon wrote: > > > In HyperKitty to be able to easily retrieve from the database all the > > threads of a given month or just all the emails of a thread, I created a > > Field in the database called ThreadID. > > When I load the archives from mailman into mongo, I look for the absence > > of the headers 'References' or 'In-Reply-To' to define an email that > > starts a new thread. > > This fails when a thread crosses channels. Eg, > > To: Pierre > From: Steve > Message-Id: > > is followed by > > To: Steve > From: Pierre > Cc: SomeList > References: > Message-Id: > > > Would anyone have an idea on how to generate a stable and delete/reload > > proof ThreadID? > > I don't see how this can be possible. Eg, in the above scenario you > construct a thread based on your reply to me. Then I go, "oh, really > I should have posted to mm-dev" and repost the thread. So the > "Message-ID of root message" fails, and I don't see an alternative > that can be predicted. So it may as well be arbitrary (eg, any > message in the thread) and stored in the database with appropriate > linkage from thread IDs to message IDs (one-to-many), and vice versa > (many-to-one). Ok, I missed a something here. So when it parses the email, it checks for 'References' or 'In-Reply-To'. - If it finds them, it looks for the preceding email - if it finds the preceding email, then the current email gets the ThreadID from the preceding email - if it does not find the preceding email, then the current email is assumed to be a new thread and thus its ThreadID is its Message-ID - if it does not find 'References' or 'In-Reply-To', then the current email is assumed to be a new thread and thus its ThreadID is its Message-ID So for the example you give, the archiver will receive your email and make a new thread out of it. > > The other solution of course being that I regenerate the thread on the > > fly based on the first email (which is still easy to find), but that > > will be a lot of db querying. > > I haven't thought about it deeply, but I would say just give the > thread an arbitrary ID in the database. Message-IDs are supposed to > universally unique, so what's wrong with keeping the thread in the > database as a tree of message IDs? Some Message-IDs will not have > corresponding messages but that's always a problem with threading (see > http://www.jwz.org/doc/threading.html, and RFC 5256). The idea of using the Message-ID for ThreadID (instead of a integer) is that, if I whether I load one months or two months of archives into the database, the link to the thread (http://mm3test.fedoraproject.org/thread/packaging at fp.o/XU7HT5JC5GND2O4JII7MTQILLTB4IN4S) will remain the same (so consistent urls). > There are other problems with threading that need to be dealt with as > well, such as References being inconsistent across messages in the > same thread and people who continue a thread with a new message, etc. For these I am not sure I can do something (at least automatically, we could always allow an admin to edit the field). Pierre From richard at NFSNet.org Thu Apr 5 22:00:24 2012 From: richard at NFSNet.org (Richard Wackerbarth) Date: Thu, 5 Apr 2012 15:00:24 -0500 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <1333651371.6278.16.camel@ambre.pingoured.fr> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <1333651371.6278.16.camel@ambre.pingoured.fr> Message-ID: <2E1CE263-285A-42DF-8841-5DB1E5633901@NFSNet.org> Pierre, There is nothing wrong with using a message ID as a thread ID. They are different namespaces (with an intuitive mapping for the first post.) The problem is only that the mapping is not stable under the "restore after deleting some messages" scenario. If you expect to be able to restore messages and keep stable thread IDs, then you will need to assure that the mapping of message to thread ID does not depend on the presence of other messages remaining in the database. Richard On Apr 5, 2012, at 1:42 PM, Pierre-Yves Chibon wrote: > The idea of using the Message-ID for ThreadID (instead of a integer) is > that, if I whether I load one months or two months of archives into the > database, the link to the thread > (http://mm3test.fedoraproject.org/thread/packaging at fp.o/XU7HT5JC5GND2O4JII7MTQILLTB4IN4S) will remain the same (so consistent urls). > >> There are other problems with threading that need to be dealt with as >> well, such as References being inconsistent across messages in the >> same thread and people who continue a thread with a new message, etc. > > For these I am not sure I can do something (at least automatically, we > could always allow an admin to edit the field). > > Pierre > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/richard%40nfsnet.org > > Security Policy: http://wiki.list.org/x/QIA9 From mark at msapiro.net Thu Apr 5 22:10:21 2012 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 5 Apr 2012 13:10:21 -0700 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <1333651371.6278.16.camel@ambre.pingoured.fr> Message-ID: Pierre-Yves Chibon wrote: > >Ok, I missed a something here. >So when it parses the email, it checks for 'References' or >'In-Reply-To'. >- If it finds them, it looks for the preceding email > - if it finds the preceding email, then the current email gets the >ThreadID from the preceding email > - if it does not find the preceding email, then the current email is >assumed to be a new thread and thus its ThreadID is its Message-ID >- if it does not find 'References' or 'In-Reply-To', then the current >email is assumed to be a new thread and thus its ThreadID is its >Message-ID This is still incomplete. One of the MUAs I use generates In-Reply-To: headers but not References: headers. Thus in cases where someone has replied to me but not included the list (and may or may not have subsequently sent the reply to the list with a different Message-ID), and I reply and include the list, the Message-ID in my In-Reply-To: is not in the archive. Another situation is someone replies to me and the list, but the list reply is greylisted and not retried for a while. Meanwhile, I reply to my copy and the Message-ID in my In-Reply-To: is not yet in the archive. Threading is not easy. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From pingou at pingoured.fr Thu Apr 5 22:20:18 2012 From: pingou at pingoured.fr (Pierre-Yves Chibon) Date: Thu, 05 Apr 2012 22:20:18 +0200 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: Message-ID: <1333657218.6278.25.camel@ambre.pingoured.fr> On Thu, 2012-04-05 at 13:10 -0700, Mark Sapiro wrote: > Pierre-Yves Chibon wrote: > > > >Ok, I missed a something here. > >So when it parses the email, it checks for 'References' or > >'In-Reply-To'. > >- If it finds them, it looks for the preceding email > > - if it finds the preceding email, then the current email gets the > >ThreadID from the preceding email > > - if it does not find the preceding email, then the current email is > >assumed to be a new thread and thus its ThreadID is its Message-ID > >- if it does not find 'References' or 'In-Reply-To', then the current > >email is assumed to be a new thread and thus its ThreadID is its > >Message-ID > > > This is still incomplete. One of the MUAs I use generates In-Reply-To: > headers but not References: headers. Thus in cases where someone has > replied to me but not included the list (and may or may not have > subsequently sent the reply to the list with a different Message-ID), > and I reply and include the list, the Message-ID in my In-Reply-To: is > not in the archive. > > Another situation is someone replies to me and the list, but the list > reply is greylisted and not retried for a while. Meanwhile, I reply to > my copy and the Message-ID in my In-Reply-To: is not yet in the > archive. > > Threading is not easy. I haven't completely read the link that Stephen sent earlier, hopefully the answer to these two points is in there :) Pierre From terri at zone12.com Thu Apr 5 23:29:26 2012 From: terri at zone12.com (Terri Oda) Date: Thu, 05 Apr 2012 17:29:26 -0400 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <1333633288.23207.26.camel@ambre.pingoured.fr> References: <1333633288.23207.26.camel@ambre.pingoured.fr> Message-ID: <4F7E0EB6.7030905@zone12.com> I haven't read the whole thread so maybe someone else has mentioned this, but we may want to take advantage of the dynamic sublists code for this, since it produces "conversations" or "topics" sublists and already has to generate and maintain a code for each. Rather than messageids these are meant to be a bit more human-readable, so they're often words with numbers suffixed. But yeah; there exists code for Mailman 2.1 that might be reusable here, and there's a GSoC project on the table to port to 3.0 so this might be a thing that we could pass to the archive utility. Terri On 12-04-05 9:41 AM, Pierre-Yves Chibon wrote: > Hi, > > In HyperKitty to be able to easily retrieve from the database all the > threads of a given month or just all the emails of a thread, I created a > Field in the database called ThreadID. > When I load the archives from mailman into mongo, I look for the absence > of the headers 'References' or 'In-Reply-To' to define an email that > starts a new thread. > Then all emails which have the header 'References' or 'In-Reply-To' will > look for the preceding email and extract the ThreadID from it. > This seems to work fine. > > At the beginning I was using a simple integer as identifier but of > course if you changed the order in which the archives are loaded or just > if you miss like one month than the ThreadID is not consistent anymore. > So I changed to use the Message-ID of the first email of the Thread as > ThreadID. > Problem is of course, if the admin removes the first email of a thread > for x or y reasons, then when reloading the archives (for z or a > reasons), we will loose the ThreadID and actually, the integrity of the > Thread (each reply to the first email will be split into their own > thread). > > Would anyone have an idea on how to generate a stable and delete/reload > proof ThreadID? > The other solution of course being that I regenerate the thread on the > fly based on the first email (which is still easy to find), but that > will be a lot of db querying. > > Thinking about it, generating the thread on the fly would also give the > possibility to regenerate the thread view from anywhere (so you could > generate a thread view for only a sub-thread). > > Do you have any suggestions/preferences ? > > Thanks, > Pierre > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/terri%40zone12.com > > Security Policy: http://wiki.list.org/x/QIA9 > From terri at zone12.com Thu Apr 5 23:40:23 2012 From: terri at zone12.com (Terri Oda) Date: Thu, 05 Apr 2012 17:40:23 -0400 Subject: [Mailman-Developers] Reminder: Get those GSoC proposals in! Message-ID: <4F7E1147.5040702@zone12.com> Just a reminder: GSoC proposals are due April 6th! The Melange system sometimes has problems on the day that things are due, so if you can get something into the Melange system now, even if it's just a draft, please do so. Google will not extend the deadline for any reason under any circumstance, so don't wait 'till the last minute! You'll be hearing from us next week as we review your proposals and ask questions, and you'll have a chance to update it then if there's something that needs adjustment! Terri From mark at msapiro.net Fri Apr 6 02:18:09 2012 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 5 Apr 2012 17:18:09 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: Message-ID: Stephen J. Turnbull wrote: > >The caveats above notwithstanding, at this point I'm definitely with >David and Bob on this issue -- +1 for including batteries. I'd like >to hear from Mark, though (even more so than from Barry; Mark is the >guy who's been guiding people through upgrades on a daily basis for >the last decade or so). I put this thread aside for "later", but just so people don't think I'm ignoring it, I'm +1 for an archiver or choice of archivers out of the box. I'd like to see a default install provide list owners with at a minimum a choice of public, private or no archives and the archives to be searchable. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From stephen at xemacs.org Fri Apr 6 05:00:32 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 6 Apr 2012 12:00:32 +0900 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <1333651371.6278.16.camel@ambre.pingoured.fr> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <1333651371.6278.16.camel@ambre.pingoured.fr> Message-ID: On Fri, Apr 6, 2012 at 3:42 AM, Pierre-Yves Chibon wrote: > So when it parses the email, it checks for 'References' or > 'In-Reply-To'. > - If it finds them, it looks for the preceding email > ? ?- if it finds the preceding email, then the current email gets the > ThreadID from the preceding email So far, so good. > ? ?- if it does not find the preceding email, then the current email is > assumed to be a new thread This is unacceptable. Mailing lists are not synchronous (eg, because of greylisting for one, but there are plenty of reasons why the mail doesn't always go through immediately). Threads must be able to integrate new messages as they arrive, even if out of order. > and thus its ThreadID is its Message-ID > - if it does not find 'References' or 'In-Reply-To', then the current > email is assumed to be a new thread and thus its ThreadID is its > Message-ID This isn't quite unacceptable, but it's clearly suboptimal. (Well-known algorithms that handle this case nicely are available.) > So for the example you give, the archiver will receive your email and > make a new thread out of it. That's an archiver that I won't use, and will strongly oppose as a candidate for the bundled archiver for Mailman (any version). >> I haven't thought about it deeply, but I would say just give the >> thread an arbitrary ID in the database. ?Message-IDs are supposed to >> universally unique, so what's wrong with keeping the thread in the >> database as a tree of message IDs? ?Some Message-IDs will not have >> corresponding messages but that's always a problem with threading (see >> http://www.jwz.org/doc/threading.html, and RFC 5256). > > The idea of using the Message-ID for ThreadID (instead of a integer) is > that, if I whether I load one months or two months of archives into the > database, the link to the thread > (http://mm3test.fedoraproject.org/thread/packaging at fp.o/XU7HT5JC5GND2O4JII7MTQILLTB4IN4S) will remain the same (so consistent urls). Sure, but this is a matter of a persistent ID in the database. When I say "arbitrary" I don't mean you can't use a message ID to represent a thread if you like, I mean that you can't algorithmically compute it in a reliable, history-independent way. From the point of view of a user, you can't even be sure that a message without References or In-Reply-To is a thread root (users will note the subject and the content, and they will be displeased with any threading algorithm that doesn't at least group subjects). I don't say you need to implement that part of the JWZ/5256 algorithm immediately, but you must not use a database schema that makes it hard to add that feature later. In most cases, users will have access to a Message-ID for some message in the thread. So I would want an URL like http://lists.example.com/archive/some-list/thread/MessageID/root/ to find the thread root for any message in the thread, not just a particular representative of the the thread. (YMMV for the URL scheme, of course.) The last component of the URL path just gives the focus (message to actually display and/or highlight in a tree widget); other useful focuses might be "latest" (a message in the thread with the most recent Date or Received header) and "self" (the message itself is the focus). More speculative focuses would be "parent" (obvious, I hope) and "node" (the most recent ancestor message with multiple children). >> There are other problems with threading that need to be dealt with as >> well, such as References being inconsistent across messages in the >> same thread and people who continue a thread with a new message, etc. > > For these I am not sure I can do something (at least automatically, we > could always allow an admin to edit the field). You must do something about inconsistent References. Suppose there is a References loop? It needs to be broken, somehow, or your program will infloop. Anyway, this is all already taken care of in Jamie's algorithm. From davidj at gmail.com Fri Apr 6 09:00:49 2012 From: davidj at gmail.com (David Jeske) Date: Fri, 6 Apr 2012 00:00:49 -0700 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <1333633288.23207.26.camel@ambre.pingoured.fr> References: <1333633288.23207.26.camel@ambre.pingoured.fr> Message-ID: On Apr 5, 2012 6:42 AM, "Pierre-Yves Chibon" wrote: > So I changed to use the Message-ID of the first email of the Thread as ThreadID. > Problem is of course, if the admin removes the first email of a thread > for x or y reasons, then when reloading the archives (for z or a > reasons), we will loose the ThreadID and actually, the integrity of the > Thread (each reply to the first email will be split into their own > thread). > > Would anyone have an idea on how to generate a stable and delete/reload proof ThreadID? I believe "deletion proof" (i.e. stable thread-ids in the case of arbitrary deletions) may be provably not possible. If you really want to be resiliant to arbitrary deletions/reloads, I think your solution is ultimately going to involve referencing more than one message in thread URLs.. For example, here is a scheme where 'messages in the thread name the thread': 1) don't publish thread-ids, but just message-ids... for example, a thread URL could be allowed to reference the message-id of 'any' message in the thread.... They could then include more than one message-id, making them resiliant to a lost messageid later. if some messageid are lost, hopefully a url someone is holding onto has another messageid that was not lost. As for how to pick the message-ids, paged display could include a messageid for a message on the page, in addition to the 'first' messageid of the thread. 2) create an 'internal only threadid' which you use to correlate messages together into a thread. (don't show this to anyone) you could generate this as a GUID, Hash, or the message-id of the message..doesn't matter, since nobody will see it... 3) when indexing messages, search in both directions (references/in-reply-to -> messageid, and vice-versa) to find out if the message belongs in a thread.. if it does, then adopt the 'internal thread id'.. if you find two different threadids in the two directions, then rewrite/combine into a single internal-thread-id -> urls can be somewhat resiliant of deleted/missing messages within a thread... and completely resilient to changes in other threads -> threads can be manually edited and merged/split after the fact, with some level of success -> could be designed to 'break down' threads that get too big, again with minimal damage, and some url compatibility From a.badger at gmail.com Fri Apr 6 19:49:47 2012 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 6 Apr 2012 10:49:47 -0700 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> Message-ID: <20120406174946.GM11151@unaka.lan> On Fri, Apr 06, 2012 at 12:00:49AM -0700, David Jeske wrote: > On Apr 5, 2012 6:42 AM, "Pierre-Yves Chibon" wrote: > > So I changed to use the Message-ID of the first email of the Thread as > ThreadID. > > Problem is of course, if the admin removes the first email of a thread > > for x or y reasons, then when reloading the archives (for z or a > > reasons), we will loose the ThreadID and actually, the integrity of the > > Thread (each reply to the first email will be split into their own > > thread). > > > > Would anyone have an idea on how to generate a stable and delete/reload > proof ThreadID? > > I believe "deletion proof" (i.e. stable thread-ids in the case of arbitrary > deletions) may be provably not possible. > > If you really want to be resiliant to arbitrary deletions/reloads, I think > your solution is ultimately going to involve referencing more than one > message in thread URLs.. > I don't see any way to make this 100% resilient against deletion + reload (where reload == from the available messages without the benefit of the old metadata) either. I think with slight modification to your steps below, we can get to resiliency against deletion or resiliency against total reload. > For example, here is a scheme where 'messages in the thread name the > thread': > > 1) don't publish thread-ids, but just message-ids... for example, a thread > URL could be allowed to reference the message-id of 'any' message in the > thread.... They could then include more than one message-id, making them > resiliant to a lost messageid later. if some messageid are lost, hopefully > a url someone is holding onto has another messageid that was not lost. > This sounds good. So instead of relying on the first message-id of the thread we internally keep a mapping of all message-ids and stableurl hashes to either an internal message-id or a tree of messages in the thread. When deleting messages, always retain the message-id and stableurl hashes for that message in the mapping. That way a url that pointed to the thread by that message-id will continue to function even though the message itself has been deleted. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From a.badger at gmail.com Fri Apr 6 19:54:00 2012 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 6 Apr 2012 10:54:00 -0700 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> Message-ID: <20120406175400.GN11151@unaka.lan> On Fri, Apr 06, 2012 at 12:10:22AM +0900, Stephen J. Turnbull wrote: > Some Message-IDs will not have > corresponding messages but that's always a problem with threading (see > http://www.jwz.org/doc/threading.html, and RFC 5256). > > There are other problems with threading that need to be dealt with as > well, such as References being inconsistent across messages in the > same thread and people who continue a thread with a new message, etc. > Looks like amk coded jqz's algorithm into a python library too: https://github.com/akuchling/jwzthreading All other links to that code that I found (on amk.ca and bitbucket) were broken so someone may want to clone that/ask andrew what's going on with it :-) -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From richard at NFSNet.org Sat Apr 7 04:48:46 2012 From: richard at NFSNet.org (Richard Wackerbarth) Date: Fri, 6 Apr 2012 21:48:46 -0500 Subject: [Mailman-Developers] [Bug 967951] The LMTP runner should reject messages with duplicate Message-IDs In-Reply-To: <20120406230612.6048.2774.launchpad@soybean.canonical.com> References: <20120329030405.10615.26178.malonedeb@soybean.canonical.com> <20120406230612.6048.2774.launchpad@soybean.canonical.com> Message-ID: <3AB034DF-713F-417A-BFE1-5522217F873C@NFSNet.org> What is the issue here? How far back in time is the runner expected to remember? On Apr 6, 2012, at 6:06 PM, Barry Warsaw wrote: > ** Changed in: mailman > Status: New => Confirmed > > ** Changed in: mailman > Importance: Undecided => High > > ** Changed in: mailman > Milestone: None => 3.0.0b2 From stephen at xemacs.org Sat Apr 7 20:48:35 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 8 Apr 2012 03:48:35 +0900 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <20120406175400.GN11151@unaka.lan> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <20120406175400.GN11151@unaka.lan> Message-ID: Bill Janssen has one too, I forget it it's based on amk's or not, but it's current. See thread in email-sig: http://mail.python.org/pipermail/email-sig/2012-January/000882.html On Sat, Apr 7, 2012 at 2:54 AM, Toshio Kuratomi wrote: > On Fri, Apr 06, 2012 at 12:10:22AM +0900, Stephen J. Turnbull wrote: >> Some Message-IDs will not have >> corresponding messages but that's always a problem with threading (see >> http://www.jwz.org/doc/threading.html, and RFC 5256). >> >> There are other problems with threading that need to be dealt with as >> well, such as References being inconsistent across messages in the >> same thread and people who continue a thread with a new message, etc. >> > Looks like amk coded jqz's algorithm into a python library too: > ?https://github.com/akuchling/jwzthreading > > All other links to that code that I found (on amk.ca and bitbucket) were > broken so someone may want to clone that/ask andrew what's going on with it > :-) > > -Toshio > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/stephen%40xemacs.org > > Security Policy: http://wiki.list.org/x/QIA9 From syst3m.w0rm at gmail.com Sun Apr 8 03:28:14 2012 From: syst3m.w0rm at gmail.com (Aamir Khan) Date: Sun, 8 Apr 2012 06:58:14 +0530 Subject: [Mailman-Developers] Integrating HyperKitty with Mailman3 Message-ID: I believe that after integrating HyperKitty with mailman, there will be archiver['hyperkitty'] which can be used to archive the messages. Am i correct? http://packages.python.org/mailman/src/mailman/archiving/docs/common.html#sending-the-message-to-the-archiver I know that mailman3 offers pluggable architecture, but still after going through some documentation it is not apparent to me how exactly HyperKitty will be integrated with mailman3. Can somebody briefly explain and point out to relevant files in source code ? -- Aamir Khan | 3rd Year | Computer Science & Engineering | IIT Roorkee From davidj at gmail.com Sun Apr 8 07:53:59 2012 From: davidj at gmail.com (David Jeske) Date: Sat, 7 Apr 2012 22:53:59 -0700 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <20120406174946.GM11151@unaka.lan> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <20120406174946.GM11151@unaka.lan> Message-ID: On Apr 6, 2012 10:49 AM, "Toshio Kuratomi" wrote: > > 1) don't publish thread-ids, but just message-ids... for example, a thread > > URL could be allowed to reference the message-id of 'any' message in the > > thread.... They could then include more than one message-id, making them > > resiliant to a lost messageid later. if some messageid are lost, hopefully > > a url someone is holding onto has another messageid that was not lost. > > > This sounds good. So instead of relying on the first message-id of the thread > we internally keep a mapping of all message-ids and stableurl hashes to > either an internal message-id or a tree of messages in the thread. I think of this as keeping a mapping from "rfc822 message-id" to "internal thread-id". I think you are using different words to say the same thing. > When deleting messages, always retain the message-id and stableurl hashes > for that message in the mapping. That way a url that pointed to the thread > by that message-id will continue to function even though the message itself > has been deleted. Perhaps I misunderstood. If you are going to have a record of the deletion (i.e. you can keep a deleted message around in some form), this problem becomes much easier. I thought this desire was to have stable urls and threads when you rebuild and a message is missing. Absolutly if there is a message 'deletion' feature, it should delete the message contents but leave a 'stub' that links the message-id and references/in-reply-to, so it can help hold the thread together during a rebuild. My memory is foggy, but I think we used a technique like this in Yahoo Groups. From stephen at xemacs.org Sun Apr 8 11:14:25 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 8 Apr 2012 18:14:25 +0900 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> <20120406174946.GM11151@unaka.lan> Message-ID: On Sun, Apr 8, 2012 at 2:53 PM, David Jeske wrote: > Perhaps I misunderstood. If you are going to have a record of the deletion > (i.e. you can keep a deleted message around in some form), this problem > becomes much easier. In practice, complete deletion is occasionally necessary. The archives should be robust to that. > Absolutly if there is a message 'deletion' feature, it should delete the > message contents but leave a 'stub' that links the message-id and > references/in-reply-to, so it can help hold the thread together during a > rebuild. If a message is actually a referencable member of a thread, there will be references to it in other messages. Only in the (increasingly rare) case of a thread held together only by In-Reply-Tos will the thread be cut by removing a message; otherwise the reference headers are enough to rebuild. I think it's reasonable to leave a stub whose only content is "This message was administratively removed" plus the References, Message-ID, and Date header fields. I don't know what to do about the required >From field, but since it's not going out on the wire in a certain technical sense the RFC doesn't apply. Alternatively, use 'From: "J. Redacted User" ". The only real problem I can see with this is that third parties who see it may go searching personal archives for a local copy of the offending message, which goes against the spirit of deletion -- better to pretend the message was never publicly posted. From barry at list.org Sun Apr 8 18:39:21 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 12:39:21 -0400 Subject: [Mailman-Developers] Integrating HyperKitty with Mailman3 In-Reply-To: References: Message-ID: <20120408123921.0ff6d14f@limelight.wooz.org> On Apr 08, 2012, at 06:58 AM, Aamir Khan wrote: >I believe that after integrating HyperKitty with mailman, there will be >archiver['hyperkitty'] which can be used to archive the messages. Am i >correct? Yes, but that's mostly an implementation detail you don't need to worry about. config.archivers is just for internal bookkeeping and use in the ArchiveRunner. >http://packages.python.org/mailman/src/mailman/archiving/docs/common.html#sending-the-message-to-the-archiver >I know that mailman3 offers pluggable architecture, but still after going >through some documentation it is not apparent to me how exactly HyperKitty >will be integrated with mailman3. Can somebody briefly explain and point out >to relevant files in source code ? It's relatively straightforward, once you understand how the configuration system works. The file src/mailman/config/schema.cfg is a kind of template for the mailman.cfg ini file. Search down for the [archive.master] section; this is a template for other [archiver.foo] sections. In here, you'll see all the default variables and their values for configuring an archiver. Look a little farther down and you'll see for example the [archiver.prototype] section which provides just the relevant overrides for the `prototype` archiver. So, to enable hyperkitty, you would have to add something like the following section in your mailman.cfg file: -----snip snip----- [archiver.hyperkitty] class: python.path.to.hyperkitty.HyperKitty -----snip snip----- Of course, you'd probably want to `enable` it too. One tricky thing here is that the `class` value names a Python dotted-module path, so the class must be importable. Ensuring that the hyperkitty module (and this is just a suggestion, YMMV) is importable by the core engine may not be fully baked. For now, just set $PYTHONPATH. The final bit of the puzzle is that the python.path.to.hyperkitty.HyperKitty class must implement the IArchiver interface, although if it's impossible to implement something like permalink(), it should just raise a NotImplementedError. Take a look at the Prototype class for an example. Note that all the magic of getting a message into the archiver happens through the archive_message() method of IArchiver. This can do anything you need to do to inject the message into the archiver. It can make direct Python calls, like the prototype archiver does, or it shell out to a command like the MHonArc archiver does, or it can send an email like the MailArchive one does. Hope that helps. -Barry From barry at list.org Sun Apr 8 19:00:06 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 13:00:06 -0400 Subject: [Mailman-Developers] [Bug 967951] The LMTP runner should reject messages with duplicate Message-IDs In-Reply-To: <3AB034DF-713F-417A-BFE1-5522217F873C@NFSNet.org> References: <20120329030405.10615.26178.malonedeb@soybean.canonical.com> <20120406230612.6048.2774.launchpad@soybean.canonical.com> <3AB034DF-713F-417A-BFE1-5522217F873C@NFSNet.org> Message-ID: <20120408130006.7903f75d@limelight.wooz.org> On Apr 06, 2012, at 09:48 PM, Richard Wackerbarth wrote: >What is the issue here? >How far back in time is the runner expected to remember? Forever? :) What I'm thinking is that early in the process we'll register every Message-ID we've seen (or maybe it should be every one we've accepted) in the message store. Then the LMTP server can check the message store before it accepts any new message. I should note though that I've tried several branches in the (distant-ish) past to implement this, and there are lots of tricky details to work out, especially around some assumptions in the test suite. I marked it High because I think it should be tackled once more before 3.0 final. Cheers, -Barry >On Apr 6, 2012, at 6:06 PM, Barry Warsaw wrote: > >> ** Changed in: mailman >> Status: New => Confirmed >> >> ** Changed in: mailman >> Importance: Undecided => High >> >> ** Changed in: mailman >> Milestone: None => 3.0.0b2 From barry at list.org Sun Apr 8 19:05:31 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 13:05:31 -0400 Subject: [Mailman-Developers] Automating Migration from MM2 to MM3 In-Reply-To: References: Message-ID: <20120408130531.15bfdf43@limelight.wooz.org> (Removing mailman-coders which should just be fore commit messages.) On Apr 06, 2012, at 09:41 PM, Richard Wackerbarth wrote: >In order to provide migration from MM2 to MM3, we will need to reformat the >list configuration, membership rosters and the user preferences. LP: #965532 >What should we do about: pending messages and the message archives? I think pending messages *could* be upgraded, but it would probably be a pain. >Is it sufficient to assume that the MM2 lists will be shut down and all >pending messages (and digests) will be delivered/flushed before restarting >the list on the MM3 server? I think this is a fair requirement. Basically, you need a clean, non-running system in order to fully upgrade. >Do we need to provide a legacy interface to pipermail, thus "kicking the >issue down the road", or Will someone migrate the legacy archives, or Will >the user be expected to maintain two archives? I think they'll probably *have* to maintain two archives, if they want to continue to support legacy URLs. Pipermail just has too many problems too ensure that you could regenerate the archive even with itself and guarantee your urls won't change. You wouldn't want to break the googles. :) So I think the safest thing for a site to do would be to leave the old Pipermail archives around, and then regenerate the new archives from the .mbox file. >Have I omitted any additional transition issues? Probably, but I can't think of what atm. :) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Sun Apr 8 19:38:16 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 13:38:16 -0400 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <4F7E0EB6.7030905@zone12.com> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <4F7E0EB6.7030905@zone12.com> Message-ID: <20120408133816.29cb74b4@limelight.wooz.org> On Apr 05, 2012, at 05:29 PM, Terri Oda wrote: >I haven't read the whole thread so maybe someone else has mentioned this, but >we may want to take advantage of the dynamic sublists code for this, since it >produces "conversations" or "topics" sublists and already has to generate and >maintain a code for each. Rather than messageids these are meant to be a bit >more human-readable, so they're often words with numbers suffixed. But yeah; >there exists code for Mailman 2.1 that might be reusable here, and there's a >GSoC project on the table to port to 3.0 so this might be a thing that we >could pass to the archive utility. Don't forget too that we have the Stable URL proposal, which turns arbitrary Message-IDs into 32 upper-case ASCII letter and digit character base 32 hashes. -Barry From richard at NFSNet.org Sun Apr 8 20:11:57 2012 From: richard at NFSNet.org (Richard Wackerbarth) Date: Sun, 8 Apr 2012 13:11:57 -0500 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <20120408133816.29cb74b4@limelight.wooz.org> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <4F7E0EB6.7030905@zone12.com> <20120408133816.29cb74b4@limelight.wooz.org> Message-ID: <5C7FAC2C-67D1-4C23-B104-F2963750E6E5@NFSNet.org> I would propose a slightly different scheme for converting messages to stable URIs.. If we create our ID by concatenation of some hash and a part of the date, then the mail server need remember only those messages that fall in the same date-sensitive part of the namespace. It can "forget" about ancient history. Further, if we maintain sufficient Hamming distance, we can perform "error correction" (mapping multiple IDs to the same canonical one)) and, thus compensate for minor encoding differences caused by timing skew. On Apr 8, 2012, at 12:38 PM, Barry Warsaw wrote: > On Apr 05, 2012, at 05:29 PM, Terri Oda wrote: > >> I haven't read the whole thread so maybe someone else has mentioned this, but >> we may want to take advantage of the dynamic sublists code for this, since it >> produces "conversations" or "topics" sublists and already has to generate and >> maintain a code for each. Rather than messageids these are meant to be a bit >> more human-readable, so they're often words with numbers suffixed. But yeah; >> there exists code for Mailman 2.1 that might be reusable here, and there's a >> GSoC project on the table to port to 3.0 so this might be a thing that we >> could pass to the archive utility. > > Don't forget too that we have the Stable URL proposal, which turns arbitrary > Message-IDs into 32 upper-case ASCII letter and digit character base 32 > hashes. > > -Barry From barry at list.org Sun Apr 8 23:35:46 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 17:35:46 -0400 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> <20120406174946.GM11151@unaka.lan> Message-ID: <20120408173546.1e04728d@limelight.wooz.org> On Apr 07, 2012, at 10:53 PM, David Jeske wrote: >Perhaps I misunderstood. If you are going to have a record of the deletion >(i.e. you can keep a deleted message around in some form), this problem >becomes much easier. I thought this desire was to have stable urls and >threads when you rebuild and a message is missing. > >Absolutly if there is a message 'deletion' feature, it should delete the >message contents but leave a 'stub' that links the message-id and >references/in-reply-to, so it can help hold the thread together during a >rebuild. My memory is foggy, but I think we used a technique like this in >Yahoo Groups. I like the scheme outlined by Toshio where (IIRC) any message-id can be used to index into its thread. I also agree with David that a deletion should keep enough of a stub around to maintain consistent thread links. I think this is also important for the end-user. Imagine you've found a particular taken-down message through a search engine cache. You then follow the url. I think it would be better to give them an informative message about the take-down rather than just 404'ing the url (although the latter or similar might also be useful for spiders so that they know the message is no longer available). Stephen observes that complete deletion is occasionally necessary. While true, I still think a placeholder/stub could be inserted to keep the thread integrity whole. Cheers, -Barry From barry at list.org Sun Apr 8 23:35:54 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 17:35:54 -0400 Subject: [Mailman-Developers] What is "GNU Mailman?" (was Re: mailman / archive-ui / licensing questions) In-Reply-To: References: Message-ID: <20120408173554.2a5a64a2@limelight.wooz.org> See what happens when you go on vacation? So many interesting issues to untangle! I'll try to catch up but my responses will no doubt be somewhat fractured. First of all, thanks David for bringing ClearSilver to our attention and offering it to the Mailman project. We can all agree that Pipermail is dated, is missing key features, and needs to be replaced. It's been removed from the lp:mailman branch. What does it mean to be part of the "GNU Mailman" project? This used to be an easy question to answer because everything was bundled. When you download the mm2.1 tarball, you get an archiver, a web ui, and an engine. Everything is GPLv2+'d with copyright owned by the FSF. Life is simple. Now we have at least two separate subprojects, the core and the web ui. It's very likely that what we'll call "GNU Mailman 3.0" will be just the core. For various reasons, this needs to be released as soon as possible, but it's important to understand that it won't be a full replacement for Mailman 2.1. I think of it more like the Python 3.0 release - a critical milestone for the project, usable on its own, but with deficiencies we both know about and don't yet know. We'll need to manage expectations, but it's also true that it just will not get a full workout until it's got that "final" stamp on it. We can't wait for Postorius or an official archiver, but both those will probably be included in future releases. One of the core principles of mm3 is that site admins get more choices. You can use Postorius as your web ui, but it's also easy to use something else, or no web ui at all. Want to use your own archiver? No problem. Want to throw all your data into PostgreSQL and drive the user database off your corporate databa$e? No problem (hopefully :). So again, in a mm3 world, what does it mean to be part of the GNU Mailman project? Here are some of my own principles, and I'd like to hear yours. No fiefdoms in the code. I much prefer projects where everyone feels a shared sense of stewardship for the code base. Of course we'll have experts in one area or another, and everyone is going to have an opinion about how things should work. Hopefully you won't depose me as BDFL just yet. But I do think that nobody should have veto power over any particular aspect of the code. An expert, or even myself, should be able to make a convincing technical argument for why something should be done or not done, and that should hold sway over a collective solution. Now, it might be because one way is the right way to do it, or because another way is the expedient way. And not everyone will agree with every decision, but I also think it's important to fight the good fight and then work hard to make this a successful collaboration. Of course, you're never forced to hack on something you disagree with. Almost above all else, this should be *fun* :). This relates to big code donations like the archiver. Once we as the Mailman project accept something under our umbrella, we all have the right and responsibility to dig in and hack on it. In Python, I don't think it's really worked out very well to have some big donated module be "owned" by one person. There are both pros and cons for getting subsumed into a project, so only do it when everyone understand and agrees to that. As I mentioned, bundling and releasing was easy in 2.1. It'll be easy in 3.0 only because that will probably just include the core. What does a release look like in Mailman 3.1 and beyond? How do we take all these disparate projects (Postorius, the API client, an archiver, etc.) and release these in an easy to download and install format? I'm still not sure, and I'm not holding up the 3.0 release to figure that out, but we will have to figure that out at some point (and probably get it wrong the first few times :). Licensing was also an easy decision. Everything was GPLv2+. Now the core is GPLv3+, as is Postorius. I'm not a licensing zealot; I'm pretty much happy to hack on anything that has a FLOSS license. I think a copyleft Python would fail miserably, but copyleft has worked well for Mailman for probably 15 years, and it's too late to change the license. I'm happy for people to make money off of Mailman, but the GPL helps ensure and encourage that folks give back to the project. GPLv3+ seems right for the core, and pretty right for the web ui (AGPL might be better, but Toshio has identified some problems in practice with it). I would certainly prefer that any archiver that gets bundled under the GNU Mailman moniker be copyleft, with copyright assigned to the FSF. The core and web-ui are already structured this way, so I think the consistency ultimately makes our lives, and more importantly the lives of our users, easier. Those are my preferences anyway. Maybe copyright assignment to the FSF isn't right for the archiver, but I need a more convincing argument than "I don't like it". Same for choice of license. From a project management perspective, consistency is a big win, so convince us why it's better, or at least okay, to have different licensing and ownership regimes under a single project's banner. Oh, one more principle I'd like to maintain: please write it in Python. No disrespect to other languages, but Python is just more fun and consistency counts. Okay, okay, you can include some JavaScript for the web bling if you must. :) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Sun Apr 8 23:48:55 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 17:48:55 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> Message-ID: <20120408174855.35a479ac@limelight.wooz.org> On Apr 02, 2012, at 08:04 PM, David Jeske wrote: >The question i "would you BUNDLE another archiver even if the licenses >don't match?" If you're donating the archiver to the GNU Mailman project, for us to maintain, release, bundle, and develop, then I think that would be a very high hurdle to overcome. Sorry, but it just is. I really don't want our developers to have to think about whether they can copy a chunk of useful code from the archiver to the core. Or whether they can refactor some web-ui code, developed under the GPLv3+, and re-use it as a library in the S-BSD licensed archiver code base. I know with absolute certainty that I personally don't want to have to think about stuff like that. Do you really want to spend your time trying to figure out all the insane legalistic conundrums that's going to bring up? >My archiver has been available for download (like many others) for ten >years. All these sites are still running a limping pipermail archive, >because it's bundled. I want to get Mailman a better bundled archive. Which is fantastic, and which I fully encourage. One of the reasons why Pipermail is so ubiquitous is that it was bundled with Mailman 2.1. But another reason is that it was so painful to replace. Mailman 3's architecture fixes the latter, and `bzr rm` fixed the former. >HOWEVER, I personally will not write GPL code. I might submit a tiny patch >or bugfix, but I'm simply opposed to restrictions on how someone uses >something that I'm trying to donate to the software community. (i.e. you're >never going to turn me into a mailman developer, the best you'd get is me >writing my own mailman-ish and releassing it under S-BSD.. if you want >that, let me know) I'm not going to spend time on this list arguing for the GPL. The bottom line is that the core, and by extension the web ui, are GPLv3+ and that cannot be changed. Having a different licensing and ownership regime for one component of the project will make our lives more difficult, and drain resources from developers who would rather hack than worry about legal crap. Probably the only way I'd change my mind about that is if RMS personally told us that we could still treat the non-copyleft donation the same way we treat all the other code, i.e. we can use the code and freely copy between them without any additional administrative overhead. Cheers, -Barry From barry at list.org Sun Apr 8 23:53:02 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 17:53:02 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120403185822.GI11151@unaka.lan> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> Message-ID: <20120408175302.6d0ef973@limelight.wooz.org> On Apr 03, 2012, at 11:58 AM, Toshio Kuratomi wrote: >Distributed/pointed to by list.org along with mailman and postorius might be >negotiable though :-) Absolutely. I'm committed to making it as easy as possible for an admin to integrate third-party FLOSS archivers with mm3. >I don't think you're going to find the will to make this sort of decision >right at this instant because what we want the archiver ecosystem to look >like for mailman3 is somewhat in the air. Do we really want an obviously >less capable archiver to be the bundled archiver? Do we want to have >a single blessed archiver (probably in a separate tarball as postorius, the >admin web ui, is separate) as an eventual goal? Do we want (at least for >a year or two) to let people go to town with their new ideas for archivers >and then see if a best-of-breed archiver is raising its head? I don't >believe any of this is decided inside of our minds yet, so, for now, people >are defaulting to wait and see. A hearty +1 to all of the above. I know for sure that 3.0 final won't be held up for lack of a robust archiver. Having this conversation now is important for future releases though. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Mon Apr 9 00:01:58 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 18:01:58 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <4F7BE7F4.7070607@zone12.com> Message-ID: <20120408180158.4e205e28@limelight.wooz.org> On Apr 03, 2012, at 11:56 PM, David Jeske wrote: >If this GPL/S-BSD issue turns out to be a blocker, then I'll just make my >own site and maintain (my version) there because I want to release my code >S-BSD. > >Also, there will be *zero* ill-will if you folks want to wrap it up in a >GPL license and stick it into mailman... i just won't be maintaining that, >or assigning copyright, and any patches I make will be into my S-BSD tree. >Perhaps not ideal, but still seems a better outcome than pipermail. David, there's one thing that's not clear to me. If you donated the code to GNU Mailman and we bundled it under our banner, would you continue to maintain, develop, and release it as a separate project? -Barry From barry at list.org Mon Apr 9 00:09:47 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 18:09:47 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> Message-ID: <20120408180947.30630aca@limelight.wooz.org> On Apr 03, 2012, at 09:33 PM, David Jeske wrote: >I'm just going to charge down the path I was on and finish up something >that's a great drop in for MM2/MM3. I'll even try to add some pipermail URL >compatibility. I think that's an excellent way to go, especially right now. >I'm a bit scared of a world where MM3 does not include any archiver. If >pipermail popularity is any indication of how often admins 'stick with the >bundled defaults', we could have an unreasonable number of MM3 lists with >no archives at all. Eventually, the GNU Mailman project will bundle a real archiver (not just the dumb prototype one). It won't happen for 3.0 so there's plenty of time for folks to charge ahead and make the case for their favorite through awesomeness. >Obviously the team is free to bless any archiver it wants, mine or others. It depends on what "bless" means. I think as a project we should make it easy to integrate Mailman with any FLOSS archiver. I'm also quite happy to include the IArchiver implementation shim in the core for any FLOSS archiver (as long as we can test it!) so that a site admin only needs to flip a few mailman.cfg switches to turn it on. It gets a lot more complicated if "bless" means to borg it into our project management, legal, release, and development structure. Cheers, -Barry From barry at list.org Mon Apr 9 00:14:46 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 18:14:46 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120404031831.M75170@nleaudio.com> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <20120404031831.M75170@nleaudio.com> Message-ID: <20120408181446.45756d7d@limelight.wooz.org> On Apr 03, 2012, at 11:21 PM, Bob Puff wrote: >I think the majority of MM users will be simply using the RPM that comes with >their distro, and there is a real benefit to stuff working right "out of the >box". This includes the Archiving functions. Distros are of course free to make their own opinionated decisions about how components work together. Think about the rest of the email stack: a distro makes decisions about MTA, antispam, IMAP/POP servers, etc. etc. including how they all work together and how much effort it takes to configure and run those services. Heck, entire businesses are springing up over service provisioning. So I have full confidence that distros will make things way more easy for people than it would be if you had to download and install all the individual upstream source packages. I think our job as a project is to make that possible, and easy. A secondary job is to make our own opinionated choices where appropriate. We're not yet there with the archiver, IMHO. -Barry From barry at list.org Mon Apr 9 00:17:12 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 18:17:12 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <20120404031831.M75170@nleaudio.com> Message-ID: <20120408181712.2e375a0d@limelight.wooz.org> On Apr 03, 2012, at 09:16 PM, David Jeske wrote: >I'd personally like to see a better archiver rolled into an MM2 point >release, as well as upcoming MM3 development. (I understand pipermail URL >compat would be nice in that case). I'd strongly oppose any change in default archiver for Mailman 2.1. I don't think it's possible to make that decision yet for Mailman 3.0. Including a default archiver for Mailman 3.1 should be a top priority. A web ui should be a priority as well! -Barry From barry at list.org Mon Apr 9 00:19:53 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 18:19:53 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: References: Message-ID: <20120408181953.0778290e@limelight.wooz.org> On Apr 05, 2012, at 05:18 PM, Mark Sapiro wrote: >I'd like to see a default install provide list owners with at a minimum >a choice of public, private or no archives and the archives to be >searchable. See also Jeff's first paragraph in comment #1 here: https://bugs.launchpad.net/mailman/+bug/967238 -Barry From barry at list.org Mon Apr 9 00:29:00 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 18:29:00 -0400 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <5C7FAC2C-67D1-4C23-B104-F2963750E6E5@NFSNet.org> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <4F7E0EB6.7030905@zone12.com> <20120408133816.29cb74b4@limelight.wooz.org> <5C7FAC2C-67D1-4C23-B104-F2963750E6E5@NFSNet.org> Message-ID: <20120408182900.7251d339@limelight.wooz.org> On Apr 08, 2012, at 01:11 PM, Richard Wackerbarth wrote: >I would propose a slightly different scheme for converting messages to stable >URIs.. > >If we create our ID by concatenation of some hash and a part of the date, >then the mail server need remember only those messages that fall in the same >date-sensitive part of the namespace. It can "forget" about ancient history. Hi Richard, We had a very lengthy discussion about the hash a year or so ago, when the current algorithm was agreed upon. I'm too swamped at the moment to dig up the links, but look for input from Jeff Breidenbach and Jeff Marshall. The conclusion was that Message-ID was both sufficient and preferable as the sole input into the X-Message-ID-Hash value used for stable URLs. Of course date information could certainly be used to determine expiration from any kind of Message-ID cache for LMTP acceptance purposes. It doesn't have to be part of the hash input for that. Note though that Mailman has long had a feature to "clobber" the Date header when forwarding the message on to the archive. In mm2.1 this was closely tied to Pipermail, but in mm3 this can be enabled for any archiver. The problem was that Date headers can get skewed enough that it would cause threading problems in Pipermail. It's probably true that most bogus Date headers come from spam (trying to get their message at the top or bottom of my date sorted inbox summary). >Further, if we maintain sufficient Hamming distance, we can perform "error >correction" (mapping multiple IDs to the same canonical one)) and, thus >compensate for minor encoding differences caused by timing skew. Hmm, I'm having trouble seeing how useful this would be if the Date is not used to calculate the stable url. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Mon Apr 9 01:00:10 2012 From: barry at list.org (Barry Warsaw) Date: Sun, 8 Apr 2012 19:00:10 -0400 Subject: [Mailman-Developers] triaging the remaining bugs for 3.0 final Message-ID: <20120408190010.4116fbb6@limelight.wooz.org> I see the light at the end of the Mailman 3.0 final release tunnel. I spent a few hours triaging all the bugs on Launchpad tagged with 'mailman3'. For those which I think are important to fix or investigate for 3.0 final, I marked with an importance of Critical or High. The difference there being Critical bugs I'd like to fix in beta2 whereas High bugs should be fixed before the final release, and may get knocked down in priority. There should be no Medium or Undecided bugs (the latter are possible if they are also Incomplete). Low bugs are those which would be nice to fix but won't block the 3.0 final release. For Status, Confirmed means I really do think it's a bug. Triaged just means I've looked at it but haven't yet decided whether it's a bug or not. Here's how to find all the relevant bugs for the mm3.0 final release: http://tinyurl.com/7799oek So, this means that if you're looking to help out, start with Critical bugs, then High bugs. Of course, if you find a Low bug that itches you, feel free to take a crack at it! Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From bwinton at mozilla.com Mon Apr 9 01:14:28 2012 From: bwinton at mozilla.com (Blake Winton) Date: Sun, 08 Apr 2012 19:14:28 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120408174855.35a479ac@limelight.wooz.org> References: <4F7A2304.5060408@zone12.com> <20120408174855.35a479ac@limelight.wooz.org> Message-ID: <4F821BD4.1080807@mozilla.com> On 08-04-12 17:48 , Barry Warsaw wrote: > On Apr 02, 2012, at 08:04 PM, David Jeske wrote: >> The question i "would you BUNDLE another archiver even if the licenses >> don't match?" > If you're donating the archiver to the GNU Mailman project, for us to > maintain, release, bundle, and develop, then I think that would be a very high > hurdle to overcome. Sorry, but it just is. Would it work for everyone if David licensed the archiver to Mailman under the GPLv3+? (There could still be a question about the license for contributed patches over whether they could be pulled back into the main tree or not, but as long as it was reasonably clear one way or the other, I don't think it would be a problem in practice. On the other hand, I am an optimist... ;) Later, Blake. -- Blake Winton Thunderbird User Experience Lead bwinton at mozilla.com From bwinton at mozilla.com Mon Apr 9 01:27:49 2012 From: bwinton at mozilla.com (Blake Winton) Date: Sun, 08 Apr 2012 19:27:49 -0400 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <4F821E2F.1010203@msapiro.net> References: <4F7A2304.5060408@zone12.com> <20120408174855.35a479ac@limelight.wooz.org> <4F821BD4.1080807@mozilla.com> <4F821E2F.1010203@msapiro.net> Message-ID: <4F821EF5.10708@mozilla.com> On 08-04-12 19:24 , Mark Sapiro wrote: > On 04/08/2012 04:14 PM, Blake Winton wrote: >> Would it work for everyone if David licensed the archiver to Mailman >> under the GPLv3+? > It won't work for David. See, e.g., > Well, that's not exactly what David said. ;) (I'm not proposing he stops releasing it under S-BSD, just that he re-licenses the copy in Mailman as GPL. So he can continue to work on the code and release it under a permissive license, but Mailman can also use and distribute it. ) Later, Blake. -- Blake Winton Thunderbird User Experience Lead bwinton at mozilla.com From mark at msapiro.net Mon Apr 9 01:24:31 2012 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 08 Apr 2012 16:24:31 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <4F821BD4.1080807@mozilla.com> References: <4F7A2304.5060408@zone12.com> <20120408174855.35a479ac@limelight.wooz.org> <4F821BD4.1080807@mozilla.com> Message-ID: <4F821E2F.1010203@msapiro.net> On 04/08/2012 04:14 PM, Blake Winton wrote: > Would it work for everyone if David licensed the archiver to Mailman > under the GPLv3+? It won't work for David. See, e.g., -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From stephen at xemacs.org Mon Apr 9 04:05:20 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 9 Apr 2012 11:05:20 +0900 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <4F821EF5.10708@mozilla.com> References: <4F7A2304.5060408@zone12.com> <20120408174855.35a479ac@limelight.wooz.org> <4F821BD4.1080807@mozilla.com> <4F821E2F.1010203@msapiro.net> <4F821EF5.10708@mozilla.com> Message-ID: On Mon, Apr 9, 2012 at 8:27 AM, Blake Winton wrote: > On 08-04-12 19:24 , Mark Sapiro wrote: >> >> On 04/08/2012 04:14 PM, Blake Winton wrote: >>> >>> Would it work for everyone if David licensed the archiver to Mailman >>> under the GPLv3+? >> >> It won't work for David. > > Well, that's not exactly what David said. ?;) No, that *is* what David said, and repeatedly. He will not license under GPL, period. What he has also said is that he would be happy to maintain his original distribution in parallel to a GPLed branch bundled with Mailman. He would be willing to do a (very) small amount of work to keep them in sync, I believe, but his releases will be under simplified BSD so any contributions that he is going to maintain must be licensed that way. This matters because, in practice, if there are significant contributions under GPL to the Mailman branch, it will become a real (though friendly) fork, and we will lose the benefit of David's maintenance because we'll have to integrate his changes into our branch. He won't do that for us any more. I personally see that as win-win. Barry doesn't, presumably because (1) to keep David as maintainer means that contributions either need to go through him (implicitly making themm BSD), or we'll need to do some legal dance to explicitly relicense every such contribution BSD (since in practice our contributor agreement will make any contribution to Mailman itself GPLv3+ only), which (2) implicitly gives David veto power over the bundled archiver. The reason I see it as win-win is that I don't think there will be a lot of contribution from the current Mailman core to David's archiver. There clearly is a lot of enthusiasm for something with social networking features, and David's archiver doesn't look like a good platform for that to me. Eventually, the recommended (and bundled) archiver will be something else. > (I'm not proposing he stops releasing it under S-BSD, just that he > re-licenses the copy in Mailman as GPL. David doesn't need to do anything. We just copy the code and release it in Mailman under the GPLv3+ like the rest of Mailman. That's just a special case of the main reason for using a BSD license. >?So he can continue to work on the > code and release it under a permissive license, but Mailman can also use and > distribute it. ) There's nothing stopping us from doing that, not even the possibility of offending David. That's *why* he uses BSD in the first place, so we can do that if we want to. But he won't do it for us. From stephen at xemacs.org Mon Apr 9 04:14:51 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 9 Apr 2012 11:14:51 +0900 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120408174855.35a479ac@limelight.wooz.org> References: <4F7A2304.5060408@zone12.com> <20120408174855.35a479ac@limelight.wooz.org> Message-ID: On Mon, Apr 9, 2012 at 6:48 AM, Barry Warsaw wrote: > On Apr 02, 2012, at 08:04 PM, David Jeske wrote: > Probably the only way I'd change my mind about that is if RMS personally told > us that we could still treat the non-copyleft donation the same way we treat > all the other code, i.e. we can use the code and freely copy between them > without any additional administrative overhead. He won't do that, because it's not possible. You cannot freely copy from a copyleft code base into a non-copyleft code base; you must indenture the latter. What we can do is branch the code, and freely copy back-and-forth between Mailman core and the code we got from the non-copyleft code base. The potential costs of that I point out in another message, so don't reply to this one. :-) From stephen at xemacs.org Mon Apr 9 04:18:49 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 9 Apr 2012 11:18:49 +0900 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <20120408182900.7251d339@limelight.wooz.org> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <4F7E0EB6.7030905@zone12.com> <20120408133816.29cb74b4@limelight.wooz.org> <5C7FAC2C-67D1-4C23-B104-F2963750E6E5@NFSNet.org> <20120408182900.7251d339@limelight.wooz.org> Message-ID: On Mon, Apr 9, 2012 at 7:29 AM, Barry Warsaw wrote: > On Apr 08, 2012, at 01:11 PM, Richard Wackerbarth wrote: > >>I would propose a slightly different scheme for converting messages to stable >>URIs.. >> >>If we create our ID by concatenation of some hash and a part of the date, >>then the mail server need remember only those messages that fall in the same >>date-sensitive part of the namespace. It can "forget" about ancient history. > > We had a very lengthy discussion about the hash a year or so ago, when the > current algorithm was agreed upon. ?I'm too swamped at the moment to dig up > the links, but look for input from Jeff Breidenbach and Jeff Marshall. I believe it's the thread including this message: http://mail.python.org/pipermail/email-sig/2012-January/000883.html I don't really see the point of not storing all the IDs, anyway. A million message IDs isn't even going to take up a gigabyte! (I think it's reasonable to reject a 1000-byte Message-ID as an attack, don't you?) Anybody who's running an archive that receives unique messages in mega-message units presumably has enough resources that they can afford the odd gigabyte (heck, even in RAM ;-) even if not all the messages are going to be stored in the archive due to expiration policies or whatever. From davidj at gmail.com Mon Apr 9 06:37:46 2012 From: davidj at gmail.com (David Jeske) Date: Sun, 8 Apr 2012 21:37:46 -0700 Subject: [Mailman-Developers] mailman / archive-ui / licensing questions In-Reply-To: <20120408180158.4e205e28@limelight.wooz.org> References: <4F7A2304.5060408@zone12.com> <20120403185822.GI11151@unaka.lan> <4F7BE7F4.7070607@zone12.com> <20120408180158.4e205e28@limelight.wooz.org> Message-ID: I think the last several messages covered whats-what pretty well. Summarizing what already seems to have been reclarified a few times excellently by others... ClearSilver List Archive is S-BSD, and will remain so. That license allows you folks to wrap it in GPLv3 if you wish, but I won't be doing so myself or assigning copyright as I don't wish those restrictions to be enforced on my code. I apologize that this license discussion has lasted as long as it has, as I'm sure we'd all rather be talking about cool archiver UI code and features. :) The only remaining question I saw was Barry's here... On Apr 8, 2012 3:02 PM, "Barry Warsaw" wrote: > David, there's one thing that's not clear to me. If > you donated the code to GNU Mailman and > we bundled it under our banner, would you continue > to maintain, develop, and release it as a separate > project? If MM bundled (some version of) the code, wrapped it in GPLv3, and maintained it, I don't anticipate I'd maintain, develop, and continue to release a separate project. I'd merely keep my webpage up distributing the S-BSD code-release. If I did make changes, I'd distribute them as S-BSD patches to my S-BSD code. However, seeing as CSLA hasn't changed in a decade, after I'm done updating it, my contributions probably wouldn't change for a decade more. By my view of this entire license and bundling discussion it seems like the most practical possibilities are: 1) If MM really likes how CSLA ends up, you folks can adopt and GPLv3 the code, effectively becoming the official maintainers of the project.. (accepting that the GPLv3 restrictions couldn't be enforced on the original code, as it's also released S-BSD) 2) If MM likes how CSLA ends up, but would rather have me maintain it... I can maintain it as a separate S-BSD project, and MM can point-to or reference it as one of the external (yet easy to install) archiver options. 3) If MM doesn't like how CSLA ends up, then we can all have a good laugh at how much time we spent in theoretical license discussions over something that didn't matter. Let's hope not #3. I'm going to have to work extra hard now to be sure that doesn't happen. :) I learned more about license nuances and general MM dev thoughts from this thread that I expected, so thanks too everyone that replied and contributed! From davidj at gmail.com Mon Apr 9 06:52:34 2012 From: davidj at gmail.com (David Jeske) Date: Sun, 8 Apr 2012 21:52:34 -0700 Subject: [Mailman-Developers] Integrating HyperKitty with Mailman3 In-Reply-To: <20120408123921.0ff6d14f@limelight.wooz.org> References: <20120408123921.0ff6d14f@limelight.wooz.org> Message-ID: Are you expecting this direct python configuration import to actually "be" an archiver, or simply to be a configuration shim to get data to an archiver? Python imports are not version-dependent (like C-shlibs are), so it seems dubious to expect an external archiver to necessarily be compatible with the same version of python that MM3 is. I know I've run into this problem in the past, especially because of how much the python MIME message classes changed over each python release (though hopefully they are more stable now) On Apr 8, 2012 9:39 AM, "Barry Warsaw" wrote: > -----snip snip----- > [archiver.hyperkitty] > class: python.path.to.hyperkitty.HyperKitty > -----snip snip----- > > Of course, you'd probably want to `enable` it too. > > One tricky thing here is that the `class` value names a Python dotted-module > path, so the class must be importable. Ensuring that the hyperkitty module > (and this is just a suggestion, YMMV) is importable by the core engine may not > be fully baked. For now, just set $PYTHONPATH. From richard at nfsnet.org Mon Apr 9 14:10:45 2012 From: richard at nfsnet.org (Richard Wackerbarth) Date: Mon, 9 Apr 2012 07:10:45 -0500 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> <4F7E0EB6.7030905@zone12.com> <20120408133816.29cb74b4@limelight.wooz.org> <5C7FAC2C-67D1-4C23-B104-F2963750E6E5@NFSNet.org> <20120408182900.7251d339@limelight.wooz.org> Message-ID: On Apr 8, 2012, at 9:18 PM, Stephen J. Turnbull wrote: > I don't really see the point of not storing all the IDs, anyway. Not only does this require excessive resources, but it requires significant infrastructure for failure recovery. (Think backups, journaling, etc.) That requirement may not be an issue for Google, but it is a significant additional burden for small operations, migrations, etc. I support the concept of Stable URI. The concept of using a hash into a large namespace is probably adequate. However, at a minimum, the URI SHOULD include an easily identifiable schema-revision indicator. That way, if the present scheme is found lacking, we can, compatibly, switch to a new schema and a new namespace. Further, by intentionally changing the namespace, based on time, it becomes reasonable to assure uniqueness in all but exceptional situations without requiring infinite perfect memory. Further, by switching namespaces, past faults in that memory become self-healing. I think that migrations, alone, justify the use of a scheme that does not require infinite preservation of all past message IDs. I would hope that the historical experience in the crypto world would convince you of the need to make provision for an unknown future. There, schemes that were thought to be "unbreakable" have been adopted and widely used. Only, well after that time, was it discovered that there was a flaw and a new scheme needed to be utilized. The use of long-lived stable URIs needs to be prepared for that eventuality. Therefore, the URI must self-identify its namespace and the namespace must not be based solely on something that can outlive the use of the namespace. From barry at list.org Mon Apr 9 16:59:26 2012 From: barry at list.org (Barry Warsaw) Date: Mon, 9 Apr 2012 10:59:26 -0400 Subject: [Mailman-Developers] Integrating HyperKitty with Mailman3 In-Reply-To: References: <20120408123921.0ff6d14f@limelight.wooz.org> Message-ID: <20120409105926.782f1e51@limelight.wooz.org> On Apr 08, 2012, at 09:52 PM, David Jeske wrote: >Are you expecting this direct python configuration import to actually "be" >an archiver, or simply to be a configuration shim to get data to an >archiver? Whatever makes the most sense for that particular archiver. The prototype archiver is so (purposely) dumb that it's implemented right in process. The Mail Archive shim just drops a copy of the message into the outgoing queue, after it calculates the appropriate recipient address. IOW, it sets the message up to be forwarded to their service over SMTP. The MHonArc shim just shells out to the appropriate command, piping the message bytes to that command's stdin. So I can turn this question around and ask, what's the best way to get messages into ClearSilver? >Python imports are not version-dependent (like C-shlibs are), so it seems >dubious to expect an external archiver to necessarily be compatible with >the same version of python that MM3 is. I know I've run into this problem >in the past, especially because of how much the python MIME message classes >changed over each python release (though hopefully they are more stable now) Well, until email6 is released, Mailman 3 is ported to Python 3, and we can all (finally) do email the right way in Python. :) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Mon Apr 9 17:16:23 2012 From: barry at list.org (Barry Warsaw) Date: Mon, 9 Apr 2012 11:16:23 -0400 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> <4F7E0EB6.7030905@zone12.com> <20120408133816.29cb74b4@limelight.wooz.org> <5C7FAC2C-67D1-4C23-B104-F2963750E6E5@NFSNet.org> <20120408182900.7251d339@limelight.wooz.org> Message-ID: <20120409111623.330b3ef9@limelight.wooz.org> On Apr 09, 2012, at 11:18 AM, Stephen J. Turnbull wrote: >On Mon, Apr 9, 2012 at 7:29 AM, Barry Warsaw wrote: >> We had a very lengthy discussion about the hash a year or so ago, when the >> current algorithm was agreed upon. ?I'm too swamped at the moment to dig up >> the links, but look for input from Jeff Breidenbach and Jeff Marshall. > >I believe it's the thread including this message: > >http://mail.python.org/pipermail/email-sig/2012-January/000883.html Shockingly, it's even older than that. I just did a quick perusal of the page in the wiki which defines this. Revision 18 dated 2008-07-03 is the first one that contain the current description of the algorithm: "X-Message-ID-Hash is calculated from the Base 32 encoded SHA 1 hash of the Message-ID header. As with RFC 2822, the angle bracket delimiters are not considered part of the Message-ID and MUST NOT contribute to the hash." http://wiki.list.org/display/DEV/Stable+URLs So yeah, it was a little more than "a year or so ago" :). >I don't really see the point of not storing all the IDs, anyway. A >million message IDs isn't even going to take up a gigabyte! (I think >it's reasonable to reject a 1000-byte Message-ID as an attack, don't >you?) Anybody who's running an archive that receives unique messages >in mega-message units presumably has enough resources that they can >afford the odd gigabyte (heck, even in RAM ;-) even if not all the >messages are going to be stored in the archive due to expiration >policies or whatever. Agreed! As I mentioned to Richard, it's not necessary that X-Message-ID-Hash be used as the thread id, in part or in whole. That's not its original purpose. The important thing is that any message in the archiver must be discoverable given its Message-ID or X-Message-ID-Hash. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Mon Apr 9 17:28:53 2012 From: barry at list.org (Barry Warsaw) Date: Mon, 9 Apr 2012 11:28:53 -0400 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: References: <1333633288.23207.26.camel@ambre.pingoured.fr> <4F7E0EB6.7030905@zone12.com> <20120408133816.29cb74b4@limelight.wooz.org> <5C7FAC2C-67D1-4C23-B104-F2963750E6E5@NFSNet.org> <20120408182900.7251d339@limelight.wooz.org> Message-ID: <20120409112853.04997e02@limelight.wooz.org> On Apr 09, 2012, at 07:10 AM, Richard Wackerbarth wrote: >I support the concept of Stable URI. The concept of using a hash into a large >namespace is probably adequate. However, at a minimum, the URI SHOULD >include an easily identifiable schema-revision indicator. That way, if the >present scheme is found lacking, we can, compatibly, switch to a new schema >and a new namespace. Should we attempt to push the stable URI concept as an RFC? Does anybody (Murray perhaps) have the interest and time to do that? I think the RFC would be pretty simple. Having an RFC would also be nice for getting rid of the X- prefix. In any event, we can declare the algorithm on our current wiki page to be version 1.0 of our stable URI definition. Archiver search algorithms can expose this version number in their URLs if they're so inclined. E.g.: http://mail.example.com/1.0/7GC2V6BEDVME27VQ34W7AXMFPA3H2YWW I should probably also be able to find the message this way: http://mail.example.com/search?message-id=%3C20120409152339.16496.75486%40foo.example.org%3E and probably http://mail.example.com/search?strict=1&message-id=20120409152339.16496.75486%40foo.example.org and maybe others. -Barry From stephen at xemacs.org Mon Apr 9 19:43:16 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 10 Apr 2012 02:43:16 +0900 Subject: [Mailman-Developers] From the creation of a ThreadID In-Reply-To: <20120409112853.04997e02@limelight.wooz.org> References: <1333633288.23207.26.camel@ambre.pingoured.fr> <4F7E0EB6.7030905@zone12.com> <20120408133816.29cb74b4@limelight.wooz.org> <5C7FAC2C-67D1-4C23-B104-F2963750E6E5@NFSNet.org> <20120408182900.7251d339@limelight.wooz.org> <20120409112853.04997e02@limelight.wooz.org> Message-ID: On Tue, Apr 10, 2012 at 12:28 AM, Barry Warsaw wrote: > On Apr 09, 2012, at 07:10 AM, Richard Wackerbarth wrote: > >>I support the concept of Stable URI. > > Should we attempt to push the stable URI concept as an RFC? ?Does anybody > (Murray perhaps) have the interest and time to do that? ?I think the RFC would > be pretty simple. I don't think we have sufficient agreement on how to implement yet. > Having an RFC would also be nice for getting rid of the X- prefix. AIUI, the X- prefix is now considered a bad idea for public protocols in any case. I don't think we need an RFC for it until we're pretty sure we have it right. > In any event, we can declare the algorithm on our current wiki page to be > version 1.0 of our stable URI definition. ?Archiver search algorithms can > expose this version number in their URLs if they're so inclined. IMHO, our stable URIs should work on any of the servers we might connect to to retrieve the message. In terms of best current practice, Gmane has offered stable URLs for about a decade now: http://msgid.gmane.org/20120409152339.16496.75486 at foo.example.org To put it on the wire to Gmane, just URL-encode the message-id and be done with it. IMO, the ideal would be just like netnews: list-archive://mailman-developers.python.org/20120409152339.16496.75486 at foo.example.org The List-ID is not entirely redundant due to cross-posting. In this scheme, it's up to the MUA to decide which archive(s) to query for this, just as with netnews looking for a newsgroup. I really don't see why the stable URI would want to be anything else. So the scheme on the wiki seems overengineered to me, with the possible exception of the "industrial-strength message IDs are too long for the footer" problem. But http://mail.example.com/1.0/7GC2V6BEDVME27VQ34W7AXMFPA3H2YWW is really too long for a footer too; what we want are tinyurls. So I think that footer URLs should be considered a different problem from the stable URI problem. From richard at nfsnet.org Mon Apr 9 22:35:08 2012 From: richard at nfsnet.org (Richard Wackerbarth) Date: Mon, 9 Apr 2012 15:35:08 -0500 Subject: [Mailman-Developers] [Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3 In-Reply-To: <20120408174323.2395.52774.launchpad@gac.canonical.com> References: <20120326174611.5464.44688.malonedeb@chaenomeles.canonical.com> <20120408174323.2395.52774.launchpad@gac.canonical.com> Message-ID: On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote: > Barry Warsaw (barry) has assigned this bug to you for GNU Mailman: > Need a script to upgrade from MM2 to MM3 > https://bugs.launchpad.net/bugs/965532 Here are some thoughts on a possible migration technique. I would request discussion and suggestions. In particular, what about the idea of converting the configuration file to HTML as an intermediate file format? Selectable css could easily render it as a viewable report. It could still be edited by hand without too much difficulty. Richard "Wacky" Wackerbarth - - - - - - Steps to migrate from MM2 to MM3 1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should include testing to assure that things are ready to create new lists. 2) Translate list configurations a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would populate configurations at the List level. It might also reformat selected parameters. In particular, various