From glibersat at sigill.org Sat Nov 1 18:36:28 2014 From: glibersat at sigill.org (Guillaume Libersat) Date: Sat, 01 Nov 2014 18:36:28 +0100 Subject: [Mailman-Developers] MM3 Sqlite to PostgreSQL migration? Message-ID: <54551A1C.5090605@sigill.org> Hi, We are currently running a MM3 instance in "production" with a SQLite backend (at ulists.org). It was at first a test, but a lot of people have already started to use the service and now we are kind of stuck (but happy ;-) ). First, I've seen a lot of DBLock issues in the logs and I think there also is a race condition happening with Held Messages that leads to subsequents 500 until you manually copy fake email files (If I'm correct, the held message file is deleted, then the SQL query is made but fails. The result is a system in an inconsistent state: there's a held message in DB, but no more on disk). Second, I've tried to manually migrate to PostgreSQL but did not succeed because of a lot typing issues and differences between these two databases. Barry told me there's an SQLAlchemy port that is soon to be released, at least in alpha stage. Do you think it could help to achieve such a migration? I'm willing to help testing this new backend and, if someone can guide me with the MM stuff, write migration scripts. Thanks and again, MM3 is really an awesome suite of software! :) Guillaume From barry at list.org Sat Nov 1 19:18:05 2014 From: barry at list.org (Barry Warsaw) Date: Sat, 1 Nov 2014 14:18:05 -0400 Subject: [Mailman-Developers] ORM layer transition Message-ID: <20141101141805.206254e4@limelight.wooz.org> Hi Developers! In honor of the penultimate Mailman Day of 2014, I wanted to announce a significant transition in the implementation of Mailman 3 and call out the contributions of two of our members. For a long while now we've used Storm as the ORM layer in MM3. ORM, meaning "object relational mapper" is the chunk of code that translates between the persistent Python objects in a running MM3 system and the underlying SQL database. Storm was a good choice many years ago because it was a nice, simple, thin layer that was easy to understand, use, and customize. In the intervening years though, Storm has shown its age. It is no longer effectively maintained upstream, and it does not support Python 3. While MM3 is not yet a Python 3 application, it is my strong desire to eventually make it so. The fact that Storm has no Python 3 port, and is unlikely to, would prevent MM3 from being ported[*]. In a survey of the alternative available standalone Python ORMs, SQLAlchemy stands out as popular, well-maintained, long stable, and Python 3 compatible, and thus I am happy to announce that the MM3 trunk has been ported to SQLAlchemy (SA). The full test suite completes against both SQLite and PostgreSQL as back-ends, and I would expect that with a little bit of work (contributions welcome!) any other back-end supported by SA should work too. In addition, rather than the ball-of-hack I wrote to migrate the database schema from one version to another, we will now be using Alembic to manage schema migrations. I must acknowledge two really spectacular contributions from our members. Abhilash Raj began the SA port almost on a dare by me on IRC and with many discussions in various forums, produced the bulk of the port. He asked great questions, answered mine own with patience, showed good diligence, and the depth of knowledge about MM3 that such a port required. I have never met Abhilash in person but the quality of his work is impressive. Aur?lien Bompard also provided invaluable contributions to this branch. Some of you may recognize Aur?lien from his work on Hyperkitty, the fabulous new archiver. Aur?lien was able to join us last year at the Pycon sprint, and it was really great to be able to hang out with him. He did an incredible amount of testing and refinement on the SA branch, feeding me patches that fixed a few of the outstanding test failures, especially on PostgreSQL. Without the contributions of both Abhilash and Aur?lien, this port would not have happened. Kudos to them, and my personal appreciation for their patience and enthusiastic good spirits in working with their picky, curmudgeonly project leader, who took much too long to review and merge their work. :) I encourage the adventurous among you to grab the bzr trunk and give it a whirl. Do let us know here or in the bug tracker about any problems you might encounter, especially if you are upgrading from an existing Storm-based instance of MM3. At least one more release of the core is planned this year. Enjoy, -Barry [*] We have one other known Python 2-only dependency blocking our porting efforts. We use the restish library as our REST/HTTP framework, and while there are many things I love about restish, it falls into the same camp as Storm. It's Python 2-only, and effectively abandoned upstream. Fortunately we have a ready alternative in the Falcon framework. I have a branch of MM3 ported to Falcon ready to go as soon as upstream releases its new version, which will have a few changes I submitted that allows us to implement restish-style object-based traversal. This means we can port to Falcon, keep largely the same REST code in place, and be ready for Python 3. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From stephen at xemacs.org Sat Nov 1 20:16:35 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 02 Nov 2014 04:16:35 +0900 Subject: [Mailman-Developers] ORM layer transition In-Reply-To: <20141101141805.206254e4@limelight.wooz.org> References: <20141101141805.206254e4@limelight.wooz.org> Message-ID: <87ioiy20ks.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > I am happy to announce that the MM3 trunk has been ported to > SQLAlchemy (SA). The full test suite completes against both SQLite and > PostgreSQL as back-ends, and I would expect that with a little bit of work > (contributions welcome!) any other back-end supported by SA should work too. Huzzah! Happy Mailman Day! (Although it isn't here any more. :) > Without the contributions of both Abhilash and Aur?lien, this port would not > have happened. Kudos to them, +1 !! Steve From victoriano at uma.es Sun Nov 2 16:46:46 2014 From: victoriano at uma.es (Victoriano Giralt) Date: Sun, 02 Nov 2014 16:46:46 +0100 Subject: [Mailman-Developers] Fwd: [sympa-users] thoughts re. DMARC impacts In-Reply-To: <544EE72C.2080708@meetinghouse.net> References: <544EE72C.2080708@meetinghouse.net> Message-ID: <545651E6.5030606@uma.es> Folks, first my excuses for the "dual posting" but I did not want to loose the odd Mailman developer that is not subscribed to the users list. This has recently circulated in the sympa-users lists. As a proud member of both communities I think that the synergy Miles proposes could really have some impact. Actually, it would be great if the possible RFC had an author from each and every list manager in existence (there are not so many :)), or at least from the big ones, including the commercial dinosaur, ListServ (I'm connected to a big instance of this beast, so I might get a direct channel to them). All this said, please, take a look at Miles message ;-) -------- Original Message -------- Subject: [sympa-users] thoughts re. DMARC impacts Date: Mon, 27 Oct 2014 20:45:32 -0400 From: Miles Fidelman Reply-To: Miles Fidelman To: sympa-users at cru.fr , dmarc at ietf.org Folks, Many of us just had to deal with the impacts of DMARC on our lists. In the aftermath, I've been participating on the dmarc-ietf email list - trying to discuss ways to "fix DMARC" to better coexist with lists and other 3rd-party services (like "send this article to...."). Unfortunately, it seems like the discussion is bogged down in two regards: - the ietf-dmarc working group is charted to "enhance DMARC" - dealing with its impacts is not really the focus - folks want a general purpose solution Personally, I'd like to see a short-term solution to make our lives easier, as list managers - sort of the way that NAT has been dealing with IPv4 address exhaustion, while we wait for IPv6 to happen. I've been thinking along the lines of an update to RFC2369 - the 16-year-old document defines the List-* headers. Say by adding a couple of headers along the lines of: List-Original-Author: List-Original-Reply-To: List-Reply-To: Seems to me that this would: 1. give us a standard way to find the original author (and for HTML mail, to reply by clicking on a mailto: URL in the header) 2. provide standardized information that could be used by MUAs for identifying, and presenting reply options (maybe leading to "reply to list" and "reply to author" buttons starting to show up on toolbars) 3. set the stage for adding some authentication mechanisms that accomplish what DMARC is intended to do (e.g. by adding a few more headers that cryptographically authenticate the new List- headers and bind them to the message body) It strikes me that this might proceed rather quickly, if incorporated into an RFC co-authored and submitted by folks from the mailing list software community (i.e., the folks who'd write the patches to Sympa, Mailman, ezmlm, etc) - particularly if some running code were to be implemented as part of the process. (Can you say, "rough consensus and running code?") Reactions? Anybody want to collaborate on a draft RFC for submission through the independent submissions path? Any thoughts on who needs to be involved from the Sympa community, and/or from the other mailing list software communities? Miles Fidelman (a frustrated sympa administrator) -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra -- Victoriano Giralt Central ICT Services Systems Manager University of Malaga +34952131415 SPAIN ================================================================== Note: signature.asc is the electronic signature of present message A: Yes. > Q: Are you sure ? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting annoying in email ? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 213 bytes Desc: OpenPGP digital signature URL: From stephen at xemacs.org Mon Nov 3 05:12:43 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 03 Nov 2014 13:12:43 +0900 Subject: [Mailman-Developers] Fwd: [sympa-users] thoughts re. DMARC impacts In-Reply-To: <545651E6.5030606@uma.es> References: <544EE72C.2080708@meetinghouse.net> <545651E6.5030606@uma.es> Message-ID: <87bnoo2a84.fsf@uwakimon.sk.tsukuba.ac.jp> Reply-To set to Mailman-Users. Although actual work on an RFC probably would be done by a developer, there's no reason to exclude site admins and list owners, and the impact of DMARC is surely apparent to developers, admins, and owners alike, as well as to our subscribers. Victoriano Giralt writes: > This has recently circulated in the sympa-users lists. As a proud member > of both communities I think that the synergy Miles proposes could really > have some impact. Thank you for posting. Mailman as an organization (through me, the more or less official delegate to the DMARC Working Group - below, just "the WG") is well aware of Miles' proposal. In short, I'm opposed to it in current form, both personally and on behalf of Mailman. That said, I welcome discussion and advocacy of his point of view, and if the Mailman community goes there, we can either delegate more people to advocate it or I can do my best to present the Mailman consensus as well as my own view (assuming it doesn't change in response to further advocacy :-). The rest is tl;dr, I suppose, but it is a complete and I hope fair summary of discussion of these proposals on the dmarc at ieft.org list: http://www.ietf.org/mail-archive/web/dmarc/ > Subject: [sympa-users] thoughts re. DMARC impacts > Date: Mon, 27 Oct 2014 20:45:32 -0400 > From: Miles Fidelman > Many of us just had to deal with the impacts of DMARC on our lists. > > In the aftermath, I've been participating on the dmarc-ietf email > list - trying to discuss ways to "fix DMARC" to better coexist with > lists and other 3rd-party services (like "send this article > to...."). > > Unfortunately, it seems like the discussion is bogged down in two regards: > - the ietf-dmarc working group is charted to "enhance DMARC" - dealing > with its impacts is not really the focus - folks want a general purpose > solution This is false. The charter is public: http://tools.ietf.org/wg/dmarc/charters In fact, Miles' proposal, and several similar ones, have received a lot of attention from the WG. Nobody has claimed it's out of WG scope or tried to shut down discussion on that basis, although it's "off current focus". It's true that two specific participants are strong proponents of "general purpose solutions", but they are "special interests" (a party of 2, or perhaps 2 parties of 1 ;-). The people who are the backbone of the WG have on average two *decades* of experience in creating email-related and other Internet standards, and they are quite happy to consider incremental improvements, two of which (the *-dkim-* Internet Drafts) are cited below. Currently, the WG's focus is on figuring out just what those impacts are, it's true, and that is a "general purpose" task. Nevertheless, many members of the WG are open to "off-focus" work as individuals, and the WG wiki is open as a forum for presenting "off-focus" ideas to be addressed ad hoc by individuals or by the WG in the future. http://tools.ietf.org/wg/dmarc/trac/wiki > Personally, I'd like to see a short-term solution to make our lives > easier, as list managers - sort of the way that NAT has been dealing > with IPv4 address exhaustion, while we wait for IPv6 to happen. This is an excellent analogy. NAT has caused almost as many problems as it solved, especially in the security area. Miles' proposal would be the same IMO. > I've been thinking along the lines of an update to RFC2369 - the > 16-year-old document defines the List-* headers. Say by adding a couple > of headers along the lines of: > List-Original-Author: > List-Original-Reply-To: These two have trivial generalizations which apply to some of the other third-party issues created by DMARC. Specifically, just drop the "List-" prefix. This generalization *will* occur -- those users will perceive the utility to themselves, and just start using it. The misnaming may not cause problems, but why invite them when we don't have to? By either name (or slight variations, such as "Original-From" and "X-Original-From" -- the latter has long been used by GMail for internal purposes, apparently), this proposal has the following design issues that must be resolved: 1. It changes the semantics of the From field defined by RFC 5322. According to RFC 5322, From is the definitive field indicating the author of the message content in a sense made somewhat precise in that RFC and other email RFCs. This proposal would add a very general "except when it isn't" clause to that definition, requiring updates to every MUA in use. It may affect many other RFCs as well -- specifically but not limited to all email authentication RFCs that propose to address the phishing problem. 2. Advocates of such headers are sharply divided on proposed semantics. Some (Google, Scott Kitterman on the WG ML) consider that the semantics should be restricted to the Reply-To function, and otherwise be invisible to end users. Miles himself clearly favors "really truly From" semantics that would encourage MUAs to make it visible. I believe that in practice Miles' interpretation would win, and in that case I'm quite sure that there would be zero-day exploits by phishers (unless the DMARC folks get there first by implementing DKIM signatures applied to empty Original-Author fields!) 3. No serious consideration has been given to potential new exploits or social engineering under either semantics, although the possibility of generalizing existing phishing techniques under the "really truly from" interpretation are painfully obvious. 4. No consideration has been given to precise semantics of Original-Reply-To. 5. No consideration has been given to alternative use of, or interaction with combined with, existing Resent-* fields. 6. No consideration has been given to multiple instances of the Original-* fields. In my opinion, some of these issues (1, 2, 3) cannot be resolved satisfactorily in a reasonable amount of time, and therefore the proposal is fatally flawed as a "fast-track" IETF document. It could proceed (as DMARC itself did!) as a private agreement between mailing lists and MUA developers. The argument that an IETF RFC would make coordination with MUA developers easier is valid but weak. The popular MUA developers do not really consider RFCs in their designs (several have told me this, including Mozilla folks). Rather, they wait for implementations to appear in wide use, and then leverage those implementations as serves their users. > List-Reply-To: I have an old proposal related to this, with the intent of mitigating the Reply-To munging issue. (I'm not sure what Miles wants it for.) I'll dig it up. N.B. I never submitted an Internet Draft, that's when the MUA authors told me they ignore such things. :-) > Seems to me that this would: > 1. give us a standard way to find the original author Correct. > (and for HTML mail, to reply by clicking on a mailto: URL in the > header) [Aside: This feature isn't at all restricted to HTML mail. It's just a function of the MUA's presentation capabilities, and some text- oriented MUAs provide hotkeys for navigating and activating links.] > 2. provide standardized information that could be used by MUAs for > identifying, and presenting reply options I don't think these fields provide any information that the RFC 5322 and RFC 2369 fields don't already contain. The problem that this proposal is intended to address is that DMARC is designed to control where that information can be provided and bound to the message content -- specifically restricting these functions to the Author Domain's MTA. Theoretically they could authorize third parties, in practice they don't, and there's good reason to believe they won't do so soon. The problem with the proposal is that any attempt to allow the good guys (lists) to provide that information bound to altered message content[1] will provoke responses from the phishers, and from the DMARC participants. > (maybe leading to "reply to list" and "reply to author" buttons > starting to show up on toolbars) They already do (eg, Mutt and KMail). In any case the required information has long been available in the RFC 2369 List-Post field -- if an MUA hasn't implemented reply-to-list by now, its developers have their reasons for not doing so. > 3. set the stage for adding some authentication mechanisms that > accomplish what DMARC is intended to do (e.g. by adding a few more > headers that cryptographically authenticate the new List- headers and > bind them to the message body) In fact, DKIM can *already* authenticate those headers. The problem posed by DMARC is that Author Domains[2] want to control the use of mailboxes in their domains in the From field[3], and do so by signing that field. This will not cause problems where mailboxes are for corporate convenience -- the corporation just tells its agents that they can't use those mailboxes to send to public mailing lists or for other third-party uses. Where it does cause problems is at those services which provide mailboxes for arbitrary personal use. It is precisely those services that balk at delegating signing authority, primarily due to the costs of investigating and approving third parties, and for some third-party authorization schemes, burdens on infrastructure (especially the DNS). (Here, corporations will have a for-profit or increased-benefit reason for delegation, and so will be perfectly happy to pay the costs of delegation when appropriate.) It is already possible for Author Domains to delegate authentication of From to mailing lists and other third-party services. There is a now-mature and very elaborate proposal for a protocol similar to DMARC itself that uses the DNS to publish such authorizations on a long-term basis: http://tools.ietf.org/html/draft-otis-tpa-label and two that extend DKIM to allow message by message authorization: http://tools.ietf.org/html/draft-kucherawy-dkim-delegate http://tools.ietf.org/html/draft-levine-dkim-conditional However, it is rather questionable whether services like Google or Yahoo! with hundreds of millions of mailboxes will be able to vet tens or hundreds of thousands of third-party services. Google and Yahoo! technical staff have already indicated on the WG list that they really would rather not. The effective response of DMARC-using Author Domains is already known: sign empty "Original-*" fields corresponding to existing fields that are known to be useful in phishing and spoofing in general. Whether they will actually do that depends on MUA take-up and whether the spoofers exploit. That seems very likely to be highly correlated with how effective these fields are at achieving their intended purpose. IMO, to make these proposals useful, proponents need to provide a plausible mechanism to prevent spoofing without preventing the intended use at the same time. > It strikes me that this might proceed rather quickly, if incorporated > into an RFC co-authored and submitted by folks from the mailing list > software community The writing of a document might go quickly, but it takes at least a year to get to Proposed Standard status, even without IETF community opposition. Cf. DMARC itself, which is actually four or five years old by now. > (Can you say, "rough consensus and running code?") Running code, yes, but that "rough consensus" would have to include (self-appointed ;-) representatives of *all* email users, not just the MLs. A rough consensus of a subset is exactly what got us DMARC. And those other "representatives" include a *lot* of people who take a very dim view of screwing with RFC 5322 semantics. > Reactions? Anybody want to collaborate on a draft RFC for > submission through the independent submissions path? Any thoughts > on who needs to be involved from the Sympa community, and/or from > the other mailing list software communities? As I say, I have an old draft for something related to List-Reply-To which I will dig up and post here for reference.[4] I won't be participating in writing up any of Miles' proposals until I've seen plausible ways to address issues 1-6 presented above. I encourage anybody who thinks that they all can be solved to do so! and to participate. Aone piece of advice: from the point of view of Miles and other proponents, the resistance from the IETF old guard is "political". But I believe it will be strong. You could try to present it as an informational RFC ("mailing lists are doing this even though it doesn't conform to applicable standards"), but I suspect the RFC editor would delay publication until (1) the impact of this practice on conformance to email RFCs in general is described (issue 1 above must be resolved), and (2) the response of phishers and DMARC participants to the practice is visible. For this reason alone, I strongly recommend creating an ad hoc "Committee for the Preservation of Mailing Lists" or something like that, and publishing your standard independently of the RFC process. There is precedent for this. Daniel Bernstein's "Mail-Followup-To" header is widely provided and respected by open source MUAs, both for news posts (the original application) and for email (primarily to mailing lists). Of course the pain of DMARC is mostly felt by users of the big mailbox providers (ie, not users of open source MUAs), but there you already have GMail (and maybe other webmail services, I haven't checked) using a variant of the proposal. (Warning: that doesn't mean that GMail would implement the features that you propose in the user-visible MUA! But at least they're not opposed to the header field itself.) Good luck! Steve Footnotes: [1] In the sense of invalidating the first-party DKIM signature. [2] By which I mean an organization such as Yahoo! that maintains a domain registration and MTA infrastructure, and provides mail transport services including mailbox addresses and storage to users. [3] They have a legitimate interest in doing so, but that discussion is long since moot -- DMARC is a fact and it's not going to go away. [4] If there's enough interest I might try to push it through the IETF process. From glibersat at sigill.org Mon Nov 3 22:59:20 2014 From: glibersat at sigill.org (Guillaume Libersat) Date: Mon, 03 Nov 2014 22:59:20 +0100 Subject: [Mailman-Developers] ORM layer transition In-Reply-To: <87ioiy20ks.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20141101141805.206254e4@limelight.wooz.org> <87ioiy20ks.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5457FAB8.6040206@sigill.org> Hi, First, thanks for this wonderful enhancement, it's gonna make a lot of things much easier and reliable! I've just tried the branch and wanted to point out that I had to add : "from __future__ import print_function" at the beginning of: - src/mailman/bin/checkdbs.py - src/mailman/bin/bumpdigests.py otherwise it would complain about using "file=" in the print function (using python2.x). It looks to run smoothly in my test environment, how crazy is it to try it on the production server? Any chance to corrupt the DB? :) I'd like to explore the possibility to write a migration script from sqlite to postgres with the new ORM (saw this: https://zignar.net/2011/11/01/copy-data-from-mysql-to-postgres-using-sqlalchemy/ ). If someone is already working on this, please tell me! Cheers, Guillaume Le 01/11/2014 20:16, Stephen J. Turnbull a ?crit : > Barry Warsaw writes: > > > I am happy to announce that the MM3 trunk has been ported to > > SQLAlchemy (SA). The full test suite completes against both SQLite and > > PostgreSQL as back-ends, and I would expect that with a little bit of work > > (contributions welcome!) any other back-end supported by SA should work too. > > Huzzah! Happy Mailman Day! (Although it isn't here any more. :) > > > Without the contributions of both Abhilash and Aur?lien, this port would not > > have happened. Kudos to them, > > +1 !! > > Steve > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/glibersat%40sigill.org > > Security Policy: http://wiki.list.org/x/QIA9 -- http://sigill.org From stephen at xemacs.org Tue Nov 4 00:57:15 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 04 Nov 2014 08:57:15 +0900 Subject: [Mailman-Developers] ORM layer transition In-Reply-To: <5457FAB8.6040206@sigill.org> References: <20141101141805.206254e4@limelight.wooz.org> <87ioiy20ks.fsf@uwakimon.sk.tsukuba.ac.jp> <5457FAB8.6040206@sigill.org> Message-ID: <87ppd3zvl0.fsf@uwakimon.sk.tsukuba.ac.jp> Guillaume Libersat writes: > It looks to run smoothly in my test environment, how crazy is it to > try it on the production server? Any chance to corrupt the DB? :) Only you can judge craziness, but I would say, yes, there is a chance of DB corruption that is higher than before the migration. No reflection on anybody's work here, but different ORMs make slightly different assumptions about the workflow and interfaces. It's not easy to get all these things absolutely correct when migrating. I don't think the risk is very high; if I were already running a production server with data I worried about (my server is production in a sense but it would be the work of 15 minutes to reproduce the subscriber list by hand :-), I'd just back up twice a day instead of once. From barry at list.org Tue Nov 4 02:01:21 2014 From: barry at list.org (Barry Warsaw) Date: Mon, 3 Nov 2014 20:01:21 -0500 Subject: [Mailman-Developers] ORM layer transition In-Reply-To: <5457FAB8.6040206@sigill.org> References: <20141101141805.206254e4@limelight.wooz.org> <87ioiy20ks.fsf@uwakimon.sk.tsukuba.ac.jp> <5457FAB8.6040206@sigill.org> Message-ID: <20141103200121.2639cf78@anarchist.wooz.org> On Nov 03, 2014, at 10:59 PM, Guillaume Libersat wrote: >I've just tried the branch and wanted to point out that I had to add : > >"from __future__ import print_function" > >at the beginning of: > - src/mailman/bin/checkdbs.py > - src/mailman/bin/bumpdigests.py > >otherwise it would complain about using "file=" in the print function >(using python2.x). Those two files haven't been ported to MM3 yet. I also haven't gotten around to deleting them. ;) Really they need to be changed into a subcommand of the 'mailman' cli. Contributions welcome! >I'd like to explore the possibility to write a migration script from sqlite >to postgres with the new ORM (saw this: >https://zignar.net/2011/11/01/copy-data-from-mysql-to-postgres-using-sqlalchemy/ >). If someone is already working on this, please tell me! Not that I know of but it would be a nice script to have. If you do come up with something let us know. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From aurelien at bompard.org Tue Nov 4 14:31:34 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Tue, 4 Nov 2014 14:31:34 +0100 Subject: [Mailman-Developers] MM3 Sqlite to PostgreSQL migration? In-Reply-To: <54551A1C.5090605@sigill.org> References: <54551A1C.5090605@sigill.org> Message-ID: Hey Guillaume, The new SQLAlchemy backend has just been merged as you've probably seen. It should upgread seamlessly from the previous Storm backend, but nothing has been done to make transition from SQLite to PostgreSQL possible. If I were you I would take the following steps: 1. upgrade to the new SQLAlchemy-based branch and let mailman do the migration 2. set your mailman config to point to the PostgreSQL DB 3. start mailman to have it create the DB schema 4. now run an SQLAlchemy-based script that would migrate the data of each table from SQLite to PostgreSQL Step 4 is of course the most complicated, but you can iterate on each table using the metadata instance. Use 'Metadata.sorted_tables' to iterate in the proper order (based on foreign key references). You should only have to create one connection for each DB, run 'Metadata.reflect()' on each, iterate on the SQLite tables and insert in the PostgreSQL. Let me know if that works. Good luck, Jim ;-) Aur?lien 2014-11-01 18:36 GMT+01:00 Guillaume Libersat : > Hi, > > We are currently running a MM3 instance in "production" with a SQLite > backend (at ulists.org). It was at first a test, but a lot of people > have already started to use the service and now we are kind of stuck > (but happy ;-) ). > > First, I've seen a lot of DBLock issues in the logs and I think there > also is a race condition happening with Held Messages that leads to > subsequents 500 until you manually copy fake email files (If I'm > correct, the held message file is deleted, then the SQL query is made > but fails. The result is a system in an inconsistent state: there's a > held message in DB, but no more on disk). > > Second, I've tried to manually migrate to PostgreSQL but did not succeed > because of a lot typing issues and differences between these two databases. > Barry told me there's an SQLAlchemy port that is soon to be released, at > least in alpha stage. Do you think it could help to achieve such a > migration? I'm willing to help testing this new backend and, if someone > can guide me with the MM stuff, write migration scripts. > > Thanks and again, MM3 is really an awesome suite of software! :) > > Guillaume > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/aurelien%40bompard.org > > Security Policy: http://wiki.list.org/x/QIA9 > From vtico at yandex.com Thu Nov 6 12:23:44 2014 From: vtico at yandex.com (Vijay Tico) Date: Thu, 06 Nov 2014 14:53:44 +0330 Subject: [Mailman-Developers] mailman-bundler-3.0b1 error - Does it have an HTTPS login page? In-Reply-To: <545124A4.7080008@yandex.com> References: <545124A4.7080008@yandex.com> Message-ID: <545B5A40.2000604@yandex.com> Hi, Bump! It's about a week, and no response! Have I posted to the right place? Do you the Postorius/Hyperkitty developers have their own mailing list? On 10/29/2014 09:02 PM, Vijay Tico wrote: > Hi, > > After installation of mailman-bundler-3.0b1 according to the > instructions in README.rst and running the web interface, the web server > runs and responds HTTP requests. The problem is that when I click the > Login button on the top right of the page, I am redirected to this url: > https://localhost:8000/archives/accounts/login > > The server naturally doesn't respond to this, as its an HTTPS request > being sent to an HTTP server. > > I tried some hacks, but could not get it working (I don't know django at > all). > > How can I solve this? Why has this happened? > > Bests > From raj.abhilash1 at gmail.com Thu Nov 6 12:51:06 2014 From: raj.abhilash1 at gmail.com (Abhilash Raj) Date: Thu, 6 Nov 2014 17:21:06 +0530 Subject: [Mailman-Developers] mailman-bundler-3.0b1 error - Does it have an HTTPS login page? In-Reply-To: <545B5A40.2000604@yandex.com> References: <545124A4.7080008@yandex.com> <545B5A40.2000604@yandex.com> Message-ID: HI Vijay, This was a bug in mailman-bundler, which is already fixed in the main trunk. You can either install mailman-bundler from the latest source or fix yourself manually. The fix is to edit 'mailman-bundler/mailman-web/development.py' and change the value of `USE_SSL` to False (It should be true in your case). thanks, Abhilash Raj On Thu, Nov 6, 2014 at 4:53 PM, Vijay Tico wrote: > Hi, > > Bump! It's about a week, and no response! Have I posted to the right > place? Do you the Postorius/Hyperkitty developers have their own mailing > list? > > On 10/29/2014 09:02 PM, Vijay Tico wrote: > > Hi, > > > > After installation of mailman-bundler-3.0b1 according to the > > instructions in README.rst and running the web interface, the web server > > runs and responds HTTP requests. The problem is that when I click the > > Login button on the top right of the page, I am redirected to this url: > > https://localhost:8000/archives/accounts/login > > > The server naturally doesn't respond to this, as its an HTTPS request > > being sent to an HTTP server. > > > > I tried some hacks, but could not get it working (I don't know django at > > all). > > > > How can I solve this? Why has this happened? > > > > Bests > > > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/raj.abhilash1%40gmail.com > > Security Policy: http://wiki.list.org/x/QIA9 > From vtico at yandex.com Thu Nov 6 20:06:02 2014 From: vtico at yandex.com (Vijay Tico) Date: Thu, 06 Nov 2014 22:36:02 +0330 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods Message-ID: <545BC69A.6090706@yandex.com> Hi, I have installed mailman-bundler-3.0b2. On the Postorius login page, by default there is the Google, persona and Yahoo authentication. I only want basic native application authentication, which I can't see in this page. Tweaking the AUTHENTICATION_BACKENDS variable in development.py, I could disable the Google and Yahoo authentication and 'django.contrib.auth.backends.ModelBackend' seemingly corresponds to the persona authentication. AUTHENTICATION_BACKENDS = ( #'social_auth.backends.google.GoogleBackend', #'social_auth.backends.yahoo.YahooBackend', #'social_auth.backends.OpenIDBackend', 'django.contrib.auth.backends.ModelBackend', ) So the question does Postorius/Hyperkitty support basic authentication at all? How can I enable it? Cheers From vtico at yandex.com Fri Nov 7 04:03:36 2014 From: vtico at yandex.com (Vijay Tico) Date: Fri, 07 Nov 2014 06:33:36 +0330 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545BC69A.6090706@yandex.com> References: <545BC69A.6090706@yandex.com> Message-ID: <545C3688.1040209@yandex.com> I can't even get persona working. After login with persona the system just behaves like it has not logged it. Please help! On 11/06/2014 10:36 PM, Vijay Tico wrote: > Hi, > > I have installed mailman-bundler-3.0b2. On the Postorius login page, by > default there is the Google, persona and Yahoo authentication. I only > want basic native application authentication, which I can't see in this > page. > > Tweaking the AUTHENTICATION_BACKENDS variable in development.py, I could > disable the Google and Yahoo authentication and > 'django.contrib.auth.backends.ModelBackend' seemingly corresponds to the > persona authentication. > > AUTHENTICATION_BACKENDS = ( > #'social_auth.backends.google.GoogleBackend', > #'social_auth.backends.yahoo.YahooBackend', > #'social_auth.backends.OpenIDBackend', > 'django.contrib.auth.backends.ModelBackend', > ) > > So the question does Postorius/Hyperkitty support basic authentication > at all? How can I enable it? > > Cheers > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/vtico%40yandex.com > > Security Policy: http://wiki.list.org/x/QIA9 > From vtico at yandex.com Fri Nov 7 21:23:42 2014 From: vtico at yandex.com (Vijay Tico) Date: Fri, 07 Nov 2014 23:53:42 +0330 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545C3688.1040209@yandex.com> References: <545BC69A.6090706@yandex.com> <545C3688.1040209@yandex.com> Message-ID: <545D2A4E.8010301@yandex.com> Is this issue resolved in the newer versions, or is this just a matter of misconfiguration? On 11/07/2014 06:33 AM, Vijay Tico wrote: > I can't even get persona working. After login with persona the system > just behaves like it has not logged it. Please help! > > On 11/06/2014 10:36 PM, Vijay Tico wrote: >> Hi, >> >> I have installed mailman-bundler-3.0b2. On the Postorius login page, by >> default there is the Google, persona and Yahoo authentication. I only >> want basic native application authentication, which I can't see in this >> page. >> >> Tweaking the AUTHENTICATION_BACKENDS variable in development.py, I could >> disable the Google and Yahoo authentication and >> 'django.contrib.auth.backends.ModelBackend' seemingly corresponds to the >> persona authentication. >> >> AUTHENTICATION_BACKENDS = ( >> #'social_auth.backends.google.GoogleBackend', >> #'social_auth.backends.yahoo.YahooBackend', >> #'social_auth.backends.OpenIDBackend', >> 'django.contrib.auth.backends.ModelBackend', >> ) >> >> So the question does Postorius/Hyperkitty support basic authentication >> at all? How can I enable it? >> >> Cheers >> _______________________________________________ >> Mailman-Developers mailing list >> Mailman-Developers at python.org >> https://mail.python.org/mailman/listinfo/mailman-developers >> Mailman FAQ: http://wiki.list.org/x/AgA3 >> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ >> Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/vtico%40yandex.com >> >> Security Policy: http://wiki.list.org/x/QIA9 >> > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/vtico%40yandex.com > > Security Policy: http://wiki.list.org/x/QIA9 > From vtico at yandex.com Sat Nov 8 17:16:27 2014 From: vtico at yandex.com (Vijay Tico) Date: Sat, 08 Nov 2014 19:46:27 +0330 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545D2A4E.8010301@yandex.com> References: <545BC69A.6090706@yandex.com> <545C3688.1040209@yandex.com> <545D2A4E.8010301@yandex.com> Message-ID: <545E41DB.9040508@yandex.com> Sorry, is this the right place to ask this question? Two days is passing and no answer yet. Bests On 11/07/2014 11:53 PM, Vijay Tico wrote: > Is this issue resolved in the newer versions, or is this just a matter > of misconfiguration? > > On 11/07/2014 06:33 AM, Vijay Tico wrote: >> I can't even get persona working. After login with persona the system >> just behaves like it has not logged it. Please help! >> >> On 11/06/2014 10:36 PM, Vijay Tico wrote: >>> Hi, >>> >>> I have installed mailman-bundler-3.0b2. On the Postorius login page, by >>> default there is the Google, persona and Yahoo authentication. I only >>> want basic native application authentication, which I can't see in this >>> page. >>> >>> Tweaking the AUTHENTICATION_BACKENDS variable in development.py, I could >>> disable the Google and Yahoo authentication and >>> 'django.contrib.auth.backends.ModelBackend' seemingly corresponds to the >>> persona authentication. >>> >>> AUTHENTICATION_BACKENDS = ( >>> #'social_auth.backends.google.GoogleBackend', >>> #'social_auth.backends.yahoo.YahooBackend', >>> #'social_auth.backends.OpenIDBackend', >>> 'django.contrib.auth.backends.ModelBackend', >>> ) >>> >>> So the question does Postorius/Hyperkitty support basic authentication >>> at all? How can I enable it? >>> >>> Cheers >>> _______________________________________________ >>> Mailman-Developers mailing list >>> Mailman-Developers at python.org >>> https://mail.python.org/mailman/listinfo/mailman-developers >>> Mailman FAQ: http://wiki.list.org/x/AgA3 >>> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ >>> Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/vtico%40yandex.com >>> >>> Security Policy: http://wiki.list.org/x/QIA9 >>> >> >> _______________________________________________ >> Mailman-Developers mailing list >> Mailman-Developers at python.org >> https://mail.python.org/mailman/listinfo/mailman-developers >> Mailman FAQ: http://wiki.list.org/x/AgA3 >> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ >> Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/vtico%40yandex.com >> >> Security Policy: http://wiki.list.org/x/QIA9 >> > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/vtico%40yandex.com > > Security Policy: http://wiki.list.org/x/QIA9 > From mark at msapiro.net Sat Nov 8 17:24:44 2014 From: mark at msapiro.net (Mark Sapiro) Date: Sat, 08 Nov 2014 08:24:44 -0800 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545E41DB.9040508@yandex.com> References: <545BC69A.6090706@yandex.com> <545C3688.1040209@yandex.com> <545D2A4E.8010301@yandex.com> <545E41DB.9040508@yandex.com> Message-ID: <545E43CC.8060001@msapiro.net> On 11/08/2014 08:16 AM, Vijay Tico wrote: > Sorry, is this the right place to ask this question? Two days is passing > and no answer yet. Yes, this is the right place. I am unable to answer your questions, and apparently those who are are otherwise occupied at the moment. Two days is not a long time. If you are impatient, you can try the #mailman irc channel on freenode.net. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From flo.fuchs at gmail.com Sat Nov 8 18:01:19 2014 From: flo.fuchs at gmail.com (Florian Fuchs) Date: Sat, 08 Nov 2014 18:01:19 +0100 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545C3688.1040209@yandex.com> References: <545BC69A.6090706@yandex.com> <545C3688.1040209@yandex.com> Message-ID: <545E4C5F.1010906@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Vijay, On 11/07/2014 04:03 AM, Vijay Tico wrote: > I can't even get persona working. After login with persona the > system just behaves like it has not logged it. Please help! A couple of questions to get a little closer to what exactly went wrong: Does "After login with persona" mean, that the persona login process itself (new browser window/tab) went OK and without any errors? Did the new browser window/tab close itself after you completed the persona login? Did you notice any errors (in the browser console for instance) during the process? Did the mailman login page reload after the persona window/tab was closed, and if so, did it redirect you correctly to the '/archives/' url? Florian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUXkxfAAoJEEceGbPdavl7sewIAKCb7gDhLvvFGdgIczr6gvC6 2m44plaI8V6tNq0pUPDLqYJHBInsGRxNpyKtsMjlYXF3/cW97PHfm8l5Ep4v5UKJ sQFbV2/Rp2thSWt0dN56QaOme1zqAUMMigrxBbKkxCVvfooGT4ciO1IotHrxkAPk x1c0d6pWckMhkLNz+KxwBVPhLcmV3uWb/W1f160bNNXn0V65/KPKweiAQfdmVcBV b2/+Cqulkfoe7hTY3SLrEpzlGhdouVHS8OiVR2jnrYK1XBOVtN+eNqQUVtm27UGv YUExS8zhQHHepEreXanZPG4pQPk9NBC7QYC8Ih7SI0ndPct4Wld6sWLohT3364E= =v5oe -----END PGP SIGNATURE----- From vtico at yandex.com Sat Nov 8 19:50:07 2014 From: vtico at yandex.com (Vijay Tico) Date: Sat, 08 Nov 2014 22:20:07 +0330 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545E4C5F.1010906@gmail.com> References: <545BC69A.6090706@yandex.com> <545C3688.1040209@yandex.com> <545E4C5F.1010906@gmail.com> Message-ID: <545E65DF.5010300@yandex.com> Hi, On 11/08/2014 08:31 PM, Florian Fuchs wrote: > Hi Vijay, > > On 11/07/2014 04:03 AM, Vijay Tico wrote: >> I can't even get persona working. After login with persona the >> system just behaves like it has not logged it. Please help! > > A couple of questions to get a little closer to what exactly went wrong: > > Does "After login with persona" mean, that the persona login process > itself (new browser window/tab) went OK and without any errors? The persona login seems to be OK, at least no error occurs. > > Did the new browser window/tab close itself after you completed the > persona login? Did you notice any errors (in the browser console for > instance) during the process? Yes it closed itself. No errors appear in the console. > > Did the mailman login page reload after the persona window/tab was > closed, and if so, did it redirect you correctly to the '/archives/' url? Yes it reloaded to '/archives/' correctly. But the new page seems NOT logged in. The Login button is still there, and the administrative options don't appear. > > Florian > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/vtico%40yandex.com > > Security Policy: http://wiki.list.org/x/QIA9 > From stephen at xemacs.org Sat Nov 8 11:56:56 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 08 Nov 2014 19:56:56 +0900 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545D2A4E.8010301@yandex.com> References: <545BC69A.6090706@yandex.com> <545C3688.1040209@yandex.com> <545D2A4E.8010301@yandex.com> Message-ID: <87tx2ax8nb.fsf@uwakimon.sk.tsukuba.ac.jp> Vijay Tico writes: > Is this issue resolved in the newer versions, or is this just a matter > of misconfiguration? Well, that's kinda hard to tell since we don't know what you've configured how. There's a good chance that it's a problem in the Mailman system since this is all quite young, but without knowing exactly what you've done it's hard to diagnose. I would suggest getting hold of terri or florianf (Postorius developers) or abompard (I pretty sure he's HyperKitty dev) on Freenode IRC #mailman channel, and having them walk you through the process. That should convince them of a bug if there is one (or maybe it's just hard to understand docs, but that should be fixed too). From flo.fuchs at gmail.com Sun Nov 9 21:09:12 2014 From: flo.fuchs at gmail.com (Florian Fuchs) Date: Sun, 09 Nov 2014 21:09:12 +0100 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545E65DF.5010300@yandex.com> References: <545BC69A.6090706@yandex.com> <545C3688.1040209@yandex.com> <545E4C5F.1010906@gmail.com> <545E65DF.5010300@yandex.com> Message-ID: <545FC9E8.6080806@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/08/2014 07:50 PM, Vijay Tico wrote: > Yes it reloaded to '/archives/' correctly. But the new page seems > NOT logged in. The Login button is still there, and the > administrative options don't appear. mailman-bundler redirects to the '/archives/' index page (hyperkitty) by default after login, but that can be changed[1]. The admin options can be found under '/mailman3/'. What happens if you access that URL? Can you see the admin options and -- hopefully -- a 'logout' link? Anyway, even if that works, it's still strange that you see the login button in the archives header after the persona dance is done... Also, we should definitely make it easier to hop from the archives to the admin pages and vice versa. Florian [1] Change the LOGIN_REDIRECT_URL to 'mailman3' in mailman_web/development.py or mailman_web/production.py if you want the login to redirect to the admin pages (postorius). -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUX8noAAoJEEceGbPdavl7xL0H+gOwCHtgk65DUYmhOJKLIR3y hOUW3UHc8BJhzCBmCRunz7pLCC1LQWn6q5k22PHiCGavz9zwFW7E2mlkXr+tPE22 pBbtEP/62DkZgGFAzekRoP0JyHvqXSoae29kwX23ZEBTv6/+bvLy/voOEn+e2EUo 44xSS0LOKvsLjOp4imtSXtvFN6hir7iSlhN9ki5Mrtm30tg/1F/KIYTMl0tkExui SPcun+50+Z8KLtxOzah99MbOnNSYC7ViV5ROiThlQkPIbYR4QsqxagAHOQ8k3a8B 6Dz+sS65HB2IxztidbEjR0FOFPXW+wBxBIcGKtucB5+8fxdBQ4s+CpFrAlQP8hs= =OXnz -----END PGP SIGNATURE----- From aurelien at bompard.org Mon Nov 10 09:19:14 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Mon, 10 Nov 2014 09:19:14 +0100 Subject: [Mailman-Developers] mailman-bundler-3.0b2 Login methods In-Reply-To: <545FC9E8.6080806@gmail.com> References: <545BC69A.6090706@yandex.com> <545C3688.1040209@yandex.com> <545E4C5F.1010906@gmail.com> <545E65DF.5010300@yandex.com> <545FC9E8.6080806@gmail.com> Message-ID: Hey! Indeed I'm the HyperKitty dev, and I wrote mailman-bundler too. Sorry I couldn't reply earlier. > Yes it reloaded to '/archives/' correctly. But the new page seems > > NOT logged in. The Login button is still there, and the > > administrative options don't appear. > Please check your webserver logs for any 500 error. Also please check that you have setup the BROWSERID_AUDIENCES variable in the configuration file (there is a link in a comment right above the variable with instructions on how to configure it). Also, we should definitely make it easier to hop from the archives to > the admin pages and vice versa. > There should be a link to Postorious in the top menu bar in HyperKitty now. Aur?lien From terri at toybox.ca Wed Nov 12 09:55:11 2014 From: terri at toybox.ca (Terri Oda) Date: Wed, 12 Nov 2014 00:55:11 -0800 Subject: [Mailman-Developers] Want Mailman 3 suite to release faster? Here's some bugs you can help with! Message-ID: <5463206F.5040602@toybox.ca> Some weeks back, Florian went through the Postorius bug queue to find a list of bugs that he thinks are blocking us from being able to have a final Mailman Suite release. I just wanted to advertise that list to you all in case anyone here is looking for some valuable work to do: https://bugs.launchpad.net/postorius/+bugs?field.tag=mailman3-suite-blocker There's only 13 of them at the moment, so that's nice. They're in a variety of different states. Some are somewhat old and we could really use someone to just try to reproduce them on the most recent version of postorius and post an update in the bug saying it's still a problem (or not!). Some have notes from me suggesting a smaller piece that could be done first by a beginner wanting a new bug to try or someone more experienced wanting a quicker task. (don't worry about saving beginner bugs at this stage; I'm more interested in seeing a release!) Some are more complex. If anyone's got time to push these forwards in any way, it would be very much appreciated! I'm even happy to send you some mailman stickers if you want a physical token of appreciation. Terri From flo.fuchs at gmail.com Wed Nov 12 10:09:20 2014 From: flo.fuchs at gmail.com (Florian Fuchs) Date: Wed, 12 Nov 2014 10:09:20 +0100 Subject: [Mailman-Developers] Want Mailman 3 suite to release faster? Here's some bugs you can help with! In-Reply-To: <5463206F.5040602@toybox.ca> References: <5463206F.5040602@toybox.ca> Message-ID: <546323C0.5010108@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/12/2014 09:55 AM, Terri Oda wrote: > Some weeks back, Florian went through the Postorius bug queue to > find a list of bugs that he thinks are blocking us from being able > to have a final Mailman Suite release. > > I just wanted to advertise that list to you all in case anyone here > is looking for some valuable work to do: > > https://bugs.launchpad.net/postorius/+bugs?field.tag=mailman3-suite-blocker Terri, > thank you for sending this around! Oh, and if anyone thinks something's missing on that list: Just file a bug adding the tag "mailman3-suite-blocker" to it, so it gets more attention. Florian P.S.: BTW, the one I am currently working on is this one: https://bugs.launchpad.net/postorius/+bug/1313912 The fix will also include an update to bootstrap version 3 and make postorius more responsive to different screen sizes. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUYyO9AAoJEEceGbPdavl7HAEH/RDpOc35cnYjr5Ixq38oPbYo 1BcPyoehW5Nh1T64sLJ16Zm7ZigFRE1btT2R5EBaGIIeENJE1M9PeBKmra5CUEmv JP95pc4hxHiv/R7otU6RZPO6JS8zRcsQesK4L+WRDbKtt+9uoMfV1Swkzk/YZ9gp jReTNKUKjUKFEH8Q4boYmaXuKG40vDCUJ5Ddc1P47AyfVv45iEBaXyzewYGBh4Ux mKVGe12hyRYcT9staWLVE2uuCn+7uCbiLIisI9ywh1utre4WkeGzwBe6lfrWRu5K ZXXCT21ehyTJF4VXn7t4h57dHZntBTYZktgDhSIcnzbdvlgUCVhfHFlTyAfcga4= =0E9B -----END PGP SIGNATURE----- From aurelien at bompard.org Wed Nov 12 13:39:21 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Wed, 12 Nov 2014 13:39:21 +0100 Subject: [Mailman-Developers] Want Mailman 3 suite to release faster? Here's some bugs you can help with! In-Reply-To: <546323C0.5010108@gmail.com> References: <5463206F.5040602@toybox.ca> <546323C0.5010108@gmail.com> Message-ID: Very cool, I'll have a look! A. From barry at list.org Wed Nov 12 16:47:33 2014 From: barry at list.org (Barry Warsaw) Date: Wed, 12 Nov 2014 10:47:33 -0500 Subject: [Mailman-Developers] Want Mailman 3 suite to release faster? Here's some bugs you can help with! In-Reply-To: <5463206F.5040602@toybox.ca> References: <5463206F.5040602@toybox.ca> Message-ID: <20141112104733.3d9223bf@anarchist.wooz.org> Thanks very much for sending this around. On the core, I don't have a formal list of blockers (I should add tags), but what I think must be done include: - Port the REST framework to falcon. This is essentially done, and is just waiting for an upstream falcon release to land on trunk. I have a branch that I'm regularly merging core trunk into and running the tests. - lp:~barry/mailman/subpolicy - this fixes the subscription policy so that it can be set through REST and actually will affect how folks can subscribe. It's a big branch that was blocked on database migration, so now that the SQLAlchemy branch has landed (yay!), I've returned to this branch. I'll go through the 96 mailman3 bugs: https://bugs.launchpad.net/mailman/+bugs?field.tag=mailman3 and tag them using mailman3-suite-blocker for anything I think has to be fixed before the core can be released. If you have your favorites, please don't hesitate to tag them, and I'll review when I triage. BTW, if you haven't seen the recent work in lp:mailman, please take a look. There's now a tox.ini file so it will make running the test suite much, much easier. The instructions for running the tests against PostgreSQL are simplified too. It's also possible to collect coverage during the test. The numbers aren't as abysmal as I was worried they'd be. ;) We get over 90% for the full code base, though several modules have rather poor coverage. It's not a top priority for suite release, but I do plan on getting the coverage up to 100% over time. Cheers, -Barry From barry at list.org Sat Nov 15 18:05:00 2014 From: barry at list.org (Barry Warsaw) Date: Sat, 15 Nov 2014 12:05:00 -0500 Subject: [Mailman-Developers] falcon branch is merged Message-ID: <20141115120500.2e89a14a@anarchist.wooz.org> I'm happy to say that with Kurt Griffiths' release of Falcon 2.0b1 to PyPI, I have merged the MM3 falcon branch to trunk. This means the Mailman internal dependency transitions are complete. Along with the SQLAlchemy transition, our trunk should now only depend on Python 3 compatible libraries. Porting MM3 to Python 3 is not on the critical path for the final release, but at least work on that can now begin. Happy hacking. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From aurelien at bompard.org Tue Nov 25 11:46:18 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Tue, 25 Nov 2014 11:46:18 +0100 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members Message-ID: Hi *, I'd like to discuss the algorithm we use to enforce a list's default moderation action. Here's what I understand: - the list's default_moderation_action setting is stored in the database and can be changed via the REST interface - when a new member is created, its moderation_action is imported from the list's default_moderation_action depending on its role (see model/member.py:84) - this member's moderation_action is stored in the database - it's the member's moderation_action that is used in the moderation rule (rules/moderation.py:52) As a result, unless I'm missing something changing, the list's default_moderation_action has no effect on existing members. Do I understand correctly? I suggest we leave the member's moderation_action on None when it's created, and modify the moderation rules to go look for the list's default setting if the member's value is None. How about that ? Other suggestions? Aur?lien From stephen at xemacs.org Tue Nov 25 17:41:31 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 26 Nov 2014 01:41:31 +0900 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: References: Message-ID: <87y4qzqll0.fsf@uwakimon.sk.tsukuba.ac.jp> Aurelien Bompard writes: > I suggest we leave the member's moderation_action on None when it's > created, and modify the moderation rules to go look for the list's default > setting if the member's value is None. I think this is a bad idea. From the subscriber's perspective, it changes the list's behavior with no action from her, for no good reason that she knows. The semantics of these variables is, and I think should be, initialization, not fallback. From Richard at Damon-Family.org Tue Nov 25 18:39:59 2014 From: Richard at Damon-Family.org (Richard Damon) Date: Tue, 25 Nov 2014 12:39:59 -0500 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: References: Message-ID: <5474BEEF.4010703@Damon-Family.org> On 11/25/14, 5:46 AM, Aurelien Bompard wrote: > Hi *, > > I'd like to discuss the algorithm we use to enforce a list's default > moderation action. Here's what I understand: > - the list's default_moderation_action setting is stored in the database > and can be changed via the REST interface > - when a new member is created, its moderation_action is imported from the > list's default_moderation_action depending on its role (see > model/member.py:84) > - this member's moderation_action is stored in the database > - it's the member's moderation_action that is used in the moderation rule > (rules/moderation.py:52) > > As a result, unless I'm missing something changing, the list's > default_moderation_action has no effect on existing members. Do I > understand correctly? > > I suggest we leave the member's moderation_action on None when it's > created, and modify the moderation rules to go look for the list's default > setting if the member's value is None. > How about that ? Other suggestions? > > Aur?lien > The only way I could see that would be to make the flag 3 valued: They are moderated They are not moderated They are in the default state of moderation, see the lists default value. -- Richard Damon From barry at list.org Tue Nov 25 22:05:13 2014 From: barry at list.org (Barry Warsaw) Date: Tue, 25 Nov 2014 16:05:13 -0500 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: References: Message-ID: <20141125160513.5146d133@anthem> On Nov 25, 2014, at 11:46 AM, Aurelien Bompard wrote: >As a result, unless I'm missing something changing, the list's >default_moderation_action has no effect on existing members. Do I >understand correctly? That is correct. As Steve points out, the intent is that this setting is initialization not fallback. I think this does work differently in MM2.1, but that version's schema isn't really sophisticated enough to do anything different. There, members have a moderation flag which gets initialized from default_member_moderation. Then, if the member's flag is set, member_moderation_action (a list setting) is taken. So I guess in the case of moderated members, there is a fallback action, and while there is no member-specific action, there is a member-specific state. MM3 collapses that so that rather than having a flag with a fallback action, each member just has an action. This allows you more flexibility in controlling an individual member's posting privileges, at the cost of having to change all members individually if a list's default changes. Can you elaborate on the use case for having a fallback action? Note that in MM3, nonmembers are technically members too, but with a 'nonmember' role, so the same moderation action flag controls their posting privileges too. This replaces MM2.1's generic_nonmember_action and the crufty lists of {accept,hold,reject,discard}_these_nonmembers. Cheers, -Barry From mark at msapiro.net Wed Nov 26 05:59:47 2014 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 25 Nov 2014 20:59:47 -0800 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: <20141125160513.5146d133@anthem> References: <20141125160513.5146d133@anthem> Message-ID: <54755E43.9070700@msapiro.net> On 11/25/2014 01:05 PM, Barry Warsaw wrote: > > I think this does work differently in MM2.1, but that version's schema isn't > really sophisticated enough to do anything different. There, members have a > moderation flag which gets initialized from default_member_moderation. Then, > if the member's flag is set, member_moderation_action (a list setting) is > taken. So I guess in the case of moderated members, there is a fallback > action, and while there is no member-specific action, there is a > member-specific state. MM 2.1 does have an additional capability in the web UI, namely the ability to set all members mod bits on or off with an off/on radio selection and a single click. Thus, an admin can change default_member_moderation to No or Yes and with a couple more clicks set all current members to match. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From raj.abhilash1 at gmail.com Wed Nov 26 10:00:59 2014 From: raj.abhilash1 at gmail.com (Abhilash Raj) Date: Wed, 26 Nov 2014 14:30:59 +0530 Subject: [Mailman-Developers] Python3 port of Mailman3 Message-ID: Hi All, I am working on porting mailman3 to python3. There are few doubts I have which may sound stupid, but I have a little knowledge about encoding and charsets which I think is important for the port. Also this is my first time working on python3 so forgive me for asking anything obvious( I did try googling first) The code is in very preliminary stages with tons of errors. Its up on launchpad. lp:~raj-abhilash1/mailman/py3 So my doubts: 1. `pkg_resources.resource_string` used to read the configuration files while initialization and testing now returns bytes instead of string(like in python2). So instead I use `pkg_resources.resource_stream` and deocode to the steam to a `utf-8` encoding(why? to avoid errors for the time being). I wanted to ask what would be the best for this case? 2. To create a hash everywhere unicode strings must be encoded. So again the same question what should I encode it to? UTF-8 or US-ASCII? Or someway to determine which encoding should be used? Also if anybody has suggestions on porting please let me know. I am online sporadically on irc as `maxking` on #mailman (I see all of the messages sent while I am away). thanks, Abhilash Raj From post-mailman-developers at partan.com Wed Nov 26 07:20:19 2014 From: post-mailman-developers at partan.com (Andrew Partan) Date: Wed, 26 Nov 2014 01:20:19 -0500 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: <20141125160513.5146d133@anthem> References: <20141125160513.5146d133@anthem> Message-ID: <20141126062019.GA52482@partan.com> On Tue, Nov 25, 2014 at 04:05:13PM -0500, Barry Warsaw wrote: > MM3 collapses that so that rather than having a flag with a fallback action, > each member just has an action. This allows you more flexibility in > controlling an individual member's posting privileges, at the cost of having > to change all members individually if a list's default changes. Can you > elaborate on the use case for having a fallback action? Say you have a list where a large argument breaks out. I want to change the list to moderated for a while until everyone calms down. Then change it back to unmoderated. There are some people who are always moderated and others who are always unmoderated; their settings should not change when the list's setting changes. Andrew Partan From mark at msapiro.net Wed Nov 26 16:18:47 2014 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 26 Nov 2014 07:18:47 -0800 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: <20141126062019.GA52482@partan.com> References: <20141125160513.5146d133@anthem> <20141126062019.GA52482@partan.com> Message-ID: <5475EF57.2090105@msapiro.net> On 11/25/2014 10:20 PM, Andrew Partan wrote: > Say you have a list where a large argument breaks out. I want to > change the list to moderated for a while until everyone calms down. > Then change it back to unmoderated. There are some people who are > always moderated and others who are always unmoderated; their > settings should not change when the list's setting changes. In MM 2.1 at least, there is a separate setting for emergency moderation to handle this situation in exactly this way. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at list.org Wed Nov 26 17:19:58 2014 From: barry at list.org (Barry Warsaw) Date: Wed, 26 Nov 2014 11:19:58 -0500 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: <54755E43.9070700@msapiro.net> References: <20141125160513.5146d133@anthem> <54755E43.9070700@msapiro.net> Message-ID: <20141126111958.7f2ddeb3@anarchist.wooz.org> On Nov 25, 2014, at 08:59 PM, Mark Sapiro wrote: >MM 2.1 does have an additional capability in the web UI, namely the >ability to set all members mod bits on or off with an off/on radio >selection and a single click. Thus, an admin can change >default_member_moderation to No or Yes and with a couple more clicks set >all current members to match. Right, though like all the mass change actions in the web ui, this just iterates through all the members, flipping their mod bits. I could certainly envision an equivalent REST API that let you set certain attributes on all members of a mailing list (perhaps matching some criteria, a la /members/find Cheers, -Barry From barry at list.org Wed Nov 26 17:21:21 2014 From: barry at list.org (Barry Warsaw) Date: Wed, 26 Nov 2014 11:21:21 -0500 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: <5475EF57.2090105@msapiro.net> References: <20141125160513.5146d133@anthem> <20141126062019.GA52482@partan.com> <5475EF57.2090105@msapiro.net> Message-ID: <20141126112121.1c53d9cf@anarchist.wooz.org> On Nov 26, 2014, at 07:18 AM, Mark Sapiro wrote: >On 11/25/2014 10:20 PM, Andrew Partan wrote: >> Say you have a list where a large argument breaks out. I want to >> change the list to moderated for a while until everyone calms down. >> Then change it back to unmoderated. There are some people who are >> always moderated and others who are always unmoderated; their >> settings should not change when the list's setting changes. > >In MM 2.1 at least, there is a separate setting for emergency moderation >to handle this situation in exactly this way. Yep, mm3 still has the emergency moderation flag, which is still how I see handling Andrew's use case. Cheers, -Barry From mark at msapiro.net Thu Nov 27 06:34:13 2014 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 26 Nov 2014 21:34:13 -0800 Subject: [Mailman-Developers] Changing the default_moderation_action has no effect on existing members In-Reply-To: <87y4qzqll0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87y4qzqll0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5476B7D5.4020602@msapiro.net> On 11/25/2014 08:41 AM, Stephen J. Turnbull wrote: > Aurelien Bompard writes: > > > I suggest we leave the member's moderation_action on None when it's > > created, and modify the moderation rules to go look for the list's default > > setting if the member's value is None. > > I think this is a bad idea. From the subscriber's perspective, it > changes the list's behavior with no action from her, for no good > reason that she knows. But moderation_action, although a per-user attribute, is not really a user setting. User's don't set their moderation_action or necessarily even know what it is. It is a list admin/moderator function to determine whether a user's posts are moderated in some way, and it is normal for this to change (e.g. from moderated to unmoderated or vice versa) depending on the user's behavior. Granted, with Aurelien's suggestion, changing the default is done without regard to the actions of some specific users, but still, moderation and moderation actions are really list admin/moderator functions, not user functions and users probably should not expect them to be constant. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at list.org Sun Nov 30 00:33:54 2014 From: barry at list.org (Barry Warsaw) Date: Sat, 29 Nov 2014 18:33:54 -0500 Subject: [Mailman-Developers] Python3 port of Mailman3 In-Reply-To: References: Message-ID: <20141129183354.08f39196@limelight.wooz.org> On Nov 26, 2014, at 02:30 PM, Abhilash Raj wrote: >I am working on porting mailman3 to python3. There are few doubts I have >which may sound stupid, but I have a little knowledge about encoding and >charsets which I think is important for the port. Also this is my first time >working on python3 so forgive me for asking anything obvious( I did try >googling first) The code is in very preliminary stages with tons of >errors. Its up on launchpad. > > lp:~raj-abhilash1/mailman/py3 Yay! Note that I also have been playing around with it a bit. I wouldn't say it's a functional branch, but you might compare notes: lp:~barry/mailman/py3 >1. `pkg_resources.resource_string` used to read the configuration files while >initialization and testing now returns bytes instead of string(like in >python2). Right. "resource_string" is really a misnomer. In another Python 3 project I work on, I use: from pkg_resource import resource_string as resource_bytes just so that the call sites are accurate. >So instead I use `pkg_resources.resource_stream` and deocode to the steam to >a `utf-8` encoding(why? to avoid errors for the time being). I wanted to ask >what would be the best for this case? You can of course also decode the bytestring that "resource_bytes" returns. That's generally what I do in that other project. >2. To create a hash everywhere unicode strings must be encoded. So again the >same question what should I encode it to? UTF-8 or US-ASCII? Or someway to >determine which encoding should be used? I think utf-8 is generally the right encoding to use, except in contexts where you are given a better hint. E.g. if you're decoding say the payload of a message with a Content-Type header, and that header has a `charset` attribute, it will name the encoding that you're supposed to use. There may be cases where us-ascii or latin-1 would be better, but I think those should be determined on a case-by-case basis. >Also if anybody has suggestions on porting please let me know. I am online >sporadically on irc as `maxking` on #mailman (I see all of the messages sent >while I am away). The trunk should now be `python2.7 -3` clean, so it's ready to be ported. I'd probably not try to run the full test suite yet, but start at the lowest level, e.g. database/model and work your way up. If there are changes that make sense even in Python 2 (i.e. for bilingual support for now) try to put those in separate branches and merge proposals, so they can be merged into trunk even before a full port. This page will be indispensable: https://wiki.python.org/moin/PortingToPy3k/BilingualQuickRef Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: