From maxking at asynchronous.in Sun Oct 1 17:12:46 2017 From: maxking at asynchronous.in (Abhilash Raj) Date: Sun, 01 Oct 2017 14:12:46 -0700 Subject: [Mailman-Developers] Migrating Postorius and Hyperkitty to Python 3 Message-ID: <1506892366.1092546.1124303592.674357F2@webmail.messagingengine.com> Hi All, Mailman uses Django web framework for the web based frontends, Postorius - The Official UI, and Hyperkitty - The official Archiver. They are both Django "apps" which means that they can be plugged into any other existing Django "project" (aka Django "installation") to work alongside other apps that people might be running. Currently, both the Django apps we have are Python 2 only, we have talked about moving to Python 3 but we decided we want it to be bilingual (support both 2 & 3). The reason we decided that was because if people would want to embed Postorius & Hyperkitty in their installations, they need to be able to run it under whatever python versions they are using. I want to revisit this assumption for being bilingual. Currently, there is no supported version of Django which doesn't support Python 3. Starting from v2.0, set to release in December 2017, Django is going to drop Python2 support. Now, that doesn't mean no one can run Django under Python2, 1.11 (LTS version) supports Python2 and will be supported probably till Python 2 is supported (April 2020 according to [1]). I believe that our (limited) development efforts would be best utilized if we just drop the support for Python 2 in Postorius & Hyperkitty instead of trying to be bilingual. Any day one of our dependencies may decide to do the same, and we would have to then use Python 3 anyway. Also, dropping Python 2 support doesn't seem like a lot of pain for anyone, you just need another instance of Django running, which is not *that* hard using uwsgi (in Emperor mode). I believe most of our dependencies should support Python 3, or should have a good enough replacement if it doesn't. Thoughts? [1]: https://www.djangoproject.com/download/ -- Abhilash Raj maxking at asynchronous.in From simon.hanna at serve-me.info Tue Oct 3 17:26:14 2017 From: simon.hanna at serve-me.info (Simon Hanna) Date: Tue, 3 Oct 2017 23:26:14 +0200 Subject: [Mailman-Developers] Migrating Postorius and Hyperkitty to Python 3 In-Reply-To: <1506892366.1092546.1124303592.674357F2@webmail.messagingengine.com> References: <1506892366.1092546.1124303592.674357F2@webmail.messagingengine.com> Message-ID: <0dc9468e-abb6-2899-ec2c-e4a599a7824d@serve-me.info> I'm in favor of dropping Python2. All distros have Python3 and afaik all dependencies are Python3 ready. So I don't see a reason why anybody would want to stick to Python2, especially since core is Python3 only... From mark at msapiro.net Sun Oct 8 16:14:42 2017 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 8 Oct 2017 13:14:42 -0700 Subject: [Mailman-Developers] Problem getting tox to use mysql Message-ID: <359150be-5322-2f3f-0524-16175407de63@msapiro.net> I'm trying to work on the issue . I've been unable to make a test fail. I finally discovered that when I run 'tox -e py35-nocov-mysql' or 'tox -e py35-nocov-mysql -- -P hold', tox installs the pymysql dependency, but SQLAlchemy is still using the sqlite dialect when the tests run. It appears I may need a [database] section in some mailman.cfg, but I don't know what I would put there for accessing the database, and I don't know where I'd put the .cfg. Also, isn't this something tox should just do? I do think that when GitLab CI runs the mysql tests it does actually use mysql as I've seen mysql related failures there, but I don't seem to have the magic to make it work locally. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From maxking at asynchronous.in Sun Oct 8 17:16:19 2017 From: maxking at asynchronous.in (Abhilash Raj) Date: Sun, 08 Oct 2017 14:16:19 -0700 Subject: [Mailman-Developers] Problem getting tox to use mysql In-Reply-To: <359150be-5322-2f3f-0524-16175407de63@msapiro.net> References: <359150be-5322-2f3f-0524-16175407de63@msapiro.net> Message-ID: <1507497379.1429795.1131926360.3A36BCB1@webmail.messagingengine.com> On Sun, Oct 8, 2017, at 01:14 PM, Mark Sapiro wrote: > I'm trying to work on the issue > . > > I've been unable to make a test fail. I finally discovered that when I > run 'tox -e py35-nocov-mysql' or 'tox -e py35-nocov-mysql -- -P hold', > tox installs the pymysql dependency, but SQLAlchemy is still using the > sqlite dialect when the tests run. > > It appears I may need a [database] section in some mailman.cfg, but I > don't know what I would put there for accessing the database, and I > don't know where I'd put the .cfg. This is the command that Gitlab CI runs, the environment variable below is what mailman uses for "extra" testing config. $ MAILMAN_EXTRA_TESTING_CFG=/home/runner/configs/mysql.cfg tox -e py35-nocov-mysql The extra mysql.cfg is what holds the database configuration. Hope that is helpful! > Also, isn't this something tox should just do? It can't without the knowledge of path to the extra configuration, hence the environment variable. > > I do think that when GitLab CI runs the mysql tests it does actually use > mysql as I've seen mysql related failures there, but I don't seem to > have the magic to make it work locally. > > -- > Mark Sapiro The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/raj.abhilash1%40gmail.com > > Security Policy: http://wiki.list.org/x/QIA9 -- Abhilash Raj maxking at asynchronous.in From maxking at asynchronous.in Sun Oct 8 17:54:10 2017 From: maxking at asynchronous.in (Abhilash Raj) Date: Sun, 08 Oct 2017 14:54:10 -0700 Subject: [Mailman-Developers] Migrating Postorius and Hyperkitty to Python 3 In-Reply-To: <917CF69B-6086-4088-8149-D54BB9279D1E@toybox.ca> References: <1506892366.1092546.1124303592.674357F2@webmail.messagingengine.com> <917CF69B-6086-4088-8149-D54BB9279D1E@toybox.ca> Message-ID: <1507499650.1436275.1131949240.15A297C0@webmail.messagingengine.com> On Sun, Oct 8, 2017, at 02:39 PM, Terri Oda wrote: > What's the state for Centos? I think it was the last likely distro we > cared about with old python needs, but I'm not sure core supports it > at all any more. I am not sure about the state of CentOS, but Core doesn't support Python 2 anymore. So, in theory, if you need to run Mailman 3, you need Python 3 anyway as the API isn't meant to be exposed outside of a single host. Also, I don't know about the state of containers support in distros, but with containers, there is always an option to run Python3 even if their distro doesn't support. > > > On October 1, 2017 2:12:46 PM PDT, Abhilash Raj > wrote:>> Hi All, >> >> >> >> Mailman uses Django web framework for the web based frontends, >> Postorius >>>> - The Official UI, and Hyperkitty - The official Archiver. They >> are both >>>> Django "apps" which means that they can be plugged into any other >>>> existing Django "project" (aka Django "installation") to work >> alongside >>>> other apps that people might be running. >> >> >> >> Currently, both the Django apps we have are Python 2 only, we have >>>> talked about moving to Python 3 but we decided we want it to be >> >> bilingual (support both 2 & 3). The reason we decided that was >> because >>>> if people would want to embed Postorius & Hyperkitty in their >> >> installations, they need to be able to run it under whatever python >>>> versions they are using. >> >> >> >> I want to revisit this assumption for being bilingual. >> Currently, there >>>> is no supported version of Django which doesn't support Python 3. >>>> Starting from v2.0, set to release in December 2017, Django is >> going to >>>> drop Python2 support. Now, that doesn't mean no one can run >> Django under >>>> Python2, 1.11 (LTS version) supports Python2 and will be supported >>>> probably till Python 2 is supported (April 2020 according to [1]). >>>> >> >> I believe that our (limited) development efforts would be best >> utilized >>>> if we just drop the support for Python 2 in Postorius & Hyperkitty >>>> instead of trying to be bilingual. Any day one of our >> dependencies may >>>> decide to do the same, and we would have to then use Python 3 anyway. >>>> Also, dropping Python 2 support doesn't seem like a lot of pain for >>>> anyone, you just need another instance of Django running, >> which is not >>>> *that* hard using uwsgi (in Emperor mode). I believe most of our >>>> dependencies should support Python 3, or should have a good enough >>>> replacement if it doesn't. >> >> >> >> >> >> Thoughts? >> >> >> >> [1]: https://www.djangoproject.com/download/ >> -- Abhilash Raj maxking at asynchronous.in From terri at toybox.ca Sun Oct 8 17:39:27 2017 From: terri at toybox.ca (Terri Oda) Date: Sun, 08 Oct 2017 14:39:27 -0700 Subject: [Mailman-Developers] Migrating Postorius and Hyperkitty to Python 3 In-Reply-To: <1506892366.1092546.1124303592.674357F2@webmail.messagingengine.com> References: <1506892366.1092546.1124303592.674357F2@webmail.messagingengine.com> Message-ID: <917CF69B-6086-4088-8149-D54BB9279D1E@toybox.ca> What's the state for Centos? I think it was the last likely distro we cared about with old python needs, but I'm not sure core supports it at all any more. On October 1, 2017 2:12:46 PM PDT, Abhilash Raj wrote: >Hi All, > >Mailman uses Django web framework for the web based frontends, >Postorius >- The Official UI, and Hyperkitty - The official Archiver. They are >both >Django "apps" which means that they can be plugged into any other >existing Django "project" (aka Django "installation") to work alongside >other apps that people might be running. > >Currently, both the Django apps we have are Python 2 only, we have >talked about moving to Python 3 but we decided we want it to be >bilingual (support both 2 & 3). The reason we decided that was because >if people would want to embed Postorius & Hyperkitty in their >installations, they need to be able to run it under whatever python >versions they are using. > >I want to revisit this assumption for being bilingual. Currently, there >is no supported version of Django which doesn't support Python 3. >Starting from v2.0, set to release in December 2017, Django is going to >drop Python2 support. Now, that doesn't mean no one can run Django >under >Python2, 1.11 (LTS version) supports Python2 and will be supported >probably till Python 2 is supported (April 2020 according to [1]). > >I believe that our (limited) development efforts would be best utilized >if we just drop the support for Python 2 in Postorius & Hyperkitty >instead of trying to be bilingual. Any day one of our dependencies may >decide to do the same, and we would have to then use Python 3 anyway. >Also, dropping Python 2 support doesn't seem like a lot of pain for >anyone, you just need another instance of Django running, which is not >*that* hard using uwsgi (in Emperor mode). I believe most of our >dependencies should support Python 3, or should have a good enough >replacement if it doesn't. > > >Thoughts? > >[1]: https://www.djangoproject.com/download/ > >-- > Abhilash Raj > maxking at asynchronous.in >_______________________________________________ >Mailman-Developers mailing list >Mailman-Developers at python.org >https://mail.python.org/mailman/listinfo/mailman-developers >Mailman FAQ: http://wiki.list.org/x/AgA3 >Searchable Archives: >http://www.mail-archive.com/mailman-developers%40python.org/ >Unsubscribe: >https://mail.python.org/mailman/options/mailman-developers/terri%40toybox.ca > >Security Policy: http://wiki.list.org/x/QIA9 From mark at msapiro.net Sun Oct 8 20:17:43 2017 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 8 Oct 2017 17:17:43 -0700 Subject: [Mailman-Developers] Problem getting tox to use mysql In-Reply-To: <1507497379.1429795.1131926360.3A36BCB1@webmail.messagingengine.com> References: <359150be-5322-2f3f-0524-16175407de63@msapiro.net> <1507497379.1429795.1131926360.3A36BCB1@webmail.messagingengine.com> Message-ID: On 10/08/2017 02:16 PM, Abhilash Raj wrote: > > This is the command that Gitlab CI runs, the environment variable below > is what > mailman uses for "extra" testing config. > > $ MAILMAN_EXTRA_TESTING_CFG=/home/runner/configs/mysql.cfg tox -e > py35-nocov-mysql > > The extra mysql.cfg is what holds the database configuration. Thanks Abhilash. What I didn't realize is since I already have a mysql server running on my dev box for other purposes, I had to use that server and create a 'mailman' database within it and then use appropriate credentials to access it. I can now actually run tests using mysql. My results in this case are different from the original issue at because my server is not running in "strict" mode., but I can now work on the original issue since I can actually test any fixes. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Oct 9 03:55:47 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 9 Oct 2017 16:55:47 +0900 Subject: [Mailman-Developers] Migrating Postorius and Hyperkitty to Python 3 In-Reply-To: <917CF69B-6086-4088-8149-D54BB9279D1E@toybox.ca> References: <1506892366.1092546.1124303592.674357F2@webmail.messagingengine.com> <917CF69B-6086-4088-8149-D54BB9279D1E@toybox.ca> Message-ID: <23003.11139.882222.192330@turnbull.sk.tsukuba.ac.jp> Terri Oda writes: > What's the state for Centos? I think it was the last likely distro > we cared about with old python needs, but I'm not sure core > supports it at all any more. Do we need to worry about current distros? We're still seeing plenty of questions about new installations of Mailman 2 on mailman-users. Mailman 3 is still a double-diamond application. As I wrote in a different thread, I don't think Mailman 3 is going to be ready for the mom & pop listowner for a while. If versions of Centos (or whatever) don't support Python 3 by then, "let them run containers". :-) Steve -- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From barry at list.org Mon Oct 9 10:03:36 2017 From: barry at list.org (Barry Warsaw) Date: Mon, 9 Oct 2017 10:03:36 -0400 Subject: [Mailman-Developers] Migrating Postorius and Hyperkitty to Python 3 In-Reply-To: <23003.11139.882222.192330@turnbull.sk.tsukuba.ac.jp> References: <1506892366.1092546.1124303592.674357F2@webmail.messagingengine.com> <917CF69B-6086-4088-8149-D54BB9279D1E@toybox.ca> <23003.11139.882222.192330@turnbull.sk.tsukuba.ac.jp> Message-ID: <3AB3C6D2-D84B-443E-A938-E24EBCAEDADF@list.org> On Oct 9, 2017, at 03:55, Stephen J. Turnbull wrote: > > Do we need to worry about current distros? Probably not, as you point out. > We're still seeing plenty of questions about new installations of > Mailman 2 on mailman-users. Mailman 3 is still a double-diamond > application. As I wrote in a different thread, I don't think Mailman > 3 is going to be ready for the mom & pop listowner for a while. If > versions of Centos (or whatever) don't support Python 3 by then, "let > them run containers". :-) Yep, and we have a good container story now, so I think that?s entirely viable. There is work being done on Debian packaging for MM3 and I think having everything on Python 3 should make that story better too. (Debian unstable does have Python3 Django). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From fmouse at fmp.com Thu Oct 12 15:37:55 2017 From: fmouse at fmp.com (Lindsay Haisley) Date: Thu, 12 Oct 2017 14:37:55 -0500 Subject: [Mailman-Developers] Extracting some Mailman code Message-ID: <1507837075.16717.90.camel@fmp.com> Hey folks, This is a request for a pointer to some code rather than an offering for Mailman, but I hope my cred with the Mailman developers will fetch me a bit of help. I can read code and find this out myself, but one of the Mailman devs can probably give me some pointers that will save me a lot of time, time being the only human resource which is truly limited! I'm running Courier-MTA, an excellent MTA around which I've built FMP's small ESP services. Courier has the ability to do a simple email redirect using an alias address in a flat file, in a special directory, containing only the email address to which email should be redirected. Courier also has the ability to interpret lines in this file starting with "|" as programs to which the body of an email can be submitted via stdin and any required processing done therein. I'm the author of courier-to-mailman.py in the contrib collection in Mailman 2's current standard code which works this way. I'm seeing increasing problems with DMARC rejection of emails sent through this simple redirection mechanism, for obvious reasons, and I'm thinking that I might borrow code from Mailman to re-write the From address just as Mailman does when handling a list with from_is_list set to "Munge From", and then pipe emails for selected ESP clients through this filter. Said filter must: * detect whether or not the sending domain publishes a DMARC "p=reject" or "p=quarantine" record * If so, parse out the From address in the email and rewrite it in the general form Mailman uses with "on behalf of ..." giving the origianl sender and specifying the mail server's DN in the sender address. ... after which the email will be sent on to the recipients _real_ address. The management of the message body, the piping and such is handled quite well by Courier. I assume that this would mitigate the DMARC issue for redirections through our mail server, just as it does for Mailman. So, if someone could give me a few pointers to the relevant code in Mailman 2, and any suggestions which might save me some time, I can take it from there. I'm python-literate and have hacked our copy of Mailman here in the past - perhaps more than is wise since every time I upgrade I need to apply a number of patches to bring my mods along with the upgrade :) Thanks for any help you can give me. If it's too much bother, tell me so and I'll put on my hacker's hat and go read code :) Ciao, -- Lindsay Haisley | "Behold! Our way lies through a FMP Computer Services | dark wood whence in which 512-259-1190 | weirdness may wallow!? http://www.fmp.com | --Beauregard From mark at msapiro.net Thu Oct 12 16:15:16 2017 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 12 Oct 2017 13:15:16 -0700 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <1507837075.16717.90.camel@fmp.com> References: <1507837075.16717.90.camel@fmp.com> Message-ID: <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> On 10/12/2017 12:37 PM, Lindsay Haisley wrote: > > So, if someone could give me a few pointers to the relevant code in > Mailman 2, and any suggestions which might save me some time, I can > take it from there. I'm python-literate and have hacked our copy of > Mailman here in the past - perhaps more than is wise since every time I > upgrade I need to apply a number of patches to bring my mods along with > the upgrade :) There are two pieces to this in Mailman (both 2.1 and 3.1). One piece is determining the DMARC policy of the From: domain. In MM 2, the code that does this is in Mailman/Utils.py beginning with the comments # The next functions read data from # https://publicsuffix.org/list/public_suffix_list.dat and implement the # algorithm at https://publicsuffix.org/list/ to find the "Organizational # Domain corresponding to a From: domain. and extending through the end of the def _DMARCProhibited(mlist, email, dmarc_domain, org=False): function. There are recent changes for MM 2.1.25. See or just look at This code is improved for MM 3. Most of the changes have to do with the organizational domain data from . in MM 2.1, this is retrieved once when first needed (first post after a (re)start of Mailman) and kept in core until the next restart which could be a long time. In MM 3 the data are cached, but the cache has a lifetime after which it is reloaded. See for the MM 3 code. The second part is the actual From: header munging. In MM 3 that's done by . In MM 2.1 it's in multiple places, but the meat is in Mailman/Handlers/CookHeaders.py. There's more to it because the actual transformations aren't done to the message until after it's been queued for the digest and the archiver, but CookHeaders.py is where the work is done. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From fmouse at fmp.com Sun Oct 15 10:35:19 2017 From: fmouse at fmp.com (Lindsay Haisley) Date: Sun, 15 Oct 2017 09:35:19 -0500 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> Message-ID: <1508078119.16717.129.camel@fmp.com> On Thu, 2017-10-12 at 13:15 -0700, Mark Sapiro wrote: > On 10/12/2017 12:37 PM, Lindsay Haisley wrote:? > > So, if someone could give me a few pointers to the relevant code in > > Mailman 2, and any suggestions which might save me some time, I can > > take it from there. I'm python-literate and have hacked our copy of > > Mailman here in the past - perhaps more than is wise since every time I > > upgrade I need to apply a number of patches to bring my mods along with > > the upgrade :) > > There are two pieces to this in Mailman (both 2.1 and 3.1). One piece is > determining the DMARC policy of the From: domain. In MM 2, the code that > does this is in Mailman/Utils.py beginning with the comments > > # The next functions read data from > # https://publicsuffix.org/list/public_suffix_list.dat?and implement the > # algorithm at https://publicsuffix.org/list/?to find the "Organizational > # Domain corresponding to a From: domain. > > and extending through the end of the > > def _DMARCProhibited(mlist, email, dmarc_domain, org=False): I'm running MM 2.1.18-1 here and find only > # This takes an email address, and returns True if DMARC policy is p=reject > # or possibly quarantine. > def IsDMARCProhibited(mlist, email): > ... etc This looks pretty straight-forward. I can dispense with code related to mlist since I need only a True|False determination of whether the sending domain publishes a DMARC "p=reject" or "p=quarantine" record. You folks are obviously up-to-speed on DMARC nuances and this code looks pretty through. I assume the reference to publicsuffix.org comes with later versions of 2.x and in MM 3 since there's none in 2.1.18-1. This must be something new in the DMARC mitigation world and I'm not familiar with it. Is there any reason to pull in a more recent MM 2 and use the DMARC detection code therein? Speed is important here since this is simply a turnaround on a single email, not dependent on any list variables. I'm reluctant to burden every redirection turnaround with an HTTP look-up. Replacement of the From header is just a matter of reading the email headers into an array, making modifications if necessary and pushing the result, followed by the message body, out to Courier's sendmail clone. Basically: if from_domain publishes bad DMARC: ? ? if Reply-To does not exist: ? ? ? ? copy From header to Reply-To ? ? Replace From with "On behalf of old_From" Feed headers and body to Courier's sendmail clone My take on it is that this should work OK. -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From mark at msapiro.net Sun Oct 15 13:24:16 2017 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 15 Oct 2017 10:24:16 -0700 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <1508078119.16717.129.camel@fmp.com> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> Message-ID: <3bf8e27e-328a-7d5b-a4f9-9e616824fe62@msapiro.net> On 10/15/2017 07:35 AM, Lindsay Haisley wrote: > > I assume the reference to publicsuffix.org comes with later versions of > 2.x and in MM 3 since there's none in 2.1.18-1. This must be something > new in the DMARC mitigation world and I'm not familiar with it. This is code that was added in 2.1.22 to deal with organizational domains. Every domain has a corresponding organizational domain which may or may not be the same as the original domain. In many cases it's simple. For example, the organizational domain for example.com is example.com and organizational domain for any.subdomain.of.example.com is example.com. The DMARC standard says check the policy of the domain, but if the domain doesn't publish a policy, check the policy of the corresponding organizational domain, so you actually need to check the organizational domain. It's even more complicated than that because the organizational domain can publish a p= policy which applies to it and any subdomains that don't publish their own policy, but it can also publish an s= policy which applies only to subdomains that don't publish their own policy but not to itself. The actual determination of the organizational domain for a given domain can be complex. For common tlds like .com, .org, .edu, .net and the like, the organizational domain is simply the next level, e.g. example.com, etc., but it can get much more complicated than that. For example, see the .jp section in the data at . > Is there any reason to pull in a more recent MM 2 and use the DMARC > detection code therein? Speed is important here since this is simply a > turnaround on a single email, not dependent on any list variables. I'm > reluctant to burden every redirection turnaround with an HTTP look-up. I think you need to deal with organizational domains. You may be able to get away with just assuming the organizational domain is the two top levels and ignoring all those cases where it isn't, but you should at least look at either or for ideas. > Replacement of the From header is just a matter of reading the email > headers into an array, making modifications if necessary and pushing > the result, followed by the message body, out to Courier's sendmail > clone. Basically: > > if from_domain publishes bad DMARC: > ? ? if Reply-To does not exist: > ? ? ? ? copy From header to Reply-To > ? ? Replace From with "On behalf of old_From" I have seen it said that email addresses in display names in From: headers are a sign of spaminess. Thus, in the above I suggest that old_From should just be the display name part of the original From: or be munged in some way (replace '@' with ' at ' or ?) so it doesn't look like an email address. > Feed headers and body to Courier's sendmail clone > > My take on it is that this should work OK. > -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From vesely at tana.it Mon Oct 16 04:17:06 2017 From: vesely at tana.it (Alessandro Vesely) Date: Mon, 16 Oct 2017 10:17:06 +0200 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <3bf8e27e-328a-7d5b-a4f9-9e616824fe62@msapiro.net> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> <3bf8e27e-328a-7d5b-a4f9-9e616824fe62@msapiro.net> Message-ID: <064ce4bf-a06e-91ca-08b4-45ae0e52e199@tana.it> On Sun 15/Oct/2017 19:24:16 +0200 Mark Sapiro wrote: >> Replacement of the From header is just a matter of reading the email >> headers into an array, making modifications if necessary and pushing >> the result, followed by the message body, out to Courier's sendmail >> clone. Basically: >> >> if from_domain publishes bad DMARC: >> ? ? if Reply-To does not exist: >> ? ? ? ? copy From header to Reply-To >> ? ? Replace From with "On behalf of old_From" > > I have seen it said that email addresses in display names in From: > headers are a sign of spaminess. Thus, in the above I suggest that > old_From should just be the display name part of the original From: or > be munged in some way (replace '@' with ' at ' or ?) so it doesn't look > like an email address. I'm unclear how adding a Reply-To:, depending on users' clients is going to affect mailing list traffic. For an alternative, when the usual action is to reply to mailing list, it seems to me to be smoother to just mangle the From: so as to make it like so: John Doe Although the addition of capitalized anti-spam diversions in the domain part of email addresses is still en vogue in several mailing list, this method never gained traction. Why? Ale -- http://www.example.org.REMOVE.THE.TRAILING.PARTS/ From mark at msapiro.net Mon Oct 16 12:31:55 2017 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 16 Oct 2017 09:31:55 -0700 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <064ce4bf-a06e-91ca-08b4-45ae0e52e199@tana.it> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> <3bf8e27e-328a-7d5b-a4f9-9e616824fe62@msapiro.net> <064ce4bf-a06e-91ca-08b4-45ae0e52e199@tana.it> Message-ID: <9e2cf1f8-8909-a42c-b759-0728c6d4db33@msapiro.net> On 10/16/2017 01:17 AM, Alessandro Vesely wrote: > > I'm unclear how adding a Reply-To:, depending on users' clients is going to > affect mailing list traffic. Had you read the whole thread from the beginning, you would know that Lindsay is not talking about mailing list mail here. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From fmouse at fmp.com Mon Oct 16 12:52:09 2017 From: fmouse at fmp.com (Lindsay Haisley) Date: Mon, 16 Oct 2017 11:52:09 -0500 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <9e2cf1f8-8909-a42c-b759-0728c6d4db33@msapiro.net> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> <3bf8e27e-328a-7d5b-a4f9-9e616824fe62@msapiro.net> <064ce4bf-a06e-91ca-08b4-45ae0e52e199@tana.it> <9e2cf1f8-8909-a42c-b759-0728c6d4db33@msapiro.net> Message-ID: <1508172729.16717.140.camel@fmp.com> On Mon, 2017-10-16 at 09:31 -0700, Mark Sapiro wrote: > On 10/16/2017 01:17 AM, Alessandro Vesely wrote: > >? > > I'm unclear how adding a Reply-To:, depending on users' clients is > going to > > affect mailing list traffic. > > > Had you read the whole thread from the beginning, you would know that > Lindsay is not talking about mailing list mail here. Alessandro, I'm stealing code ;) DMARC is a hassle for those of us who operate legitimate email redirection services, e.g. for customers with custom domain names who get at least some of their email someplace such as Gmail. My MTA (Courier) allows the introduction of an arbitrary program into the mail pipe, which in this case is something I'm putting together to do a header munge a la Mailman. Mark has been most helpful and the piece is almost done. Python just WORKS :) -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From fmouse at fmp.com Mon Oct 16 16:06:44 2017 From: fmouse at fmp.com (Lindsay Haisley) Date: Mon, 16 Oct 2017 15:06:44 -0500 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <9e2cf1f8-8909-a42c-b759-0728c6d4db33@msapiro.net> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> <3bf8e27e-328a-7d5b-a4f9-9e616824fe62@msapiro.net> <064ce4bf-a06e-91ca-08b4-45ae0e52e199@tana.it> <9e2cf1f8-8909-a42c-b759-0728c6d4db33@msapiro.net> Message-ID: <1508184404.16717.146.camel@fmp.com> On Mon, 2017-10-16 at 09:31 -0700, Mark Sapiro wrote: > On 10/16/2017 01:17 AM, Alessandro Vesely wrote: > > > > > > I'm unclear how adding a Reply-To:, depending on users' clients is > > going to > > affect mailing list traffic. > > Had you read the whole thread from the beginning, you would know that > Lindsay is not talking about mailing list mail here. I've ported much of the MM 2 DMARC mitigation code to an email processor for the Courier MTA. The principles I've used could probably be applied to many MTAs. If anyone is interested, I'll be happy to post my code (dmarc_shield.py) to this list as an attachment, and if anyone has the time and inclination to critique it, or point out potential bugs or problems I'd be happy to receive them. Many thanks to Mark for taking the time to go over the salient points in the Mailman code in email. -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Oct 16 23:38:26 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 17 Oct 2017 12:38:26 +0900 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <1508078119.16717.129.camel@fmp.com> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> Message-ID: <23013.31538.287126.725058@turnbull.sk.tsukuba.ac.jp> I gather you have completed your program already, but I had this in the works and it might be useful for people doing similar things. Lindsay Haisley writes: > Is there any reason to pull in a more recent MM 2 and use the DMARC > detection code therein? Speed is important here since this is simply a > turnaround on a single email, not dependent on any list variables. I'm > reluctant to burden every redirection turnaround with an HTTP > look-up. You may want to consider improving performance by caching DNS results by domain. This should be reasonable space as long as you do this check after spam elimination. If expiries are fixed you will need to have a reasonably short expiry on negative (p=none) results (fails nasty -- bounces from receivers), but could have a pretty long one on positive results. You could also get the actual TTL out of the DNS reply for more accurate expiry. Of course the effectiveness of caching depends heavily on the actual pattern of mail received at the domains in question. You don't need to burden each redirection with an HTTP lookup. There's only one publicsuffix list, which you can download occasionally. I would guess once a day would be more than enough and no burden at all; you could even do it asynchronously in a cron job. > if from_domain publishes bad DMARC: > ? ? if Reply-To does not exist: > ? ? ? ? copy From header to Reply-To > ? ? Replace From with "On behalf of old_From" > Feed headers and body to Courier's sendmail clone > > My take on it is that this should work OK. I think you should consider adding an else to the inner if: else: append From to Reply-To as Mailman does. Otherwise the user has to copy/paste the address if they really want to reply to the author rather than the Reply-To for some reason, and it may not be present at all if you follow Mark's advice to not copy it to the display name in From. I'm with Mark on everything else. I don't think the probability you need to deal with organizational domains is that high, but the costs are potentially high (collateral damage = disabled or unsubscribed users). From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Oct 16 23:39:42 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 17 Oct 2017 12:39:42 +0900 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <064ce4bf-a06e-91ca-08b4-45ae0e52e199@tana.it> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> <3bf8e27e-328a-7d5b-a4f9-9e616824fe62@msapiro.net> <064ce4bf-a06e-91ca-08b4-45ae0e52e199@tana.it> Message-ID: <23013.31614.482701.684317@turnbull.sk.tsukuba.ac.jp> Alessandro Vesely writes: > I'm unclear how adding a Reply-To:, depending on users' clients is > going to affect mailing list traffic. For an alternative, when the > usual action is to reply to mailing list, it seems to me to be > smoother to just mangle the From: so as to make it like so: > > John Doe > > Although the addition of capitalized anti-spam diversions in the > domain part of email addresses is still en vogue in several mailing > list, this method never gained traction. Why? I would guess the main reasons are that it's ugly and makes the convenient automatic field-filling features of MUAs inaccurate, especially those MUAs that don't actually present the address (which are very common these days). Steve -- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From fmouse at fmp.com Tue Oct 17 02:23:47 2017 From: fmouse at fmp.com (Lindsay Haisley (linode)) Date: Tue, 17 Oct 2017 01:23:47 -0500 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <23013.31538.287126.725058@turnbull.sk.tsukuba.ac.jp> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> <23013.31538.287126.725058@turnbull.sk.tsukuba.ac.jp> Message-ID: <3EB9F11F-6FB8-4D9D-82BE-F26087ABBE52@fmp.com> On Oct 16, 2017, at 10:38 PM, Stephen J. Turnbull wrote: > > I gather you have completed your program already, but I had this in > the works and it might be useful for people doing similar things. Good points, Stephen. Thanks. Yes, I?ve got the basic code well honed to cooperate with Courier but have close to a week of hard work ahead of me on totally unrelated tasks, so I?m putting this project on the shelf for the moment. I?ll put proper caching on the short to-do list. This puts me in mind of the old adage, the only way to get a program finished is to kill the programmer :) From fmouse at fmp.com Tue Oct 17 02:47:53 2017 From: fmouse at fmp.com (Lindsay Haisley (linode)) Date: Tue, 17 Oct 2017 01:47:53 -0500 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <23013.31614.482701.684317@turnbull.sk.tsukuba.ac.jp> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> <3bf8e27e-328a-7d5b-a4f9-9e616824fe62@msapiro.net> <064ce4bf-a06e-91ca-08b4-45ae0e52e199@tana.it> <23013.31614.482701.684317@turnbull.sk.tsukuba.ac.jp> Message-ID: <8D8A401A-15C4-43D6-B528-33576C6E93AE@fmp.com> Sent from my iPhone > On Oct 16, 2017, at 10:39 PM, Stephen J. Turnbull wrote: > > I would guess the main reasons are that it's ugly and makes the > convenient automatic field-filling features of MUAs inaccurate, > especially those MUAs that don't actually present the address (which > are very common these days). What I did, at Mark?s suggestion, Was to convert the ?@? symbol in the original From to the word ?at? and add the whole she-bang to the new From address comment. I?m not sure about the usefulness of tacking the old From address onto an existing Reply?To address. Yes, it preserves choice, but it?s a choice that few, if any of FMP?s customers/users would really understand or make use of. Given the simplistic design of most modern MUAs, and the general lack of knowledge about the inner workings of email clients (and the possibilities therein), most folks, including me, just hit the reply button and take it from there. Sent from my iPhone From fmouse at fmp.com Tue Oct 17 14:28:34 2017 From: fmouse at fmp.com (Lindsay Haisley) Date: Tue, 17 Oct 2017 13:28:34 -0500 Subject: [Mailman-Developers] Extracting some Mailman code In-Reply-To: <23013.31538.287126.725058@turnbull.sk.tsukuba.ac.jp> References: <1507837075.16717.90.camel@fmp.com> <3f39383e-f2c2-ae67-b21c-d946a4cea405@msapiro.net> <1508078119.16717.129.camel@fmp.com> <23013.31538.287126.725058@turnbull.sk.tsukuba.ac.jp> Message-ID: <1508264914.41533.42.camel@fmp.com> On Tue, 2017-10-17 at 12:38 +0900, Stephen J. Turnbull wrote: > I gather you have completed your program already, but I had this in > the works and it might be useful for people doing similar things. If anyone is interested in my DMARC mitigation code for Courier-MTA it's online at . There are lots of usage notes and such in the code comments at the beginning of the program. Stephen, as a first step to speeding this piece up, I've made the Organizational Domain database lookup local to the machine hosting the MTA so it's very fast and doesn't load the publicsuffix.org web server. -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From fmouse at fmp.com Thu Oct 19 10:32:50 2017 From: fmouse at fmp.com (Lindsay Haisley) Date: Thu, 19 Oct 2017 09:32:50 -0500 Subject: [Mailman-Developers] DMARC mitigation code question Message-ID: <1508423570.47036.208.camel@fmp.com> Mark, et al While configuring the MM 2.1 DMARC mitigation code for use in Courier- MTA I note that the return from?_DMARCProhibited(..) for a detected policy of "p=none" is True, just as it is for "p=quarantine" and "p=reject". Why is this? I would assume that if this code is reached, and p=none, that no From munging would be required. -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From mark at msapiro.net Thu Oct 19 21:13:24 2017 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 19 Oct 2017 18:13:24 -0700 Subject: [Mailman-Developers] DMARC mitigation code question In-Reply-To: <1508423570.47036.208.camel@fmp.com> References: <1508423570.47036.208.camel@fmp.com> Message-ID: <25e999ab-eed5-fdc5-6da7-48cce57683d5@msapiro.net> On 10/19/2017 07:32 AM, Lindsay Haisley wrote: > Mark, et al > > While configuring the MM 2.1 DMARC mitigation code for use in Courier- > MTA I note that the return from?_DMARCProhibited(..) for a detected > policy of "p=none" is True, just as it is for "p=quarantine" and > "p=reject". Why is this? I would assume that if this code is reached, > and p=none, that no From munging would be required. I think the code you're looking at is conditional on dmarc_none_moderation_action being True which is neither a default nor a recommended setting. It was added (reluctantly) in response to and probably should not have been. It is not in Mailman 3. If you have time and interest, read the bug report (and all 11 comments). -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From fmouse at fmp.com Fri Oct 20 12:24:44 2017 From: fmouse at fmp.com (Lindsay Haisley) Date: Fri, 20 Oct 2017 11:24:44 -0500 Subject: [Mailman-Developers] DMARC mitigation code question In-Reply-To: <25e999ab-eed5-fdc5-6da7-48cce57683d5@msapiro.net> References: <1508423570.47036.208.camel@fmp.com> <25e999ab-eed5-fdc5-6da7-48cce57683d5@msapiro.net> Message-ID: <1508516684.4881.10.camel@ubuntu.fmp.com> On Thu, 2017-10-19 at 18:13 -0700, Mark Sapiro wrote: > On 10/19/2017 07:32 AM, Lindsay Haisley wrote: > > Mark, et al > > > > While configuring the MM 2.1 DMARC mitigation code for use in Courier- > > MTA I note that the return from _DMARCProhibited(..) for a detected > > policy of "p=none" is True, just as it is for "p=quarantine" and > > "p=reject". Why is this? I would assume that if this code is reached, > > and p=none, that no From munging would be required. > > > I think the code you're looking at is conditional on > dmarc_none_moderation_action being True which is neither a default nor a > recommended setting. It was added (reluctantly) in response to > and probably should > not have been. It is not in Mailman 3. > > If you have time and interest, read the bug report (and all 11 comments). I read the comments on the bug report and looked further at the original code at . It looks as if munging on a policy of "p=none" is a per-list or per-site choice. I cut out code related to the mlist object from my application so this remained as a default. I've set line 1359 to "return False", but will give the issue some thought and possibly make it a configuration option from a comment-annotated variable definitions near the start of the code. I'm trying to keep this piece as simple as possible. For anyone interested, I've posted the dmarc_shield.py publicly at . -- Lindsay Haisley | "Everything works if you let it" FMP Computer Services | 512-259-1190 | - The Roadie http://www.fmp.com | From mark at msapiro.net Fri Oct 20 14:15:03 2017 From: mark at msapiro.net (Mark Sapiro) Date: Fri, 20 Oct 2017 11:15:03 -0700 Subject: [Mailman-Developers] DMARC mitigation code question In-Reply-To: <1508516684.4881.10.camel@ubuntu.fmp.com> References: <1508423570.47036.208.camel@fmp.com> <25e999ab-eed5-fdc5-6da7-48cce57683d5@msapiro.net> <1508516684.4881.10.camel@ubuntu.fmp.com> Message-ID: <6a87a779-606f-45c0-8315-8e0bf3785283@msapiro.net> On 10/20/2017 09:24 AM, Lindsay Haisley wrote: > It looks as if munging on a policy of "p=none" is a per-list or per-site > choice. I cut out code related to the mlist object from my application > so this remained as a default. That's the issue. The stanza beginning at requires three conditions to return True on p=none. They are all conditions on list settings: dmarc_none_moderation_action - True dmarc_quarantine_moderation_action - True dmarc_moderation_action - Munge From or Wrap Message. This is consistent with the web UI help text at > I've set line 1359 to "return False", but > will give the issue some thought and possibly make it a configuration > option from a comment-annotated variable definitions near the start of > the code. I'm trying to keep this piece as simple as possible. For > anyone interested, I've posted the dmarc_shield.py publicly at > . I recommend just deleting the entire stanza at lines 1352-1360 which will fall through to the "return False" at line 1361. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From fmouse at fmp.com Fri Oct 20 15:20:38 2017 From: fmouse at fmp.com (Lindsay Haisley) Date: Fri, 20 Oct 2017 14:20:38 -0500 Subject: [Mailman-Developers] DMARC mitigation code question In-Reply-To: <6a87a779-606f-45c0-8315-8e0bf3785283@msapiro.net> References: <1508423570.47036.208.camel@fmp.com> <25e999ab-eed5-fdc5-6da7-48cce57683d5@msapiro.net> <1508516684.4881.10.camel@ubuntu.fmp.com> <6a87a779-606f-45c0-8315-8e0bf3785283@msapiro.net> Message-ID: <1508527238.3456.0.camel@ubuntu.fmp.com> On Fri, 2017-10-20 at 11:15 -0700, Mark Sapiro wrote: > I recommend just deleting the entire stanza at lines 1352-1360 which > will fall through to the "return False" at line 1361. > Probably a wise application of the KISS principle as well :) -- Lindsay Haisley | "Everything works if you let it" FMP Computer Services | 512-259-1190 | - The Roadie http://www.fmp.com | From aurelien at bompard.org Mon Oct 23 11:48:42 2017 From: aurelien at bompard.org (Aurelien Bompard) Date: Mon, 23 Oct 2017 17:48:42 +0200 Subject: [Mailman-Developers] Renaming a list Message-ID: Hey folks! I wanted to rename a list (only the list_name, not the list_id) and I figured I would make the transition smoother by adding an "AcceptableAlias" with the old name. I did that and realized that the acceptable aliases are not added to the postfix_lmtp file, as a result emails sent to the old address are just bounced at the Postfix level. Is this the intended behavior ? BTW, changing the list_name attribute is sufficient to rename a list, right? I didn't find anything else to change since everything seems to be keyed to the list_id but I thought I'd check with you. Thanks! Aur?lien From barry at list.org Mon Oct 23 11:59:55 2017 From: barry at list.org (Barry Warsaw) Date: Mon, 23 Oct 2017 11:59:55 -0400 Subject: [Mailman-Developers] Renaming a list In-Reply-To: References: Message-ID: On Oct 23, 2017, at 11:48, Aurelien Bompard wrote: > > I wanted to rename a list (only the list_name, not the list_id) and I > figured I would make the transition smoother by adding an "AcceptableAlias" > with the old name. I did that and realized that the acceptable aliases are > not added to the postfix_lmtp file, as a result emails sent to the old > address are just bounced at the Postfix level. > Is this the intended behavior ? It is. The thinking is that whatever email addresses end up funneling to a list is really outside of Mailman?s control. That?s really part of your MTA?s configuration, and acceptable aliases are just a way of telling Mailman to expand its implicit address logic. > BTW, changing the list_name attribute is sufficient to rename a list, > right? I didn't find anything else to change since everything seems to be > keyed to the list_id but I thought I'd check with you. Correct. List name is mutable but list-id is not, even if you move it to a different domain! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From maxking at asynchronous.in Mon Oct 23 20:02:27 2017 From: maxking at asynchronous.in (Abhilash Raj) Date: Mon, 23 Oct 2017 17:02:27 -0700 Subject: [Mailman-Developers] Renaming a list In-Reply-To: References:

Message-ID: <1508803347.2964661.1148632464.3D92DC2B@webmail.messagingengine.com> On Mon, Oct 23, 2017, at 08:59 AM, Barry Warsaw wrote: > On Oct 23, 2017, at 11:48, Aurelien Bompard wrote: > > > > I wanted to rename a list (only the list_name, not the list_id) and I > > figured I would make the transition smoother by adding an "AcceptableAlias" > > with the old name. I did that and realized that the acceptable aliases are > > not added to the postfix_lmtp file, as a result emails sent to the old > > address are just bounced at the Postfix level. > > Is this the intended behavior ? > > It is. The thinking is that whatever email addresses end up funneling to > a list is really outside of Mailman?s control. That?s really part of > your MTA?s configuration, and acceptable aliases are just a way of > telling Mailman to expand its implicit address logic. As I understand, AcceptableAliases are defined as regular expressions, what if Mailman put the values that are a valid list name to transport_maps? Converting regex to a listname doesn't sound like a straight forward thing to me though. This most definitely adds some complexity for unique checks of Listnames. I am not sure if we already check acceptable alias rules for existing mailing lists to make sure they don't match any new list created. We'd have to do it if we start putting acceptable aliases in transport maps. As a feature, it could be helpful to have multiple posting addresses for a single List. Are there any major downsides to doing this? -- Abhilash Raj maxking at asynchronous.in From mark at msapiro.net Mon Oct 23 20:20:23 2017 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 23 Oct 2017 17:20:23 -0700 Subject: [Mailman-Developers] Renaming a list In-Reply-To: <1508803347.2964661.1148632464.3D92DC2B@webmail.messagingengine.com> References:

<1508803347.2964661.1148632464.3D92DC2B@webmail.messagingengine.com> Message-ID: On 10/23/2017 05:02 PM, Abhilash Raj wrote: > On Mon, Oct 23, 2017, at 08:59 AM, Barry Warsaw wrote: >> On Oct 23, 2017, at 11:48, Aurelien Bompard wrote: >>> >>> I wanted to rename a list (only the list_name, not the list_id) and I >>> figured I would make the transition smoother by adding an "AcceptableAlias" >>> with the old name. I did that and realized that the acceptable aliases are >>> not added to the postfix_lmtp file, as a result emails sent to the old >>> address are just bounced at the Postfix level. >>> Is this the intended behavior ? >> >> It is. The thinking is that whatever email addresses end up funneling to >> a list is really outside of Mailman?s control. That?s really part of >> your MTA?s configuration, and acceptable aliases are just a way of >> telling Mailman to expand its implicit address logic. > > As I understand, AcceptableAliases are defined as regular expressions, > what if Mailman put the values that are a valid list name to > transport_maps? Converting regex to a listname doesn't sound like a > straight forward thing to me though. I think that's going to be too complex and error prone. > This most definitely adds some complexity for unique checks of > Listnames. I am not sure if we already check acceptable alias rules for > existing mailing lists to make sure they don't match any new list > created. We'd have to do it if we start putting acceptable aliases in > transport maps. We don't and it doesn't matter for what acceptable_aliases is intended to be used for. The background is a list can be configured (and is by default) to require_explicit_destination. which is intended as an anti-spam measure. I.e. it requires the list posting address to appear in To: or Cc: of the message and stops "To: undisclosed recipients:;" type of mail. > As a feature, it could be helpful to have multiple posting addresses for > a single List. Are there any major downsides to doing this? You can, either by aliases, .forwards or other MTA type configurations. And that's exactly what acceptable_aliases is for - to accept those alternate posting addresses as valid explicit destinations. But, I think it should be up to the installation to configure their MTAs or whatever for this. I don't thing we should be trying to generate MTA configurations for alternate list names bases on acceptable_aliases. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From aurelien at bompard.org Tue Oct 24 04:38:03 2017 From: aurelien at bompard.org (Aurelien Bompard) Date: Tue, 24 Oct 2017 10:38:03 +0200 Subject: [Mailman-Developers] Renaming a list In-Reply-To: References:

<1508803347.2964661.1148632464.3D92DC2B@webmail.messagingengine.com> Message-ID: > > > As I understand, AcceptableAliases are defined as regular expressions, > > what if Mailman put the values that are a valid list name to > > transport_maps? Converting regex to a listname doesn't sound like a > > straight forward thing to me though. > > I think that's going to be too complex and error prone. > Agreed. But, I think it should be up to the installation to configure their MTAs > or whatever for this. I don't thing we should be trying to generate MTA > configurations for alternate list names bases on acceptable_aliases. > Yeah, I'll just use a separate hash of renamed lists in Postfix. Thanks. Aur?lien From maxking at asynchronous.in Tue Oct 24 16:52:46 2017 From: maxking at asynchronous.in (Abhilash Raj) Date: Tue, 24 Oct 2017 13:52:46 -0700 Subject: [Mailman-Developers] Signing commits with gpg Message-ID: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> Hi All, Gitlab now supports verification of commit signatures and it would be awesome if we start signing commits. It is a relatively painless process and happens automatically with little configuration. Spoofing authors in git is quite easy, actually provided as a command line option (--author, --reset-author), and I believe it would be a good practice to sign all the commits (even outside of Mailman). Here are steps for how you can do that: 1. Add your public key to Gitlab (https://gitlab.com/profile/gpg_keys) 2. Commit with `-S` (capital S) Here is the relevant section of `.gitconfig` to auto-sign every commit you make (no need for step 2 if you do this): ``` [user] name = Abhilash Raj email = raj.abhilash1 at gmail.com signingkey = 541EA0448453394FF77A0ECC9D9B2BA061D0A67C [commit] gpgsign = true ``` Once you have pushed a signed commit to Gitlab and have uploaded your gpg public key, you will see a green "Verified" button alongside every commit. (See https://gitlab.com/maxking/mailman/commits/msapiro/mailman-pending) -- Abhilash Raj maxking at asynchronous.in From barry at list.org Tue Oct 24 17:18:14 2017 From: barry at list.org (Barry Warsaw) Date: Tue, 24 Oct 2017 17:18:14 -0400 Subject: [Mailman-Developers] Signing commits with gpg In-Reply-To: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> References: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> Message-ID: <0BEF3E3E-57D8-4A49-A66C-13A48D7F166D@list.org> On Oct 24, 2017, at 16:52, Abhilash Raj wrote: > > Gitlab now supports verification of commit signatures and it would be > awesome if we start signing commits. It is a relatively painless process > and happens automatically with little configuration. Very cool that GL has enabled this! Thanks for sending the recipe too. I definitely encourage folks (especially core devs) to start signing commits. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mark at msapiro.net Tue Oct 24 18:56:23 2017 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 24 Oct 2017 15:56:23 -0700 Subject: [Mailman-Developers] Signing commits with gpg In-Reply-To: <0BEF3E3E-57D8-4A49-A66C-13A48D7F166D@list.org> References: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> <0BEF3E3E-57D8-4A49-A66C-13A48D7F166D@list.org> Message-ID: On 10/24/2017 02:18 PM, Barry Warsaw wrote: > On Oct 24, 2017, at 16:52, Abhilash Raj wrote: >> >> Gitlab now supports verification of commit signatures and it would be >> awesome if we start signing commits. It is a relatively painless process >> and happens automatically with little configuration. > > Very cool that GL has enabled this! Thanks for sending the recipe too. I definitely encourage folks (especially core devs) to start signing commits. I have set my .gitconfig to automatically sign commits (I already had my signingkey in the [user] section, but I didn't have [commit] gpgsign = true which I now do). I remember looking into signing commits when we first switched from bzr to git because I was used to signing all commits. At that time, it seemed controversial. See, e.g., where linus argues that "Signing each commit is totally stupid." and that you should sign tags but not commits. I don't know enough about the internals of this to have an opinion, and as I said I will be signing my commits going forward, and the post I link to is over 8 years old and things might have changed, but there it is for what it's worth. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From barry at list.org Wed Oct 25 11:40:04 2017 From: barry at list.org (Barry Warsaw) Date: Wed, 25 Oct 2017 11:40:04 -0400 Subject: [Mailman-Developers] Signing commits with gpg In-Reply-To: References: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> <0BEF3E3E-57D8-4A49-A66C-13A48D7F166D@list.org> Message-ID: On Oct 24, 2017, at 18:56, Mark Sapiro wrote: > > I remember looking into signing commits when we first switched from bzr > to git because I was used to signing all commits. At that time, it > seemed controversial. See, e.g., > > where linus argues that "Signing each commit is totally stupid." and > that you should sign tags but not commits. > > I don't know enough about the internals of this to have an opinion, and > as I said I will be signing my commits going forward, and the post I > link to is over 8 years old and things might have changed, but there it > is for what it's worth. I?m not sure that any of the points Linus brings up in that thread have changed, but I?m also not sure how relevant they are to our workflow. It?s interesting enough that Gitlab is now showing the verified tag for signed commits, although TBH, I?m also not sure how much that buys us in practice. Still, it?s easy enough to experiment with, so let?s do it and see if it has any practical impact on us, either pro or con. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From simon at hannaweb.eu Wed Oct 25 12:14:23 2017 From: simon at hannaweb.eu (Simon Hanna) Date: Wed, 25 Oct 2017 18:14:23 +0200 Subject: [Mailman-Developers] Signing commits with gpg In-Reply-To: References: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> <0BEF3E3E-57D8-4A49-A66C-13A48D7F166D@list.org>

Message-ID: I guess more important would be to sign the releases. At least archlinux likes to have signatures for source archives and often requests upstream projects to add this. For me as a user it would be more interesting to have a verified release signed by one key that's static rather than a commit history that is signed by many different keys that I don't know. I guess the single commit signature is more relevant to other developers, so we know who actually committed something. However, if all commits to the master branches come from merge requests, you already use gitlabs verification. It's not as good as gpg signatures, but in the end you have to trust gitlab to a certain degree anyway... Another thing that just came to mind: how does commit squashing work? You'll probably have to do that offline and not use gitlabs autosmashing... I don't have anything against it and I can also rather easily start doing that. (I will have to have my keychain nearby, as I don't have my keys stored on my machines...) -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From barry at list.org Wed Oct 25 12:55:56 2017 From: barry at list.org (Barry Warsaw) Date: Wed, 25 Oct 2017 12:55:56 -0400 Subject: [Mailman-Developers] Signing commits with gpg In-Reply-To: References: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> <0BEF3E3E-57D8-4A49-A66C-13A48D7F166D@list.org>

Message-ID: <6D3B17EC-9339-4B81-8C36-AFC7AC2EBAF6@list.org> On Oct 25, 2017, at 12:14, Simon Hanna wrote: > > I guess more important would be to sign the releases. At least archlinux likes to have signatures for source archives and often requests upstream projects to add this. Definitely. I (try to remember to) sign both tags and releases for Core. > Another thing that just came to mind: how does commit squashing work? You'll probably have to do that offline and not use gitlabs autosmashing? I would think that squash merges would destroy the record of any intermediate signed commits. Core doesn?t have a firm policy either way; sometimes I squash merge sometimes not. I?m philosophical opposed to squash merging, but git often really makes me want to do it anyway. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From dkg at fifthhorseman.net Wed Oct 25 19:11:39 2017 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Wed, 25 Oct 2017 19:11:39 -0400 Subject: [Mailman-Developers] Signing commits with gpg In-Reply-To: References: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> <0BEF3E3E-57D8-4A49-A66C-13A48D7F166D@list.org>

Message-ID: <87fua6yg9g.fsf@fifthhorseman.net> On Wed 2017-10-25 18:14:23 +0200, Simon Hanna wrote: > For me as a user it would be more interesting to have a verified > release signed by one key that's static rather than a commit history > that is signed by many different keys that I don't know. this is not an "either/or" thing. it's a "both, and!" thing. software provenance works at multiple levels, and for software that we care about, we should have a cryptographic path on as many of them as possible. --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Oct 26 01:25:07 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 26 Oct 2017 14:25:07 +0900 Subject: [Mailman-Developers] Signing commits with gpg In-Reply-To: References: <1508878366.822863.1149764296.597023D9@webmail.messagingengine.com> <0BEF3E3E-57D8-4A49-A66C-13A48D7F166D@list.org> Message-ID: <23025.29107.838288.194844@turnbull.sk.tsukuba.ac.jp> Mark Sapiro writes: > where linus argues that "Signing each commit is totally stupid." and > that you should sign tags but not commits. I agree with Linus that signing all commits is probably unnecessary because of the SHA1 chain, but I disagree with signing only tags. I think that the theoretical sweet spot is signing merge commits (or branch head in case of a fast-forward) at push time. But pragmatically that's too annoying (requires user decision AFAIK, easy to omit, etc), so autosigning every commit FTW IMHO. Steve From vesely at tana.it Thu Oct 26 14:20:38 2017 From: vesely at tana.it (Alessandro Vesely) Date: Thu, 26 Oct 2017 20:20:38 +0200 Subject: [Mailman-Developers] Use of the public suffix list Message-ID: Hi all, I noticed (from a DMARC mitigation utility that Lindsay extracted) that Mailman features its own approach to using the PSL. Of course, development must go on, and sometimes it is a waste of time trying to make a super-duper scaffolding for a job that can be carried out complying to the KISS principle. At any rate, what is the future of DMARC lookups in Mailman? * The specs say that "DMARC should be amended to use [a method better than PSL] as soon as it is generally available" [1]. I believe that sentence refers to RDAP, which was released more or less at the same time (March 2015) [2]. [1] https://tools.ietf.org/html/rfc7489#appendix-A.6 [2] https://datatracker.ietf.org/wg/weirds/documents/ * There are various Python packages for domain name splitting. They obviously use the PSL, but supposedly would transparently switch to a better method in case. If Mailman used one such package, a practical advantage for users would be to update the PSL in only one place, if they happened to use the same dependency. I found six packages. tldextract [3] is the only one of them which caches a JSON object rather than the original textual representation of the list. It uses a frozenset. tld [4] and publicsuffixlist [5] also build a set. publicsuffix[6] and publicsuffix2 [7] build lists of nested dictionaries from all the labels. dnspy [8] builds a dictionary of FQDNs, somewhat like Mailman. How does the time to build the structure compare with the time taken by DSN queries? [3] https://pypi.python.org/pypi/tldextract [4] https://pypi.python.org/pypi/tld [5] https://pypi.python.org/pypi/publicsuffixlist [6] https://pypi.python.org/pypi/publicsuffix [7] https://pypi.python.org/pypi/publicsuffix2 [8] https://pypi.python.org/pypi/dnspy * Debian distributes a publicsuffix package which brings a textual version of the list. Since stretch, it also brings a "dafsa" version. Nowadays, most C implementations (Firefox, Chromium) use dafsa. They build the structure using offsets rather than pointers, so that the blob can be defined in a source file as a literal static array of chars, in order to minimize loading time. That strategy works well as long as the relevant package is upgraded more frequently than the PSL. Otherwise, as for libpsl, one ends up using obsolete data. Surprisingly, the publisuffix package itself is not upgraded as frequently as the PSL. This bug [9] is what prompted me to write this message. I guess you, as Mailman developers, have pondered this subject and I'd be interested to know what you think. [9] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879008 TIA for any reply Ale -- From mark at msapiro.net Thu Oct 26 18:09:28 2017 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 26 Oct 2017 15:09:28 -0700 Subject: [Mailman-Developers] Mailman 2.1.25 released Message-ID: <102d4978-61d0-67ac-d5ed-7c1603c86fd8@msapiro.net> I am pleased to announce the release of Mailman 2.1.25. Python 2.4 is the minimum supported, but Python 2.7 is strongly recommended. This is a routine bug fix release with a minor new feature and some accessibility improvements for screen readers. See the attached README.txt for details. Mailman is free software for managing email mailing lists and e-newsletters. Mailman is used for all the python.org and SourceForge.net mailing lists, as well as at hundreds of other sites. For more information, please see our web site at one of: http://www.list.org https://www.gnu.org/software/mailman http://mailman.sourceforge.net/ https://mirror.list.org/ Mailman 2.1.25 can be downloaded from https://launchpad.net/mailman/2.1/ https://ftp.gnu.org/gnu/mailman/ https://sourceforge.net/projects/mailman/ -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- 2.1.25 (26-Oct-2017) New Features - The admindb held subscriptions listing now includes the date of the most recent request from the address. (LP: #1697097) Accessibility - The admin Membership List now includes text for screen readers which identifies the function of each checkbox. CSS is added to the page to visually hide the text but still allow screen readers to read it. Similar text has been added to some radio buttons on the admindb pages. i18n - The Russian translation has been updated by Sergey Matveev. (LP:#1708016) Bug fixes and other patches - Thanks to Jim Popovitch, certain failures in DNS lookups of DMARC policy will now result in mitigations being applied. (LP: #1722013) - The default DMARC reject reason now properly replaces %(listowner)s. (LP: #1718962) - The web roster page now shows case preserved email addresses. (LP: #1707447) - Changed the SETGID wrappers to only pass those items in the environment that are needed by the called scripts. (LP: #1705736) - Fixed MTA/Postfix.py to ensure that created aliases(.db) and virtual-mailman(.db) files are readable by Postfix and the .db files are owned by the Mailman user. (LP: #1696066) - Defended against certain web attacks that cause exceptions and "we hit a bug" responses when POST data or query fragments contain multiple values for the same parameter. (LP: #1695667) - The fix for LP: #1614841 caused a regression in the options CGI. This has been fixed. (LP: #1602608) - Added a -a option to the (e)grep commands in contrib/mmdsr to account for logs that may have non-ascii and be seen as binary. - Fixed the -V option to bin/list_lists to not show lists whose host is a subdomain of the given domain. (LP: #1695610) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: