From aditi.medha96 at gmail.com Sun Jul 2 16:04:40 2017 From: aditi.medha96 at gmail.com (ADITI GUPTA) Date: Mon, 3 Jul 2017 01:34:40 +0530 Subject: [Mailman-Developers] Error in configuring hyperkitty Message-ID: Hello, I tried to inject an email through command line to the archive queue but am unable to do so. I am using this command "mailman inject -q archive -f email.txt gsoc-demo at systers.org" . When I try to inject in "bad" queue, I am able to see the email in the bad queue folder. How can I see the email in the hyperkitty ui? I am getting this on injecting email. "mailman.interfaces.configuration.MissingConfigurationFileError: ~/Documents/dev/mailman/mailman-hyperkitty/mailman-hyperkitty.cfg " . I am trying this in mailman.cfg file : [archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration:~/Documents/dev/mailman/mailman-hyperkitty/mailman-hyperkitty.cfg I am not able to figure out that why it is not able to find the configuration file. Thank You, Aditi ? From mark at msapiro.net Sun Jul 2 16:13:10 2017 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 2 Jul 2017 13:13:10 -0700 Subject: [Mailman-Developers] Error in configuring hyperkitty In-Reply-To: References: Message-ID: <7b45e957-6dd3-6d8e-1ef4-1a0ca7ac8a7a@msapiro.net> On 7/2/17 1:04 PM, ADITI GUPTA wrote: > > I am getting this on injecting email. > "mailman.interfaces.configuration.MissingConfigurationFileError: > ~/Documents/dev/mailman/mailman-hyperkitty/mailman-hyperkitty.cfg " . > > I am trying this in mailman.cfg file : > [archiver.hyperkitty] > class: mailman_hyperkitty.Archiver > enable: yes > configuration:~/Documents/dev/mailman/mailman-hyperkitty/mailman-hyperkitty.cfg > > I am not able to figure out that why it is not able to find the > configuration file. Perhaps because its home directory is not yours. Try an absolute path without ~ -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From johny at neuromancer.sk Mon Jul 3 10:04:45 2017 From: johny at neuromancer.sk (Jan Jancar) Date: Mon, 3 Jul 2017 16:04:45 +0200 Subject: [Mailman-Developers] Pluggable workflows Message-ID: <66c3fe4f-531a-b4bf-c53d-e035c51113b1@neuromancer.sk> Hi all! As I note in one of my GSoC project updates, the PGP-enabled mailing lists require two new features from Mailman Core, both related to workflows. The first is to be able to inject steps into a subscription workflow to request the users pubkey before moderation checks. The second it to be able to require confirmation on all commands (to prevent replay attacks), although this could be avoided by storing the hashes from all the user sent signed commands. In this message I propose a way of refactoring the workflows into pluggable components. This post is also on my blog, for inline images I recommend going there: https://neuromancer.sk/article/10 Current workflows ----------------- Both of these requirements require changes to the way [Un]SubscriptionWorkflows and workflows in general work. Currently the subscription policy of a mailing list is determined by its `subscription_policy` attribute of the `SubscriptionPolicy` enum. That policy is enforced in the `SubscriptionWorkflow` state machine, which curently works with a queue of states, and looks like this: [See attached subscription_workflow.png] It is quite complex, and the `UnsubscriptionWorkflow` has much of the same code except the verification and moderation parts. Proposal -------- What I propose is to make the [Un]SubscriptionWorkflows, more generally workflows, pluggable components which would work on a stack state machine and be composed of mixins like so: [See attached subscription_workflow_split.png] This way a mailing list will have the name of its `SubscriptionWorkflow` component in the `subscription_policy` attribute. Which would mean for example an `SubscriptionPolicy.open` would be transformed into a class like so: @public @implementer(ISubscriptionWorkflow) class OpenSubscriptionPolicy(SubscriptionBase, VerificationMixin): """""" name = 'policy-open' description = 'An open subscription policy, only requires verification.' initial_state = 'prepare' def _step_prepare(self): self.push('do_subscription') self.push('verification_checks') self.push('sanity_checks') I believe this simplifies the policies and also allows chaining and composing of workflow steps, which would enable plugins to add custom subscription policies enforcing more steps/checks during the [un]subscription process. For example an organization-only mailing list could add a check that checks for the subscribing address in the organization database, therefore removing the need for manual moderation of subscriptions. This should also allow for requiring confirmation of all commands, by attaching a workflow instance to them. For the membership commands, this would be the corresponding [Un]SubscriptionWorkflows, and for other commands it would be currently disabled/empty workflow. Then a plugin could set the confirm workflow on commands it needs to. Comments? --------- I am currently working on this as a MR [1], although I am not sure if this is the right track to go on with workflows, so feedback/help is appreciated. I think something to split the monolithic [Un]SubscriptionWorkflows would be a nice thing in Mailman Core, as they share a lot of the same code and are overly complex for what they are, a linear series of checks with some skipping possible. [1]: https://gitlab.com/mailman/mailman/merge_requests/299 Cheers, -- Jan ______________________________________________________ /\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ # -------------- next part -------------- A non-text attachment was scrubbed... Name: subscription_workflow_split.png Type: image/png Size: 154466 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: subscription_workflow.png Type: image/png Size: 88638 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 862 bytes Desc: OpenPGP digital signature URL: From johny at neuromancer.sk Fri Jul 7 17:52:24 2017 From: johny at neuromancer.sk (Jan Jancar) Date: Fri, 7 Jul 2017 23:52:24 +0200 Subject: [Mailman-Developers] [GSoC] Encrypted mailing lists - update v8 In-Reply-To: References: Message-ID: Hi all! https://neuromancer.sk/article/11 Another update ============== Just a quick update about the state of the project before I disappear from civilization for this weekend. Pluggable workflows ------------------- https://gitlab.com/mailman/mailman/merge_requests/299 I have now finally got the pluggable workflows branch to test succesfully! With workflows being loaded dynamically, with workflow steps being saved completely, full backward compatibility, also database migrations and REST backward compatibility. That branch is still missing more tests for diffcov to pass. https://gitlab.com/J08nY/mailman/pipelines/9645615 ^^python 3.4 CI envs are currently broken, since `aiosmtpd 1.1` is python 3.5 only, so don't mind that. Key management -------------- https://gitlab.com/J08nY/mailman-pgp/tree/feature/workflows Building up on the pluggable workflows I have a base implementation of how the subscribers key gets to the mailing list and its moderator and also of how it's confirmed. SMTPS + STARTTLS ---------------- https://gitlab.com/mailman/mailman/merge_requests/286 During this week I also got to add SMTP over SSL/TLS support to `aiosmtpd`, a lib which Mailman uses for it's SMTP tests, so with that the SMTPS+STARTTLS MR has tests which pass and I think is ready for merging. Thanks Barry for the quick responses and help debugging over at `aiosmtpd`! Next up ------- For next week I plan on cleaning up the pluggable workflows branch, finishing the pre-subscription key management, implementing after-subscription key management as well as getting back on track with documenting the mailman-pgp architecture and changes at the pgpmailman-proposal repo, which has been lacking. Cheers, -- Jan ______________________________________________________ /\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ # -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 862 bytes Desc: OpenPGP digital signature URL: From galaxyjustice2 at gmail.com Tue Jul 11 20:50:59 2017 From: galaxyjustice2 at gmail.com (Erin Test) Date: Tue, 11 Jul 2017 17:50:59 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API Message-ID: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> We are currently using Mailman 2 for a number of mailing lists on our website (PHP based). We have mailing lists but need to have these people grouped in the same groups on our website and don't want to have to maintain 2 lists of the same people. I'd like to move us to Mailman 3 and take advantage of the REST API and write my own PHP code in interact with it. All of that being said I'm missing something in getting my basic setup finalized. I have a web server (Apache) running and have installed mailman core. When I try to access the REST API I get some variation of connection refused depending on what I'm using (python command line, curl, etc). I'm feeling like apache doesn't know where to point port 8001, and I don't know what to tell it. For the previous version of mailman you had to put and Alias and a Script Alias into the apache config. I've seen some documentation about what to put in for Postorius, but haven't found anything for just mailman. Any thoughts and/or pointers would be greatly appreciated. -Erin From mark at msapiro.net Tue Jul 11 23:48:58 2017 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 11 Jul 2017 20:48:58 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> Message-ID: <5206e820-d38a-6e87-c1d4-fd6b376b5d17@msapiro.net> On 07/11/2017 05:50 PM, Erin Test wrote: > > All of that being said I'm missing something in getting my basic setup finalized. I have a web server (Apache) running and have installed mailman core. When I try to access the REST API I get some variation of connection refused depending on what I'm using (python command line, curl, etc). I'm feeling like apache doesn't know where to point port 8001, and I don't know what to tell it. You really can't make use of the REST API via a web browser or through a normal web server like Apache. Web browsers basically do HTTP GET and can submit forms via HTTP POST. The REST API requires other HTTP methods for full functionality and returns responses that are not particularly human friendly and are definitely not HTML. You should be able to access the core REST API via curl. See for more information. > For the previous version of mailman you had to put and Alias and a Script Alias into the apache config. I've seen some documentation about what to put in for Postorius, but haven't found anything for just mailman. The rest API in Mailman core is not a web UI. That's what Postorius is for. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at list.org Wed Jul 12 12:16:43 2017 From: barry at list.org (Barry Warsaw) Date: Wed, 12 Jul 2017 12:16:43 -0400 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> Message-ID: <20170712121643.549aff28@presto.wooz.org> On Jul 11, 2017, at 05:50 PM, Erin Test wrote: >All of that being said I'm missing something in getting my basic setup >finalized. I have a web server (Apache) running and have installed mailman >core. When I try to access the REST API I get some variation of connection >refused depending on what I'm using (python command line, curl, etc). I'm >feeling like apache doesn't know where to point port 8001, and I don't know >what to tell it. To supplement what Mark said, there are a few things to keep in mind with the REST API. First, you really don't need to put Apache in front of it, as Mailman Core already provides an HTTP server for the REST API [*]. Second, you definitely do *not* want to expose the REST API on the public internet. It's an administrative API so it has full access to you list server. It only provides very basic authentication and no user based authorization, so any process with the basic auth keys can do anything. We have a *very* nascent project called Lemme which is geared toward a public facing authenticating REST proxy. While we have ideas for how it would work, we haven't had many resources available to work on it. Contributions of course would be greatly welcomed! https://gitlab.com/mailman/lemme/blob/master/README.rst Cheers, -Barry [*] By default we use the stdlib simple wsgi server, which is convenient but makes no claims of being very performant. Hopefully for 3.2 I will switch to aiohttp, uvloop, or some other better performing substrate. From galaxyjustice2 at gmail.com Wed Jul 12 14:57:52 2017 From: galaxyjustice2 at gmail.com (Erin Test) Date: Wed, 12 Jul 2017 11:57:52 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: <20170712121643.549aff28@presto.wooz.org> References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> Message-ID: <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com> Hi Barry and Mark Thanks for your responses. I have no desire to open the REST API to the public, I'm just trying to access it at all and assumed I needed to provided a web server, but that's apparently not the case. Other than mailman start is there something I'm missing that is needed to start the Mailman included web server? To access the REST API I've tried both command line curl calls: curl -I -u restadmin http://localhost:8001/3.1/system/versions Enter host password for user 'restadmin': curl: (7) Failed to connect to localhost port 8001: Connection refused And calls from within python: >>> import requests >>> response = requests.get( ... 'http://localhost:8001/3.1/system/versions', ... auth=('restadmin', 'restpass')) Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection raise err File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 61] Connection refused During handling of the above exception, another exception occurred: ... more errors presented after this but I'm assuming they relate to the Connection refused. Thanks, -Erin From mark at msapiro.net Wed Jul 12 15:04:30 2017 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 12 Jul 2017 12:04:30 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com> References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com> Message-ID: On 07/12/2017 11:57 AM, Erin Test wrote: > > To access the REST API I've tried both command line curl calls: > curl -I -u restadmin http://localhost:8001/3.1/system/versions > Enter host password for user 'restadmin': > curl: (7) Failed to connect to localhost port 8001: Connection refused Is something else listening on port 8001 (like maybe Apache). Do you have iptables or some other firewall and if so have you opened port 8001. What does sudo netstat -l -n -t -p say? Is something listening on 8001 and if so, what? -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at list.org Wed Jul 12 15:10:23 2017 From: barry at list.org (Barry Warsaw) Date: Wed, 12 Jul 2017 15:10:23 -0400 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com> References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com> Message-ID: <20170712151024.413ad607@presto.wooz.org> Hi Erin, On Jul 12, 2017, at 11:57 AM, Erin Test wrote: >To access the REST API I've tried both command line curl calls: >curl -I -u restadmin http://localhost:8001/3.1/system/versions >Enter host password for user 'restadmin': >curl: (7) Failed to connect to localhost port 8001: Connection refused What does `mailman info` say? That will print the hostname:port that the REST server is listening on, and the basic auth credentials in use. All of those are of course configurable in your mailman.cfg file. If your curl arguments match what `mailman info` says, then there must be something else going on. Maybe there's a firewall issue? You can also check the Mailman logs to see if you're getting some exception or other in the server, and of course you can also adjust the logging level in the mailman.cfg file. Cheers, -Barry From galaxyjustice2 at gmail.com Wed Jul 12 16:11:15 2017 From: galaxyjustice2 at gmail.com (Erin Test) Date: Wed, 12 Jul 2017 13:11:15 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com> Message-ID: I'm not aware of anything else using that port. sudo netstat -l -n -t -p 8001 8001: unknown or uninstrumented protocol I'm on a mac and I read the options for netstat can be different between OSX/BSD and Linux, so I verified using lsof (8889 is the port mysql is using) leaves:~ ejustice$ lsof -n -i4TCP:8001 | grep LISTEN leaves:~ ejustice$ lsof -n -i4TCP:8889 | grep LISTEN mysqld 84142 ejustice 17u IPv6 0x7164e83ec7901d4b 0t0 TCP *:ddi-tcp-2 (LISTEN) Also I've tried changing the default port in /etc/mailman.cfg (from 8001 to 9001 and a couple other randomly chosen ports) and I get the same Connection Refused. When I try to use the port mysql is using 8889 I get sudo curl -I -u restadmin http://localhost:8889/3.1/system/versions Enter host password for user 'restadmin': curl: (8) Weird server reply So I don't think it's a conflict with a port that is in use. -Erin > On Jul 12, 2017, at 12:04 PM, Mark Sapiro wrote: > > On 07/12/2017 11:57 AM, Erin Test wrote: >> >> To access the REST API I've tried both command line curl calls: >> curl -I -u restadmin http://localhost:8001/3.1/system/versions >> Enter host password for user 'restadmin': >> curl: (7) Failed to connect to localhost port 8001: Connection refused > > > Is something else listening on port 8001 (like maybe Apache). Do you > have iptables or some other firewall and if so have you opened port 8001. > > What does > > sudo netstat -l -n -t -p > > say? Is something listening on 8001 and if so, what? From galaxyjustice2 at gmail.com Wed Jul 12 16:27:31 2017 From: galaxyjustice2 at gmail.com (Erin Test) Date: Wed, 12 Jul 2017 13:27:31 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: <20170712151024.413ad607@presto.wooz.org> References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com> <20170712151024.413ad607@presto.wooz.org> Message-ID: Everything matches between mailman info and what I'm putting into the curl call. Also mailman info & mailman conf are showing changes when I make them so I know /etc/mailman.cfg file is being read. As to the logs I've got the logging level set at debug for all the logging categories but I'm not seeing anything in the log files. -Erin > On Jul 12, 2017, at 12:10 PM, Barry Warsaw wrote: > > Hi Erin, > > On Jul 12, 2017, at 11:57 AM, Erin Test wrote: > >> To access the REST API I've tried both command line curl calls: >> curl -I -u restadmin http://localhost:8001/3.1/system/versions >> Enter host password for user 'restadmin': >> curl: (7) Failed to connect to localhost port 8001: Connection refused > > What does `mailman info` say? That will print the hostname:port that the REST > server is listening on, and the basic auth credentials in use. All of those > are of course configurable in your mailman.cfg file. > > If your curl arguments match what `mailman info` says, then there must be > something else going on. Maybe there's a firewall issue? You can also check > the Mailman logs to see if you're getting some exception or other in the > server, and of course you can also adjust the logging level in the mailman.cfg > file. > > Cheers, > -Barry > From mark at msapiro.net Wed Jul 12 16:34:15 2017 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 12 Jul 2017 13:34:15 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com>

Message-ID: <27fb1846-ed12-9052-6ddd-af98568c2a68@msapiro.net> On 07/12/2017 01:11 PM, Erin Test wrote: > I'm not aware of anything else using that port. > > sudo netstat -l -n -t -p 8001 > 8001: unknown or uninstrumented protocol sudo netstat -l -n -t -p no 8001. -p means list the names of the processes listining. The output will be a list of all the processes listening on TCP ports. Look for port 8001 in that list. If it's not there, nothing is listening on 8001. If it is, the name of the process will be listed. Here's a sample output: > $ sudo netstat -l -n -t -p > Active Internet connections (only servers) > Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name > tcp 0 0 192.168.5.13:8301 0.0.0.0:* LISTEN 1465/consul > tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 24794/nginx > tcp 0 0 127.0.0.1:8400 0.0.0.0:* LISTEN 1465/consul > tcp 0 0 127.0.0.1:8500 0.0.0.0:* LISTEN 1465/consul > tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1508/sshd > tcp 0 0 127.0.0.1:8024 0.0.0.0:* LISTEN 1220/python3 > tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1796/postgres > tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN 1465/consul > tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 1814/master > tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 24794/nginx > tcp 0 0 127.0.0.1:8891 0.0.0.0:* LISTEN 1711/opendkim > tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 24368/python > tcp 0 0 127.0.0.1:8001 0.0.0.0:* LISTEN 1221/python3 This is mailman listening on port 8001 > tcp6 0 0 :::22 :::* LISTEN 1508/sshd > tcp6 0 0 :::25 :::* LISTEN 1814/master > > I'm on a mac and I read the options for netstat can be different between OSX/BSD and Linux, so I verified using lsof (8889 is the port mysql is using) > > leaves:~ ejustice$ lsof -n -i4TCP:8001 | grep LISTEN So maybe nothing is listening on 8001. > leaves:~ ejustice$ lsof -n -i4TCP:8889 | grep LISTEN > mysqld 84142 ejustice 17u IPv6 0x7164e83ec7901d4b 0t0 TCP *:ddi-tcp-2 (LISTEN) > > Also I've tried changing the default port in /etc/mailman.cfg (from 8001 to 9001 and a couple other randomly chosen ports) and I get the same Connection Refused. When I try to use the port mysql is using 8889 I get > > sudo curl -I -u restadmin http://localhost:8889/3.1/system/versions > Enter host password for user 'restadmin': > curl: (8) Weird server reply > > So I don't think it's a conflict with a port that is in use. The conflict would be if some other process was listening on 8001, Mailman wouldn't be able to. Have you started mailman (mailman start)? -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From galaxyjustice2 at gmail.com Wed Jul 12 16:42:47 2017 From: galaxyjustice2 at gmail.com (Erin Test) Date: Wed, 12 Jul 2017 13:42:47 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: <27fb1846-ed12-9052-6ddd-af98568c2a68@msapiro.net> References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com>

<27fb1846-ed12-9052-6ddd-af98568c2a68@msapiro.net> Message-ID: > On Jul 12, 2017, at 1:34 PM, Mark Sapiro wrote: > > Have you started mailman (mailman start)? I thought I did, but when I tried to stop it just now it said it wasn't running. And when I try it now I'm seeing sudo mailman start Password: Starting Mailman's master runner /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python: can't open file '/sbin/master': [Errno 2] No such file or directory Thanks, -Erin From mark at msapiro.net Wed Jul 12 18:49:17 2017 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 12 Jul 2017 15:49:17 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com>

<27fb1846-ed12-9052-6ddd-af98568c2a68@msapiro.net> Message-ID: <30ff24d6-ae39-dc6c-7492-3377b283b282@msapiro.net> On 07/12/2017 01:42 PM, Erin Test wrote: > > I thought I did, but when I tried to stop it just now it said it wasn't running. And when I try it now I'm seeing > sudo mailman start > Password: > Starting Mailman's master runner > /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python: can't open file '/sbin/master': [Errno 2] No such file or directory How did you install Mailman? You might find this thread interesting . -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From galaxyjustice2 at gmail.com Wed Jul 12 19:39:36 2017 From: galaxyjustice2 at gmail.com (Erin Test) Date: Wed, 12 Jul 2017 16:39:36 -0700 Subject: [Mailman-Developers] Telling Apache about Mailman Rest API In-Reply-To: <30ff24d6-ae39-dc6c-7492-3377b283b282@msapiro.net> References: <61DE3F75-A6BB-443F-BB37-B101BF80848B@gmail.com> <20170712121643.549aff28@presto.wooz.org> <40FBC603-994C-48E5-AC10-F1B79773C7FD@gmail.com>

<27fb1846-ed12-9052-6ddd-af98568c2a68@msapiro.net> <30ff24d6-ae39-dc6c-7492-3377b283b282@msapiro.net> Message-ID: <82717BD6-4C49-4B84-ACF8-DD7BE7CCE239@gmail.com> > On Jul 12, 2017, at 3:49 PM, Mark Sapiro wrote: > You might find this thread interesting > >. That link pointed me in the right direction. It was the following change I'd added to the mailman.cfg. [mailman] # Which paths.* filesystem layout to use. layout: fhs while following the instructions here: http://docs.mailman3.org/en/latest/prodsetup.html?highlight=apache#setting-up-mailman-core Once I commented that line out I'm able to follow the examples https://mailman.readthedocs.io/en/latest/src/mailman/rest/docs/basic.html without issue. Thank you Mark and Barry! -Erin From johny at neuromancer.sk Fri Jul 14 18:18:51 2017 From: johny at neuromancer.sk (Jan Jancar) Date: Sat, 15 Jul 2017 00:18:51 +0200 Subject: [Mailman-Developers] [GSoC] Encrypted mailing lists In-Reply-To: References: Message-ID: <0984f48f-4c12-fbfd-7c23-2988d8b72c54@neuromancer.sk> Hi all! Checking in with progress around the halfway point of the GSoC 2017 timeline. https://neuromancer.sk/article/12 ====================== The point of no return ====================== This week is around the halfway of GSoC 2017 project timeline, hence the title. Key management ============== As I set out to do, I have now implemented most of the after subscription key management. The `key change` command. The pre subscription key management is done via a custom dynamically loaded subscription policy/workflow. This is possible with my `pluggable-workflows` branch/MR which introduces dynamically loaded workflows, which subscription and unsubscription policies are a part of. Subscription ------------ On subscription the PGP enabled mailing list needs a users public key. An example conversation establishing that follows (shows a list with moderation + pgp enabled subscription policy): Mailman User ------- ---- list at example.com anne at example.com <- subscribe 1. 2. key set "token1" -> + instructions to reply and attach user key (signed by list) <- key set "token1" 3. (key attached as per RFC3156(PGP/MIME) or inline and we do our best to find it.) 4. key confirm "token2" -> + statement containing "token2" and user key fingerprint in body, with instructions to sign it (signed by list, encrypted to user key) <- key confirm "token2" 5. + must contain signed statement from step 4 (required to be signed by user key) 6. ... the rest of the subscription steps (moderation) ... Steps 3 and 5 can be optionally encrypted to the list key. This workflow gives "some" assurance that the user really has the key she is subscribing with, however the trust in the address/user/key has to come from the list moderator and both my proposal and implementation rely on that. This workflow only proves that Mailman is communicating with someone that has the user key (can decrypt `"token2"` and sign the `statement`), not that that user is `anne at example.com`. Key change ---------- Works similar to the `key set` command on subscription. Works via a custom workflow, but not a subscription workflow/policy, just a workflow. Mailman User ------- ---- list at example.com anne at example.com <- key change 1. (new! key attached as per RFC3156(PGP/MIME) or inline and we do our best to find it.) 2. key confirm "token1" -> + statement containing "token1" and new! user key fingerprint in body, with instructions to sign it (signed by list, encrypted to new! user key) <- key confirm "token1" 3. + must contain signed statement from step 2. (required to be signed by old! user key) Steps 1 and 3 can again be optionally encrypted. This ensures that only someone with possesion of the old user key can change it. It requires the user to decrypt with the new key and sign a statement with the new key fingerprint as well as the token in it with the old key. PyPi package ============ [mailman-pgp @ PyPi](https://pypi.python.org/pypi/mailman-pgp/0.1) I have now created a PyPi package for the mailman-pgp plugin as it's maturing, and while still unusable I wanted the name. Test coverage ============= I got test coverage of mailman-pgp to 95% which now actually means something as large parts of the codebase are implemented and not just stubs. Since this plugin requires features not merged into Mailman Core (or PGPy) for pipelines to work I maintain forks of both with branches that merge all required branches into a `plugin` branch for Mailman Core and `dev` branch for PGPy. Next up ======= Outgoing -------- I had a look at working in outgoing encryption of messages for pgp enabled mailing lists. Encryption cannot be done as a handler, as it requires to process the messages per-recipient or per-batch of recipients. The only configurable/dynamic option currently in Mailman Core is the `[mta].outgoing` callable. Which is not ideal since it itself has more functionality than simply dispatching to a correct `Delivery` instance based on the list configuration, that would have to be duplicated in mailman-pgp. The `IMailTransportAgentDelivery` interface looks promising though, so I'm thinking about refactoring the outgoing callable a bit to allow setting a custom one without the duplicities. Signature hash tracking ----------------------- To prevent replay attacks for things where always sending and requiring a signed confirmation token in response is not practically doable, the pgp plugin will keep signature hashes with timestamps and posssibly key fingerprints in a DB/some other quickly searchable structure, and will check signatures it receives against the hashes in that structure. It will then act accordingly, for example not allowing a user to post the same message (with the same signature) again, since it could have been replayed. Documentation ------------- I wanted to get the pgpmailman-proposal repository updated with current state/MRs but didn't get to it this week, so might find time for it over the weekend as well as for some code documentation. Cheers, -- Jan ______________________________________________________ /\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ # -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 862 bytes Desc: OpenPGP digital signature URL: From johny at neuromancer.sk Thu Jul 20 13:39:04 2017 From: johny at neuromancer.sk (Jan Jancar) Date: Thu, 20 Jul 2017 19:39:04 +0200 Subject: [Mailman-Developers] [GSoC] Encrypted mailing lists - update v10 In-Reply-To: References: Message-ID: Hi all! Another week, another project update. https://neuromancer.sk/article/13 ==================== Post title goes here ==================== Signature hash tracking ----------------------- It would be relatively easy to replay a signed message to a mailing list by a user as no kind of challenge-response is done on posting. While signature replay checking is usually done on the end users point with against his keyring and messages he has so far received and their context, I think it is kind of expected of PGP enabled Mailman to also do this as it relays the messages. On a successful posting to a PGP enabled mailing list (`AcceptedEvent` to be precise), the message is searched for PGP signatures and their digests and key fingerprints are added into the `sighash` table. The signature rule then checks the digests from a posting against those in the `sighash` table. If it finds it has previously accepted a message with any of those hashes it takes the `duplicate_sig_action` which is per-list configurable. This of course means that if the `duplicate_sig_action` is not set to `Action.defer`, which means the message gets rejected or dropped, that a signature that was sent to a list cannot be sent again. Of course if a user wants to send a signed message again, he can just resign it and send it again, the hashes won't match. However sometimes it might be useful to post the message as signed originally, for example to prove something. However, I think it is worth it to keep this as a configurable option. Maybe with another option disabling the collection of signature hashes, for performance reasons. Outgoing processing ------------------- Since a the plugin needs to process the messages for outgoing encryption on a per-recipient or per-chunk basis, it couldn't be implemented as a Handler, I thought about implementing it as a custom OutgoingRunner but that didn't work out either. However the `IMailTransportAgentDelivery` interface is great for what the plugin needs to do. So there are two custom PGP enabled delivery classes that get selected by a custom `[mta] outgoing` callable, one for bulk delivery and other for individual delivery. Documentation ------------- I got the `pgpmailman-proposal` repository almost completely up to date on changes to the plugin and my current proposed Core/Client/Postorius/HyperKitty changes and MRs. Next up ======= More testing ------------ As I remembered to ignore coverage of tests it dropped to 88% and I think it is currently in order to add more comprehensive tests for features implemented that check the edge cases which currently remain. Setup live development instance ------------------------------- I believe I got pretty far into the development without having a proper live development instance of Mailman Core + plugins and Postorius + HyperKitty + Client, so I'm going to set that up now and test manually. With time I might set this up as a public site/mailing list server, to demonstrate the features of the PGP plugin. Archivers --------- Currently thinking about implementing two encrypted archivers, one local, one remote. The local one would be similar to the prototype archiver but store messages encrypted, TBD how. The remote one will send the messages encrypted to a receiving archiver, the django-pgpmailman instance running next to HyperKitty. django-pgpmailman ----------------- With most of the essential stuff in the core plugin done, the web ui part of development can begin. Cheers, -- Jan ______________________________________________________ /\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ # -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 862 bytes Desc: OpenPGP digital signature URL: From barry at list.org Thu Jul 20 16:56:24 2017 From: barry at list.org (Barry Warsaw) Date: Thu, 20 Jul 2017 15:56:24 -0500 Subject: [Mailman-Developers] Click CLI branch Message-ID: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> Just a quick note to mention that my big branch to adopt click for command line option parsing should now be done. https://gitlab.com/mailman/mailman/merge_requests/292 This will close #319 and #346 and make adding new `mailman` subcommands much easier. (We still need the bits to define additional search paths, and probably some better documentation that would be part of a general ?Extending Mailman? section.) Along the way I think I?ve made several other improvements, including (I hope!) reducing or eliminating the occasional hangs we see on CI, speeding up the test suite a bit, and making things more robust. Please feel free to review it and play with it. It?s finishing CI now but I?ll hold off on merging it for a day or two. I?m especially interested to hear what Jan thinks for the plugin work he?s doing. The big downside could be that because this is such a big change, existing MPs may have to be rebased. It?s a big branch with lots of little sweater threads that took longer than I expected, but I should be done now, and I think it will be a good improvement to the code. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mark at msapiro.net Thu Jul 20 17:12:24 2017 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 20 Jul 2017 14:12:24 -0700 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> Message-ID: <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> On 07/20/2017 01:56 PM, Barry Warsaw wrote: > Just a quick note to mention that my big branch to adopt click for command line option parsing should now be done. > > https://gitlab.com/mailman/mailman/merge_requests/292 ... > Please feel free to review it and play with it. It?s finishing CI now but I?ll hold off on merging it for a day or two. I?m especially interested to hear what Jan thinks for the plugin work he?s doing. > > The big downside could be that because this is such a big change, existing MPs may have to be rebased. Thank Barry, I'll look at it. I'll also rebase !301 and !202 after it lands, and I'll now delete my branch as this supercedes that. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From mark at msapiro.net Thu Jul 20 17:26:57 2017 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 20 Jul 2017 14:26:57 -0700 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> Message-ID: <7f28dfe9-6681-70bf-7107-82cd5a60a0f8@msapiro.net> On 07/20/2017 02:12 PM, Mark Sapiro wrote: > On 07/20/2017 01:56 PM, Barry Warsaw wrote: >> Just a quick note to mention that my big branch to adopt click for command line option parsing should now be done. >> >> https://gitlab.com/mailman/mailman/merge_requests/292 > > I'll look at it. ... The first thing I notice right away is the help text doesn't fill. E.g., > $ ./mailman --help > Usage: mailman [OPTIONS] COMMAND [ARGS]... > > The GNU Mailman mailing list management system Copyright 1998-2017 by the > Free Software Foundation, Inc. http://www.list.org > > Options: > -C, --config PATH Configuration file to use. If not given, the environment > variable > MAILMAN_CONFIG_FILE is consulted and used if > set. If neither are given, a > default configuration file > is loaded. > --version Show the version and exit. > --help Show this message and exit. > > Commands: > ... and > $ ./mailman shell --help > Usage: mailman shell [OPTIONS] [RUN_ARGS]... > > Operate on a mailing list. > > For detailed help, see --details > > Options: > -i, --interactive Leaves you at an interactive prompt after all other > processing is complete. > This is the default unless the > --run option is given. > -r, --run TEXT Run a script on a mailing list. The argument is the > module path to a > callable. This callable will be > imported and then called with the mailing > list as the > first argument. If additional arguments are given at > the end > of the command line, they are passed as > subsequent positional arguments to > the callable. For > additional help, see --details. > --details Print detailed instructions and exit. > -l, --listspec TEXT A specification of the mailing list to operate on. > This may be the posting > address of the list, or its > List-ID. The argument can also be a Python > regular > expression, in which case it is matched against both > the posting > address and List-ID of all mailing lists. > To use a regular expression, > LISTSPEC must start with a > ^ (and the matching is done with re.match(). > LISTSPEC > cannot be a regular expression unless --run is given. > --help Show this message and exit. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From johny at neuromancer.sk Thu Jul 20 17:33:25 2017 From: johny at neuromancer.sk (Jan Jancar) Date: Thu, 20 Jul 2017 23:33:25 +0200 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> Message-ID: On 07/20/2017 10:56 PM, Barry Warsaw wrote: > Just a quick note to mention that my big branch to adopt click for command line option parsing should now be done. > > https://gitlab.com/mailman/mailman/merge_requests/292 > > This will close #319 and #346 and make adding new `mailman` subcommands much easier. (We still need the bits to define additional search paths, and probably some better documentation that would be part of a general ?Extending Mailman? section.) > > Along the way I think I?ve made several other improvements, including (I hope!) reducing or eliminating the occasional hangs we see on CI, speeding up the test suite a bit, and making things more robust. > > Please feel free to review it and play with it. It?s finishing CI now but I?ll hold off on merging it for a day or two. I?m especially interested to hear what Jan thinks for the plugin work he?s doing. > > The big downside could be that because this is such a big change, existing MPs may have to be rebased. > > It?s a big branch with lots of little sweater threads that took longer than I expected, but I should be done now, and I think it will be a good improvement to the code. > > Cheers, > -Barry > > Nice stuff! Looking at the dynamic loading of subcommands, plugin subcommands can be easily integrated. Pretty much a one line change on top of this and !288: - add_components('mailman.commands', ICLISubCommand, self._commands) + add_pluggable_components('commands', ICLISubCommand, self._commands) Will rebase the plugin branch on top of this once it lands. Cheers, -- Jan ______________________________________________________ /\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ # -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 862 bytes Desc: OpenPGP digital signature URL: From mark at msapiro.net Thu Jul 20 18:11:43 2017 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 20 Jul 2017 15:11:43 -0700 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: <7f28dfe9-6681-70bf-7107-82cd5a60a0f8@msapiro.net> References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> <7f28dfe9-6681-70bf-7107-82cd5a60a0f8@msapiro.net> Message-ID: On 07/20/2017 02:26 PM, Mark Sapiro wrote: > > The first thing I notice right away is the help text doesn't fill. E.g., > >> $ ./mailman --help >> Usage: mailman [OPTIONS] COMMAND [ARGS]... >> >> The GNU Mailman mailing list management system Copyright 1998-2017 by the >> Free Software Foundation, Inc. http://www.list.org >> >> Options: >> -C, --config PATH Configuration file to use. If not given, the environment >> variable >> MAILMAN_CONFIG_FILE is consulted and used if >> set. If neither are given, a >> default configuration file >> is loaded. >> --version Show the version and exit. >> --help Show this message and exit. ... I've looked at the click docs and I see that text from docstrings is wrapped and filled as is the text > """\ > The GNU Mailman mailing list management system > Copyright 1998-2017 by the Free Software Foundation, Inc. > http://www.list.org > """ which wraps and fills as above, nd there is even a control to turn that off paragraph by paragraph, but the 'help=' text for an option doesn't seem to be handled that way. A possible workaround is to replace things like > help=_("""\ > Configuration file to use. If not given, the environment variable > MAILMAN_CONFIG_FILE is consulted and used if set. If neither are given, a > default configuration file is loaded.""")) with > help=_( > 'Configuration file to use. If not given, the environment variable ' > 'MAILMAN_CONFIG_FILE is consulted and used if set. If neither are given, ' > 'a default configuration file is loaded.')) This seems to work to produce > Usage: mailman [OPTIONS] COMMAND [ARGS]... > > The GNU Mailman mailing list management system Copyright 1998-2017 by the > Free Software Foundation, Inc. http://www.list.org > > Options: > -C, --config PATH Configuration file to use. If not given, the environment > variable MAILMAN_CONFIG_FILE is consulted and used if > set. If neither are given, a default configuration file > is loaded. > --version Show the version and exit. > --help Show this message and exit. > -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From mark at msapiro.net Thu Jul 20 20:22:10 2017 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 20 Jul 2017 17:22:10 -0700 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> <7f28dfe9-6681-70bf-7107-82cd5a60a0f8@msapiro.net> Message-ID: On 07/20/2017 03:11 PM, Mark Sapiro wrote: One more thing. '-h' doesn't invoke help. See for how to enable this. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From barry at list.org Thu Jul 20 23:25:46 2017 From: barry at list.org (Barry Warsaw) Date: Thu, 20 Jul 2017 23:25:46 -0400 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> Message-ID: <20170720232546.0826df28@presto.wooz.org> On Jul 20, 2017, at 11:33 PM, Jan Jancar wrote: >Looking at the dynamic loading of subcommands, plugin subcommands can be >easily integrated. Pretty much a one line change on top of this and !288: > > - add_components('mailman.commands', ICLISubCommand, self._commands) > + add_pluggable_components('commands', ICLISubCommand, self._commands) > >Will rebase the plugin branch on top of this once it lands. Very cool. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From barry at list.org Thu Jul 20 23:26:26 2017 From: barry at list.org (Barry Warsaw) Date: Thu, 20 Jul 2017 23:26:26 -0400 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> Message-ID: <20170720232626.28db61fe@presto.wooz.org> On Jul 20, 2017, at 02:12 PM, Mark Sapiro wrote: >I'll look at it. I'll also rebase !301 and !202 after it lands, and I'll >now delete my branch >as this supercedes that. Thanks Mark! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From barry at list.org Fri Jul 21 10:23:43 2017 From: barry at list.org (Barry Warsaw) Date: Fri, 21 Jul 2017 10:23:43 -0400 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> <7f28dfe9-6681-70bf-7107-82cd5a60a0f8@msapiro.net> Message-ID: <20170721102343.5f04e08c@presto> On Jul 20, 2017, at 02:26 PM, Mark Sapiro wrote: >The first thing I notice right away is the help text doesn't fill. E.g., Now that you remind me, I noticed this too and I was going to file an upstream issue on this, but I forgot. I did some pdb tracing through the click source and figured out the problem. https://github.com/pallets/click/issues/834 TL;DR: Your workaround provided the essential clue. The main difference between the original code and your workaround is that the latter omits the embedded newlines. Click apparently dedents the text but doesn't remove the newlines and that confuses click's formatting code. In the meantime, I have a workaround that I'm in the process of implementing. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From barry at list.org Fri Jul 21 11:23:43 2017 From: barry at list.org (Barry Warsaw) Date: Fri, 21 Jul 2017 11:23:43 -0400 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> <9c6a773d-c1f7-0e86-928c-3edd3d12fd1f@msapiro.net> <7f28dfe9-6681-70bf-7107-82cd5a60a0f8@msapiro.net>

Message-ID: <20170721112343.1fa78da8@presto> On Jul 20, 2017, at 05:22 PM, Mark Sapiro wrote: >On 07/20/2017 03:11 PM, Mark Sapiro wrote: > >One more thing. '-h' doesn't invoke help. See > >for how to enable this. Thanks, fixed! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From mark at msapiro.net Fri Jul 21 12:46:24 2017 From: mark at msapiro.net (Mark Sapiro) Date: Fri, 21 Jul 2017 09:46:24 -0700 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> Message-ID: <7d4cc33d-39d6-7c60-365b-c3e8f55c1b5b@msapiro.net> Another thing I noticed is the help for the withlist --run option says in part: If additional arguments are given at the end of the command line, they are passed as subsequent positional arguments to the callable. For additional help, see --details. The additional arguments are not actually passed as subsequent positional arguments to the callable. They are passed as a single positional argument which is a tuple of the additional arguments. --details is correct in its example showing def change(mlist, args): mlist.display_name = args[0] but the --run description makes me think it should be def change(mlist, name): mlist.display_name = name -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From barry at list.org Fri Jul 21 15:08:34 2017 From: barry at list.org (Barry Warsaw) Date: Fri, 21 Jul 2017 14:08:34 -0500 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: <7d4cc33d-39d6-7c60-365b-c3e8f55c1b5b@msapiro.net> References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> <7d4cc33d-39d6-7c60-365b-c3e8f55c1b5b@msapiro.net> Message-ID: <87D9E33F-6934-4367-9DB3-8ADDA0903776@list.org> On Jul 21, 2017, at 11:46, Mark Sapiro wrote: > > Another thing I noticed is the help for the withlist --run option says > in part: > > If additional arguments are given at the end of the command line, they > are passed as subsequent positional arguments to the callable. For > additional help, see --details. > > The additional arguments are not actually passed as subsequent > positional arguments to the callable. They are passed as a single > positional argument which is a tuple of the additional arguments. > > --details is correct in its example showing > > def change(mlist, args): > mlist.display_name = args[0] > > but the --run description makes me think it should be > > def change(mlist, name): > mlist.display_name = name Thanks Mark. I?d like to preserve the API of Mailman 2.1, so I?m changing that back to passing them in as positional arguments (i.e. to match the ?run description). Just testing that change locally now. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From barry at python.org Fri Jul 21 23:03:48 2017 From: barry at python.org (Barry Warsaw) Date: Fri, 21 Jul 2017 23:03:48 -0400 Subject: [Mailman-Developers] Click CLI branch References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> Message-ID: <20170721230348.5e7e6955@presto> On Jul 20, 2017, at 03:56 PM, Barry Warsaw wrote: >This will close #319 and #346 and make adding new `mailman` subcommands much >easier. This is now merged! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From mark at msapiro.net Fri Jul 21 23:53:46 2017 From: mark at msapiro.net (Mark Sapiro) Date: Fri, 21 Jul 2017 20:53:46 -0700 Subject: [Mailman-Developers] Click CLI branch In-Reply-To: <20170721230348.5e7e6955@presto> References: <847BBC05-664B-42D8-B838-7E59B817D1C5@list.org> <20170721230348.5e7e6955@presto> Message-ID: <4b343cc8-c18e-2141-a323-18439a7482be@msapiro.net> On 07/21/2017 08:03 PM, Barry Warsaw wrote: > On Jul 20, 2017, at 03:56 PM, Barry Warsaw wrote: > >> This will close #319 and #346 and make adding new `mailman` subcommands much >> easier. > > This is now merged! And MRs !202 and !301 have been rebased and CI is running. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From vbnlohani at gmail.com Tue Jul 25 10:15:36 2017 From: vbnlohani at gmail.com (Vaibhav Lohani) Date: Tue, 25 Jul 2017 19:45:36 +0530 Subject: [Mailman-Developers] Discourse Integration In-Reply-To: <22717.13813.915523.526404@turnbull.sk.tsukuba.ac.jp> References: <22717.13813.915523.526404@turnbull.sk.tsukuba.ac.jp> Message-ID: Stephen J. Turnbull wrote: > I don't know of anyone who is currently working on it, nor do we have > specific plans at the moment. Is there anyone interested to work on this as of now ? I was thinking to start working on this integration. If anyone is interested and we can integrate it together, it would be great. > What sort of integration do you have in mind? The idea is to have a two way binding sort of integration, where the conversations can be carried out in discourse as well as mailing list mode. Both python and discourse have their rest apis. Discourse has its front end for the api in ember.js ( A javascript framework ) that works in the browser. So, we can have a delivery queue for discourse like we have other delivery queues and a processor for that queue that hits the discourse api. Also, we will have to tweak a little bit on the ember front end so that it hits the mailman rest api every time a new message is sent from discourse, or else, if the mailman rest api should not be opened publicly, then we will have to tweak on the server side at discourse api, as both mailman and discourse will be hosted on the same server. Thanks and Regards, Vaibhav Lohani From johny at neuromancer.sk Tue Jul 25 20:12:16 2017 From: johny at neuromancer.sk (Jan Jancar) Date: Wed, 26 Jul 2017 02:12:16 +0200 Subject: [Mailman-Developers] [GSoC] Encrypted mailing lists - second evaluation In-Reply-To: References: Message-ID: Hi all! https://neuromancer.sk/article/14 ================= Second evaluation ================= The second evaluation period came quite fast after the first one, nonetheless the project advanced much further so quick recap of its current state is in order. Since first evaluation ====================== SMTPS + STARTTLS ---------------- [MR 286](https://gitlab.com/mailman/mailman/merge_requests/286) Finally with working tests after upstream fixes in aiosmtpd. Pluggable components (plugins) ------------------------------ [MR 288](https://gitlab.com/mailman/mailman/merge_requests/288) Rebased after the Click CLI processing branch was merged and got it to work nicely in the end. There was an issue with config processing, as plugins can now supply their own CLI commands and plugins are configured/specified in the Mailman core config, which can be supplied by a CLI option. So that created a bit of an argument processing/lazy-loading issue but looking some more at how Click works I was able to quickly resolve it. Config option is now an [eager](http://click.pocoo.org/5/options/#callbacks-and-eager-options) one and initializes Mailman before commands need to be listed/used. Pluggable workflows ------------------- [MR 299](https://gitlab.com/mailman/mailman/merge_requests/299) Also rebased and added tests to get diffcov to 100% and fix one of the migrations in that branch, as it was broken before. Key management (plugin) ----------------------- Implemented email commands for managing per-address PGP keys in plugin. It uses pluggable workflows to plug into the subscription process of a PGP enabled mailing list and requests the user key, also does mailback confirmation of it. The key is then available to the list moderator during subscription moderation, and more generally to the plugin for verifying signatures and encrypting. After subscription key management is also implemented where a user can change his set key, provided he can sign a challenge provided by the plugin with the old key. Key revocation handling is also necessary, but not yet done. Outgoing processing (plugin) ---------------------------- Implemented custom Bulk and Individual delivery classes for PGP enabled lists in plugin. These deliveries optionally encrypt the mail to subscribers keys, and/or sign with the list key. The bulk one retains anonymity of subscribers as the keyids are zeroed out of the PKESK(Public Key Encrypted Symmetric Key) packets which OpenPGP implementations should handle as a wildcard keyid and try decrypting with all usable private keys. It is also almost as efficient as it can be, as it only encrypts the message with one session key per chunk and then encrypts said session key to recipients in the chunk. The signing is also configurable. Signature hash tracking (plugin) -------------------------------- Implemented store of signature hashes from successful postings to a PGP enabled list which are then (optionally) used to stop replay of the same signature by Mailman in a future posting. PyPI package (plugin) --------------------- [mailman-pgp @ PyPI](https://pypi.python.org/pypi/mailman-pgp) Created the PyPI package for the PGP plugin. Overall ======= I would like to be a bit further in the project at this point, however I am very optimistic about the next days and weeks. After resolving some issues and TODOs I have for the current plugin implementation and setting up the live Mailman instance with the PGP plugin, which should be up by the end of the week. I believe I can start work on the other side of the REST API, of somehow hooking PGP enabled archives/lists to Postorius and HyperKitty. Cheers, -- Jan ______________________________________________________ /\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ # -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 862 bytes Desc: OpenPGP digital signature URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Jul 26 14:36:04 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 27 Jul 2017 03:36:04 +0900 Subject: [Mailman-Developers] Discourse Integration In-Reply-To: References: <22717.13813.915523.526404@turnbull.sk.tsukuba.ac.jp> Message-ID: <22904.57620.954910.92582@turnbull.sk.tsukuba.ac.jp> Vaibhav Lohani writes: > Stephen J. Turnbull wrote: > > What sort of integration do you have in mind? > > The idea is to have a two way binding sort of integration, where the > conversations can be carried out in discourse as well as mailing list > mode. The big issue here is that Discourse provides a centralized server, which means that conversations can be serialized properly, and you can even be notified if somebody else is replying at the same time. Mailing lists are asynchronous, so that posts often "cross in the mail". While these two modes are *fundamentally* incompatible, so cannot be perfectly merged, it's important that the two systems behave as much like each other as possible. In particular, you can't assume that mailing list subscribers will be using a browser. Many people use dedicated mail clients. Also, once you've got the basic syncing functionality down, it would be interesting to see if you could provide a mail protocol for manipulating Discourse features (likes, badges, etc) via the email interface. (This might be almost trivial if Discourse already provides such a protocol for email.) > the mailman rest api should not be opened publicly, The Mailman REST API is there to be used. The problem is that access is not authenticated at this point: anyone who can access it has the run of the installation. So everybody who has access to the REST API is effectively the site manager. Since you are planning to host the Discourse and Mailman instances on the same server, this should not be a problem. But you should be aware that that Mailman API will be firewalled from the Internet in any public instance for the foreseeable future. Hope this helps. Steve