From mark at msapiro.net Sat Sep 2 11:50:18 2017 From: mark at msapiro.net (Mark Sapiro) Date: Sat, 2 Sep 2017 08:50:18 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot Message-ID: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net> I have encountered a strange issue on both lists.mailman3.org and mail.python.org. Namely, Mailman3 is not started upon reboot. If I reboot the server Mailman 3 is shut down normally but is never started when the server comes up. A typical mailman.log excerpt is > Sep 02 10:12:06 2017 (20345) nntp runner caught SIGTERM. Stopping. > Sep 02 10:12:06 2017 (20345) %s runner exiting. > Sep 02 10:12:06 2017 (20342) command runner caught SIGTERM. Stopping. > Sep 02 10:12:06 2017 (20342) %s runner exiting. > Sep 02 10:12:07 2017 (20348) %s runner exiting. > Sep 02 10:12:07 2017 (20318) Master stopped > Sep 02 10:21:24 2017 (17278) Master started > Sep 02 10:21:29 2017 (17306) %s runner started. > Sep 02 10:21:29 2017 (17301) %s runner started. > Sep 02 10:21:29 2017 (17311) %s runner started. > Sep 02 10:21:29 2017 (17299) %s runner started. (the '%s' should be fixed by https://gitlab.com/mailman/mailman/merge_requests/311). The entries from 10:12:07 are the normal shutdown from the reboot. The very next thing is at 10:21:24 when I manually did 'service mailman start'. The /etc/init.d/mailman3 script is attached as mailman.txt. This has not changed recently. The only thing that has changed is the /etc/init.d/qcluster script (attached as qcluster.txt). This existed before, but was changed to use start-stop-daemon and to add a 'stop' function. I also did 'update-rc.d qcluster defaults' to create the rc*.d entries for qcluster which I had neglected to do before. The entries for mailman3 are there (unchanged in over a year). Does anyone have an idea why Mailman 3 is not starting on reboot? -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- ### BEGIN INIT INFO # Provides: GNU Mailman # Required-Start: $local_fs $network # Required-Stop: $local_fs $network # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Mailman Service # Description: service control for Mailman ### END INIT INFO PATH=/opt/mailman/mailman-bundler/bin:/usr/sbin:/usr/bin:/bin:/sbin: DESC="GNU Mailman service" DAEMON=/opt/mailman/mailman-bundler/bin/mailman NAME=mailman USER=mailman GROUP=mailman # Exit if the package is not installed [ -x "$DAEMON" ] || exit 0 # Load the VERBOSE setting and other rcS variables . /lib/init/vars.sh # Define LSB log_* functions. # Depend on lsb-base (>= 3.2-14) to ensure that this file is present # and status_of_proc is working. . /lib/lsb/init-functions case "$1" in start) [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME" # use --force to remove a stale lock. sudo -u $USER $DAEMON start --force ;; stop) [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME" sudo -u $USER $DAEMON stop ;; status) sudo -u $USER $DAEMON status ;; reopen) sudo -u $USER $DAEMON reopen ;; restart) log_daemon_msg "Restarting $DESC" "$NAME" sudo -u $USER $DAEMON restart ;; *) echo "Usage: $SCRIPTNAME {start|stop|status|reopen|restart}" >&2 exit 3 ;; esac -------------- next part -------------- ### BEGIN INIT INFO # Provides: HyperKitty async tasks runner # Required-Start: $local_fs $network # Required-Stop: $local_fs $network # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: HyperKitty async tasks runner # Description: HyperKitty async tasks runner ### END INIT INFO PATH=/opt/mailman/mailman-bundler/bin:/usr/sbin:/usr/bin:/bin:/sbin: DESC="HyperKitty async tasks runner" DAEMON=/opt/mailman/mailman-bundler/bin/mailman-web-django-admin NAME=qcluster USER=mailman GROUP=mailman PIDFILE=/var/run/qcluster.pid # Exit if the package is not installed [ -x "$DAEMON" ] || exit 0 # Load the VERBOSE setting and other rcS variables . /lib/init/vars.sh # Define LSB log_* functions. # Depend on lsb-base (>= 3.2-14) to ensure that this file is present # and status_of_proc is working. . /lib/lsb/init-functions case "$1" in start) log_daemon_msg "Starting $DESC" "$NAME" start-stop-daemon --start --quiet \ --pidfile $PIDFILE --make-pidfile --background \ --chuid $USER:$GROUP --exec $DAEMON -- qcluster ;; stop) log_daemon_msg "Stopping $DESC" "$NAME" start-stop-daemon --stop --pidfile $PIDFILE rm $PIDFILE ;; *) echo "Usage: $SCRIPTNAME {start|stop}" >&2 exit 3 ;; esac From mark at msapiro.net Sat Sep 2 13:31:17 2017 From: mark at msapiro.net (Mark Sapiro) Date: Sat, 2 Sep 2017 10:31:17 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot [addendum] In-Reply-To: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net> References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net> Message-ID: On 09/02/2017 08:50 AM, Mark Sapiro wrote: > The only thing that has changed is the > /etc/init.d/qcluster script (attached as qcluster.txt). This existed > before, but was changed to use start-stop-daemon and to add a 'stop' > function. > > I also did 'update-rc.d qcluster defaults' to create the rc*.d entries > for qcluster which I had neglected to do before. The entries for > mailman3 are there (unchanged in over a year). The issue seems unrelated to qcluster. As a test, I stopped qcluster, removed /etc/init.d/qcluster and the /etc/rc*.d entries and rebooted the server and mailman still wasn't started on reboot. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at list.org Sat Sep 2 15:30:06 2017 From: barry at list.org (Barry Warsaw) Date: Sat, 2 Sep 2017 15:30:06 -0400 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot [addendum] In-Reply-To: References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net> Message-ID: It would be interesting to know if the problem is in the init script or Mailman. You may have to crank up logging of both to find out. Also, I wonder if the same problem occurs if you use systemd? Other than that I don?t have any ideas. -Barry > On Sep 2, 2017, at 13:31, Mark Sapiro wrote: > > On 09/02/2017 08:50 AM, Mark Sapiro wrote: >> The only thing that has changed is the >> /etc/init.d/qcluster script (attached as qcluster.txt). This existed >> before, but was changed to use start-stop-daemon and to add a 'stop' >> function. >> >> I also did 'update-rc.d qcluster defaults' to create the rc*.d entries >> for qcluster which I had neglected to do before. The entries for >> mailman3 are there (unchanged in over a year). > > > The issue seems unrelated to qcluster. As a test, I stopped qcluster, > removed /etc/init.d/qcluster and the /etc/rc*.d entries and rebooted the > server and mailman still wasn't started on reboot. > > -- > Mark Sapiro The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/barry%40list.org > > Security Policy: http://wiki.list.org/x/QIA9 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mark at msapiro.net Sat Sep 2 21:27:44 2017 From: mark at msapiro.net (Mark Sapiro) Date: Sat, 2 Sep 2017 18:27:44 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot [addendum] In-Reply-To: References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net>

Message-ID: <451e5d99-9cf2-52f1-384a-c06a89a1a2a7@msapiro.net> On 09/02/2017 12:30 PM, Barry Warsaw wrote: > It would be interesting to know if the problem is in the init script or Mailman. You may have to crank up logging of both to find out. Also, I wonder if the same problem occurs if you use systemd? It appears to be in Mailman. I added some logging to the init.d/mailman script as follows: --- /etc/init.d/mailman.bak 2017-09-03 00:55:54.720770063 +0000 +++ /etc/init.d/mailman 2017-09-03 00:41:49.569395653 +0000 @@ -15,6 +15,8 @@ USER=mailman GROUP=mailman +date >> /etc/mm_script +echo $@ >> /etc/mm_script # Exit if the package is not installed [ -x "$DAEMON" ] || exit 0 @@ -28,9 +30,11 @@ case "$1" in start) + echo 'starting' >> /etc/mm_script [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME" # use --force to remove a stale lock. sudo -u $USER $DAEMON start --force + echo 'started' >> /etc/mm_script ;; stop) [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME" Then I rebooted and got this in /etc/mm_script Sun Sep 3 00:42:25 UTC 2017 stop Sun Sep 3 00:43:26 UTC 2017 start starting started but Mailman wasn't running so I did 'sudo service mailman start' and got an additional Sun Sep 3 00:44:38 UTC 2017 start starting started in /etc/mm_script, after which Mailman was running. mailman.log has ... Sep 03 00:42:33 2017 (2636) nntp runner caught SIGTERM. Stopping. Sep 03 00:42:33 2017 (2636) nntp runner exiting. Sep 03 00:42:33 2017 (2639) rest runner caught SIGTERM. Stopping. Sep 03 00:42:34 2017 (2639) rest runner exiting. Sep 03 00:42:34 2017 (2616) Master stopped Sep 03 00:44:44 2017 (2572) Master started Sep 03 00:44:50 2017 (2590) in runner started. Sep 03 00:44:50 2017 (2596) retry runner started. Sep 03 00:44:51 2017 (2595) rest runner started. Sep 03 00:44:51 2017 (2588) bounces runner started. ... note that the 'stop' from the reboot produced messages ending with Sep 03 00:42:34 2017 (2616) Master stopped and the next message in the log is Sep 03 00:44:44 2017 (2572) Master started from the manual start. This issue began after I installed changes including commit ae0042a90220119414f61aeb20c6b58bfacb8af2 affecting mailman/bin/mailman.py and mailman/bin/master.py. I haven't gone through those changes in detail, but I suspect they have something to do with it. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From simon.hanna at serve-me.info Mon Sep 4 17:46:27 2017 From: simon.hanna at serve-me.info (Simon Hanna) Date: Mon, 4 Sep 2017 23:46:27 +0200 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot In-Reply-To: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net> References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net> Message-ID: > Does anyone have an idea why Mailman 3 is not starting on reboot? I'm using systemd and it has been working for me the last couple of reboots. I don't have one coming up on my prod setup until end of this month, I can check again then. I once had issues with stale locks, because the processes sometimes get killed, leaving locks in place. But you seem to --force it, so that shouldn't be the problem... From mark at msapiro.net Mon Sep 4 20:45:35 2017 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 4 Sep 2017 17:45:35 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot In-Reply-To: References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net> Message-ID: On 09/04/2017 02:46 PM, Simon Hanna wrote: > >> Does anyone have an idea why Mailman 3 is not starting on reboot? > I'm using systemd and it has been working for me the last couple of > reboots. I don't have one coming up on my prod setup until end of this > month, I can check again then. What are you running? I think the issue is caused by commit ae0042a90220119414f61aeb20c6b58bfacb8af2 are you running with that? > I once had issues with stale locks, because the processes sometimes get > killed, leaving locks in place. But you seem to --force it, so that > shouldn't be the problem... That isn't the issue here. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From mark at msapiro.net Mon Sep 4 23:59:20 2017 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 4 Sep 2017 20:59:20 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot [addendum] In-Reply-To: <451e5d99-9cf2-52f1-384a-c06a89a1a2a7@msapiro.net> References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net>

<451e5d99-9cf2-52f1-384a-c06a89a1a2a7@msapiro.net> Message-ID: On 09/02/2017 06:27 PM, Mark Sapiro wrote: > On 09/02/2017 12:30 PM, Barry Warsaw wrote: >> It would be interesting to know if the problem is in the init script or Mailman. You may have to crank up logging of both to find out. Also, I wonder if the same problem occurs if you use systemd? > > > It appears to be in Mailman. This get's stranger and stranger. I removed the logging from the init.d script which invokes /opt/mailman/mailman-bundler/bin/mailman with the appropriate start/stop option. I then added logging to it. Here it is: #!/bin/bash unset PYTHONSTARTUP echo `date` At bundler/bin/mailman >> /opt/mailman/temp_log exec /opt/mailman/mailman-bundler/venv-3.4/bin/mailman \ -C "/opt/mailman/mailman-bundler/deployment/mailman.cfg" \ $@ so when invoked it prints a timestamp and 'At bundler/bin/mailman'. I also added logging to /opt/mailman/mailman-bundler/venv-3.4/bin/mailman as follows: #!/usr/local/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'mailman==3.2.0a1','console_scripts','mailman' __requires__ = 'mailman==3.2.0a1' import re import sys import time from pkg_resources import load_entry_point if __name__ == '__main__': with open('/opt/mailman/temp_log', 'a') as fp: print('{} At venv bin/mailman __main__'.format(time.asctime()), file=fp) sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit( load_entry_point('mailman==3.2.0a1', 'console_scripts', 'mailman')() ) so it prints a time stamp and 'At venv bin/mailman __main__' in the if __name__ == '__main__': block. Finally, I added a bit to the main() function in mailman/bin/mailman.py so it reads (a bunch of decorators omitted): def main(ctx, config_file): # XXX https://github.com/pallets/click/issues/303 """\ The GNU Mailman mailing list management system Copyright 1998-2017 by the Free Software Foundation, Inc. http://www.list.org """ # click handles dispatching to the subcommand via the Subcommands class. import time with open('/opt/mailman/temp_log', 'a') as fp: print('{} At mailman.main'.format(time.asctime()), file=fp) Then I rebooted and subsequently manually ran sudo service mailman start. I got this in temp_log Tue Sep 5 03:10:57 UTC 2017 At bundler/bin/mailman Tue Sep 5 03:10:58 2017 At venv bin/mailman __main__ Tue Sep 5 03:11:57 UTC 2017 At bundler/bin/mailman Tue Sep 5 03:11:58 2017 At venv bin/mailman __main__ Tue Sep 5 03:15:17 UTC 2017 At bundler/bin/mailman Tue Sep 5 03:15:18 2017 At venv bin/mailman __main__ Tue Sep 5 03:15:22 2017 At mailman.main The 03:10:.. entries are from the 'stop' from the reboot. The absence of 'At mailman.main' may be significant. All the normal shutdown messages are in mailman.log about 6-7 seconds later. But, if I do 'sudo service mailman stop' there is an 'At mailman.main' message logged. Then the 3:11:.. entries are from init running the init.d/mailman script and indicate that /opt/mailman/mailman-bundler/bin/mailman and /opt/mailman/mailman-bundler/venv-3.4/bin/mailman were both invoked, but there is nothing in mailman.log and no 'At mailman.main' message. Then the 03:15:.. entries are from when I manually started Mailman. It appears that load_entry_point('mailman==3.2.0a1', 'console_scripts', 'mailman')() is being executed by /opt/mailman/mailman-bundler/venv-3.4/bin/mailman, but either somehow the 'start' argument is lost or something after that fails. I don't know how the 'start' argument could be getting lost, but I don't know what else could be failing either. Any hints? -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From mark at msapiro.net Tue Sep 5 12:19:40 2017 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 5 Sep 2017 09:19:40 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot [addendum] In-Reply-To: References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net>

<451e5d99-9cf2-52f1-384a-c06a89a1a2a7@msapiro.net> Message-ID: <6b6f00c2-a70d-4638-3a8c-1ffcbf9fb761@msapiro.net> I'm beginning to develop some understanding of this. I think it is not related to the changes for plugins, but rather is related to the use of click and possibly something to do with run levels. I have put logging in /opt/mailman/mailman-bundler/venv-3.4/bin/mailman to report the command that invoked it before it in turn invokes mailman.bin.mailman.main(), and I put logging there to report the fact that control got there. The log looks like Tue Sep 5 15:42:57 2017 ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'stop'] Tue Sep 5 15:42:58 2017 mailman.py - main() Tue Sep 5 15:43:08 2017 ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'start', '--force'] Tue Sep 5 15:43:09 2017 mailman.py - main() Tue Sep 5 15:47:16 2017 ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'stop'] Tue Sep 5 15:48:13 2017 ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'start', '--force'] Tue Sep 5 15:58:39 2017 ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'start', '--force'] Tue Sep 5 15:58:47 2017 mailman.py - main() The 15:42:.. entries are from a manual stop. The 15:43:.. entries are from a manual start. The 15:47:16 entry is the 'stop' from a reboot - note no 'mailman.py - main()' here, and the 15:48:13 entry is the 'start' from the reboot - again no 'mailman.py - main()' here. Finally, the 15:58:.., entries are from a manual start. There are several 'click' decorators on the mailman.bin.mailman.main() function, and I suspect something there is detecting the run level and just quitting if a shutdown or startup is in progress. I would like to try systemd instead, but both lists.mailman3.org and mail.python.org are on Ubuntu 14.04 which doesn't support systemd. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From raj.abhilash1 at gmail.com Tue Sep 5 15:42:17 2017 From: raj.abhilash1 at gmail.com (Abhilash Raj) Date: Tue, 05 Sep 2017 12:42:17 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot [addendum] In-Reply-To: <6b6f00c2-a70d-4638-3a8c-1ffcbf9fb761@msapiro.net> References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net>

<451e5d99-9cf2-52f1-384a-c06a89a1a2a7@msapiro.net> <6b6f00c2-a70d-4638-3a8c-1ffcbf9fb761@msapiro.net> Message-ID: <1504640537.785760.1096211544.690E9A0D@webmail.messagingengine.com> Hi Mark, I did just a quick search and read about issues related to Click and init scripts. See below. On Tue, Sep 5, 2017, at 09:19 AM, Mark Sapiro wrote: > I'm beginning to develop some understanding of this. I think it is not > related to the changes for plugins, but rather is related to the use of > click and possibly something to do with run levels. > > I have put logging in /opt/mailman/mailman-bundler/venv-3.4/bin/mailman > to report the command that invoked it before it in turn invokes > mailman.bin.mailman.main(), and I put logging there to report the fact > that control got there. > > The log looks like > > > Tue Sep 5 15:42:57 2017 > ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', > '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'stop'] > Tue Sep 5 15:42:58 2017 mailman.py - main() > Tue Sep 5 15:43:08 2017 > ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', > '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'start', > '--force'] > Tue Sep 5 15:43:09 2017 mailman.py - main() > Tue Sep 5 15:47:16 2017 > ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', > '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'stop'] > Tue Sep 5 15:48:13 2017 > ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', > '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'start', > '--force'] > Tue Sep 5 15:58:39 2017 > ['/opt/mailman/mailman-bundler/venv-3.4/bin/mailman', '-C', > '/opt/mailman/mailman-bundler/deployment/mailman.cfg', 'start', > '--force'] > Tue Sep 5 15:58:47 2017 mailman.py - main() > > > The 15:42:.. entries are from a manual stop. The 15:43:.. entries are > from a manual start. The 15:47:16 entry is the 'stop' from a reboot - > note no 'mailman.py - main()' here, and the 15:48:13 entry is the > 'start' from the reboot - again no 'mailman.py - main()' here. Finally, > the 15:58:.., entries are from a manual start. > > There are several 'click' decorators on the mailman.bin.mailman.main() > function, and I suspect something there is detecting the run level and > just quitting if a shutdown or startup is in progress. > > I would like to try systemd instead, but both lists.mailman3.org and > mail.python.org are on Ubuntu 14.04 which doesn't support systemd. I *looks* like your observation about the click *might* be correct, I'd have to test more to find out which can't for next few days. Here is an excerpt from the click documentation[1]: --- Click in Python 3 does all the Unicode handling in the standard library and is subject to its behavior. In Python 2, Click does all the Unicode handling itself, which means there are differences in error behavior. The most glaring difference is that in Python 2, Unicode will ?just work?, while in Python 3, it requires extra care. The reason for this is that in Python 3, the encoding detection is done in the interpreter, and on Linux and certain other operating systems, its encoding handling is problematic. The biggest source of frustration is that Click scripts invoked by init systems (sysvinit, upstart, systemd, etc.), deployment tools (salt, puppet), or cron jobs (cron) will refuse to work unless a Unicode locale is exported. If Click encounters such an environment it will prevent further execution to force you to set a locale. This is done because Click cannot know about the state of the system once it?s invoked and restore the values before Python?s Unicode handling kicked in. --- It does mention that an error trace will show up when there are locale problems with Python 3, but I guess you don't see any of that. Maybe it is suppressed somewhere? [1]: http://click.pocoo.org/5/python3/#python-3-surrogate-handling > -- > Mark Sapiro The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/raj.abhilash1%40gmail.com > > Security Policy: http://wiki.list.org/x/QIA9 > Email had 1 attachment: > + signature.asc > 1k (application/pgp-signature) From mark at msapiro.net Tue Sep 5 15:54:54 2017 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 5 Sep 2017 12:54:54 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot [addendum] In-Reply-To: <1504638818.780239.1096177528.1DF61953@webmail.messagingengine.com> References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net>

<451e5d99-9cf2-52f1-384a-c06a89a1a2a7@msapiro.net> <6b6f00c2-a70d-4638-3a8c-1ffcbf9fb761@msapiro.net> <1504638818.780239.1096177528.1DF61953@webmail.messagingengine.com> Message-ID: <42f51722-2f53-6647-8fb4-75372ee513bc@msapiro.net> On 09/05/2017 12:13 PM, Abhilash Raj wrote: > > I did just a quick search and read about issues related to Click and > init scripts. See below. Thank you Abhilash. I had started looking through the click docs, but hadn't found the relevant stuff yet. ... > The biggest source of frustration is that Click scripts invoked by init > systems (sysvinit, upstart, systemd, etc.), deployment tools (salt, > puppet), or cron jobs (cron) will refuse to work unless a Unicode locale > is exported. (Thank you!)*10. All it needed was the addition of export LANG=en_US.UTF-8 to the init.d script. It now works. > It does mention that an error trace will show up when there are locale > problems with Python 3, but I guess you don't see any of that. Maybe it > is suppressed somewhere? I never saw it and I just grepped all the logs, and it's not there. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From barry at python.org Tue Sep 5 17:25:32 2017 From: barry at python.org (Barry Warsaw) Date: Tue, 5 Sep 2017 14:25:32 -0700 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot [addendum] In-Reply-To: <42f51722-2f53-6647-8fb4-75372ee513bc@msapiro.net> References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net>

<451e5d99-9cf2-52f1-384a-c06a89a1a2a7@msapiro.net> <6b6f00c2-a70d-4638-3a8c-1ffcbf9fb761@msapiro.net> <1504638818.780239.1096177528.1DF61953@webmail.messagingengine.com> <42f51722-2f53-6647-8fb4-75372ee513bc@msapiro.net> <413a2b4b-84e8-ecb0-f4e4-5c5b2ff4c2bc@msapiro.net> Message-ID: <085A1D15-C849-4300-96D5-3CF72E835B2D@python.org> On Sep 5, 2017, at 14:51, Mark Sapiro wrote: > > On 09/05/2017 02:25 PM, Barry Warsaw wrote: >> >> Can you suggest a good place to document this? > > > We really don't say anything about running Mailman as a service. I guess > we think that people know how to do that. > > The now defunct mailman-bundler has a sample systemd mailman3.service > file but that's it. > > This click issue is a definite gotcha however and should be mentioned. > Perhaps an additional section at > could mention it. It wouldn't necessarily need to give entire scripts, > but just mention that click requires that a minimal locale setting be > exported before issuing 'mailman' commands. See what you think about https://gitlab.com/mailman/mailman/merge_requests/316 Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 273 bytes Desc: Message signed with OpenPGP URL: From JACKSON at encompasserve.org Thu Sep 7 08:31:45 2017 From: JACKSON at encompasserve.org (noskcaJ leahciM) Date: Thu, 07 Sep 2017 08:31:45 -0400 (EDT) Subject: [Mailman-Developers] GDPR Message-ID: <01QITR1BQCOO0002JD@Encompasserve.org> GDPR is nearing the last 7 months of its 2 year transitional implementation before becoming part of the law in EU countries (inc., despite Brexit, the UK), affecting 0.5bn citizens together with US enterprises in under Privacy Shield (replacing Safe Harbour) as well as enterprises in EEA member countries and, outside of the EEA, countries whose privacy laws have been assessed for adequacy. Operators of lists such as MM are as much called-to-account as are the vast corporations behind popular social media that currently absolve themselves as being mere publishers. Below is an initial impact assessment (vs. MM 2.1.14) of high priority impacts in bullet-point form for immediate consideration based on my current understanding. Where changes are required, they are required for continued use of MM in EU countries and in adequate countries (inc. EU-US Privacy Shield) where one or more list members is an identifiable EU citizen. 1. The term consent has a specific meaning within GDPR (i.e. one of a number of lawful bases on which personal data may be processed). The subscribe web flow and subscribe/confirm email exchanges should be edited to use the term "consent" rather than "confirm" where appropriate to make that consent explicit. 2. The term restriction has a specific meaning within GDPR (in general it means suspending any use (processing) or disclosure of personal data. Nomail could usefully be renamed as "Restriction" and the Nomail functionality extended made to operate both ways, i.e. restricting members from posting and restricting (suppresing) retrieval of a restricted member's details in a list of [subscribed] members. 3. GDPR places strong record-keeping obligations on data controllers. It implies that audit trails of consent be maintained. A per-subscriber report that shows how they were subscribed, keeping details of ip addresses of browser used if by web etc., log of all preference changes made etc. is a Should-Have requirement. A per-list-report perhaps showing less detail (e.g. method of subscription and time and date of consent) is a a Could-Have requirement. 4. GDPR provides that the right of erasure (that has coloquially become known as the "right to be forgotten) be provided upon request. In addition ot unsubscribing from a list, a list subscriber Should be able to initiate automated erasure from the site archive of all posts of and from a given subscriber. There seems no reason for erasure to be moderated. Using the existing password authentication or a confirm string, subscribers could be provided with the ability themselves to erase their personal details from the site, including all archived posts, without administrative intervention or incurring the delays inherent with moderation. GDPR doesn't provide for selective erasure, for example in the case of a non-factual, inflamatory or libelous post that a subscriber wishes to have removed; all-or-nothing will suffice for compliance with GDPR. Erasure is required to uphold a right so is a Should-Have requirement. 5. GDPR enshrines 7 principles and 8, legally-enforceable, rights. While "Privacy by Design and by Default" is not actually one of the principles it is a mandatory approach. The default for list settings should be to i) to close lists, making posting to lists restricted to members; ii) to hide or disguise identities in automatically-added elements of list members' posts (e.g. in "From:"); iii) to hide or disguise identities inadvertently disclosed in list members' posts e.g. from quoting others etc. (My interpretation is that no risk arises to the operator of the list where member 2 quoting text of member 1 where member 1 intentionally included their contact details in a post, for example as a phone number or as "me -at- here"; meaning that it is sufficient to suppres/disguise headers and email addresses entered un-disguised by posters) iv) either to disallow retrieval of list members to [even] subscribed members, or to hide/disguise addresses in retrieved lists; v) to archive privately, requiring authentication and and keeping records (logging) of to whom information on the archive was disclosed; vi) on closed lists (the default), disallow the posting of HTML by which otherwise members might seek to obtain the identity of others through inclusion of web bugs/beacons. From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Sep 9 03:05:35 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 9 Sep 2017 16:05:35 +0900 Subject: [Mailman-Developers] GDPR In-Reply-To: <01QITR1BQCOO0002JD@Encompasserve.org> References: <01QITR1BQCOO0002JD@Encompasserve.org> Message-ID: <22963.37567.882163.454340@turnbull.sk.tsukuba.ac.jp> tl;dr I don't think this is worth thinking about until we get a request from users who actually are threatened with liability for their Mailman installations. Suggestions from where we can obtain funding to do this stuff? The harder parts are non-trivial. noskcaJ leahciM writes: > 1. The term consent has a specific meaning within GDPR (i.e. one of a > number of lawful bases on which personal data may be processed). > The subscribe web flow and subscribe/confirm email exchanges should > be edited to use the term "consent" rather than "confirm" where > appropriate to make that consent explicit. I don't think this is something we should do blindly. If a list owner or site affected by GDPR wants to consult a lawyer and contribute that knowledge, I think we should follow up with changes, but I can see issues that would involves potential liability for our users if we use a word that has legal meaning and our procedures aren't up to the standard. > 2. The term restriction has a specific meaning within GDPR (in general > it means suspending any use (processing) or disclosure of personal > data. Nomail could usefully be renamed as "Restriction" and No, it can't. I see no good reason to include both nomail and privacy protection in a single option. Nomail is very useful without being tied to privacy options. > Nomail functionality extended made to operate both ways, i.e. > restricting members from posting This is not what nomail does. That's moderation. Nomail prevents a subscriber from receiving posts, and is primarily intended as a user option, not an admin option. > and restricting (suppresing) retrieval of a restricted member's > details in a list of [subscribed] members. There's a separate user option for this, IIRC, as well as a global option restricting availability of lists to the list and site admins. I don't see a strong argument for combining these options, or combining retrieval suppression with moderation. > 3. GDPR places strong record-keeping obligations on data controllers. > It implies that audit trails of consent be > maintained. Patches welcome. I don't think this is reasonable to ask of most sites at present, and the security implications of "audit" (identifying and authenticating users and crypto signing said "trails" etc) seem daunting. > 4. GDPR provides that the right of erasure (that has coloquially become > known as the "right to be forgotten) be provided upon request. In > addition ot unsubscribing from a list, a list subscriber Should be > able to initiate automated erasure from the site archive of all > posts of and from a given subscriber. This is simply not feasible to guarantee in Mailman 3, since we deliberately separated the archiver and sites (even individual lists) may supply their own. This also raises hairy issues about authentication. > There seems no reason for erasure to be moderated. Using the > existing password authentication or a confirm string, subscribers > could be provided with the ability themselves to erase their > personal details from the site, This is not necessarily under control of Mailman. > including all archived posts, without administrative > intervention or incurring the delays inherent with moderation. I don't think the existing authentication features are sufficiently reliable for this. Specifically, I would imagine that the rights embodied in GDPR inhere in *human persons*, not in email mailboxes. Yet in fact in all current authentication done by Mailman only mailboxes are identified, not persons. > GDPR doesn't provide for selective erasure, for example in the case > of a non-factual, inflamatory or libelous post that a subscriber > wishes to have removed; all-or-nothing will suffice for compliance > with GDPR. Erasure is required to uphold a right so is a > Should-Have requirement. > > 5. GDPR enshrines 7 principles and 8, legally-enforceable, rights. > While "Privacy by Design and by Default" is not actually one of the > principles it is a mandatory approach. The default for list > settings should be to > > i) to close lists, making posting to lists restricted to members; This is something we should do anyway, since restricting posting to members is generally a regrettable but necessary spam-control practice. > ii) to hide or disguise identities in automatically-added elements > of list members' posts (e.g. in "From:"); That's not reasonable. The list does not "own" or add From (or To or Cc, for that matter); the author does. In the general case, identity can be concealed by the poster if they want to by acquiring an additional mailbox. Some effort to prevent address harvesting from archives by spammers is reasonable, but hiding identity is another matter entirely. > iii) to hide or disguise identities inadvertently disclosed in list > members' posts e.g. from quoting others etc. That's not reasonable. As above, we don't own the text, the poster does. Again, discouraging harvesting is reasonable, but hiding identities in text is not a reasonable requirement. > iv) either to disallow retrieval of list members to [even] > subscribed members, or to hide/disguise addresses in retrieved > lists; Options are present. Changing defaults if necessary is not a big deal. > v) to archive privately, requiring authentication I don't like this (in my use cases, public archives are usually desirable), but it might be reasonable as a default. > and and keeping records (logging) of to whom information > on the archive was disclosed; This is unreasonable. Of course HTTP accesses are logged, and with access limited to authenticated subscribers the relevant list user is known, but in general there is no *identification*. Furthermore, this is invading the *reader's* privacy in any case where personal information intended to be closely held is not disclosed. > vi) on closed lists (the default), disallow the posting of HTML by > which otherwise members might seek to obtain the identity of > others through inclusion of web bugs/beacons. We already have an option to refuse HTML. But users don't like that. Accurately restricting what can be posted sounds difficult to me, but patches welcome. Steve From simon.hanna at serve-me.info Mon Sep 11 05:17:51 2017 From: simon.hanna at serve-me.info (Simon Hanna) Date: Mon, 11 Sep 2017 11:17:51 +0200 Subject: [Mailman-Developers] Mailman 3 not restarting after reboot In-Reply-To: References: <5ffdb0f5-f188-4396-7dc8-4ecabc70d68f@msapiro.net>

Message-ID: On 09/05/2017 02:45 AM, Mark Sapiro wrote: > On 09/04/2017 02:46 PM, Simon Hanna wrote: >>> Does anyone have an idea why Mailman 3 is not starting on reboot? >> I'm using systemd and it has been working for me the last couple of >> reboots. I don't have one coming up on my prod setup until end of this >> month, I can check again then. > > What are you running? I think the issue is caused by > > commit ae0042a90220119414f61aeb20c6b58bfacb8af2 > > are you running with that? I'm currently still running 3.1.0 (Using my archlinux packages) From JACKSON at encompasserve.org Fri Sep 15 19:07:19 2017 From: JACKSON at encompasserve.org (noskcaJ leahciM) Date: Fri, 15 Sep 2017 19:07:19 -0400 (EDT) Subject: [Mailman-Developers] GDPR Message-ID: <01QJ5JK26GXI0005CZ@Encompasserve.org> | tl;dr Too long: The regulation and articles are even longer. Didn't Read: But you replied none-the-less ;-) ! I don't think this is worth thinking about until we get a | request from users who actually are threatened with liability for | their Mailman installations. There will be installations and use cases that are not impacted. GDPR is European law and failing to comply with it is not an option. It's generally recognised that for all impacted, work may be required to attain compliance. Awareness-raising should not, however, result in messengers being shot. | | Suggestions from where we can obtain funding to do this stuff? | The harder parts are non-trivial. | | noskcaJ leahciM writes: | | > 1. The term consent has a specific meaning within GDPR (i.e. one of a | > number of lawful bases on which personal data may be processed). | > The subscribe web flow and subscribe/confirm email exchanges should | > be edited to use the term "consent" rather than "confirm" where | > appropriate to make that consent explicit. | | I don't think this is something we should do blindly. If a list owner | or site affected by GDPR wants to consult a lawyer and contribute that | knowledge, I think we should follow up with changes, but I can see | issues that would involves potential liability for our users if we use | a word that has legal meaning and our procedures aren't up to the | standard. Art.7, Rec.32,42. Do not require that the word "consent" be used rather than "confirm". However, consent is one of the lawful bases and being unambiguous that consent is being requested seemed a reasonable suggestion for the effort required to make the change. | | > 2. The term restriction has a specific meaning within GDPR (in general | > it means suspending any use (processing) or disclosure of personal | > data. Nomail could usefully be renamed as "Restriction" and | | No, it can't. I see no good reason to include both nomail and privacy | protection in a single option. Nomail is very useful without being tied | to privacy options. | | > Nomail functionality extended made to operate both ways, i.e. | > restricting members from posting | | This is not what nomail does. That's moderation. Nomail prevents a | subscriber from receiving posts, and is primarily intended as a user | option, not an admin option. The separate nomain and moderation settings can be kept, along with their traditional meaning. However Art.18 and particularly Rec.67. imply that a single "restriction" setting (that sets nomail + moderation) that shows as "restriction" (and without the reverse implication) is required. | | > and restricting (suppresing) retrieval of a restricted member's | > details in a list of [subscribed] members. | | There's a separate user option for this, IIRC, as well as a global | option restricting availability of lists to the list and site admins. | I don't see a strong argument for combining these options, or | combining retrieval suppression with moderation. Please see clarification/compromise above. | | > 3. GDPR places strong record-keeping obligations on data controllers. | > It implies that audit trails of consent be | > maintained. | | Patches welcome. I don't think this is reasonable to ask of most | sites at present, and the security implications of "audit" | (identifying and authenticating users and crypto signing said "trails" | etc) seem daunting. Rec.13 to Art.30 provides for national for organisations under 250 persons. Even Rec.82 does not refer to explicitly to "audit" trails so this need not be over-egged. However it does seem reasonable that MM maintain records of the date and time of subscription/un-subscription and method (email/web). A record of subscription/un-subscription can be had as it stands from emails notifying the list manager; it's just that this isn't as convenient. | | > 4. GDPR provides that the right of erasure (that has coloquially become | > known as the "right to be forgotten) be provided upon request. In | > addition ot unsubscribing from a list, a list subscriber Should be | > able to initiate automated erasure from the site archive of all | > posts of and from a given subscriber. | | This is simply not feasible to guarantee in Mailman 3, since we | deliberately separated the archiver and sites (even individual lists) | may supply their own. [Following text moved to below] Art.17 (b) allows this right to be asserted at a whim; however it does refer to taking account of available technology and the cost of implementation and requiring (only) "reasonable" steps (which was reflected in what was said below). | | > There seems no reason for erasure to be moderated. Using the | > existing password authentication or a confirm string, subscribers | > could be provided with the ability themselves to erase their | > personal details from the site, | | This is not necessarily under control of Mailman. Where it is, then it should be provided. | This also raises hairy issues about authentication. The existing authentication should be good-enough. It isn't envisaged that authentication be improved *just* to protect against other people deleting a subscriber's posts. | | > including all archived posts, without administrative | > intervention or incurring the delays inherent with moderation. | | I don't think the existing authentication features are sufficiently | reliable for this. Specifically, I would imagine that the rights | embodied in GDPR inhere in *human persons*, not in email mailboxes. | Yet in fact in all current authentication done by Mailman only | mailboxes are identified, not persons. Please see above proportionate and reasonable measures. | | > GDPR doesn't provide for selective erasure, for example in the case | > of a non-factual, inflamatory or libelous post that a subscriber | > wishes to have removed; all-or-nothing will suffice for compliance | > with GDPR. Erasure is required to uphold a right so is a | > Should-Have requirement. | > | > 5. GDPR enshrines 7 principles and 8, legally-enforceable, rights. | > While "Privacy by Design and by Default" is not actually one of the | > principles it is a mandatory approach. The default for list | > settings should be to | > | > i) to close lists, making posting to lists restricted to members; | | This is something we should do anyway, since restricting posting to | members is generally a regrettable but necessary spam-control | practice. Fortunately, anti-spam measures hapilly collide with "Privacy by Design and by Default" -- see also below. | | > ii) to hide or disguise identities in automatically-added elements | > of list members' posts (e.g. in "From:"); | | That's not reasonable. The list does not "own" or add From (or To or | Cc, for that matter); the author does. In the general case, identity | can be concealed by the poster if they want to by acquiring an | additional mailbox. Some effort to prevent address harvesting from | archives by spammers is reasonable, but hiding identity is another | matter entirely. What was referred-to primarily was automatically-added elements. Some effort to prevent address-harvesting is performed by other mailers/forum software accepting that its scope goes beyond owned, automatically-added elements to envelope and body owned by the subscriber. Mailers/forum software that does that already helps subscribers not-to leak personal data. That surely is the point rather than some technical distinction between which elements are owned by whom. | | > iii) to hide or disguise identities inadvertently disclosed in list | > members' posts e.g. from quoting others etc. | | That's not reasonable. As above, we don't own the text, the poster | does. Again, discouraging harvesting is reasonable, but hiding | identities in text is not a reasonable requirement. | Some already do this, but don't try to go-beyond replacing '@' signs etc. There is precedent. The practice is already accepted and collides with the interests of anti-harvesting. | > iv) either to disallow retrieval of list members to [even] | > subscribed members, or to hide/disguise addresses in retrieved | > lists; | | Options are present. Changing defaults if necessary is not a big deal. Changing the default is consistent with "Privacy by Design and by Default". Art.25. | | > v) to archive privately, requiring authentication | | I don't like this (in my use cases, public archives are usually | desirable), but it might be reasonable as a default. | | > and and keeping records (logging) of to whom information | > on the archive was disclosed; | | This is unreasonable. Of course HTTP accesses are logged, and with | access limited to authenticated subscribers the relevant list user is | known, but in general there is no *identification*. Furthermore, this | is invading the *reader's* privacy in any case where personal | information intended to be closely held is not disclosed. The reader's privacy is a valid counter-argument. However data breaches (think Equifax) are reportable within 72 hours to authorities (Art.33) and possibly to those affected (Art.34). It would be helpful to know to whom data that should not have been disclosed had been disclosed. In the absence of information about accesses to an archive page before it was taken-down that disclosed data that it should not have, and assuming the private archive members and list members coincided, one would have to assume the entire list. As this still allows the controller to give a reasonable report of to whom data has been disclosed, your counter-argument should be accepted. | | > vi) on closed lists (the default), disallow the posting of HTML by | > which otherwise members might seek to obtain the identity of | > others through inclusion of web bugs/beacons. | | We already have an option to refuse HTML. But users don't like that. | Accurately restricting what can be posted sounds difficult to me, but | patches welcome. The existing detection measurement is proportionate and reasonable. What was said was that for closed lists, that the refuse-HTML measure be invoked by default by virtue of the list being designated as closed. No additional difficult technology or patches required. | | Steve leahciM From barry at python.org Fri Sep 15 19:44:39 2017 From: barry at python.org (Barry Warsaw) Date: Fri, 15 Sep 2017 16:44:39 -0700 Subject: [Mailman-Developers] GDPR In-Reply-To: <01QITR1BQCOO0002JD@Encompasserve.org> References: <01QITR1BQCOO0002JD@Encompasserve.org> Message-ID: noskcaJ leahciM wrote: > GDPR is nearing the last 7 months of its 2 year transitional > implementation before becoming part of the law in EU countries (inc., > despite Brexit, the UK), affecting 0.5bn citizens together with US > enterprises in under Privacy Shield (replacing Safe Harbour) as well as > enterprises in EEA member countries and, outside of the EEA, countries > whose privacy laws have been assessed for adequacy. Operators of lists > such as MM are as much called-to-account as are the vast corporations > behind popular social media that currently absolve themselves as being > mere publishers. GDRP may be well-intentioned, and may even be a good idea, but I know for a fact that many organizations both commercial and volunteer are struggling mightily to comply within the required timeframe. I suspect that many such organization will simply not be able to comply. Realistically, there is no way the GNU Mailman project can comply given our available resources. One outcome could be that our downstream consumers take over that responsibility. Another is that volunteers in our community step up with offers to provide us with their expertise, guidance, and code. We will welcome such volunteers and help ensure that the legal requirements align with project goals, sensibilities, and timelines. So, if you want to see a GDPR compliant GNU Mailman, please find some people to help us. Cheers, -Barry From JACKSON at encompasserve.org Fri Sep 15 20:25:31 2017 From: JACKSON at encompasserve.org (noskcaJ leahciM) Date: Fri, 15 Sep 2017 20:25:31 -0400 (EDT) Subject: [Mailman-Developers] GDPR Message-ID: <01QJ5MB0W0OE0005CZ@Encompasserve.org> Barry Warsaw wrote: | noskcaJ leahciM wrote: | > GDPR is nearing the last 7 months of its 2 year transitional | > implementation before becoming part of the law in EU countries (inc., | > despite Brexit, the UK), affecting 0.5bn citizens together with US | > enterprises in under Privacy Shield (replacing Safe Harbour) as well as | > enterprises in EEA member countries and, outside of the EEA, countries | > whose privacy laws have been assessed for adequacy. Operators of lists | > such as MM are as much called-to-account as are the vast corporations | > behind popular social media that currently absolve themselves as being | > mere publishers. | | GDRP may be well-intentioned, and may even be a good idea, but I know | for a fact that many organizations both commercial and volunteer are | struggling mightily to comply within the required timeframe. I suspect | that many such organization will simply not be able to comply. No disagreement there then. (I've written a paper predicting it based on an estimate of current non-compliance with the existing Data Protection Act in the UK, 20 years after it came onto the UK's statute book. German-speaking countries have had existing tighter rules so will be in better-shape, though I haven't attempted an international comparison.) | Realistically, there is no way the GNU Mailman project can comply given | our available resources. One outcome could be that our downstream | consumers take over that responsibility. Another is that volunteers in | our community step up with offers to provide us with their expertise, | guidance, Well, hopefully I've provided a little input and am happy to continue to. | and code. We will welcome such volunteers and help ensure | that the legal requirements align with project goals, sensibilities, and | timelines. | | So, if you want to see a GDPR compliant GNU Mailman, please find some | people to help us. I'm less pessimistic. For those that have seen the requirements for the first time, it's entirely understandable to react to it coming-in left-of-field and see it as daunting. However some of the changes required may, with a little consideration, be seen to be not as extensive as first feared; and others, to be a good idea. | | Cheers, | -Barry | leahciM From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Sep 17 14:31:37 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 18 Sep 2017 03:31:37 +0900 Subject: [Mailman-Developers] GDPR In-Reply-To: <01QJ5JK26GXI0005CZ@Encompasserve.org> References: <01QJ5JK26GXI0005CZ@Encompasserve.org> Message-ID: <22974.49033.263692.239256@turnbull.sk.tsukuba.ac.jp> noskcaJ leahciM writes: > | tl;dr > > Too long: The regulation and articles are even longer. > > Didn't Read: But you replied none-the-less ;-) I read EVERYTHING. Above is advice to my readers: "I read, so you don't have to". ;-) I feel sorry for anyone who hangs out in places where "executive summary" is not the normal semantics of "tl;dr"! > There will be installations and use cases that are not > impacted. GDPR is European law and failing to comply with it is not > an option. Of course failing to comply is an option. Whether that leaves our European users with the option to use Mailman is another matter. > Awareness-raising should not, however, result in messengers being > shot. I took aim at the message, which is based on legislation that seems to be difficult to comply with, not at the messenger. > Art.7, Rec.32,42. Do not require that the word "consent" be used > rather than "confirm". However, consent is one of the lawful bases > and being unambiguous that consent is being requested seemed a > reasonable suggestion for the effort required to make the change. If and when we comply, "this is fine". I worry that it looks like an easy change we could do now, but until we are in compliance, which I would suppose includes making it clear at subscription time what the users are "consenting" to, it's a bad idea. At minimum it's unethical, at worst there could be legal liability if users thought they were "consenting" to a GDPR-compliant list configuration. No? > The separate nomain and moderation settings can be kept, along with > their traditional meaning. However Art.18 and particularly > Rec.67. imply that a single "restriction" setting (that sets nomail > + moderation) that shows as "restriction" (and without the reverse > implication) is required. I have no idea what that means? As implemented in Mailman, "nomail" means the list functions as usual but the user gets no mail, and "moderation" means the list functions as usual but the user can't post. Neither protects the user's private data in any way that I understand. Could you explain what is mandated here again? > The existing authentication [by email confirmation] should be > good-enough. It isn't envisaged that authentication be improved > *just* to protect against other people deleting a subscriber's > posts. You miss the point. We don't care what the law mandates, except insofar as our users do. Our responsibility is to our users (the list admins of several flavors) and to their users (the subscribers and posters). Given the guarantees the law describes as its goals, if we say we comply with the law, our users will expect us to implement it as *they* want, not to whatever the minimal level the law prescribes. (Of course if the minimal level exceeds what users expect, then we have to do it anyway, but that's unusual in my experience.) > Please see above proportionate and reasonable measures. Nothing about GDPR is proportionate or reasonable in the context of email and mailing lists. GDPR fails both on grounds of what is reasonable to implement in email, and on grounds of actually protecting users' privacy, as far as I can tell from your description. > That surely is the point rather than some technical distinction > between which elements are owned by whom. It's not "some technical distinction." It's "you can expect that many features of the mailing list will fail to work as they should" -- including the new privacy features mandated by the law. This is what I mean by "nothing is proportionate or reasonable." I don't know about other communication systems that will be affected by this legislation, but as described the folks who wrote the law did not understand or care how it would work with mailing lists and similar protocols (netnews, for example). > | > iii) to hide or disguise identities inadvertently disclosed > | > in list members' posts e.g. from quoting others etc. > | > | That's not reasonable. As above, we don't own the text, the poster > | does. Again, discouraging harvesting is reasonable, but hiding > | identities in text is not a reasonable requirement. > > Some already do this, but don't try to go-beyond replacing '@' > signs etc. Just so. But this does not satisfy "hide or disguise identity" as I understand it. Then there are things like ".signatures" -- but many MUAs and/or users use this feature in a "technically incorrect" way, and of course they are frequently hidden -- and munged -- in quoted passages and the like. I think that users will reasonably expect more than a token effort to implement a hide-or-disguise requirement. >From the point of view of mailing list implementers, GDPR is a hot mess. I agree with Barry, that some of the features that actually protect privacy are going to require a lot of effort to implement. In any case, to serve our users (and theirs) well, specification and implementation should be done by somebody with access to a lawyer to figure out (1) what the regulatory requirements are and (2) whether there might be liability over and above regulation if personal information is leaked in a way that is compliant with the regulation but not expected by the user who "consented" to having their personal data added to the application's database, or if personal information is deleted without the permission of the person involved. IMO "security" means anticipating what can go wrong as far as our expertise permits, and then designing the system to prevent or mitigate it, or warn the user that we were unable to prevent or mitigate. This is going to be difficult at best given the difficulties that even commercial shops are having, according to Barry. Steve -- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From JACKSON at encompasserve.org Sun Sep 17 18:56:17 2017 From: JACKSON at encompasserve.org (noskcaJ leahciM) Date: Sun, 17 Sep 2017 18:56:17 -0400 (EDT) Subject: [Mailman-Developers] GDPR Message-ID: <01QJ8BR3NJ06000NVI@Encompasserve.org> | noskcaJ leahciM writes: | | > | tl;dr | > | > Too long: The regulation and articles are even longer. | > | > Didn't Read: But you replied none-the-less ;-) | | I read EVERYTHING. Above is advice to my readers: "I read, so you | don't have to". ;-) | | I feel sorry for anyone who hangs out in places where "executive | summary" is not the normal semantics of "tl;dr"! | | > There will be installations and use cases that are not | > impacted. GDPR is European law and failing to comply with it is not | > an option. | | Of course failing to comply is an option. Whether that leaves our | European users with the option to use Mailman is another matter. | | > Awareness-raising should not, however, result in messengers being | > shot. | | I took aim at the message, which is based on legislation that seems to | be difficult to comply with, not at the messenger. It's the law. Some of us have to deal with it; like it or not. | | > Art.7, Rec.32,42. Do not require that the word "consent" be used | > rather than "confirm". However, consent is one of the lawful bases | > and being unambiguous that consent is being requested seemed a | > reasonable suggestion for the effort required to make the change. | | If and when we comply, "this is fine". I worry that it looks like an | easy change we could do now, but until we are in compliance, which I | would suppose includes making it clear at subscription time what the | users are "consenting" to, it's a bad idea. At minimum it's | unethical, at worst there could be legal liability if users thought | they were "consenting" to a GDPR-compliant list configuration. No? GDPR merely calls for explicit consent (where appropriate). E.g. "By replying or clicking this link, you are consenting to your email address being subscribed to the closed list xyz. You then receive emails sent by other members subscribed to the list and emails that you send to this list will be sent to the other subscribers of the list. Emails you send to the list will also be archived on a website to which you and other members of the list have a login and password to view." | | > The separate nomain and moderation settings can be kept, along with | > their traditional meaning. However Art.18 and particularly | > Rec.67. imply that a single "restriction" setting (that sets nomail | > + moderation) that shows as "restriction" (and without the reverse | > implication) is required. | | I have no idea what that means? As implemented in Mailman, "nomail" | means the list functions as usual but the user gets no mail, and | "moderation" means the list functions as usual but the user can't | post. Neither protects the user's private data in any way that I | understand. Could you explain what is mandated here again? A picture is worth a thousand words: digraph statechart { size="2.91,2.75"; // Produces 280x70 pixel image rankdir=LR; edge [splines="curved", fontname=Helvetica, fontsize=11 rotate=90]; node [fixedsize=true, style=filled, color=black, fillcolor="#CFE7F5", fontname=Helvetica, fontsize=11]; start [shape=doublecircle, style="filled,bold" color=black, fillcolor=white, width=0.45]; start -> idle; idle -> idle [label="rectify"]; idle -> erased [label="erase"]; end [shape=doublecircle, style="filled,bold" color=black, width=0.4]; erased -> end; idle -> restricted [label="challenge,\nobjection,\nDSAR"]; restricted -> erased [label="erase"]; restricted -> idle [label="rectify"]; restricted -> idle [label="stet"]; } As the graphviz dot graph shows, restricted is a "holding bay" for people's data. As soon as any issue or query is raised about your processing of someone's data, you're supposed to restrict or suspend processing (using) it. You're not to delete it, just stop using it. (That implies send/receive). And, in your system it should show as restricted. | > The existing authentication [by email confirmation] should be | > good-enough. It isn't envisaged that authentication be improved | > *just* to protect against other people deleting a subscriber's | > posts. | | You miss the point. We don't care what the law mandates, except | insofar as our users do. Our responsibility is to our users (the list | admins of several flavors) and to their users (the subscribers and | posters). Given the guarantees the law describes as its goals, if we | say we comply with the law, our users will expect us to implement it | as *they* want, not to whatever the minimal level the law prescribes. | (Of course if the minimal level exceeds what users expect, then we | have to do it anyway, but that's unusual in my experience.) You sound angry. | | > Please see above proportionate and reasonable measures. | | Nothing about GDPR is proportionate or reasonable in the context of | email and mailing lists. GDPR fails both on grounds of what is | reasonable to implement in email, and on grounds of actually | protecting users' privacy, as far as I can tell from your description. | | > That surely is the point rather than some technical distinction | > between which elements are owned by whom. | | It's not "some technical distinction." It's "you can expect that many | features of the mailing list will fail to work as they should" -- | including the new privacy features mandated by the law. This is what | I mean by "nothing is proportionate or reasonable." I don't know | about other communication systems that will be affected by this | legislation, but as described the folks who wrote the law did not | understand or care how it would work with mailing lists and similar | protocols (netnews, for example). Stephen, look at it another way. There have been, and continue to be be gross enfringements. Examples are deliberately confusing people by mixing opt-in and opt-out check boxes presented, and not declaring that details provided for one purpose will be used (or sold) for another. The likes of mailchimp have reacted by pushing-back responsibility on their Customers. See this, written within three weeks ago: https://kb.mailchimp.com/accounts/management/about-mailchimp-and-eu-safe-harbor I hope that you will come-round to seeing that there are changes that MM could make that are proportionate and reasonable. | | > | > iii) to hide or disguise identities inadvertently disclosed | > | > in list members' posts e.g. from quoting others etc. | > | | > | That's not reasonable. As above, we don't own the text, the poster | > | does. Again, discouraging harvesting is reasonable, but hiding | > | identities in text is not a reasonable requirement. | > | > Some already do this, but don't try to go-beyond replacing '@' | > signs etc. | | Just so. But this does not satisfy "hide or disguise identity" as I | understand it. Then there are things like ".signatures" -- but many | MUAs and/or users use this feature in a "technically incorrect" way, | and of course they are frequently hidden -- and munged -- in quoted | passages and the like. I think that users will reasonably expect more | than a token effort to implement a hide-or-disguise requirement. What was called-for was not token. What was called-for was: a) Never add-in stuff that could disclose b) So far as is practical, reasonable and proportionate, help subscribers not to /inadvertently/ leak personal data, whether theirs or someone else's. | | From the point of view of mailing list implementers, GDPR is a hot | mess. I agree with Barry, that some of the features that actually | protect privacy are going to require a lot of effort to implement. | | In any case, to serve our users (and theirs) well, specification and | implementation should be done by somebody with access to a lawyer to | figure out (1) what the regulatory requirements are and (2) whether | there might be liability over and above regulation if personal | information is leaked in a way that is compliant with the regulation | but not expected by the user who "consented" to having their personal | data added to the application's database, or if personal information | is deleted without the permission of the person involved. | | IMO "security" means anticipating what can go wrong as far as our | expertise permits, and then designing the system to prevent or | mitigate it, or warn the user that we were unable to prevent or | mitigate. This is going to be difficult at best given the | difficulties that even commercial shops are having, according to | Barry. Sections 15,16 of the GNU GPL v3.0 state (in all caps) no warranty. MM's developers are neither the controller nor processor in respect of others' use of MM. You can, however, have regard to law regulating use of personal data. leahciM | | Steve | | -- | Associate Professor Division of Policy and Planning Science | http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information | Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba | Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN | From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Sep 18 04:19:51 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 18 Sep 2017 17:19:51 +0900 Subject: [Mailman-Developers] GDPR In-Reply-To: <01QJ8BR3NJ06000NVI@Encompasserve.org> References: <01QJ8BR3NJ06000NVI@Encompasserve.org> Message-ID: <22975.33191.653668.224893@turnbull.sk.tsukuba.ac.jp> noskcaJ leahciM writes: > It's the law. Some of us have to deal with it; like it or not. > GDPR merely calls for explicit consent (where appropriate). E.g. > A picture is worth a thousand words: None of this is helpful to me; as far as I can tell it's not responsive to what I wrote or asked. > You sound angry. Yup. You do not display an understanding of what I wrote, and presume that *I* misunderstand the need and oppose your program. I do understand privacy, and although my values differ from the EU's, I do not oppose dealing with GDPR "someday". The questions are "when" and "with what resources" and "what is an accurate specification of what needs to be done". Your answers are "soon" and "it's easy" and "don't worry about it", and I don't think any of those are valid or useful. > Stephen, look at it another way. I'm already looking at it that way, and telling you it will be expensive to deal with it properly in Mailman, and in similar applications. Mailman currently likely does not have the resources to do so in the next two years. > You can, however, have regard to law regulating use of personal > data. Once again, I don't think that is a useful way to think about software development (don't we all just love DOD-STD-2167A?), and I suspect my feelings are pretty representative of OSS volunteer developers. I assure you (and everybody else who worries about GDPR) that we *will* have regard for *our* (European) *users'* *needs* vs. the law, and *their* *preferences* vs. privacy that may not be defined by law. The mere existence of a law? "Frankly, my dear, I don't give a damn." We'll do what we do, and do it well, when the time comes that we believe it serves our users better than alternative tasks do. For example, better installation procedures and Mailman 2to3 migration automation are *definitely* going to come before GDPR mitigation. Without those, there won't be very many users in Europe (or anywhere) to care about GDPR. Almost certainly, better authentication for the core will come before, too. Otherwise we won't be able to protect from some important threats to privacy, and this is something we've been thinking about for quite a while. And so on. So until GDPR's turn comes, "patches, real legal advice, and money welcome." I REALLY would like to hear from a EU information rights lawyer who's active in the personal data privacy field, or somebody facing imminent application of the law at their Mailman site who can get precise information about how regulators are going to interpret these laws, from lawyers or regulators. Steve -- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From johnl at taugh.com Sun Sep 24 16:33:06 2017 From: johnl at taugh.com (John Levine) Date: 24 Sep 2017 20:33:06 -0000 Subject: [Mailman-Developers] GDPR In-Reply-To: <01QJ5JK26GXI0005CZ@Encompasserve.org> Message-ID: <20170924203306.10102.qmail@ary.lan> In article <01QJ5JK26GXI0005CZ at Encompasserve.org> you write: >| tl;dr > >Too long: The regulation and articles are even longer. > >Didn't Read: But you replied none-the-less ;-) Having been talking to some actual lawyers about GDPR compliance, I find this analysis absurd. Specifically about the right to be forgotten, it means that you have to be able to unsubscribe, i.e., the list operator forgets that you subscribed, but it does not mean that everyone has an arbitrary right to unsay anything they might later regret. I note that the putative analysis of Art. 17 skips over the exceptions which include archiving in the public interest. There might be some tweaks to Mailman to make it easier to document who signed up and when, but for the most part it's not a big deal. Remember that when GDPR analyses talk about mailing lists, what they have in mind are broadcast lists (which is what 99% of lists are), not the discussion lists that Mailman runs. R's, John From JACKSON at encompasserve.org Sun Sep 24 18:29:41 2017 From: JACKSON at encompasserve.org (noskcaJ leahciM) Date: Sun, 24 Sep 2017 18:29:41 -0400 (EDT) Subject: [Mailman-Developers] GDPR Message-ID: <01QJI2VIQKRQ000NVI@Encompasserve.org> You might need to pay for a consultation this time. There is indeed a lawful basis for processing that is archiving in the public interest. But you simply misunderstand those provisions if you believe they give any public or private operator of every archived mail list scope to deny the right to be forgotten. leahciM ----- Original Message ----- In article <01QJ5JK26GXI0005CZ at Encompasserve.org> you write: >| tl;dr > >Too long: The regulation and articles are even longer. > >Didn't Read: But you replied none-the-less ;-) Having been talking to some actual lawyers about GDPR compliance, I find this analysis absurd. Specifically about the right to be forgotten, it means that you have to be able to unsubscribe, i.e., the list operator forgets that you subscribed, but it does not mean that everyone has an arbitrary right to unsay anything they might later regret. I note that the putative analysis of Art. 17 skips over the exceptions which include archiving in the public interest. There might be some tweaks to Mailman to make it easier to document who signed up and when, but for the most part it's not a big deal. Remember that when GDPR analyses talk about mailing lists, what they have in mind are broadcast lists (which is what 99% of lists are), not the discussion lists that Mailman runs. R's, John _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/jackson%40eisner.decus.org Security Policy: http://wiki.list.org/x/QIA9 From johnl at taugh.com Mon Sep 25 02:17:34 2017 From: johnl at taugh.com (John R Levine) Date: 24 Sep 2017 23:17:34 -0700 Subject: [Mailman-Developers] GDPR In-Reply-To: <01QJI2VIQKRQ000NVI@Encompasserve.org> References: <01QJI2VIQKRQ000NVI@Encompasserve.org> Message-ID: On Sun, 24 Sep 2017, noskcaJ leahciM wrote: > You might need to pay for a consultation this time. We did. > There is indeed a lawful basis for processing that is archiving in > the public interest. But you simply misunderstand those provisions > if you believe they give any public or private operator of every > archived mail list scope to deny the right to be forgotten. In view of our competent legal advice, we'll take our chances. Nobody's denying the right to be forgotten, but some of us are familiar with the case law and have some idea what it means. Regards, John Levine, johnl at taugh.com, Taughannock Networks, Trumansburg NY Please consider the environment before reading this e-mail. https://jl.ly