From mark at msapiro.net  Wed Feb  1 00:28:17 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Tue, 31 Jan 2017 21:28:17 -0800
Subject: [Mailman-Developers] Serious issue with Mailman MR 238, PEP 475
Message-ID: <93c42974-7237-4d44-e315-5cceff8083e9@msapiro.net>

It appears that the subject fix has the following nasty effect, at least
with Python 3.4

> Jan 29 00:00:13 2017 (7099) Master watcher caught SIGHUP.  Re-opening log files.
> Jan 29 00:00:13 2017 (8568) in runner caught SIGHUP.  Reopening logs.
> Jan 29 00:00:13 2017 (8566) pipeline runner caught SIGHUP.  Reopening logs.
> Jan 29 00:00:13 2017 (8568) in runner exiting.
> Jan 29 00:00:13 2017 (8566) pipeline runner exiting.
> Jan 29 00:00:13 2017 (8569) retry runner caught SIGHUP.  Reopening logs.
> Jan 29 00:00:13 2017 (8569) retry runner exiting.
...

I.e. when logrotate SIGHUPs the master to reopen logs, all the runners
except for 'rest' exit.

Ordinarily, I'd report this on GitLab, but they are having problems of
their own right now.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From barry at list.org  Wed Feb  1 01:16:21 2017
From: barry at list.org (Barry Warsaw)
Date: Wed, 1 Feb 2017 01:16:21 -0500
Subject: [Mailman-Developers] Serious issue with Mailman MR 238, PEP 475
In-Reply-To: <93c42974-7237-4d44-e315-5cceff8083e9@msapiro.net>
References: <93c42974-7237-4d44-e315-5cceff8083e9@msapiro.net>
Message-ID: <20170201011621.25383730@subdivisions.wooz.org>

On Jan 31, 2017, at 09:28 PM, Mark Sapiro wrote:

>I.e. when logrotate SIGHUPs the master to reopen logs, all the runners
>except for 'rest' exit.

Can you try this?
-Barry

diff --git a/src/mailman/core/runner.py b/src/mailman/core/runner.py
index 9f80941..707ad5e 100644
--- a/src/mailman/core/runner.py
+++ b/src/mailman/core/runner.py
@@ -92,23 +92,24 @@ class Runner:
             signal.SIGINT: 'SIGINT',
             signal.SIGUSR1: 'SIGUSR1',
             }.get(signum, signum)
-        if signum in (signal.SIGTERM, signal.SIGINT, signal.SIGUSR1):
+        if signum == signal.SIGHUP:
+            reopen()
+            rlog.info('%s runner caught SIGHUP.  Reopening logs.', self.name)
+        elif signum in (signal.SIGTERM, signal.SIGINT, signal.SIGUSR1):
             self.stop()
             self.status = signum
             rlog.info('%s runner caught %s.  Stopping.', self.name, signame)
-        elif signum == signal.SIGHUP:
-            reopen()
-            rlog.info('%s runner caught SIGHUP.  Reopening logs.', self.name)
-        # As of Python 3.5, PEP 475 gets in our way.  Runners with long
-        # time.sleep()'s in their _snooze() method (e.g. the retry runner) will
-        # have their system call implemented time.sleep() automatically retried
-        # at the C layer.  The only reliable way to prevent this is to raise an
-        # exception in the signal handler.  The standard run() method
-        # automatically suppresses this exception, meaning, it's caught and
-        # ignored, but effectively breaks the run() loop, which is just what we
-        # want.  Runners which implement their own run() method must be
-        # prepared to catch RunnerInterrupts, usually also ignoring them.
-        raise RunnerInterrupt
+            # As of Python 3.5, PEP 475 gets in our way.  Runners with long
+            # time.sleep()'s in their _snooze() method (e.g. the retry runner)
+            # will have their system call implemented time.sleep()
+            # automatically retried at the C layer.  The only reliable way to
+            # prevent this is to raise an exception in the signal handler.  The
+            # standard run() method automatically suppresses this exception,
+            # meaning, it's caught and ignored, but effectively breaks the
+            # run() loop, which is just what we want.  Runners which implement
+            # their own run() method must be prepared to catch
+            # RunnerInterrupts, usually also ignoring them.
+            raise RunnerInterrupt
 
     def set_signals(self):
         """See `IRunner`."""

From mark at msapiro.net  Wed Feb  1 11:13:38 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Wed, 1 Feb 2017 08:13:38 -0800
Subject: [Mailman-Developers] Serious issue with Mailman MR 238, PEP 475
In-Reply-To: <20170201011621.25383730@subdivisions.wooz.org>
References: <93c42974-7237-4d44-e315-5cceff8083e9@msapiro.net>
 <20170201011621.25383730@subdivisions.wooz.org>
Message-ID: <dc9a6eaf-36ef-2aea-ea29-01b888f75ff9@msapiro.net>

On 01/31/2017 10:16 PM, Barry Warsaw wrote:
> On Jan 31, 2017, at 09:28 PM, Mark Sapiro wrote:
> 
>> I.e. when logrotate SIGHUPs the master to reopen logs, all the runners
>> except for 'rest' exit.
> 
> Can you try this?
> -Barry
> 
> diff --git a/src/mailman/core/runner.py b/src/mailman/core/runner.py
> index 9f80941..707ad5e 100644
...


Yes, that seems good.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20170201/663527f3/attachment.sig>

From barry at list.org  Wed Feb  1 11:18:15 2017
From: barry at list.org (Barry Warsaw)
Date: Wed, 1 Feb 2017 11:18:15 -0500
Subject: [Mailman-Developers] Serious issue with Mailman MR 238, PEP 475
In-Reply-To: <dc9a6eaf-36ef-2aea-ea29-01b888f75ff9@msapiro.net>
References: <93c42974-7237-4d44-e315-5cceff8083e9@msapiro.net>
 <20170201011621.25383730@subdivisions.wooz.org>
 <dc9a6eaf-36ef-2aea-ea29-01b888f75ff9@msapiro.net>
Message-ID: <20170201111815.3803fa2d@subdivisions.wooz.org>

On Feb 01, 2017, at 08:13 AM, Mark Sapiro wrote:

>Yes, that seems good.

Thanks Mark.  When GitLab returns, I'll get that patch landed.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20170201/a8cbbb9b/attachment.sig>

From diptanshuj at gmail.com  Wed Feb  1 06:57:36 2017
From: diptanshuj at gmail.com (Diptanshu Jamgade)
Date: Wed, 01 Feb 2017 11:57:36 +0000
Subject: [Mailman-Developers] Development Enviromnent setup
Message-ID: <CA+tqw3MyyAA+0bF=nWgCyjwDZTrZdv-Wmye083r6OO12akh8GQ@mail.gmail.com>

Hello,
I have clonned the mailman gitlab repo(forked version) and created a
python3 virtualenv using `virtualenv p- python3 <virtualenv dir>`. Then I
activated the virtualenv and did `python3 setup.py develop`. On completion
I did `mailman info -v`. This returned me with this (error)[
http://pastebin.com/wLzDQwAu], Kindly help.
-- 
Diptanshu Jamgade
13EC35021
4th Year Undergraduate Student
Department of Electronics and Electrical Communication Engineering, IIT
Kharagpur
Ph. No.- +91-77-97-436-712

From barry at list.org  Wed Feb  1 12:09:55 2017
From: barry at list.org (Barry Warsaw)
Date: Wed, 1 Feb 2017 12:09:55 -0500
Subject: [Mailman-Developers] Development Enviromnent setup
In-Reply-To: <CA+tqw3MyyAA+0bF=nWgCyjwDZTrZdv-Wmye083r6OO12akh8GQ@mail.gmail.com>
References: <CA+tqw3MyyAA+0bF=nWgCyjwDZTrZdv-Wmye083r6OO12akh8GQ@mail.gmail.com>
Message-ID: <20170201120955.5bd9619e@subdivisions.wooz.org>

On Feb 01, 2017, at 11:57 AM, Diptanshu Jamgade wrote:

>I have clonned the mailman gitlab repo(forked version) and created a
>python3 virtualenv using `virtualenv p- python3 <virtualenv dir>`. Then I
>activated the virtualenv and did `python3 setup.py develop`. On completion
>I did `mailman info -v`. This returned me with this (error)[
>http://pastebin.com/wLzDQwAu], Kindly help.

That makes me think that your Python installation is broken.  Can you tell
us more about what OS you're on, what version of Python you're using, and how
Python was built/installed on it?

FWIW, I tried this on Ubuntu 17.04 (devel) and it works fine.  I build my venv
a little differently, but that shouldn't matter:

$ python3 -m venv /tmp/mm3
$ source /tmp/mm3/bin/activate
$ python setup.py develop
$ mailman info -v
<expected output>

I'll also mention that very often I don't create a separate venv, but just use
the ones that tox builds, since I almost always have them around by running
the test suite.  If you want to build those without running the tests, you can
do something like this:

$ tox -e py35-nocov --notest -r
$ .tox/py35-nocov/bin/mailman info -v

You don't need to activate the venv to use it.

All that said, if you can't import _sqlite3, then your Python installation is
messed up.  That's a stdlib module so it should always be around.

Cheers,
-Barry

From mark at msapiro.net  Wed Feb  1 12:11:42 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Wed, 1 Feb 2017 09:11:42 -0800
Subject: [Mailman-Developers] Development Enviromnent setup
In-Reply-To: <CA+tqw3MyyAA+0bF=nWgCyjwDZTrZdv-Wmye083r6OO12akh8GQ@mail.gmail.com>
References: <CA+tqw3MyyAA+0bF=nWgCyjwDZTrZdv-Wmye083r6OO12akh8GQ@mail.gmail.com>
Message-ID: <73c5093c-58af-2aa0-23ea-4ee0e6f04a2b@msapiro.net>

On 02/01/2017 03:57 AM, Diptanshu Jamgade wrote:
> Hello,
> I have clonned the mailman gitlab repo(forked version) and created a
> python3 virtualenv using `virtualenv p- python3 <virtualenv dir>`. Then I
> activated the virtualenv and did `python3 setup.py develop`. On completion
> I did `mailman info -v`. This returned me with this (error)[
> http://pastebin.com/wLzDQwAu], Kindly help.
> 


With your Python3 venv active, do

pip install pysqlite

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From turnbull.stephen.fw at u.tsukuba.ac.jp  Wed Feb  1 21:50:28 2017
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Thu, 2 Feb 2017 11:50:28 +0900
Subject: [Mailman-Developers] Development Enviromnent setup
In-Reply-To: <20170201120955.5bd9619e@subdivisions.wooz.org>
References: <CA+tqw3MyyAA+0bF=nWgCyjwDZTrZdv-Wmye083r6OO12akh8GQ@mail.gmail.com>
 <20170201120955.5bd9619e@subdivisions.wooz.org>
Message-ID: <22674.40564.757489.906461@turnbull.sk.tsukuba.ac.jp>

Barry Warsaw writes:

 > All that said, if you can't import _sqlite3, then your Python
 > installation is messed up.  That's a stdlib module so it should
 > always be around.

But distros!  Probably @ncoghlan_dev's fault. ;-)



From turnbull.stephen.fw at u.tsukuba.ac.jp  Wed Feb  1 21:52:05 2017
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Thu, 2 Feb 2017 11:52:05 +0900
Subject: [Mailman-Developers]  Development Enviromnent setup
In-Reply-To: <CA+tqw3MyyAA+0bF=nWgCyjwDZTrZdv-Wmye083r6OO12akh8GQ@mail.gmail.com>
References: <CA+tqw3MyyAA+0bF=nWgCyjwDZTrZdv-Wmye083r6OO12akh8GQ@mail.gmail.com>
Message-ID: <22674.40661.22774.829001@turnbull.sk.tsukuba.ac.jp>

Diptanshu Jamgade writes:

 > I did `mailman info -v`. This returned me with this (error)[
 > http://pastebin.com/wLzDQwAu], Kindly help.

In the future, when you don't have any idea what your problem is,
please report as a bug at GitLab (right now GitLab is down I hear?),
or attach the trace to your mail rather than using pastebin, so that
it is archived and searchable.  No action needed this time.

pastebin is an excellent service for on-the-fly discussions, such as
draft patches, but by design it is not archival.  Your report is
wasted for others searching for a solution if the traceback goes away,
and probably isn't very searchable at pastebin even while it's stored
there.

Thanks in advance!

Steve

From mark at msapiro.net  Thu Feb  9 10:38:51 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Thu, 9 Feb 2017 07:38:51 -0800
Subject: [Mailman-Developers] Fwd: Mailman | The unsubscribe process using
 the -leave address does not work because of an `assert` in the code (#310)
In-Reply-To: <issue_4316886@gitlab.com>
References: <issue_4316886@gitlab.com>
Message-ID: <ff984dc2-1a18-399d-f468-baa548564aa2@msapiro.net>

@abompard I don't know what's going here. I received the notice below
from GitLab. I tried to go there to comment that this looks like a
duplicate or at least very similar to
<https://gitlab.com/mailman/mailman/issues/294> which was fixed about 3
weeks ago, but GitLab gives me a 404 and says there is no issue #310.


-------- Forwarded Message --------
Subject:  Mailman | The unsubscribe process using the -leave
address does not work because of an `assert` in the code (#310)
Date: 	Thu, 09 Feb 2017 10:31:52 +0000
From: 	Aur?lien Bompard <gitlab at mg.gitlab.com>
Reply-To: 	mailman / Mailman
<incoming+36519312af637e6be0deecee3e4f348b at gitlab.com>
To: 	mark at msapiro.net



GitLab

Here's the situation that I have confirmed on my servers:

  * a members sends an email to the |-leave| address to unsubscribe
  * they recieve a confirmation email
  * they reply to this email-
  * |SubscriptionManager.confirm()| get called, the
    |UnSubscriptionWorkflow| gets restored and run, the goodbye email
    gets sent to the user.
  * at the end of the |UnSubscriptionWorkflow| in
    |._step_do_unsubscription()|, the |.member| attribute gets set to None
  * the |.confirm()| command returns |workflow.token|,
    |workflow.token_owner|, and |workflow.member| to the caller
  * on line 56, |commands.eml_confirm.Confirm| has an |assert member is
    not None|, which causes an AssertionError. As a result, the database
    transaction is rollbacked and the unsubscription is forgotten.

I suspect this |assert| comes from the time where the -confirm command
was only used for subscription confirmations, not unsubscription
confirmation. Removing it solves the problem.

I think this is important and simple enough to fix it for 3.1.

?
Reply to this email directly or view it on GitLab
<https://gitlab.com/mailman/mailman/issues/310>.


-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From barry at list.org  Thu Feb  9 11:28:54 2017
From: barry at list.org (Barry Warsaw)
Date: Thu, 9 Feb 2017 11:28:54 -0500
Subject: [Mailman-Developers] Fwd: Mailman | The unsubscribe process
 using the -leave address does not work because of an `assert` in the code
 (#310)
In-Reply-To: <ff984dc2-1a18-399d-f468-baa548564aa2@msapiro.net>
References: <issue_4316886@gitlab.com>
 <ff984dc2-1a18-399d-f468-baa548564aa2@msapiro.net>
Message-ID: <20170209112854.77b3f649@subdivisions.wooz.org>

On Feb 09, 2017, at 07:38 AM, Mark Sapiro wrote:

>@abompard I don't know what's going here. I received the notice below from
>GitLab. I tried to go there to comment that this looks like a duplicate or at
>least very similar to <https://gitlab.com/mailman/mailman/issues/294> which
>was fixed about 3 weeks ago, but GitLab gives me a 404 and says there is no
>issue #310.

I'm not positive, but I have a theory.  GL says there are only 309 issues of
both closed and open state on the mailman project.  It's possible that this
one got caught up in their database incident:

https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/

and the email was queued up before then, since it's date of 09-Feb-2017 is
well after the incident was resolved.  It's known that about 6 hours of
issues, MRs, comments, etc. were irretrievably lost.

I did move an issue from mailman to postorius, but I believe that what happens
in that case is that a new issue is created in the target project (with
duplicate contents but a different issue number in sequence with the target),
while the source project's issue just gets closed.  I can't find that one
right now to confirm.

Anyway, Aurelien would probably need to say whether his issue falls within the
timeline of the GL database snafu or not.  I think there's no option but to
try to re-submit the issue if that's the case.

Cheers,
-Barry

From mark at msapiro.net  Thu Feb  9 11:39:19 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Thu, 9 Feb 2017 08:39:19 -0800
Subject: [Mailman-Developers] Fwd: Mailman | The unsubscribe process
 using the -leave address does not work because of an `assert` in the code
 (#310)
In-Reply-To: <20170209112854.77b3f649@subdivisions.wooz.org>
References: <issue_4316886@gitlab.com>
 <ff984dc2-1a18-399d-f468-baa548564aa2@msapiro.net>
 <20170209112854.77b3f649@subdivisions.wooz.org>
Message-ID: <615f9d05-ebf8-6007-064d-3504603909a7@msapiro.net>

On 02/09/2017 08:28 AM, Barry Warsaw wrote:
> 
> I'm not positive, but I have a theory.  GL says there are only 309 issues of
> both closed and open state on the mailman project.  It's possible that this
> one got caught up in their database incident:


Except that Issue #309 was opened yesterday, well after the GL database
incident. So I would think that Issue #310 would have to have been
created after that.

It seems that something erased or lost issue #310 after it was posted,
but I don't think it's related to the database rollback.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From barry at list.org  Thu Feb  9 12:05:03 2017
From: barry at list.org (Barry Warsaw)
Date: Thu, 9 Feb 2017 12:05:03 -0500
Subject: [Mailman-Developers] Fwd: Mailman | The unsubscribe process
 using the -leave address does not work because of an `assert` in the code
 (#310)
In-Reply-To: <615f9d05-ebf8-6007-064d-3504603909a7@msapiro.net>
References: <issue_4316886@gitlab.com>
 <ff984dc2-1a18-399d-f468-baa548564aa2@msapiro.net>
 <20170209112854.77b3f649@subdivisions.wooz.org>
 <615f9d05-ebf8-6007-064d-3504603909a7@msapiro.net>
Message-ID: <20170209120503.72ec24c3@subdivisions.wooz.org>

On Feb 09, 2017, at 08:39 AM, Mark Sapiro wrote:

>Except that Issue #309 was opened yesterday, well after the GL database
>incident. So I would think that Issue #310 would have to have been
>created after that.
>
>It seems that something erased or lost issue #310 after it was posted,
>but I don't think it's related to the database rollback.

Oh that's scary then!  I'll see if I can track someone at Gitlab down.

-Barry

From vsukeeshbabu at gmail.com  Sat Feb 11 03:54:11 2017
From: vsukeeshbabu at gmail.com (Sukeesh V)
Date: Sat, 11 Feb 2017 14:24:11 +0530
Subject: [Mailman-Developers] Google Summer of Code
Message-ID: <CAMPbAvAE35OUjh881hLHkvK7LK48wZ0Vq1yQLPxCJ-Obujy8iw@mail.gmail.com>

Hi,

Myself Sukeesh, an undergraduate student. I am interested in contributing
to mailman. I have setup the development enviroment as mentioned here
<https://wiki.list.org/DEV/SetupDevEnvironment>.

When I tried to run postorius django server. I got this error ( Screenshot
attached ).  I am running on Arch Linux.

Can someone please help?

Thanks,
Sukeesh
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot_2017-02-11_14-21-19.png
Type: image/png
Size: 146080 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20170211/c186afa3/attachment-0001.png>

From mark at msapiro.net  Sat Feb 11 10:08:23 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Sat, 11 Feb 2017 07:08:23 -0800
Subject: [Mailman-Developers] Google Summer of Code
In-Reply-To: <CAMPbAvAE35OUjh881hLHkvK7LK48wZ0Vq1yQLPxCJ-Obujy8iw@mail.gmail.com>
References: <CAMPbAvAE35OUjh881hLHkvK7LK48wZ0Vq1yQLPxCJ-Obujy8iw@mail.gmail.com>
Message-ID: <3b643de4-8c6b-881a-f47b-21ad05b884ae@msapiro.net>

On 02/11/2017 12:54 AM, Sukeesh V wrote:
> 
> When I tried to run postorius django server. I got this error ( Screenshot
> attached ).  I am running on Arch Linux.


Postorius can't connect to Mailman core. Mailman core must be running
and the port defined in the [webservice] section of var/etc/mailman.cfg
must match that in the definition of MAILMAN_REST_API_URL in whatever
Postorius settings file you're using.	

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From mark at msapiro.net  Sun Feb 12 18:58:23 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Sun, 12 Feb 2017 15:58:23 -0800
Subject: [Mailman-Developers] What characters should be allowed in listnames
Message-ID: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>

I have just files <https://gitlab.com/mailman/mailman/issues/311> which
is about Mailman core allowing list's to be created with a slash '/' in
the name.

Core validates listnames by ensuring the fqdn_listname is a valid email
address. This is too liberal. RFC 5321 allows many characters in the
local part of a list name. We don't allow quite all of them, but we
allow this set [-0-9a-z!#$%&'*+./=?@_`{}~].

Since list names form parts of a URI, both in Postorius/HyperKitty and
in the Core's REST API, it is clear that characters that will cause
problems there should not be allowed. These include [#%&/?] and maybe
others. Additionally, I don't think we want @ in an email address local
part and + and = might cause problems with VERP which whittles it down
to [-0-9a-z!$'*._`{}~], but I'm thinking of being even more conservative
and going with just [-0-9a-z._].

I don't intend to change the email address validation except maybe to
remove @, but the code is such that an address with multiple @ won't
validate anyway.

I'd like feedback on this. What are your thoughts on what characters
should be allowed in list names?

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From barry at list.org  Sun Feb 12 20:27:49 2017
From: barry at list.org (Barry Warsaw)
Date: Sun, 12 Feb 2017 20:27:49 -0500
Subject: [Mailman-Developers] What characters should be allowed in
 listnames
In-Reply-To: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
References: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
Message-ID: <20170212202749.49fc5dcb@subdivisions.wooz.org>

On Feb 12, 2017, at 03:58 PM, Mark Sapiro wrote:

>Core validates listnames by ensuring the fqdn_listname is a valid email
>address. This is too liberal. RFC 5321 allows many characters in the
>local part of a list name. We don't allow quite all of them, but we
>allow this set [-0-9a-z!#$%&'*+./=?@_`{}~].
>
>Since list names form parts of a URI, both in Postorius/HyperKitty and
>in the Core's REST API, it is clear that characters that will cause
>problems there should not be allowed. These include [#%&/?] and maybe
>others.

I suppose if we did continue to allow them, they would have to be escaped in
the URL.  I'm not sure how much that helps, or even whether it should be part
of our decision to allow them or not.

>Additionally, I don't think we want @ in an email address local
>part and + and = might cause problems with VERP which whittles it down
>to [-0-9a-z!$'*._`{}~], but I'm thinking of being even more conservative
>and going with just [-0-9a-z._].

I think it's entirely reasonable for Mailman to narrow the list of allowable
characters in the local part of list names.  We already make some opinionated
decisions about allowable email addresses; for example, we support
case-preserving, case-insensitive addresses so we treat bob at example.com and
BOB at example.com as identical.

I'm amenable to the conservative set you propose (obviously, case
insensitive), although I have some concerns about how dots in the local part
would interfere with any List-ID operations.  E.g. foo.bar at example.com becomes
List-ID: foo.bar.example.com.  As an identifier with comparison rules
according to RFC 2919 I think it's fine (it just has to be unique).  I'm not
sure whether in practice it would cause problems with the core.

The other question is whether we're unfairly closing the door on i18n list
names.  OTOH, we haven't yet had any requests for that afaict.

>I don't intend to change the email address validation except maybe to
>remove @, but the code is such that an address with multiple @ won't
>validate anyway.
>
>I'd like feedback on this. What are your thoughts on what characters
>should be allowed in list names?

Certainly some narrowing is appropriate.  We could just clamp it down as you
suggest, understanding that there may already be lists in existence that use
the more liberal character set, and acknowledging that we may want to relax
the set based on future bug reports.

What about this: come up with an absolute black list set, e.g. the ones that
will break Mailman.  Come up with a second set of discouraged but allowed
characters, and a third set which is the narrow list you propose.  Then make
the allowable set configurable, except that the black list characters are
always disallowed.  Now, that might be too complicated, so I'm also fine with
making it narrow now, and letting the set relax based on user feedback.

Cheers,
-Barry

From mark at msapiro.net  Sun Feb 12 22:23:28 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Sun, 12 Feb 2017 19:23:28 -0800
Subject: [Mailman-Developers] What characters should be allowed in
 listnames
In-Reply-To: <20170212202749.49fc5dcb@subdivisions.wooz.org>
References: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
 <20170212202749.49fc5dcb@subdivisions.wooz.org>
Message-ID: <c5776af4-cbdb-b880-2250-312df61241ec@msapiro.net>

On 02/12/2017 05:27 PM, Barry Warsaw wrote:
> 
> Certainly some narrowing is appropriate.  We could just clamp it down as you
> suggest, understanding that there may already be lists in existence that use
> the more liberal character set, and acknowledging that we may want to relax
> the set based on future bug reports.
> 
> What about this: come up with an absolute black list set, e.g. the ones that
> will break Mailman.  Come up with a second set of discouraged but allowed
> characters, and a third set which is the narrow list you propose.  Then make
> the allowable set configurable, except that the black list characters are
> always disallowed.  Now, that might be too complicated, so I'm also fine with
> making it narrow now, and letting the set relax based on user feedback.


Thanks Barry. FWIW, MM 2.1 has an ACCEPTABLE_LISTNAME_CHARACTERS config
setting which defaults to '[-+_.=a-z0-9]'. I don't really like the + and
= in that list because of their possible interaction with VERP. I have a
WIP MR at <https://gitlab.com/mailman/mailman/merge_requests/248> that
allows only [-_.a-z0-9] (IGNORECASE) and has no config override.

The narrow, overridable config combined with a blacklist or some kind of
limitation on the overrides would be the most flexible. I'll look at
adding that to the MR. Basically, I'm thinking of a fixed list of
allowed characters which is liberal, testing that first and if that
passes, testing the config set.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From turnbull.stephen.fw at u.tsukuba.ac.jp  Tue Feb 14 13:52:46 2017
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Wed, 15 Feb 2017 03:52:46 +0900
Subject: [Mailman-Developers] What characters should be allowed in listnames
In-Reply-To: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
References: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
Message-ID: <22691.20990.436506.905909@turnbull.sk.tsukuba.ac.jp>

Mark Sapiro writes:

 > I'd like feedback on this. What are your thoughts on what characters
 > should be allowed in list names?

Uh, RFC 6532 ....

Probably that can wait for when we actually support it :-), but while
you're doing this we should (= I should when life gets sane ;-) make
sure that whatever that restriction is, is encapsulated in one place.

Steve


From mark at msapiro.net  Tue Feb 14 21:33:33 2017
From: mark at msapiro.net (Mark Sapiro)
Date: Tue, 14 Feb 2017 18:33:33 -0800
Subject: [Mailman-Developers] What characters should be allowed in
 listnames
In-Reply-To: <22691.20990.436506.905909@turnbull.sk.tsukuba.ac.jp>
References: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
 <22691.20990.436506.905909@turnbull.sk.tsukuba.ac.jp>
Message-ID: <54758298-243f-3d20-4c01-1aa43866d124@msapiro.net>

On 02/14/2017 10:52 AM, Stephen J. Turnbull wrote:
> Mark Sapiro writes:
> 
>  > I'd like feedback on this. What are your thoughts on what characters
>  > should be allowed in list names?
> 
> Uh, RFC 6532 ....


That doesn't really address my question. That has to do with
internationalized email addresses. Granted the listname must be a valid
local part of an email address, but that doesn't mean every valid local
part has to be a valid list name.

In particular, the issue that raises this question is a list name
containing a slash. while 'my/list at example.com' is a legal email
address, https://lists.example.com/mailman3/lists/my/list.example.com/
is not a URL which will work in Postorius.


> Probably that can wait for when we actually support it :-), but while
> you're doing this we should (= I should when life gets sane ;-) make
> sure that whatever that restriction is, is encapsulated in one place.


I think I've done that in my current MR on this at
<https://gitlab.com/mailman/mailman/merge_requests/248> which implements
a [mailman] config setting described in schema.cfg as

> # Specify what characters are allowed in list names.  Characters outside of
> # the class [-_.+=!$*{}~0-9a-z] matched case insensitively are never allowed,
> # but this specifies a subset as the only allowable characters.
> listname_chars: [-_.0-9a-z]

The only questions are whether these are the right sets for the "outside
this are not allowed" class and the default listname_chars class.


-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From terri at toybox.ca  Tue Feb 14 23:33:42 2017
From: terri at toybox.ca (Terri Oda)
Date: Wed, 15 Feb 2017 10:03:42 +0530
Subject: [Mailman-Developers] What characters should be allowed in
 listnames
In-Reply-To: <54758298-243f-3d20-4c01-1aa43866d124@msapiro.net>
References: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
 <22691.20990.436506.905909@turnbull.sk.tsukuba.ac.jp>
 <54758298-243f-3d20-4c01-1aa43866d124@msapiro.net>
Message-ID: <504D7BD2-4A1A-49FF-9C12-13FE7CF5E617@toybox.ca>



On February 15, 2017 8:03:33 AM GMT+05:30, Mark Sapiro <mark at msapiro.net> wrote:
>On 02/14/2017 10:52 AM, Stephen J. Turnbull wrote:
>> Mark Sapiro writes:
>> 
>>  > I'd like feedback on this. What are your thoughts on what
>characters
>>  > should be allowed in list names?
>> 
>> Uh, RFC 6532 ....
>
>
>That doesn't really address my question. That has to do with
>internationalized email addresses. Granted the listname must be a valid
>local part of an email address, but that doesn't mean every valid local
>part has to be a valid list name.
>
>In particular, the issue that raises this question is a list name
>containing a slash. while 'my/list at example.com' is a legal email
>address, https://lists.example.com/mailman3/lists/my/list.example.com/
>is not a URL which will work in Postorius.
>
My gut feeling is that allowing slashes is going to lead to confusing side effects with mail clients and even potential security bugs down the road, and we should just not do it unless there's a clear need to support slashes.

But that's really just a feeling based on exploit experience, where "be conservative in what you expect" is best practice. It doesn't speak to what the standards say or what's doable.

 Terri


From turnbull.stephen.fw at u.tsukuba.ac.jp  Sun Feb 19 01:12:52 2017
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Sun, 19 Feb 2017 15:12:52 +0900
Subject: [Mailman-Developers] What characters should be allowed in
 listnames
In-Reply-To: <54758298-243f-3d20-4c01-1aa43866d124@msapiro.net>
References: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
 <22691.20990.436506.905909@turnbull.sk.tsukuba.ac.jp>
 <54758298-243f-3d20-4c01-1aa43866d124@msapiro.net>
Message-ID: <22697.14180.824752.392728@turnbull.sk.tsukuba.ac.jp>

Mark Sapiro writes:

 > That doesn't really address my question. That has to do with
 > internationalized email addresses. Granted the listname must be a valid
 > local part of an email address, but that doesn't mean every valid local
 > part has to be a valid list name.

The problem that I thought we may face is internationalized mailboxes
and domain names *are still ASCII* which encodes Unicode.

OK, I looked it up, and I was almost certainly wrong.  The relevant
RFCs are actually 6531 (SMTPUTF8 extension), 5890 (IDNA), and 3492
(Punycode).  IDNA allows "U-labels" (UTF-8), which we can gracefully
extend to, and "A-labels" (encoded using Punycode, which uses the
ASCII repertoire).  AFAICT (haven't really looked carefully), Punycode
uses only letters, digits, and the hyphen ("-").

So I withdraw the comment.

Thanks for your work on this, Mark!

Steve

From su at elandnews.com  Sun Feb 19 03:30:31 2017
From: su at elandnews.com (SM)
Date: Sun, 19 Feb 2017 00:30:31 -0800
Subject: [Mailman-Developers] What characters should be allowed in
 listnames
In-Reply-To: <22697.14180.824752.392728@turnbull.sk.tsukuba.ac.jp>
References: <32a74e31-855d-4398-1cc1-36e036af5bef@msapiro.net>
 <22691.20990.436506.905909@turnbull.sk.tsukuba.ac.jp>
 <54758298-243f-3d20-4c01-1aa43866d124@msapiro.net>
 <22697.14180.824752.392728@turnbull.sk.tsukuba.ac.jp>
Message-ID: <6.2.5.6.2.20170219002126.09e86588@elandnews.com>

Hi Stephen,
At 10:12 PM 2/18/2017, Stephen J. Turnbull wrote:
>The problem that I thought we may face is internationalized mailboxes
>and domain names *are still ASCII* which encodes Unicode.
>
>OK, I looked it up, and I was almost certainly wrong.  The relevant
>RFCs are actually 6531 (SMTPUTF8 extension), 5890 (IDNA), and 3492
>(Punycode).  IDNA allows "U-labels" (UTF-8), which we can gracefully
>extend to, and "A-labels" (encoded using Punycode, which uses the
>ASCII repertoire).  AFAICT (haven't really looked carefully), Punycode
>uses only letters, digits, and the hyphen ("-").
>
>So I withdraw the comment.

RFC 5890 can be used for the domain name part.  The issue would be 
about what to do for the local part.  If I recall correctly, the 
question was not discussed as part of RFC 6531.  I suggest taking a 
look at RFC 7564.

Regards,
-sm 


From vbnlohani at gmail.com  Wed Feb 22 12:36:29 2017
From: vbnlohani at gmail.com (Vaibhav Lohani)
Date: Wed, 22 Feb 2017 23:06:29 +0530
Subject: [Mailman-Developers] Discourse Integration
Message-ID: <CAD8Kh54WnHh8ufBQw31UcdkCxWRGaNWGAyfRY16gFUR0-QT8Ew@mail.gmail.com>

Hello All,

There was a mention of integrating mailman 3 with discourse some times
back. I am interested in working on it. Is someone else interested in it or
already working on it ?

Regards,
Vaibhav Lohani