From 266464 at bugs.launchpad.net Wed Nov 16 15:57:53 2016 From: 266464 at bugs.launchpad.net (Uli) Date: Wed, 16 Nov 2016 20:57:53 -0000 Subject: [Bug 266464] Re: Subscriber "disappears" after subscription References: <20080905193221.27052.50410.launchpad@forster.canonical.com> Message-ID: <20161116205754.17359.97132.malone@soybean.canonical.com> The bug was not completly fixed in Mailman 2.1.15. self.__timestamp is set after the loading AND parsing of the file. With large files the parsing may need 1-2 seconds and in the meantime a new file, but older then the self.__timestamp set after the parsing ist written: (1) Nov 16 21:02:54 2016 (57765) Try-Load config file /var/lib/mailman/lists/XY/config.pck -> Load (2) Nov 16 21:02:54 2016 (57765) 1479326518 mtime(/var/lib/mailman/lists/XY/config.pck) -> Check mtime (3) Nov 16 21:02:54 2016 (57765) 0 (self.__timestamp) -> self.__timestamp is not set -> Load-File (4) Nov 16 21:02:54 2016 (57765) 1479326574 (Time before Load of config.pck) -> Timestamp int(time.time()) before the loading of the pickle file. (5) Nov 16 21:02:55 2016 (57765) 1479326575 (Time after Loading unpickling File = self.__timestamp) -> Timestamp after the Loading of the pickle file. The unpickling needs some time due to large List with 200.000 subscribers The parsing needs 1 second, timestamps differ. (6) Nov 16 21:02:55 2016 (57765) Try-Loaded config file /var/lib/mailman/lists/neuerscheinungen/config.pck -> Reload/Check Pickle-File (7) Nov 16 21:02:55 2016 (57765) 1479326574 mtime(/var/lib/mailman/lists/XY/config.pck) -> Check mtime (file is older than self.__timestamp), but nervertheless changed after the Loading in (5) (8) Nov 16 21:02:55 2016 (57765) 1479326575 SelfTimestamp -> Wrong Data saved -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/266464 Title: Subscriber "disappears" after subscription To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/266464/+subscriptions From mark at msapiro.net Wed Nov 16 19:00:21 2016 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 17 Nov 2016 00:00:21 -0000 Subject: [Bug 266464] Re: Subscriber "disappears" after subscription References: <20080905193221.27052.50410.launchpad@forster.canonical.com> Message-ID: <20161117000022.8516.18502.malone@gac.canonical.com> Thanks for the report. Fixed at http://bazaar.launchpad.net/~mailman- coders/mailman/2.1/revision/1683 ** Changed in: mailman Status: Fix Released => Fix Committed ** Changed in: mailman Milestone: 2.1.15 => 2.1.24 -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/266464 Title: Subscriber "disappears" after subscription To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/266464/+subscriptions From 1251495 at bugs.launchpad.net Fri Nov 18 04:53:29 2016 From: 1251495 at bugs.launchpad.net (ChristianEhrhardt) Date: Fri, 18 Nov 2016 09:53:29 -0000 Subject: [Bug 1251495] Re: Lists with topics enabled can throw unexpected keyword argument 'Delete' exception. References: <20131115020913.18410.66201.malonedeb@wampee.canonical.com> Message-ID: <20161118095329.5218.36164.malone@chaenomeles.canonical.com> Fixed upstream, but needs backport to Trusty - all other releases are already including the fix. The diff this is about is: @@ -71,7 +73,7 @@ if hits: msgdata['topichits'] = hits.keys() change_header('X-Topics', NLTAB.join(hits.keys()), - mlist, msg, msgdata, Delete=False) + mlist, msg, msgdata, delete=False) ** Also affects: mailman (Ubuntu) Importance: Undecided Status: New ** Also affects: mailman (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: mailman (Ubuntu) Status: New => Fix Released ** Changed in: mailman (Ubuntu Trusty) Status: New => Triaged ** Changed in: mailman (Ubuntu Trusty) Importance: Undecided => Medium ** Changed in: mailman (Ubuntu Trusty) Importance: Medium => High -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1251495 Title: Lists with topics enabled can throw unexpected keyword argument 'Delete' exception. To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1251495/+subscriptions From 1251495 at bugs.launchpad.net Fri Nov 18 04:56:43 2016 From: 1251495 at bugs.launchpad.net (ChristianEhrhardt) Date: Fri, 18 Nov 2016 09:56:43 -0000 Subject: [Bug 1251495] Re: Lists with topics enabled can throw unexpected keyword argument 'Delete' exception. References: <20131115020913.18410.66201.malonedeb@wampee.canonical.com> Message-ID: <20161118095643.8842.42164.malone@gac.canonical.com> FYI - it was reported that: "This is a high importance bug that can cause email to "disappear" since Mailman gets an exception and just puts the email into the shunt directory." by Phillip Colmer -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1251495 Title: Lists with topics enabled can throw unexpected keyword argument 'Delete' exception. To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1251495/+subscriptions From futatuki at poem.co.jp Sat Nov 19 12:07:50 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Sat, 19 Nov 2016 17:07:50 -0000 Subject: [Bug 1643210] [NEW] 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char Message-ID: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Public bug reported: If from_is_list feature is used, From: header's `realname' field is composed by original realname and translation of '%(realname)s via %(lrn)s' which may contain non-ascii character. The realname field is encoded before compose if nessesary, but translation part is not. So From header may contain raw non-ascii character. To fix this, do RFC 2047 encode after compose. (There is another bug..., if servers language setting and mail list preferred language is differ, translation has taken from servers language, not from mail list one. Attached patch contains fix of it) ** Affects: mailman Importance: Undecided Status: New ** Attachment added: "CookHeaders.py.diff.txt" https://bugs.launchpad.net/bugs/1643210/+attachment/4780159/+files/CookHeaders.py.diff.txt ** Branch linked: lp:mailman/2.1 -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From futatuki at poem.co.jp Sat Nov 19 12:33:17 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Sat, 19 Nov 2016 17:33:17 -0000 Subject: [Bug 1643215] [NEW] Some Handlers get translation of language unsupposed Message-ID: <20161119173317.8910.64826.malonedeb@gac.canonical.com> Public bug reported: Take a look at Mailman/Handlers/*.py, some of handlers seems to get translation of server language when mail list preferred language or user preferred language context is needed, for lack of set_language() (or set_translation). (Some handlers like Hold.py seems to switch language context correctly.) I found it in CookHeaders.py at first, while I try to fix bug #1643210 (and patch for fix attached there). I cannot determine which of those is correct and is incorrect, so I report it incompletely, sorry. ** Affects: mailman Importance: Undecided Status: New ** Branch linked: lp:mailman/2.1 -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643215 Title: Some Handlers get translation of language unsupposed To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643215/+subscriptions From mark at msapiro.net Sat Nov 19 14:55:53 2016 From: mark at msapiro.net (Mark Sapiro) Date: Sat, 19 Nov 2016 19:55:53 -0000 Subject: [Bug 1643215] Re: Some Handlers get translation of language unsupposed References: <20161119173317.8910.64826.malonedeb@gac.canonical.com> Message-ID: <20161119195554.8577.44108.malone@soybean.canonical.com> Whenever there is a list context, IncomingRunner (actually Runner of which IncomingRunner is a sub-class) sets the i18n language to the message sender's preferred language for the list or the list's preferred language if the sender doesn't have one before processing the message through the handler pipeline. Hold is special in this regard because it can send notices to the list admin in the list's preferred language and to the message sender in the sender's preferred language and these may be different. So I'm not sure that there is any issue here. If you think there is an issue, please provide more detail as to how and in what Handler it arises. (I have questions about your patch in https://bugs.launchpad.net/bugs/1643210 too - I will comment there when I finish reviewing it.) ** Changed in: mailman Status: New => Incomplete -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643215 Title: Some Handlers get translation of language unsupposed To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643215/+subscriptions From futatuki at poem.co.jp Sat Nov 19 21:07:14 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Sun, 20 Nov 2016 02:07:14 -0000 Subject: [Bug 1643215] Re: Some Handlers get translation of language unsupposed References: <20161119173317.8910.64826.malonedeb@gac.canonical.com> Message-ID: <20161120020714.8377.27496.malone@gac.canonical.com> I'm sorry I had misunderstood about this mechanism.(My test case was server's lang = sender's lang...) This affects only when sender's preferred language and list's one is deffer and some of headers to be encoded with RFC 2047 manner. In this case, the translation is sender's preferred language and is encoded to list language. So this seems to affect only CookHeaders.py. (And if list language's charset/encoding is UTF-8, this may not be a problem except few case the header string contains character which don't have Unicode mapping.) -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643215 Title: Some Handlers get translation of language unsupposed To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643215/+subscriptions From futatuki at poem.co.jp Sun Nov 20 01:12:16 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Sun, 20 Nov 2016 06:12:16 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161120061216.6057.53071.malone@chaenomeles.canonical.com> My #1 patch try to adjust to charset/encoding list's preferred. But after I realize my misunderstanding, it is better to adjust to sender's preference. Anyways original senders `realname' charset and translation of '%(realname)s via %(lrn)s' charset and list's preference can differ each other. ** Attachment added: "CookHeaders.py.diff.txt" https://bugs.launchpad.net/mailman/+bug/1643210/+attachment/4780389/+files/CookHeaders.py.diff.txt -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From mark at msapiro.net Sun Nov 20 13:09:36 2016 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 20 Nov 2016 18:09:36 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161120180936.8444.81250.malone@soybean.canonical.com> I'm having trouble understanding what the problem is. There are 3 possible sources of realname. In order of preference, the display name in the message's From: header if any; if not and the From: address is a list member, the list member's username if any, and if not, the local part of the sender's email address. In the first case, the real name should already be RFC 2047 encoded in the incoming message and that will be the resultant value in the munged From: header. In the other cases if the name contains non-ascii, it will be RFC 2047 encoded in the character set of the list's preferred language (or maybe utf-8 if the real name is a unicode). It seems all this should be OK. Please provide and actual From: header and possibly relevant list settings that illustrate the problem. Or is the issue that the list's real_name (lrn in the code)is not rfc 2047 encoded. I see that, but I think the fix is simply to replace the one line lrn = mlist.real_name with lrn = str(uheader(mlist, mlist.real_name)) ** Changed in: mailman Status: New => Incomplete -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From mark at msapiro.net Sun Nov 20 15:18:06 2016 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 20 Nov 2016 20:18:06 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161120201806.5239.46898.malone@wampee.canonical.com> I see some issues with the simple 'fix' I suggested above. Namely the translation of 'via' is not RFC 2047 encoded and there would probably be missing whitespace issues due to mixing of text and RFC 2047 encoded words, but it still seems to me that something like the attached should do. ** Attachment added: "Suggested patch." https://bugs.launchpad.net/mailman/+bug/1643210/+attachment/4780663/+files/CookHeaders.py.diff.txt -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From mark at msapiro.net Sun Nov 20 19:37:24 2016 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 21 Nov 2016 00:37:24 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161121003724.6026.32556.malone@chaenomeles.canonical.com> Further testing of my suggested patch shows it doesn't work in all cases. I'll continue to look at this. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From futatuki at poem.co.jp Sun Nov 20 19:37:28 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Mon, 21 Nov 2016 00:37:28 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161121003728.8337.76515.malone@gac.canonical.com> Suppose sender is a member who set preference language ja, the list language is french and original From: is 'From: =?UTF-8?B?5LqM5pyoIOmdluS7gQ==?= '. The translation of '%(realname)s via %(lrn)s' is taken from sender's language context, ja, and its charset is euc-jp. The display name of original from cannot encode to iso-8859-1 entirely and translation of 'via' part is miss interpreted as iso-8859-1. For the latter problem, we should encode it to encoding of sender's preference language (at least the translation of 'via' part). I don't want to break the realname even if it cannot be encoded to his/her preferred language's encoding, so I select to abandon to translate 'via' part for simple fix. This will occur in language settings that of charset/encoding is not UTF-8. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From futatuki at poem.co.jp Sun Nov 20 19:47:56 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Mon, 21 Nov 2016 00:47:56 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161121004756.8646.62132.malone@gac.canonical.com> The lrn part is always us-ascii so it is vallid string in all encodings that currently supported by Mailman. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From futatuki at poem.co.jp Sun Nov 20 20:10:05 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Mon, 21 Nov 2016 01:10:05 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161121011005.8444.67779.malone@soybean.canonical.com> It seems my patch in #2 also doesn't work if sender is not a list member. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From mark at msapiro.net Sun Nov 20 20:25:07 2016 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 21 Nov 2016 01:25:07 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161121012507.5051.64386.malone@wampee.canonical.com> I need to look at this further, and I won't have time for a day or two, but I will. Note however that the sender's display name in the message is already RFC 2047 encoded in some character set which may not be either of the character sets of the list's preferred language or the sender's preferred language if the sender is even a list member. I will be looking at this further when I have time. I appreciate your input and we will get it right. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From mark at msapiro.net Wed Nov 23 16:20:58 2016 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 23 Nov 2016 21:20:58 -0000 Subject: [Bug 1644356] [NEW] UnicodeError when attempting to send digests Message-ID: <20161123212058.8902.24251.malonedeb@soybean.canonical.com> Public bug reported: This may not occur in a normal installation, but on a test list I had an accumulated digest.mbox with non-ascii characters in some message subjects. I then changed the list's preferred_language from English to Japanese and during some subsequent testing, a digest was triggered. The creation of the digest would throw UnicodeError in attempting to create the MIME toc part and in attempting to set the plain (RFC 1133) digest payload. This may be due only to the change to a language with a different character set or it may be due to the fact that Mailman's character set for Japanese is euc-jp, but the email modules output characterset for euc-jp is iso-2022-jp. The exception thrown was due to unicodes not representable in iso-2022-jp. ** Affects: mailman Importance: Low Assignee: Mark Sapiro (msapiro) Status: Fix Committed -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1644356 Title: UnicodeError when attempting to send digests To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1644356/+subscriptions From mark at msapiro.net Wed Nov 23 16:28:07 2016 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 23 Nov 2016 21:28:07 -0000 Subject: [Bug 1644356] Re: UnicodeError when attempting to send digests References: <20161123212058.8902.24251.malonedeb@soybean.canonical.com> Message-ID: <20161123212808.5656.8079.launchpad@wampee.canonical.com> ** Description changed: This may not occur in a normal installation, but on a test list I had an accumulated digest.mbox with non-ascii characters in some message subjects. I then changed the list's preferred_language from English to Japanese and during some subsequent testing, a digest was triggered. The creation of the digest would throw UnicodeError in attempting to create the MIME toc part and in attempting to set the plain (RFC 1133) digest payload. This may be due only to the change to a language with a different character set or it may be due to the fact that Mailman's character set for Japanese is euc-jp, but the email modules output characterset for - euc-jp is iso-2022-jp. The exception thrown was due to unicodes nt + euc-jp is iso-2022-jp. The exception thrown was due to unicodes not representable in iso-2022-jp. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1644356 Title: UnicodeError when attempting to send digests To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1644356/+subscriptions From 1644356 at bugs.launchpad.net Wed Nov 23 16:28:36 2016 From: 1644356 at bugs.launchpad.net (Launchpad Bug Tracker) Date: Wed, 23 Nov 2016 21:28:36 -0000 Subject: [Bug 1644356] Re: UnicodeError when attempting to send digests References: <20161123212058.8902.24251.malonedeb@soybean.canonical.com> Message-ID: <20161123212838.3004.70099.launchpad@ackee.canonical.com> ** Branch linked: lp:mailman/2.1 -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1644356 Title: UnicodeError when attempting to send digests To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1644356/+subscriptions From mark at msapiro.net Wed Nov 23 16:30:06 2016 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 23 Nov 2016 21:30:06 -0000 Subject: [Bug 1644356] Re: UnicodeError when attempting to send digests References: <20161123212058.8902.24251.malonedeb@soybean.canonical.com> Message-ID: <20161123213007.5208.28088.launchpad@wampee.canonical.com> ** Changed in: mailman Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1644356 Title: UnicodeError when attempting to send digests To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1644356/+subscriptions From mark at msapiro.net Wed Nov 23 17:42:51 2016 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 23 Nov 2016 22:42:51 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161123224251.8273.31541.malone@gac.canonical.com> I have attached the results of my latest effort. For the From:, this gets the i18n translation of the '%(realname)s via %(lrn)s' string with dummy substitutions, converts it to unicode and substitutes unicode values for the substitutions and encodes it all as utf-8. I have tested this to some extent and I think it's good, but I would appreciate additional testing. Please try this and see if it works in your environment or report any issues. As far as the encoding of _('(no subject)') is concerned, my change ensures this is translated in the list's language, not the poster's. Again, thanks for your help with this. ** Attachment added: "Another attempt at a fix." https://bugs.launchpad.net/mailman/+bug/1643210/+attachment/4782228/+files/CookHeaders.py.diff.txt ** Changed in: mailman Status: Incomplete => In Progress ** Changed in: mailman Assignee: (unassigned) => Mark Sapiro (msapiro) -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From futatuki at poem.co.jp Wed Nov 23 23:13:17 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Thu, 24 Nov 2016 04:13:17 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161124041317.5758.12652.malone@chaenomeles.canonical.com> Thank you for your better fix. The fix in #10 also works fine for my environment, except a small issue that it always encodes non-ascii to UTF-8 even if sender's preferred language is same as list's but its encoding is not UTF-8. A test case. list's language : fr (iso-8859-1) sender's language : fr (iso-8859-1) sender's display name : =?iso-8859-1?q?G=E9n=E9rales?= (results) From: =?utf-8?q?G=C3=A9n=C3=A9rales_via_Mailman-test?= <...> Another case. list's language : ja (euc-jp, out going messages are encoded to iso-2022-jp) sender's language : ja (euc-jp, out going messages are encoded to iso-2022-jp) sender's display name : =?ISO-2022-JP?B?GyRCRnNMWkx3P04bKEI=?= (results) From: =?utf-8?b?5LqM5pyo6Z2W5LuBIChNYWlsbWFuLXRlc3Qg57WM55SxKQ==?= <...> It seems to be no problem for almost all MUAs nowadays except some l10n MUAs (and those MUAs will treat such encoded strings as raw ascii string, as discribed in RFC, so I think the problem is small). -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From futatuki at poem.co.jp Thu Nov 24 01:01:08 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Thu, 24 Nov 2016 06:01:08 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161124060108.8376.6835.malone@soybean.canonical.com> How about using "dn = str(Header(uvia, lcs))" instead of "dn = str(Header(uvia, 'utf-8'))" ? As variable uvia is always unicode, there is no afraid to be mistaken encodings. Header() treats charset parameter only for a hint, so it uses 'utf-8' as the fall back if it fail to encode to lcs. test case 1. list's language : fr (iso-8859-1) sender's language : fr (iso-8859-1) sender's display name : =?iso-8859-1?q?G=E9n=E9rales?= (results) From: =?iso-8859-1?q?G=E9n=E9rales_via_Mailman-test?= <...> test case 2. list's language : ja (euc-jp, out going messages are encoded to iso-2022-jp) sender's language : ja (euc-jp, out going messages are encoded to iso-2022-jp) sender's display name : =?ISO-2022-JP?B?GyRCRnNMWkx3P04bKEI=?= (results) From: =?iso-2022-jp?b?GyRCRnNMWkx3P04bKEIgKE1haWxtYW4tdGVzdCAbJEI3UE0zGyhCKQ==?= <...> test case 3. list's language : en (us-ascii) sender's language : en (us-ascii) sender's display name : Yasuhito FUTATSUKI (results) From: Yasuhito FUTATSUKI via Mailman-test <...> test case 4. list's language : fr (iso-8859-1) sender's language : ja (euc-jp, out going messages are encoded to iso-2022-jp) sender's display name : =?UTF-8?B?5LqM5pyoIOmdluS7gQ==?= (results) From: =?utf-8?b?5LqM5pyoIOmdluS7gSB2aWEgTWFpbG1hbi10ZXN0?= <...> in all of above, it looks fine. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From mark at msapiro.net Thu Nov 24 01:10:40 2016 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 24 Nov 2016 06:10:40 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161124061040.5449.90226.malone@chaenomeles.canonical.com> This is an area where there is no one right answer. I have the display name as a unicode, so how do I encode it for the header. I don't think it should ever be encoded in the character set of the poster's language because this is for a message to be sent to all list members, plus there is no guarantee that the poster's display name as encoded by the sending MUA was even encoded in Mailman's charset for the poster's language if the poster is even a member, and there is no guarantee that the translation of the 'via' can even be properly encoded in the charset of the poster's language. Further, there is no guarantee that the poster's display name can be properly encoded in the charset of the list's preferred language either. The most reasonable encoding of unicode that guarantees no loss of information is utf-8, and any MUA that recognizes RFC 2047 encodings at all should be able to handle utf-8 encodings. Even if there are MUA's that can properly decode RFC 2047 encodings in, e.g., iso-2022-jp but not utf-8, I think there are as many problems with trying to encode the original display name in the list's charset as there are with utf-8 encoding. I recognize that what I've done is a compromise, but I think it's as good as any. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From mark at msapiro.net Thu Nov 24 01:16:09 2016 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 24 Nov 2016 06:16:09 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161124061610.5307.13079.malone@wampee.canonical.com> I wrote https://bugs.launchpad.net/mailman/+bug/1643210/comments/13 before I saw https://bugs.launchpad.net/mailman/+bug/1643210/comments/12. It looks like your suggestion is good. I'll investigate that. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From futatuki at poem.co.jp Thu Nov 24 01:18:08 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Thu, 24 Nov 2016 06:18:08 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161124061808.5614.5102.malone@wampee.canonical.com> test case 5. list's language : fr (iso-8859-1) sender's language : ja (euc-jp, out going messages are encoded to iso-2022-jp) sender's display name: =?ISO-2022-JP?B?GyRCRnNMWkx3P04bKEI=?= (results) From: =?utf-8?b?5LqM5pyo6Z2W5LuBIHZpYSBNYWlsbWFuLXRlc3Q=?= <...> is a rational results, I think. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From mark at msapiro.net Thu Nov 24 11:43:51 2016 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 24 Nov 2016 16:43:51 -0000 Subject: [Bug 1643210] Re: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char References: <20161119170750.8717.22114.malonedeb@soybean.canonical.com> Message-ID: <20161124164352.26178.13857.malone@wampee.canonical.com> I have committed a fix which is essentially https://bugs.launchpad.net/mailman/+bug/1643210/+attachment/4782228/+files/CookHeaders.py.diff.txt with "dn = str(Header(uvia, lcs))" as suggested at https://bugs.launchpad.net/mailman/+bug/1643210/comments/12. Thanks very much to Yasuhito FUTATSUKI for the report and all the helpful suggestions. ** Changed in: mailman Importance: Undecided => High ** Changed in: mailman Status: In Progress => Fix Committed ** Changed in: mailman Milestone: None => 2.1.24 -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643210 Title: 'from_is_list' does not RFC2047 encode correctly when translation contains non-ascii char To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643210/+subscriptions From futatuki at poem.co.jp Thu Nov 24 12:41:33 2016 From: futatuki at poem.co.jp (Yasuhito FUTATSUKI at POEM) Date: Thu, 24 Nov 2016 17:41:33 -0000 Subject: [Bug 1643215] Re: Some Handlers get translation of language unsupposed References: <20161119173317.8910.64826.malonedeb@gac.canonical.com> Message-ID: <20161124174133.26407.5448.malone@wampee.canonical.com> The only handler seems to be affected by this, CookeHeaders.py, had been fixed with Bug #1643210. -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643215 Title: Some Handlers get translation of language unsupposed To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643215/+subscriptions From mark at msapiro.net Thu Nov 24 13:47:23 2016 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 24 Nov 2016 18:47:23 -0000 Subject: [Bug 1643215] Re: Some Handlers get translation of language unsupposed References: <20161119173317.8910.64826.malonedeb@gac.canonical.com> Message-ID: <20161124184723.28962.44483.launchpad@gac.canonical.com> ** Changed in: mailman Status: Incomplete => Invalid -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1643215 Title: Some Handlers get translation of language unsupposed To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1643215/+subscriptions From rob-launchpad.net at tigertech.com Tue Nov 29 18:02:01 2016 From: rob-launchpad.net at tigertech.com (Robert Mathews) Date: Tue, 29 Nov 2016 23:02:01 -0000 Subject: [Bug 1645901] [NEW] DKIM signatures stripped from -owner messages with anonymous lists Message-ID: <20161129230201.28287.10424.malonedeb@soybean.canonical.com> Public bug reported: If a list is set to be an anonymous list, and a message is sent to the -owner address, Mailman strips any existing DKIM header. This means that if someone from a DMARC-restricted address (e.g. yahoo.com) is sending a message that would get forwarded to an owner at a DMARC-checking ISP (e.g. yahoo.com), the message is rejected: it fails the DMARC check due to non-matching SPF and missing DKIM headers. If the DKIM header was left intact, it should work, since Mailman didn't modify the body for an -owner message. It makes privacy sense to always strip DKIM headers on messages that will be posted to an anonymous list. And it can work out okay because DMARC munging mitigation can be applied afterwards. But it doesn't seem to make sense to do the same for -owner messages on anonymous lists. Mailman doesn't apply other anonymous list modifications, like hiding the "From:" header, for -owner messages, as far as I can tell. This happens, by the way, due to Defaults.py: # This is the pipeline which messages sent to the -owner address go through OWNER_PIPELINE = [ 'SpamDetect', 'Replybot', 'CleanseDKIM', 'OwnerRecips', 'ToOutgoing', ] Is 'CleanseDKIM' really helpful in this -owner flow? Removing it would solve this problem. Alternately, perhaps CleanseDKIM could be taught to exempt -owner addresses on anonymous lists. ** Affects: mailman Importance: Undecided Status: New ** Tags: dkim dmarc -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1645901 Title: DKIM signatures stripped from -owner messages with anonymous lists To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1645901/+subscriptions From 1645901 at bugs.launchpad.net Tue Nov 29 18:41:02 2016 From: 1645901 at bugs.launchpad.net (Launchpad Bug Tracker) Date: Tue, 29 Nov 2016 23:41:02 -0000 Subject: [Bug 1645901] Re: DKIM signatures stripped from -owner messages with anonymous lists References: <20161129230201.28287.10424.malonedeb@soybean.canonical.com> Message-ID: <20161129234105.26619.87841.launchpad@ackee.canonical.com> ** Branch linked: lp:mailman/2.1 -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1645901 Title: DKIM signatures stripped from -owner messages with anonymous lists To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1645901/+subscriptions From mark at msapiro.net Tue Nov 29 18:41:49 2016 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 29 Nov 2016 23:41:49 -0000 Subject: [Bug 1645901] Re: DKIM signatures stripped from -owner messages with anonymous lists References: <20161129230201.28287.10424.malonedeb@soybean.canonical.com> Message-ID: <20161129234149.28028.65129.malone@soybean.canonical.com> Thanks for your report. I agree that CleanseDKIM should not be in OWNER_PIPELINE and I have removed it for the next release. In the mean time, you can always put OWNER_PIPELINE.remove('CleanseDKIM') in mm_cfg.py. ** Changed in: mailman Importance: Undecided => Medium ** Changed in: mailman Status: New => Fix Committed ** Changed in: mailman Milestone: None => 2.1.24 ** Changed in: mailman Assignee: (unassigned) => Mark Sapiro (msapiro) -- You received this bug notification because you are a member of Mailman Coders, which is subscribed to GNU Mailman. https://bugs.launchpad.net/bugs/1645901 Title: DKIM signatures stripped from -owner messages with anonymous lists To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/1645901/+subscriptions