From aurelien at bompard.org Mon Dec 1 17:45:58 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Mon, 1 Dec 2014 17:45:58 +0100 Subject: [Mailman-Developers] About Mailman's unicode-enabled Message subclass Message-ID: Hi * ! I'm trying to fix bug 1060951 (that I reported), and no matter how I poke at it, I can't find the proper way to solve it, so I'd like to share my thoughts with you. https://bugs.launchpad.net/mailman/+bug/1060951 The issue is with our unicode-enabled subclass of Python's email.message.Message class. Our subclass converts headers to unicode in the get() and __getitem__() methods. However, when an email has an attachment, one of the ways of representing the filename is to urlencode it (RFC 2231). The urldecoding is thus done on an unicode string, which is then passed to email.utils.collapse_rfc2231_value and decoded to unicode again. Of course, this fails, an unicode string can't be decoded twice. I tried re-implementing more of the original Message class to avoid this second decoding, but then I get another problem: the filename has already been urldecoded. Here's an example to make it clearer: >>> m1 = email.message.Message() >>> m1["content-disposition"] = "attachment; filename*=UTF-8''d%C3%A9jeuner.txt" >>> m1.get_param("filename", header="content-disposition") ('UTF-8', '', 'd\xc3\xa9jeuner.txt') If I use mailman's Message implementation, I get: >>> m2 = mailman.email.message.Message() >>> m2["content-disposition"] = "attachment; filename*=UTF-8''d%C3%A9jeuner.txt" >>> m2.get_param("filename", header="content-disposition") (u'UTF-8', u'', u'd\xc3\xa9jeuner.txt') The only difference is that the resulting tuple contains unicode strings. Now when I try to pass this result to email.utils.collapse_rfc2231_value, like Message.get_filename() does: >>> p1 = m1.get_param("filename", header="content-disposition") >>> email.utils.collapse_rfc2231_value(p1) u'd\xe9jeuner.txt' >>> p2 = m2.get_param("filename", header="content-disposition") >>> email.utils.collapse_rfc2231_value(p2) TypeError: decoding Unicode is not supported And if I suppress the final decoding from what's already a unicode string, I get u'd\xc3\xa9jeuner.txt' And, as you can see, the original u'd\xe9jeuner.txt' is not the same as u'd\xc3\xa9jeuner.txt' (the second one is encoded twice). In the end, our subclass of Message can't extract non-ascii filenames. The bug 1060951 contains a testcase for this, it's a replacement for the src/mailman/email/tests/test_message.py : https://bugs.launchpad.net/mailman/+bug/1060951 I'm really interested in any insight on this issue. Thanks for reading all that :-) Aur?lien From stephen at xemacs.org Mon Dec 1 19:47:07 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 02 Dec 2014 03:47:07 +0900 Subject: [Mailman-Developers] About Mailman's unicode-enabled Message subclass In-Reply-To: References: Message-ID: <87bnnnmclw.fsf@uwakimon.sk.tsukuba.ac.jp> Aurelien Bompard writes: > I'm really interested in any insight on this issue. Thanks for reading all > that :-) RFC 2047 and RFC 2231 should be treated as wire protocol, and handled at the bytes level. I haven't looked at this in a while, but I'm pretty sure Python 3's email module does this correctly. I would say that the routines that do this decoding should return bytes, and there should be called by the get() and __getitem__() methods. I don't have time to take a close look at the code until the weekend at least. Steve From barry at list.org Tue Dec 2 02:29:18 2014 From: barry at list.org (Barry Warsaw) Date: Mon, 1 Dec 2014 20:29:18 -0500 Subject: [Mailman-Developers] About Mailman's unicode-enabled Message subclass In-Reply-To: References: Message-ID: <20141201202918.3b35b605@limelight.wooz.org> On Dec 01, 2014, at 05:45 PM, Aurelien Bompard wrote: >I'm really interested in any insight on this issue. Thanks for reading all >that :-) To be honest, mailman.email.message.Message should really go away. It's a terrible, and old, hack. A few things *are* useful, such as pickle support (required for the qfiles to work), and the convenience properties .sender and .senders. Of course, the convenience subclasses (e.g. UserNotification and OwnerNotification) are helpful too, but in a different way. This is yet another reason why I'd like to get the core on Python 3. Then we can ditch most of this hack of a subclass and rely on the behavior of the Python 3 email package, which should DTRT. Not sure that's helpful for your immediate problem though. ;) Cheers, -Barry From mark at msapiro.net Tue Dec 2 05:43:57 2014 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 01 Dec 2014 20:43:57 -0800 Subject: [Mailman-Developers] About Mailman's unicode-enabled Message subclass In-Reply-To: References: Message-ID: <547D438D.4040301@msapiro.net> On 12/01/2014 08:45 AM, Aurelien Bompard wrote: > > I'm really interested in any insight on this issue. Thanks for reading all > that :-) I just took a quick look, but I think the problem is likely that mailman.email.message.Message overrides email.message.Message.__getitem__() as follows: def __getitem__(self, key): # Ensure that header values are unicodes. value = email.message.Message.__getitem__(self, key) if isinstance(value, str): return unicode(value, 'ascii') return value If we trace back a bit in the code, email.message.get_param(...) does for k, v in self._get_params_preserve(failobj, header): and _get_params_preserve(...) does value = self.get(header, missing) which because of our override decodes the entire "attachment; filename*=UTF-8''d%C3%A9jeuner.txt" string as ascii into u"attachment; filename*=UTF-8''d%C3%A9jeuner.txt" which is then parsed into the tuple (u'UTF-8', u'', u'd\xc3\xa9jeuner.txt'). It seems to me that any way to fix this is a horrible kludge. In particular, the value portion of the tuple has already been garbled, and I don't see a good way to fix it once it's been returned. I've looked, again briefly because after a short time my head spun out of control, at trying to fix it by overriding additional methods like _get_params_preserve(), but I didn't get far. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From aurelien at bompard.org Tue Dec 2 11:26:36 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Tue, 2 Dec 2014 11:26:36 +0100 Subject: [Mailman-Developers] About Mailman's unicode-enabled Message subclass In-Reply-To: <547D438D.4040301@msapiro.net> References: <547D438D.4040301@msapiro.net> Message-ID: Thanks to you all ! I've looked, again briefly because after a short time my head spun out > of control, at trying to fix it by overriding additional methods like > _get_params_preserve(), but I didn't get far. > Yeah, I tried that too, not much more luck... > To be honest, mailman.email.message.Message should really go away. I kinda think so too, so I tried removing the conversion to unicode in get() and __getitem__(): I have about 20 failures in the whole test suite, and a couple UnicodeWarnings from SQLAlchemy. I'm going to try and fix these failures and the code that expects unicode headers, and send you a MP How about that? A. From barry at list.org Tue Dec 2 11:42:35 2014 From: barry at list.org (Barry Warsaw) Date: Tue, 2 Dec 2014 05:42:35 -0500 Subject: [Mailman-Developers] About Mailman's unicode-enabled Message subclass In-Reply-To: References: <547D438D.4040301@msapiro.net> Message-ID: <20141202054235.03c6c529@limelight.wooz.org> On Dec 02, 2014, at 11:26 AM, Aurelien Bompard wrote: >I kinda think so too, so I tried removing the conversion to unicode in get() >and __getitem__(): I have about 20 failures in the whole test suite, and a >couple UnicodeWarnings from SQLAlchemy. I'm going to try and fix these >failures and the code that expects unicode headers, and send you a MP How >about that? +1; I look forward to seeing it. I suspect those will be useful also with my experimental py3 branch. :) Cheers, -Barry From aurelien at bompard.org Tue Dec 2 15:58:13 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Tue, 2 Dec 2014 15:58:13 +0100 Subject: [Mailman-Developers] About Mailman's unicode-enabled Message subclass In-Reply-To: <20141202054235.03c6c529@limelight.wooz.org> References: <547D438D.4040301@msapiro.net> <20141202054235.03c6c529@limelight.wooz.org> Message-ID: Hmm, that was easier than I thought : https://code.launchpad.net/~abompard/mailman/bug-1060951/+merge/243401 All tests pass, and no unicode warning from SQLAlchemy in the tests. A. 2014-12-02 11:42 GMT+01:00 Barry Warsaw : > On Dec 02, 2014, at 11:26 AM, Aurelien Bompard wrote: > > >I kinda think so too, so I tried removing the conversion to unicode in > get() > >and __getitem__(): I have about 20 failures in the whole test suite, and a > >couple UnicodeWarnings from SQLAlchemy. I'm going to try and fix these > >failures and the code that expects unicode headers, and send you a MP How > >about that? > > +1; I look forward to seeing it. I suspect those will be useful also with > my > experimental py3 branch. :) > > Cheers, > -Barry > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/aurelien%40bompard.org > > Security Policy: http://wiki.list.org/x/QIA9 > From andrew.stuart at supercoders.com.au Thu Dec 4 00:58:30 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 4 Dec 2014 10:58:30 +1100 Subject: [Mailman-Developers] Where can I find the Mailman 3 REST API specification? Message-ID: is it documented anywhere? I?m happy to dig through code or tests as long as someone can point me in the direction of a definitive list of REST API methods. thanks From barry at list.org Thu Dec 4 01:23:00 2014 From: barry at list.org (Barry Warsaw) Date: Wed, 3 Dec 2014 19:23:00 -0500 Subject: [Mailman-Developers] Where can I find the Mailman 3 REST API specification? In-Reply-To: References: Message-ID: <20141203192300.232b1dc9@anarchist.wooz.org> On Dec 04, 2014, at 10:58 AM, Andrew Stuart wrote: >is it documented anywhere? Yes! >I?m happy to dig through code or tests as long as someone can point me in the >direction of a definitive list of REST API methods. All the documentation, including the REST API, is available online at http://pythonhosted.org/mailman/ This is somewhat out of date though, because it describes 3.0b4, but as I expect to release b5 before the EOY, it will be updated to 3.0b5. In the meantime, you can branch the trunk[*] and view the documentation in-tree. It's also not wonderfully organized - contributions welcome! Cheers, -Barry [*] $ bzr branch lp:mailman From andrew.stuart at supercoders.com.au Thu Dec 4 02:17:50 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 4 Dec 2014 12:17:50 +1100 Subject: [Mailman-Developers] Mailman keeps creating "var" directories Message-ID: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> If I run the mailman command it keeps creating ?var? directories in whatever the current directory is. Is that intentional? thanks From andrew.stuart at supercoders.com.au Thu Dec 4 02:18:34 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 4 Dec 2014 12:18:34 +1100 Subject: [Mailman-Developers] Is cron used in Mailman 3? Message-ID: Where can I find the documentation that explains the required configuration for cron in Mailman 3? thanks From rajeevs1992 at gmail.com Thu Dec 4 04:24:07 2014 From: rajeevs1992 at gmail.com (rajeevs1992 at gmail.com) Date: Thu, 04 Dec 2014 08:54:07 +0530 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> Message-ID: <20141204032407.5918864.34651.1735@gmail.com> Hi, Yes, that's intentional. The var directory setting is set to current directory by default. You can set it to a fixed directory by modifying the var_dir setting in mailman.cfg. ? Sent from my BlackBerry 10 smartphone. ? Original Message ? From: Andrew Stuart Sent: Thursday 4 December 2014 06:47 To: mailman-developers at python.org Subject: [Mailman-Developers] Mailman keeps creating "var" directories If I run the mailman command it keeps creating ?var? directories in whatever the current directory is. Is that intentional? thanks _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/rajeevs1992%40gmail.com Security Policy: http://wiki.list.org/x/QIA9 From adam at iris.washington.edu Sat Dec 6 04:02:29 2014 From: adam at iris.washington.edu (Adam Clark) Date: Fri, 5 Dec 2014 19:02:29 -0800 Subject: [Mailman-Developers] Using alembic autogenerate Message-ID: Is there a procedure for autogenerating a database update using alembic? I haven't used alembic before, but through trial and error I arrived at this: > alembic -c mailman/config/alembic.cfg revision --autogenerate This fails with File "/workspace/test/mailman/libs/mailman/src/mailman/config/config.py", line 98, in __getattr__ return getattr(self._config, name) AttributeError: 'NoneType' object has no attribute 'database' So apparently mailman config wasn't being loaded. I added it to mailman/database/alembic/env.py: def run_migrations_online(): config.load() # Added url = expand(config.database.url, config.paths) ... This succeeded in producing a version file, but that version would delete everything: def upgrade(): ### commands auto generated by Alembic - please adjust! ### op.drop_table('message') op.drop_table('address') op.drop_table('autoresponserecord') op.drop_table('member') op.drop_table('ban') ... The root cause appeared to be that Model.metadata.sorted_tables returned an empty list. Manually triggering model reflection seemed to improve things; again in env.py: def run_migrations_online(): config.load() # Added url = expand(config.database.url, config.paths) engine = create_engine(url) Model.metadata.reflect(bind=engine) # Added connection = engine.connect() ... This now produces a correctly empty update against the existing code; the problem is if I change the code (and I verified that Model.metadata.sorted_tables reflects the changed model information) this *still* produces an empty update. I suspect that I have this all backwards, that this whole process is actually intended to run from within mailman rather than via the alembic command line, but I can't find where this would be. Thanks, Adam From andrew.stuart at supercoders.com.au Sat Dec 6 07:34:59 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Sat, 6 Dec 2014 17:34:59 +1100 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: <20141204032407.5918864.34651.1735@gmail.com> References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> <20141204032407.5918864.34651.1735@gmail.com> Message-ID: <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> When I run ?mailman info?, it creates a ?var? directory in the current directory. You can see in the following that I have mailman.cfg in my /etc You can also see that /etc/mailman.cfg defines locations for the various directories. You can see that prior to running mailman info there is no ?var? directory, but there is afterwards. I?m curious to understand why it is doing this when apparently mailman is picking up the configuration file OK from /etc but ignoring its contents and then setting up a new var directory in the current directory. Any ideas welcome. (venv2.7)ubuntu at server01:~$ ls blankontheme.zip mailman mailman.client postorius postorius_standalone temp venv2.7 (venv2.7)ubuntu at server01:~$ grep _dir /etc/mailman.cfg #var_dir: var archive_dir = /home/ubuntu/mailman/var/archives bin_dir = /home/ubuntu/venv2.7/bin data_dir = /home/ubuntu/mailman/var/data etc_dir = /etc ext_dir = /home/ubuntu/mailman/var/ext list_data_dir = /home/ubuntu/mailman/var/lists lock_dir = /home/ubuntu/mailman/var/locks log_dir = /home/ubuntu/mailman/var/logs messages_dir = /home/ubuntu/mailman/var/messages queue_dir = /home/ubuntu/mailman/var/queue template_dir = /home/ubuntu/mailman/var/templates var_dir = /home/ubuntu/mailman/var (venv2.7)ubuntu at server01:~$ ls blankontheme.zip mailman mailman.client postorius postorius_standalone temp venv2.7 (venv2.7)ubuntu at server01:~$ mailman info GNU Mailman 3.0.0b4+ (Carve Away The Stone) Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] config file: /etc/mailman.cfg db url: postgresql://mailman:password at localhost/mailman devmode: DISABLED REST root url: http://localhost:8001/3.0/ REST credentials: restadmin:restpass (venv2.7)ubuntu at server01:~$ ls blankontheme.zip mailman mailman.client postorius postorius_standalone temp var venv2.7 (venv2.7)ubuntu at server01:~$ On 4 Dec 2014, at 2:24 pm, rajeevs1992 at gmail.com wrote: Hi, Yes, that's intentional. The var directory setting is set to current directory by default. You can set it to a fixed directory by modifying the var_dir setting in mailman.cfg. Sent from my BlackBerry 10 smartphone. Original Message From: Andrew Stuart Sent: Thursday 4 December 2014 06:47 To: mailman-developers at python.org Subject: [Mailman-Developers] Mailman keeps creating "var" directories If I run the mailman command it keeps creating ?var? directories in whatever the current directory is. Is that intentional? thanks _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/rajeevs1992%40gmail.com Security Policy: http://wiki.list.org/x/QIA9 From raj.abhilash1 at gmail.com Sat Dec 6 08:31:19 2014 From: raj.abhilash1 at gmail.com (Abhilash Raj) Date: Sat, 6 Dec 2014 13:01:19 +0530 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> <20141204032407.5918864.34651.1735@gmail.com> <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> Message-ID: Hi Andrew, On Sat, Dec 6, 2014 at 12:04 PM, Andrew Stuart < andrew.stuart at supercoders.com.au> wrote: > When I run ?mailman info?, it creates a ?var? directory in the current > directory. > Yes this is the default behavior when you don't specify the paths explicitly in your mailman.cfg > > You can see in the following that I have mailman.cfg in my /etc > > You can also see that /etc/mailman.cfg defines locations for the various > directories. > > You can see that prior to running mailman info there is no ?var? > directory, but there is afterwards. > > I?m curious to understand why it is doing this when apparently mailman is > picking up the configuration file OK from /etc but ignoring its contents > and then setting up a new var directory in the current directory. Any > ideas welcome. > > (venv2.7)ubuntu at server01:~$ ls > blankontheme.zip mailman mailman.client postorius > postorius_standalone temp venv2.7 > (venv2.7)ubuntu at server01:~$ grep _dir /etc/mailman.cfg > #var_dir: var > archive_dir = /home/ubuntu/mailman/var/archives > bin_dir = /home/ubuntu/venv2.7/bin > data_dir = /home/ubuntu/mailman/var/data > etc_dir = /etc > ext_dir = /home/ubuntu/mailman/var/ext > list_data_dir = /home/ubuntu/mailman/var/lists > lock_dir = /home/ubuntu/mailman/var/locks > log_dir = /home/ubuntu/mailman/var/logs > messages_dir = /home/ubuntu/mailman/var/messages > queue_dir = /home/ubuntu/mailman/var/queue > template_dir = /home/ubuntu/mailman/var/templates > var_dir = /home/ubuntu/mailman/var > Where did you get this config from? Mailman supports INI-style config only. My bet is that mailman is able to find your config file but is not able to parse it. What actually happens is that mailman has predefined paths for different scenarios which can be configured with layout variable in [mailman] section inside your mailman.cfg. The default is `dev` layout which creates the `var` directory in the current directory. You can either change the default `var_dir` in `dev` layout, below is a sample config for that: ``` [paths.dev] var_dir : ``` or you can use some other paths defined in src/mailman/config/mailman.cfg under [paths.local], [paths.master], [paths.fhs] and change the default layout in your mailman.cfg like this: ``` [mailman] layout: local ``` Also my guess is that you can create your own path structure inside your mailman.cfg and use that layout with mailman. How to create that can be seen in `src/mailman/config/mailman.cfg`. thanks, Abhilash Raj From andrew.stuart at supercoders.com.au Sat Dec 6 08:48:04 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Sat, 6 Dec 2014 18:48:04 +1100 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> <20141204032407.5918864.34651.1735@gmail.com> <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> Message-ID: Hi Abhilash, >>>>> Where did you get this config from? Mailman supports INI-style config only. Actually I copied the directory specifications from the output of mailman info -v I can see now that isn?t a good idea as the format is not the same. Don?t know why I assumed it would be. Also I can see now the second part of my issue was that even after setting the directories correctly, I also has to specify which section of mailman.cfg should be used to define the directories. It?s working now thanks. Andrew On 6 Dec 2014, at 6:31 pm, Abhilash Raj wrote: Hi Andrew, On Sat, Dec 6, 2014 at 12:04 PM, Andrew Stuart wrote: When I run ?mailman info?, it creates a ?var? directory in the current directory. Yes this is the default behavior when you don't specify the paths explicitly in your mailman.cfg You can see in the following that I have mailman.cfg in my /etc You can also see that /etc/mailman.cfg defines locations for the various directories. You can see that prior to running mailman info there is no ?var? directory, but there is afterwards. I?m curious to understand why it is doing this when apparently mailman is picking up the configuration file OK from /etc but ignoring its contents and then setting up a new var directory in the current directory. Any ideas welcome. (venv2.7)ubuntu at server01:~$ ls blankontheme.zip mailman mailman.client postorius postorius_standalone temp venv2.7 (venv2.7)ubuntu at server01:~$ grep _dir /etc/mailman.cfg #var_dir: var archive_dir = /home/ubuntu/mailman/var/archives bin_dir = /home/ubuntu/venv2.7/bin data_dir = /home/ubuntu/mailman/var/data etc_dir = /etc ext_dir = /home/ubuntu/mailman/var/ext list_data_dir = /home/ubuntu/mailman/var/lists lock_dir = /home/ubuntu/mailman/var/locks log_dir = /home/ubuntu/mailman/var/logs messages_dir = /home/ubuntu/mailman/var/messages queue_dir = /home/ubuntu/mailman/var/queue template_dir = /home/ubuntu/mailman/var/templates var_dir = /home/ubuntu/mailman/var Where did you get this config from? Mailman supports INI-style config only. My bet is that mailman is able to find your config file but is not able to parse it. What actually happens is that mailman has predefined paths for different scenarios which can be configured with layout variable in [mailman] section inside your mailman.cfg. The default is `dev` layout which creates the `var` directory in the current directory. You can either change the default `var_dir` in `dev` layout, below is a sample config for that: ``` [paths.dev] var_dir : ``` or you can use some other paths defined in src/mailman/config/mailman.cfg under [paths.local], [paths.master], [paths.fhs] and change the default layout in your mailman.cfg like this: ``` [mailman] layout: local ``` Also my guess is that you can create your own path structure inside your mailman.cfg and use that layout with mailman. How to create that can be seen in `src/mailman/config/mailman.cfg`. thanks, Abhilash Raj From stephen at xemacs.org Sat Dec 6 17:48:04 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 07 Dec 2014 01:48:04 +0900 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> <20141204032407.5918864.34651.1735@gmail.com> <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> Message-ID: <87oargk9mj.fsf@uwakimon.sk.tsukuba.ac.jp> Andrew Stuart writes: > Actually I copied the directory specifications from the output of > mailman info -v I can see now that isn?t a good idea as the format > is not the same. Don?t know why I assumed it would be. Well, it is in Mailman 2 because in Mailman 2 those variables would be specified in Python. @Barry: would it be impossible/a bad idea to have mailman info -v output usable config files? From andrew.stuart at supercoders.com.au Sat Dec 6 22:08:16 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Sun, 7 Dec 2014 08:08:16 +1100 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> <20141204032407.5918864.34651.1735@gmail.com> <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> Message-ID: <53CF5AFF-EE8C-4109-8222-799800C684AF@supercoders.com.au> Hi Abhilash, I very much like the capability of mailman to create its own ?var? directory structure - most open source systems require me to spend hours with strace trying to work out what directories and permission an application needs in what location - I?m heartily sick of having my time wasted like that so this is very refreshing functionality. On the other hand it?s very confusing when a misconfiguration in the mailman.cfg results in a ?var? directory being created in the current directory **which then gets picked up and used as the configuration**, so correcting your configuration in /etc/mailman.cfg then leads to a new problem, which is that your /etc/mailman.cfg is no longer being used *at all*. I love the creation of the var directories by mailman but it feels like a bit of a hand grenade with a short fuse. Whilst I was setting up my configuration I ended up with var directories all over the file system, without any understanding of why they were there. A trap for Mailman newbies. Perhaps it?s worth considering making creation of the ?var? directory structure could be the result of an explicit option rather than a default action. thanks On 6 Dec 2014, at 6:31 pm, Abhilash Raj wrote: Hi Andrew, On Sat, Dec 6, 2014 at 12:04 PM, Andrew Stuart wrote: When I run ?mailman info?, it creates a ?var? directory in the current directory. Yes this is the default behavior when you don't specify the paths explicitly in your mailman.cfg You can see in the following that I have mailman.cfg in my /etc You can also see that /etc/mailman.cfg defines locations for the various directories. You can see that prior to running mailman info there is no ?var? directory, but there is afterwards. I?m curious to understand why it is doing this when apparently mailman is picking up the configuration file OK from /etc but ignoring its contents and then setting up a new var directory in the current directory. Any ideas welcome. (venv2.7)ubuntu at server01:~$ ls blankontheme.zip mailman mailman.client postorius postorius_standalone temp venv2.7 (venv2.7)ubuntu at server01:~$ grep _dir /etc/mailman.cfg #var_dir: var archive_dir = /home/ubuntu/mailman/var/archives bin_dir = /home/ubuntu/venv2.7/bin data_dir = /home/ubuntu/mailman/var/data etc_dir = /etc ext_dir = /home/ubuntu/mailman/var/ext list_data_dir = /home/ubuntu/mailman/var/lists lock_dir = /home/ubuntu/mailman/var/locks log_dir = /home/ubuntu/mailman/var/logs messages_dir = /home/ubuntu/mailman/var/messages queue_dir = /home/ubuntu/mailman/var/queue template_dir = /home/ubuntu/mailman/var/templates var_dir = /home/ubuntu/mailman/var Where did you get this config from? Mailman supports INI-style config only. My bet is that mailman is able to find your config file but is not able to parse it. What actually happens is that mailman has predefined paths for different scenarios which can be configured with layout variable in [mailman] section inside your mailman.cfg. The default is `dev` layout which creates the `var` directory in the current directory. You can either change the default `var_dir` in `dev` layout, below is a sample config for that: ``` [paths.dev] var_dir : ``` or you can use some other paths defined in src/mailman/config/mailman.cfg under [paths.local], [paths.master], [paths.fhs] and change the default layout in your mailman.cfg like this: ``` [mailman] layout: local ``` Also my guess is that you can create your own path structure inside your mailman.cfg and use that layout with mailman. How to create that can be seen in `src/mailman/config/mailman.cfg`. thanks, Abhilash Raj From raj.abhilash1 at gmail.com Sun Dec 7 08:53:42 2014 From: raj.abhilash1 at gmail.com (Abhilash Raj) Date: Sun, 7 Dec 2014 13:23:42 +0530 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: <53CF5AFF-EE8C-4109-8222-799800C684AF@supercoders.com.au> References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> <20141204032407.5918864.34651.1735@gmail.com> <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> <53CF5AFF-EE8C-4109-8222-799800C684AF@supercoders.com.au> Message-ID: Hi Andrew, On Sun, Dec 7, 2014 at 2:38 AM, Andrew Stuart < andrew.stuart at supercoders.com.au> wrote: > Hi Abhilash, > > I very much like the capability of mailman to create its own ?var? > directory structure - most open source systems require me to spend hours > with strace trying to work out what directories and permission an > application needs in what location - I?m heartily sick of having my time > wasted like that so this is very refreshing functionality. > > On the other hand it?s very confusing when a misconfiguration in the > mailman.cfg results in a ?var? directory being created in the current > directory **which then gets picked up and used as the configuration**, so > correcting your configuration in /etc/mailman.cfg then leads to a new > problem, which is that your /etc/mailman.cfg is no longer being used *at > all*. > Its not like your configuration at /etc/mailman.cfg is not used at all, its like mailman since cannot read your settings uses its own default ones. Is that not clean and sane enough? And coming to your problem, since mailman is still in development the default layout is `dev` which helps developers as `var` directory is stored right in front of them. Once it is packaged and released for production you can expect some default whose behavior is different, and there are already many options that I mentioned before. > > I love the creation of the var directories by mailman but it feels like a > bit of a hand grenade with a short fuse. Whilst I was setting up my > configuration I ended up with var directories all over the file system, > without any understanding of why they were there. A trap for Mailman > newbies. > That may be because of the lack of explicit documentation for production. I hope once that comes up it will be clearer for newbies and also for others as well. > > Perhaps it?s worth considering making creation of the ?var? directory > structure could be the result of an explicit option rather than a default > action. > > thanks > thanks, Abhilash Raj From barry at list.org Tue Dec 9 02:54:31 2014 From: barry at list.org (Barry Warsaw) Date: Mon, 8 Dec 2014 20:54:31 -0500 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> <20141204032407.5918864.34651.1735@gmail.com> <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> <53CF5AFF-EE8C-4109-8222-799800C684AF@supercoders.com.au> Message-ID: <20141208205431.52d7a066@anarchist.wooz.org> On Dec 07, 2014, at 01:23 PM, Abhilash Raj wrote: >And coming to your problem, since mailman is still in development the default >layout is `dev` which helps developers as `var` directory is stored right in >front of them. Once it is packaged and released for production you can expect >some default whose behavior is different, and there are already many options >that I mentioned before. Exactly so. It turns out that during development, it's pretty useful for the default layout to be 'dev' and for the 'dev' layout to put the var directory in the current working directory. I fully expect that system packagers would create an /etc/mailman.cfg that sets the default layout to something else, e.g. fhs on Linux systems. Andrew, I think this is the source of the confusion, i.e. the fact that 'dev' is the default layout. >That may be because of the lack of explicit documentation for production. I >hope once that comes up it will be clearer for newbies and also for others >as well. Agreed. I would love to see some better organization to the documentation too. This would be a great opportunity for someone to contribute. It's pretty easy to learn reStructuredText and Sphinx. Cheers, -Barry From barry at list.org Tue Dec 9 02:57:26 2014 From: barry at list.org (Barry Warsaw) Date: Mon, 8 Dec 2014 20:57:26 -0500 Subject: [Mailman-Developers] Mailman keeps creating "var" directories In-Reply-To: <87oargk9mj.fsf@uwakimon.sk.tsukuba.ac.jp> References: <7C880714-CB0A-492C-8F74-285C0F04F38B@supercoders.com.au> <20141204032407.5918864.34651.1735@gmail.com> <3DA27080-70C9-43D1-9FF8-5AF5E46980E8@supercoders.com.au> <87oargk9mj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141208205726.0b692c4f@anarchist.wooz.org> On Dec 07, 2014, at 01:48 AM, Stephen J. Turnbull wrote: >@Barry: would it be impossible/a bad idea to have mailman info -v >output usable config files? Not `mailman info` but for `mailman conf` it would make a lot of sense to have a switch to dump an ini-file compatible format. Cheers, -Barry From andrew.stuart at supercoders.com.au Thu Dec 11 03:41:26 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Thu, 11 Dec 2014 13:41:26 +1100 Subject: [Mailman-Developers] Will the next beta release have both Falcon for REST and SQLAlchemy? Message-ID: thanks Andrew From barry at list.org Thu Dec 11 03:44:32 2014 From: barry at list.org (Barry Warsaw) Date: Wed, 10 Dec 2014 21:44:32 -0500 Subject: [Mailman-Developers] Will the next beta release have both Falcon for REST and SQLAlchemy? In-Reply-To: References: Message-ID: <20141210214432.5261d4e8@anarchist.wooz.org> Yes. Both ports have landed in trunk. Cheers, -Barry From andrew.stuart at supercoders.com.au Thu Dec 11 21:08:24 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Fri, 12 Dec 2014 07:08:24 +1100 Subject: [Mailman-Developers] =?utf-8?q?Error_installing_mailman=2Eclient_?= =?utf-8?b?ZHVlIHRvICJBdXLDqWxpZW4i?= Message-ID: <011086AF-1A85-40F7-9D86-760DD7B83975@supercoders.com.au> Hello, Referring to transcript below, the word Aur?lien seems to be causing a unicode error when setting up mailman client. When I cut it out, it works. thanks (venv2.7)ubuntu at server01:~$ bzr branch lp:mailman.client You have not informed bzr of your Launchpad ID, and you must do this to write to Launchpad or access private data. See "bzr help launchpad-login". Branched 58 revisions. (venv2.7)ubuntu at server01:~$ cd mailman.client/ (venv2.7)ubuntu at server01:~/mailman.client$ ls bin COPYING.LESSER distribute_setup.py Makefile MANIFEST.in README.txt setup.cfg setup_helpers.py setup.py src _static template.py (venv2.7)ubuntu at server01:~/mailman.client$ python setup.py develop Traceback (most recent call last): File "setup.py", line 40, in 'src/mailmanclient/NEWS.txt'), File "/home/ubuntu/mailman.client/setup_helpers.py", line 138, in long_description if not value.endswith(NL): UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 162: ordinal not in range(128) (venv2.7)ubuntu at server01:~/mailman.client$ vi src/mailmanclient/NEWS.txt (venv2.7)ubuntu at server01:~/mailman.client$ cat src/mailmanclient/NEWS.txt ======================= NEWS for mailman.client ======================= 1.0.0b1 (xxxx-xx-xx) * Addedd an improved test harness using WebTest. Contributed by Aur?lien Abompard. 1.0.0a1 (2014-03-15) ==================== * Initial release. (venv2.7)ubuntu at server01:~/mailman.client$ vi src/mailmanclient/NEWS.txt (venv2.7)ubuntu at server01:~/mailman.client$ cat src/mailmanclient/NEWS.txt ======================= NEWS for mailman.client ======================= 1.0.0b1 (xxxx-xx-xx) 1.0.0a1 (2014-03-15) ==================== * Initial release. (venv2.7)ubuntu at server01:~/mailman.client$ From andrew.stuart at supercoders.com.au Thu Dec 11 21:52:06 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Fri, 12 Dec 2014 07:52:06 +1100 Subject: [Mailman-Developers] How can I run the REST tests against live servers? Message-ID: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> I want to run the REST tests against my live Mailman server with its Postfix mail server and Postgres database, via: nose2 -v -P rest After many (many) hours of digging deep and figuring out how it works, I suspect at this stage that maybe the tests set up their own mock servers? If so, I?m guessing these mock servers are getting in the way of using the live servers and thus preventing the tests from running? Assuming it is correct that the tests are setting up their own mock http smtp ltmp responders, is there a way to turn this behaviour off so the tests will run against my live REST/Postgres/Postfix? thanks Andrew From barry at list.org Thu Dec 11 22:27:52 2014 From: barry at list.org (Barry Warsaw) Date: Thu, 11 Dec 2014 16:27:52 -0500 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> Message-ID: <20141211162752.504ed099@limelight.wooz.org> On Dec 12, 2014, at 07:52 AM, Andrew Stuart wrote: >I want to run the REST tests against my live Mailman server with its Postfix >mail server and Postgres database, via: > >nose2 -v -P rest > >After many (many) hours of digging deep and figuring out how it works, I >suspect at this stage that maybe the tests set up their own mock servers? If >so, I?m guessing these mock servers are getting in the way of using the live >servers and thus preventing the tests from running? > >Assuming it is correct that the tests are setting up their own mock http smtp >ltmp responders, is there a way to turn this behaviour off so the tests will >run against my live REST/Postgres/Postfix? Hi Andrew, You're right that the test suite starts its own http and lmtp servers. Take a look at src/mailman/testing/layers.py for how this is controlled, and review src/mailman/testing/nose.py and this documentation on nose2 support for Zope testrunner-style layers: https://nose2.readthedocs.org/en/latest/plugins/layers.html I think even if you could somehow substitute real servers for the testing ones, you'll still have a problem running the test suite against them, to any semblance of success. The tests are dependent on specific arrangements of sample data in order to run successfully. E.g. you'll see that the ConfigLayer, which is the "base class"[1] creates an example.com domain, so that all a test needs to do is create a mailing list in that domain without first having to create the domain. The test suite is also very careful to restore pristine state between individual tests; look for reset_the_world() which is run after every test, and does exactly what it describes. This ensures that no state from one test can affect the state of another test. In the real world testing you propose above, you would have quite a bit of bleed-through I suspect. Can you describe why you want to run the test suite against a live Mailman server? I do think it would be interesting to build an integration test suite that would talk to real MTAs and databases, but that would take a fair bit of work and probably be separate from the unittests, which aim to gather 100% code coverage. Cheers, -Barry [1] Although layers don't really support normal Python class hierarchies. From andrew.stuart at supercoders.com.au Thu Dec 11 22:35:31 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Fri, 12 Dec 2014 08:35:31 +1100 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: <20141211162752.504ed099@limelight.wooz.org> References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> <20141211162752.504ed099@limelight.wooz.org> Message-ID: Hi Barry I?m writing an authenticating proxy for the Mailman REST API and want to make sure everything works as expected against real infrastructure (Postgres/Postfix/Sqlalchemy/Falcon). I was hoping that just pointing the REST tests request port to my proxy would work but no dice - lots of failures - trying to diagnose the failures has led me back to trying to get the test suite to work against the live servers. I?m wondering if maybe the shortest route here might not be for me to write a new set of tests which purely talk REST and don?t touch the back end directly at all. Except that?s not a short route. :-) Trying to bend the existing tests to meet my needs is looking harder than I have time to deal with. Andrew On 12 Dec 2014, at 8:27 am, Barry Warsaw wrote: On Dec 12, 2014, at 07:52 AM, Andrew Stuart wrote: > I want to run the REST tests against my live Mailman server with its Postfix > mail server and Postgres database, via: > > nose2 -v -P rest > > After many (many) hours of digging deep and figuring out how it works, I > suspect at this stage that maybe the tests set up their own mock servers? If > so, I?m guessing these mock servers are getting in the way of using the live > servers and thus preventing the tests from running? > > Assuming it is correct that the tests are setting up their own mock http smtp > ltmp responders, is there a way to turn this behaviour off so the tests will > run against my live REST/Postgres/Postfix? Hi Andrew, You're right that the test suite starts its own http and lmtp servers. Take a look at src/mailman/testing/layers.py for how this is controlled, and review src/mailman/testing/nose.py and this documentation on nose2 support for Zope testrunner-style layers: https://nose2.readthedocs.org/en/latest/plugins/layers.html I think even if you could somehow substitute real servers for the testing ones, you'll still have a problem running the test suite against them, to any semblance of success. The tests are dependent on specific arrangements of sample data in order to run successfully. E.g. you'll see that the ConfigLayer, which is the "base class"[1] creates an example.com domain, so that all a test needs to do is create a mailing list in that domain without first having to create the domain. The test suite is also very careful to restore pristine state between individual tests; look for reset_the_world() which is run after every test, and does exactly what it describes. This ensures that no state from one test can affect the state of another test. In the real world testing you propose above, you would have quite a bit of bleed-through I suspect. Can you describe why you want to run the test suite against a live Mailman server? I do think it would be interesting to build an integration test suite that would talk to real MTAs and databases, but that would take a fair bit of work and probably be separate from the unittests, which aim to gather 100% code coverage. Cheers, -Barry [1] Although layers don't really support normal Python class hierarchies. _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 From aurelien at bompard.org Fri Dec 12 00:00:27 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Fri, 12 Dec 2014 00:00:27 +0100 Subject: [Mailman-Developers] =?iso-8859-1?q?Error_installing_mailman=2Ecl?= =?iso-8859-1?q?ient_due_to_=22Aur=E9lien=22?= In-Reply-To: <011086AF-1A85-40F7-9D86-760DD7B83975@supercoders.com.au> References: <011086AF-1A85-40F7-9D86-760DD7B83975@supercoders.com.au> Message-ID: At first I thought I had introduced a regression.... but it's actually just my name :-) Don't blame me, I didn't choose it! (Florian, can you fix that? And by the way my last name is "Bompard", without the "A") Thanks ! A. 2014-12-11 21:08 GMT+01:00 Andrew Stuart : > Hello, > > Referring to transcript below, the word Aur?lien seems to be causing a > unicode error when setting up mailman client. > > When I cut it out, it works. > > thanks > > > (venv2.7)ubuntu at server01:~$ bzr branch lp:mailman.client > You have not informed bzr of your Launchpad ID, and you must do this to > write to Launchpad or access private data. See "bzr help launchpad-login". > Branched 58 revisions. > (venv2.7)ubuntu at server01:~$ cd mailman.client/ > (venv2.7)ubuntu at server01:~/mailman.client$ ls > bin COPYING.LESSER distribute_setup.py Makefile MANIFEST.in > README.txt setup.cfg setup_helpers.py setup.py src _static template.py > (venv2.7)ubuntu at server01:~/mailman.client$ python setup.py develop > Traceback (most recent call last): > File "setup.py", line 40, in > 'src/mailmanclient/NEWS.txt'), > File "/home/ubuntu/mailman.client/setup_helpers.py", line 138, in > long_description > if not value.endswith(NL): > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 162: > ordinal not in range(128) > (venv2.7)ubuntu at server01:~/mailman.client$ vi src/mailmanclient/NEWS.txt > (venv2.7)ubuntu at server01:~/mailman.client$ cat src/mailmanclient/NEWS.txt > ======================= > NEWS for mailman.client > ======================= > > 1.0.0b1 (xxxx-xx-xx) > > * Addedd an improved test harness using WebTest. Contributed by Aur?lien > Abompard. > > > 1.0.0a1 (2014-03-15) > ==================== > > * Initial release. > (venv2.7)ubuntu at server01:~/mailman.client$ vi src/mailmanclient/NEWS.txt > (venv2.7)ubuntu at server01:~/mailman.client$ cat src/mailmanclient/NEWS.txt > ======================= > NEWS for mailman.client > ======================= > > 1.0.0b1 (xxxx-xx-xx) > > > > 1.0.0a1 (2014-03-15) > ==================== > > * Initial release. > (venv2.7)ubuntu at server01:~/mailman.client$ > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/aurelien%40bompard.org > > Security Policy: http://wiki.list.org/x/QIA9 From barry at list.org Fri Dec 12 01:34:26 2014 From: barry at list.org (Barry Warsaw) Date: Thu, 11 Dec 2014 19:34:26 -0500 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> <20141211162752.504ed099@limelight.wooz.org> Message-ID: <20141211193426.3c93c7b1@marathon> On Dec 12, 2014, at 08:35 AM, Andrew Stuart wrote: >I?m writing an authenticating proxy for the Mailman REST API and want to make >sure everything works as expected against real infrastructure >(Postgres/Postfix/Sqlalchemy/Falcon). We definitely want such a proxy. It's always been our intent that the core's REST API is an adminipstrative API with full access. It should only be exposed on localhost or other highly secure interface. A proxy server such as you describe would be available on a public interface and would allow people to script the management of their list memberships and list administrations. >I was hoping that just pointing the REST tests request port to my proxy would >work but no dice - lots of failures - trying to diagnose the failures has led >me back to trying to get the test suite to work against the live servers. > >I?m wondering if maybe the shortest route here might not be for me to write a >new set of tests which purely talk REST and don?t touch the back end directly >at all. Except that?s not a short route. :-) > >Trying to bend the existing tests to meet my needs is looking harder than I >have time to deal with. Even in this case, I wouldn't expect the standard unittests to work. For example, if a normal user were to try to access the list admin API, you'd expect that to fail. I think one approach would be to continue to use the test servers, but have a backdoor connection which would let you set up the state before each test and reset the state after each test. Then the tests themselves would use the authenticated API to flex the things it can and shouldn't be able to do. It would be possible for example to expose some helper resources in the admin API which would not normally be exposed in production. Something under a top level /tests resource perhaps. Cheers, -Barry From stephen at xemacs.org Fri Dec 12 02:45:20 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 12 Dec 2014 10:45:20 +0900 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> <20141211162752.504ed099@limelight.wooz.org> Message-ID: <87y4qdfxov.fsf@uwakimon.sk.tsukuba.ac.jp> Andrew Stuart writes: > I?m writing an authenticating proxy for the Mailman REST API and > want to make sure everything works as expected against real > infrastructure (Postgres/Postfix/Sqlalchemy/Falcon). Determining that "things work as expected against real infrastructure" is not what the test suite does. AFAIK we don't have any such tests, and I don't really see how the tests we do have can be adapted to "tests against real infrastructure". The problem is specifying what to expect of real infrastructure. You could, for example, write tests that insert users, query for them, and then delete them, but this has several problems. Do you want to do these tests on "live" servers that might actually be in use? -- Probably not, but how do you prevent that? Attributes of the various objects created by the system may depend on context. -- The test needs to ignore any differences due merely to different context. Coverage is necessarily imperfect in the sense that some internal state changes can't be checked since the "before" state is indeterminate in a live system. And so on. From andrew.stuart at supercoders.com.au Fri Dec 12 05:11:33 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Fri, 12 Dec 2014 15:11:33 +1100 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: <20141211193426.3c93c7b1@marathon> References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> <20141211162752.504ed099@limelight.wooz.org> <20141211193426.3c93c7b1@marathon> Message-ID: <01358FDC-D28D-466B-9C86-42BC296A1460@supercoders.com.au> I have managed to make the existing tests work through my proxy but it was challenging for a few reasons. The tests are hard coded to transmit their HTTP requests on port 9001. The assertEqual in the tests sometimes expect to see JSON data that includes, in some cases, absolute URLs with hard coded port numbers for example: "self_link": "http://localhost:9001/3.0/lists/bee.example.com" To make the tests work I put my proxy at port 7001, changed the hard coded test requests to be transmitted to port 7001 instead of port 9001. I set testing.cfg to [webservice] port: 9001 Inside my proxy I configured all inbound queries to go to port 9001. So the server returns data including port 9001 in the self link URLs, and the tests successfully find the expected port number. Should ?self links? to anything be hard coded to a host and port number? I did wonder whether the API should be returning self links as just relative URL (i.e. no http://hostname:port ) and it is up to something else to work out the host that the self link lives on. If self links include http://hostname:port then any proxy would need to transform the data on the way back top the client to remove or correct the host and port number. Andrew On 12 Dec 2014, at 11:34 am, Barry Warsaw wrote: On Dec 12, 2014, at 08:35 AM, Andrew Stuart wrote: > I?m writing an authenticating proxy for the Mailman REST API and want to make > sure everything works as expected against real infrastructure > (Postgres/Postfix/Sqlalchemy/Falcon). We definitely want such a proxy. It's always been our intent that the core's REST API is an adminipstrative API with full access. It should only be exposed on localhost or other highly secure interface. A proxy server such as you describe would be available on a public interface and would allow people to script the management of their list memberships and list administrations. > I was hoping that just pointing the REST tests request port to my proxy would > work but no dice - lots of failures - trying to diagnose the failures has led > me back to trying to get the test suite to work against the live servers. > > I?m wondering if maybe the shortest route here might not be for me to write a > new set of tests which purely talk REST and don?t touch the back end directly > at all. Except that?s not a short route. :-) > > Trying to bend the existing tests to meet my needs is looking harder than I > have time to deal with. Even in this case, I wouldn't expect the standard unittests to work. For example, if a normal user were to try to access the list admin API, you'd expect that to fail. I think one approach would be to continue to use the test servers, but have a backdoor connection which would let you set up the state before each test and reset the state after each test. Then the tests themselves would use the authenticated API to flex the things it can and shouldn't be able to do. It would be possible for example to expose some helper resources in the admin API which would not normally be exposed in production. Something under a top level /tests resource perhaps. Cheers, -Barry _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 From andrew.stuart at supercoders.com.au Fri Dec 12 09:36:46 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Fri, 12 Dec 2014 19:36:46 +1100 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: <87y4qdfxov.fsf@uwakimon.sk.tsukuba.ac.jp> References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> <20141211162752.504ed099@limelight.wooz.org> <87y4qdfxov.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Hi Stephen Thanks for the guidance - I?ll have to admit I don?t know much about how to do such things right so the more guidance you are willing to give the better. Most of my code so far has been for me and no-one else so my next task is to learn how to write tests properly. thanks Andrew On 12 Dec 2014, at 12:45 pm, Stephen J. Turnbull wrote: Andrew Stuart writes: > I?m writing an authenticating proxy for the Mailman REST API and > want to make sure everything works as expected against real > infrastructure (Postgres/Postfix/Sqlalchemy/Falcon). Determining that "things work as expected against real infrastructure" is not what the test suite does. AFAIK we don't have any such tests, and I don't really see how the tests we do have can be adapted to "tests against real infrastructure". The problem is specifying what to expect of real infrastructure. You could, for example, write tests that insert users, query for them, and then delete them, but this has several problems. Do you want to do these tests on "live" servers that might actually be in use? -- Probably not, but how do you prevent that? Attributes of the various objects created by the system may depend on context. -- The test needs to ignore any differences due merely to different context. Coverage is necessarily imperfect in the sense that some internal state changes can't be checked since the "before" state is indeterminate in a live system. And so on. From flo.fuchs at gmail.com Tue Dec 16 12:35:11 2014 From: flo.fuchs at gmail.com (Florian Fuchs) Date: Tue, 16 Dec 2014 12:35:11 +0100 Subject: [Mailman-Developers] =?windows-1252?q?Error_installing_mailman=2E?= =?windows-1252?q?client_due_to_=22Aur=E9lien=22?= In-Reply-To: References: <011086AF-1A85-40F7-9D86-760DD7B83975@supercoders.com.au> Message-ID: <549018EF.4000401@gmail.com> Am 12.12.2014 um 00:00 schrieb Aurelien Bompard: > At first I thought I had introduced a regression.... but it's actually > just my name :-) > Don't blame me, I didn't choose it! That's a great example why French accents are great: First, in French you always know how a word is pronounced. And second, in Python they shame you into correctly decoding/encoding your I/O. :-) > (Florian, can you fix that? And by the way my last name is "Bompard", > without the "A") A fix is in the trunk. And, argh!, sorry I used your IRC nick as your last name! ;-) Florian From stephen at xemacs.org Tue Dec 16 14:08:07 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 16 Dec 2014 22:08:07 +0900 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> <20141211162752.504ed099@limelight.wooz.org> <87y4qdfxov.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87mw6neo94.fsf@uwakimon.sk.tsukuba.ac.jp> Andrew Stuart writes: > Thanks for the guidance - I?ll have to admit I don?t know much > about how to do such things right so the more guidance you are > willing to give the better. Most of my code so far has been for me > and no-one else so my next task is to learn how to write tests > properly. Well, since you got the existing tests to run against live servers, I'm pretty impressed. It certainly proves the concept" It might be worthwhile to provide a guide on using them as the last step of setting up a server, though I'm not sure how easy that would be (the easiest thing is to say "edit the code", but that's not a great way to impress new users). Did they leave any "residue" in the databases that you can detect? (That's the most important convenience of a one-shot "test server" -- you just tear it down and throw it away.) On a live server that could be bad. Eg, if a test creates a database user with a scripted password and leaves it behind, anybody that knows the script can access (some part of) the database! Steve From andrew.stuart at supercoders.com.au Tue Dec 16 21:02:26 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Wed, 17 Dec 2014 07:02:26 +1100 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: <87mw6neo94.fsf@uwakimon.sk.tsukuba.ac.jp> References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> <20141211162752.504ed099@limelight.wooz.org> <87y4qdfxov.fsf@uwakimon.sk.tsukuba.ac.jp> <87mw6neo94.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Just to clarify, I didn?t get the tests going against live servers, but I did manage to get my proxy server inserted into the ?normal? Mail man REST testing process. It?s a first step for me in getting authentication stuff going. Do ASCII diagrams work in email? Your email reader will need to use a fixed width fonr to see this: ??????????????***normal?test?configuration***???????????????????????? ????????????????????????????????????????????????????????????????????? ???????????+-------------------+????????????????????????????????????? ???????????|?Mailman?test?code?|?+--+?Sends?tests?(HTTP?requests)???? ???????????+--------+----------+????|???????????????????????????????? ????????????????????|???????????????|???????????????????????????????? Creates?HTTP?server?|???????????????|???????????????????????????????? then?tears?it?down??|???????????????|???????????????????????????????? ????????????????????|???????????????|???????????????????????????????? ??????????+---------+-----------+???|???????????????????????????????? ??????????|?????????????????????|???|???????????????????????????????? ??????????|?Mailman?HTTP?server?|?<-+???????????????????????????????? ??????????|?????????????????????|???????????????????????????????????? ??????????+---------------------+???????????????????????????????????? ????????????????????????????????????????????????????????????????????? ?????????????***my?test?configuration***????????????????????????????? ????????????????????????????????????????????????????????????????????? ?????????????+-------------------+??????????????????????????????????? ?????????????|?Mailman?test?code?|?+--+?Sends?tests?(HTTP?requests)?? ?????????????+--------+----------+????|?????????????????????????????? ??????????????????????|?????????????+-+-----------------------+?????? ??Creates?HTTP?server?|?????????????|?????????????????????????|?????? ??then?tears?it?down??|?????????????|?My proxy?server?????????|?????? ??????????????????????|?????????????|?????????????????????????|?????? ????????????+---------+-----------+?+-+-----------------------+?????? ????????????|?????????????????????|???|?????????????????????????????? ????????????|?Mailman?HTTP?server?|?<-+?????????????????????????????? ????????????|?????????????????????|?????????????????????????????????? ????????????+---------------------+?????????????????????????????????? ????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????? On 17 Dec 2014, at 12:08 am, Stephen J. Turnbull wrote: Andrew Stuart writes: > Thanks for the guidance - I?ll have to admit I don?t know much > about how to do such things right so the more guidance you are > willing to give the better. Most of my code so far has been for me > and no-one else so my next task is to learn how to write tests > properly. Well, since you got the existing tests to run against live servers, I'm pretty impressed. It certainly proves the concept" It might be worthwhile to provide a guide on using them as the last step of setting up a server, though I'm not sure how easy that would be (the easiest thing is to say "edit the code", but that's not a great way to impress new users). Did they leave any "residue" in the databases that you can detect? (That's the most important convenience of a one-shot "test server" -- you just tear it down and throw it away.) On a live server that could be bad. Eg, if a test creates a database user with a scripted password and leaves it behind, anybody that knows the script can access (some part of) the database! Steve From andrew.stuart at supercoders.com.au Tue Dec 16 23:29:40 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Wed, 17 Dec 2014 09:29:40 +1100 Subject: [Mailman-Developers] Some raw thinking on user level authorization and authentication Message-ID: This is as much gathering my thoughts on research. Anyone with questions or comments or ideas welcome. Just raw ideas at this stage. The goal is to build user level REST authentication for Mailman on top of the existing REST API. Mailman user REST authentication/authorization. Use Falcon/Talons wherever appropriate for consistency with Mailman. Authentication - establish the identity of the user Authorization - allow or deny access to specific Mailman resources based on the user identity ################Authentication & authorization process overview: step 1: authentication ?- HTTP client provides username and password ?- username and password sent to Mailman REST server ?- if Mailman REST server authenticates username and password: ?- authentication token returned to client step 2: authorization -- examines inbound requests ?- if request contains valid authentication information, continue ?- examine requested Mailman resource -? send REST request to Mailman REST server to find out if this user is allowed to do this resource request ?- if user is allowed access to resource, goto step 3 step 3: action At this stage I have three candidates for what happens next: action 1: ?- treat the inbound request as an external proxy authentication subrequest action 2: ?- directly proxy the request action 3: -- wsgi drop-through - just let the request pass through to the next level in the wsgi stack ################Possible actions action 1: ?- treat the inbound request as an external proxy authentication subrequest In which the inbound HTTP request is first handled by an external proxy server (for example Nginx), which first sends a ?subrequest? to our authentication/authorization server. If our authentication/authorization server approves the request, it responses 200 OK to the external proxy server which then directly sends the request to the Mailman REST API ** downside of subrequest based proxying ?- depends on external proxy (i.e. Nginx) that suports subrequests based authentication http://nginx.org/en/docs/http/ngx_http_auth_request_module.html ?- it would be possible to write the pure Python proxy to user this mechanism, making an external subrequesting authenticating proxy an option rather than requirement ** upside of subrequest based proxying: ? once the request is authenticated/authorized, it makes alot of sense for a specialised, high performance proxy to handle the proxying directly with the back end. All proxying edges cases and gotchas are left to the specialised proxy (i.e. Nginx) to handle, and proxy performance is as fast as the external proxy server and the Mailman REST API can talk to each other. action 2: ?- directly proxy the request In which the request is then sent on to the Mailman REST API and the response returned to the HTTP client **downside of proxying: ?- proxying is a minefield of gotchas and edge cases ?- places the authenticating proxy in the path of all requests, certainly slower than raw access to Mailman REST API **upside of proxying: ?- can be done as pure Python, no reliance on the behaviour of external proxy servers ?- the are probably fewer gotchas when dealing with one know API (i.e. Mailman REST API) versus trying to implement a generalised proxy server action 3: -- wsgi drop-through - just let the request pass through to the next level in the wsgi stack In which the inbound HTTP request comes first through the authorization/authentication wsgi layers then directly to the existing Mailman API via wsgi ** downside of wsgi drop through: ?- requires execution of the Mailman REST API as a wsgi application, which is potentially complex and more tightly coupled than desirable and would require implementation into the Mailman code base ?- put another way, the auth functions will operate entirely via queries to the Mailman REST API, and such decoupling of auth functions makes for a simpler system From stephen at xemacs.org Wed Dec 17 02:36:42 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Dec 2014 10:36:42 +0900 Subject: [Mailman-Developers] How can I run the REST tests against live servers? In-Reply-To: References: <2D09B264-75B1-48C8-9A03-084738B57144@supercoders.com.au> <20141211162752.504ed099@limelight.wooz.org> <87y4qdfxov.fsf@uwakimon.sk.tsukuba.ac.jp> <87mw6neo94.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87iohbdplh.fsf@uwakimon.sk.tsukuba.ac.jp> Andrew Stuart writes: > Do ASCII diagrams work in email? Your email reader will need to use > a fixed width fonr to see this: Thanks for the pictures! :-) They mke things very clear. Yes, for *this* list for "most" users I think it will probably work well enough. From barry at list.org Mon Dec 22 19:47:34 2014 From: barry at list.org (Barry Warsaw) Date: Mon, 22 Dec 2014 13:47:34 -0500 Subject: [Mailman-Developers] Python 3 Message-ID: <20141222134734.698db176@anthem> One of my top priorities has been to port Mailman 3 (core) to Python 3. This work is now complete, and ready to be merged into trunk. No doubt bugs still lurk, but at least the entire test suite is passing. My intent is to merge this to trunk and do another beta release before the end of the year. If you want to take a look at the branch, try this: $ bzr branch lp:~barry/mailman/py3 The diff is available online via the merge proposal: https://code.launchpad.net/~barry/mailman/py3/+merge/245313 It's a big diff. $ bzr diff . --old ../3.0 | diffstat -s 166 files changed, 1881 insertions(+), 1538 deletions(-) Python 3.4 will be the minimum required version. (It make work with earlier Python 3's but I will not officially support anything earlier.) There are a few things to keep an eye out for, especially if you are currently experimenting with Mailman 3 in production: * A few database columns have changed from LargeBinary to Unicode. I have not yet figured out how to add the Alembic schema migrations to handle this, but I'll do so before the release. * The internal format of the requests table has changed. While the column type is a string, the internal format was a sort of home baked pickle-like format. This turned out to be incompatible with Python 3, due to some assumptions made about the format. I've changed this to using a JSON representation, which is safe and platform independent. This means that any held requests (subscriptions, unsubscriptions, held posts) must be cleared before switching to the Python 3 code. I don't think it's worth trying to migrate this internal format. * The new email API in Python 3 is really wonderful, but I am mostly not yet taking advantage of it. The one area that required a significant rewrite is in the Subject prefix munging code. Some Python 2 behavior is different in Python 3 and the old algorithms had to be rewritten. I've added a lot of tests, but there are probably bugs in the new code, especially when various mixed charsets appear in the Subject header. Please do test, review, grouse, compliment, curse, rejoice, and enjoy. Over the holidays I'll spend some time cleaning a few things up (especially the documentation), then I'll merge this to trunk and do a 3.0b5 release as a Python 3-only application. Mailman 3 will not be bilingual so Python 2 support will be dropped. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From stephen at xemacs.org Tue Dec 23 05:18:47 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 23 Dec 2014 13:18:47 +0900 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141222134734.698db176@anthem> References: <20141222134734.698db176@anthem> Message-ID: <21656.60711.446815.736564@uwakimon.sk.tsukuba.ac.jp> Congratulations! Wow, Merry Christmas[1] to us! Barry Warsaw writes: > One of my top priorities has been to port Mailman 3 (core) to Python 3. This > work is now complete, and ready to be merged into trunk. Footnotes: [1] Season to taste. From aurelien at bompard.org Fri Dec 26 16:38:08 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Fri, 26 Dec 2014 16:38:08 +0100 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141222134734.698db176@anthem> References: <20141222134734.698db176@anthem> Message-ID: > One of my top priorities has been to port Mailman 3 (core) to Python 3. > This > work is now complete, and ready to be merged into trunk. No doubt bugs > still > lurk, but at least the entire test suite is passing. Congrats ! * A few database columns have changed from LargeBinary to Unicode. I have > not > yet figured out how to add the Alembic schema migrations to handle this, > but > I'll do so before the release. > This should be pretty easy, something like the following command should work: alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m "python3 port" [...] then I'll merge this to trunk and do a 3.0b5 release as a > Python 3-only application. Mailman 3 will not be bilingual so Python 2 > support will be dropped. > Wow. I am very surprised. So we went from "Python3 compatibility is not on the blocker list for Mailman 3.0" to "Mailman 3.0 will only work on Python 3". That's quite a change. This means I must now port HyperKitty and Kittystore to Python3, check that we don't use Py2-only libraries, etc. And mailman.client must be ported too, since it does import stuff from mailman for testing purposes. I think Postorious is safe, but that must be verified (I believe it imports the testing framework from mailman.client and thus must be ported too). And I thought the 3.0 release was almost there. I must say this is rather discouraging. Aur?lien From pingou at pingoured.fr Fri Dec 26 18:49:56 2014 From: pingou at pingoured.fr (Pierre-Yves Chibon) Date: Fri, 26 Dec 2014 18:49:56 +0100 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> Message-ID: <20141226174956.GN24736@carmine.pingoured.fr> On Fri, Dec 26, 2014 at 04:38:08PM +0100, Aurelien Bompard wrote: > [...] then I'll merge this to trunk and do a 3.0b5 release as a > > Python 3-only application. Mailman 3 will not be bilingual so Python 2 > > support will be dropped. > > > > Wow. I am very surprised. > So we went from "Python3 compatibility is not on the blocker list for > Mailman 3.0" to "Mailman 3.0 will only work on Python 3". > That's quite a change. > This means I must now port HyperKitty and Kittystore to Python3, check that > we don't use Py2-only libraries, etc. > And mailman.client must be ported too, since it does import stuff from > mailman for testing purposes. I think Postorious is safe, but that must be > verified (I believe it imports the testing framework from mailman.client > and thus must be ported too). This also means that mailman 3 will pretty much only not work on server-oriented distro (except probably Debian that likely ships a python 3 version, although not being involved in Debian, I do not know for sure). For the record, neither RHEL7 nor SL7 nor CentOS7 currently have a python3 stack... > And I thought the 3.0 release was almost there. I must say this is rather > discouraging. Same for me. Pierre From geoff at QuiteLikely.com Fri Dec 26 19:48:01 2014 From: geoff at QuiteLikely.com (Geoff Shang) Date: Fri, 26 Dec 2014 20:48:01 +0200 (IST) Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141226174956.GN24736@carmine.pingoured.fr> References: <20141222134734.698db176@anthem> <20141226174956.GN24736@carmine.pingoured.fr> Message-ID: On Fri, 26 Dec 2014, Pierre-Yves Chibon wrote: > This also means that mailman 3 will pretty much only not work on server-oriented > distro (except probably Debian that likely ships a python 3 version, although > not being involved in Debian, I do not know for sure). Debian Wheezy(aka Stable) ships Python3, but it's Python 3.2.3, and IIRC, Barry said Mailman3 now needs Python 3.4 in order to be fully supported. FWIW, Debian Jessy (aka Testing/Frozen?) has Python 3.4.2. HTH, Geoff. From barry at list.org Fri Dec 26 22:11:28 2014 From: barry at list.org (Barry Warsaw) Date: Fri, 26 Dec 2014 16:11:28 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> Message-ID: <20141226161128.408d9ead@anarchist.wooz.org> On Dec 26, 2014, at 04:38 PM, Aurelien Bompard wrote: >This should be pretty easy, something like the following command should work: >alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m "python3 >port" I'm having some trouble with this because the system alembic doesn't have access to the mailman source tree, which it needs (you get import errors). I tried to run alembic out of the .tox virtualenv, but that fails because the mailman config system needs to be initialized. Early on in the SA branch, there was an alembic subcommand to the mailman command. I don't think I want that, but I think we need something that calls into alembic programmatically, but with sys.path set up correctly, and the mailman system properly initialized. Maybe a 'withlist' script will do the trick. Anyway, I'll spend some time working on this and follow up here if needed. > >[...] then I'll merge this to trunk and do a 3.0b5 release as a > Python >3-only application. Mailman 3 will not be bilingual so Python 2 > support >will be dropped. > >This means I must now port HyperKitty and Kittystore to Python3 Oh gosh, no! I'm sorry I was imprecise, but I meant only that Mailman 3 *core* would be Python 3. I certainly would never make you guys do all that busy work without consulting you first. So for the record, HyperKitty and Postorious can absolutely port on their own time frame, and that need not at all block a release of the full suite. One of the advantages of accessing the core through the REST API is that it doesn't matter what clients like HyperKitty and Postorious are written in. >verified (I believe it imports the testing framework from mailman.client and >thus must be ported too). We've always said that mailman.client should be bilingual. It should be usable in either Python 2 or Python 3 scripts. I took a quick look at the current mailman.client trunk, and yes its test suite will have to be fixed, but I will take it upon myself to do that. If that's the only hangup, then I'm not worried. >And I thought the 3.0 release was almost there. I must say this is rather >discouraging. Hopefully the above provides some clarity and dis-disencouragement ;). Cheers, -Barry From barry at list.org Fri Dec 26 22:25:57 2014 From: barry at list.org (Barry Warsaw) Date: Fri, 26 Dec 2014 16:25:57 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> <20141226174956.GN24736@carmine.pingoured.fr> Message-ID: <20141226162557.576e549b@anarchist.wooz.org> On Dec 26, 2014, at 08:48 PM, Geoff Shang wrote: >On Fri, 26 Dec 2014, Pierre-Yves Chibon wrote: > >> This also means that mailman 3 will pretty much only not work on >> server-oriented distro (except probably Debian that likely ships a python 3 >> version, although not being involved in Debian, I do not know for sure). It looks like the latest Fedora has Python 3.4, though I'm not sure what version is available in the latest RedHat. Don't RH/Fedora users have something called Software Collections that can provide Python 3.4 even on older versions? >Debian Wheezy(aka Stable) ships Python3, but it's Python 3.2.3, and IIRC, >Barry said Mailman3 now needs Python 3.4 in order to be fully supported. > >FWIW, Debian Jessy (aka Testing/Frozen?) has Python 3.4.2. Yep, Jessie will have 3.4, and Ubuntu has had it since Trusty Tahr (14.04 LTS). I don't know about other distros. There's always an intricate version dance we play, but I really think 3.4 is the best base Python version to support. I would be open to merge proposals that help make it possible to run MM3 on older versions of Python 3 (but for sure nothing before 3.2), however everything that I run already has 3.4. Python 3.4 is 9 months old now and has seen two point releases, so while it make take some time for some distros to catch up, I think the release is plenty mature to use as a baseline. Cheers, -Barry From andrew.stuart at supercoders.com.au Fri Dec 26 22:26:19 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Sat, 27 Dec 2014 08:26:19 +1100 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> Message-ID: Is there anything I can do to help with these? If there?s a task list maybe I can work out what I could do. >>This means I must now port HyperKitty and Kittystore to Python3, check that >>we don't use Py2-only libraries, etc. >>And mailman.client must be ported too, since it does import stuff from >>mailman for testing purposes. I think Postorious is safe, but that must be >>verified (I believe it imports the testing framework from mailman.client >>and thus must be ported too). On 27 Dec 2014, at 2:38 am, Aurelien Bompard wrote: > One of my top priorities has been to port Mailman 3 (core) to Python 3. > This > work is now complete, and ready to be merged into trunk. No doubt bugs > still > lurk, but at least the entire test suite is passing. Congrats ! * A few database columns have changed from LargeBinary to Unicode. I have > not > yet figured out how to add the Alembic schema migrations to handle this, > but > I'll do so before the release. > This should be pretty easy, something like the following command should work: alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m "python3 port" [...] then I'll merge this to trunk and do a 3.0b5 release as a > Python 3-only application. Mailman 3 will not be bilingual so Python 2 > support will be dropped. > Wow. I am very surprised. So we went from "Python3 compatibility is not on the blocker list for Mailman 3.0" to "Mailman 3.0 will only work on Python 3". That's quite a change. This means I must now port HyperKitty and Kittystore to Python3, check that we don't use Py2-only libraries, etc. And mailman.client must be ported too, since it does import stuff from mailman for testing purposes. I think Postorious is safe, but that must be verified (I believe it imports the testing framework from mailman.client and thus must be ported too). And I thought the 3.0 release was almost there. I must say this is rather discouraging. Aur?lien _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 From barry at list.org Fri Dec 26 22:28:16 2014 From: barry at list.org (Barry Warsaw) Date: Fri, 26 Dec 2014 16:28:16 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> Message-ID: <20141226162816.78247d9c@anarchist.wooz.org> On Dec 27, 2014, at 08:26 AM, Andrew Stuart wrote: >Is there anything I can do to help with these? If there?s a task list maybe I >can work out what I could do. Hi Andrew, if you're interested in helping port Postorious and HyperKitty, I'll defer to Aurelien and Florian. Cheers, -Barry From andrew.stuart at supercoders.com.au Fri Dec 26 22:21:38 2014 From: andrew.stuart at supercoders.com.au (Andrew Stuart) Date: Sat, 27 Dec 2014 08:21:38 +1100 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> <20141226174956.GN24736@carmine.pingoured.fr> Message-ID: <8869DE25-02A6-4A22-ADC7-84A27E9BDA3D@supercoders.com.au> If it is released to version 1.0 with support for Python 2 then it will need to support Python 2 for the forseeable future. It?ll be deployed all over the place with Python 2 - when would the Python 2 be removed ever after 1.0? David Murray has made substantial changes to email in Py3 - supporting those in both 2 and 3 could be an open question. A Python 3-only version 1.0 has the longer term gain of never having to write mixed 2/3 code after 1.0. On 27 Dec 2014, at 5:48 am, Geoff Shang wrote: On Fri, 26 Dec 2014, Pierre-Yves Chibon wrote: > This also means that mailman 3 will pretty much only not work on server-oriented > distro (except probably Debian that likely ships a python 3 version, although > not being involved in Debian, I do not know for sure). Debian Wheezy(aka Stable) ships Python3, but it's Python 3.2.3, and IIRC, Barry said Mailman3 now needs Python 3.4 in order to be fully supported. FWIW, Debian Jessy (aka Testing/Frozen?) has Python 3.4.2. HTH, Geoff. _______________________________________________ Mailman-Developers mailing list Mailman-Developers at python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 From aurelien at bompard.org Fri Dec 26 22:32:04 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Fri, 26 Dec 2014 22:32:04 +0100 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141226161128.408d9ead@anarchist.wooz.org> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> Message-ID: > I'm having some trouble with this because the system alembic doesn't have > access to the mailman source tree, which it needs (you get import > errors). I > tried to run alembic out of the .tox virtualenv, but that fails because the > mailman config system needs to be initialized. > Right, I'm having issues too with the config file not being initalized. I'll have a closer look. Oh gosh, no! I'm sorry I was imprecise, but I meant only that Mailman 3 > *core* would be Python 3. I certainly would never make you guys do all > that > busy work without consulting you first. So for the record, HyperKitty and > Postorious can absolutely port on their own time frame, and that need not > at > all block a release of the full suite. One of the advantages of accessing > the > core through the REST API is that it doesn't matter what clients like > HyperKitty and Postorious are written in. > I'm not exclusively using the REST API though. I'm importing a couple interfaces, mostly the archiver interface. I'm also using the custom Message class a lot in the tests. And I'm importing a couple database types for compatibility in my own schemas (Enum and UUID) But I think the main problem is the import of mailman's config object in the class that implements the IArchiver interface. I don't believe there's another way to get the configuration. And now that I think of it, the archiver interface will be imported by Mailman core, and will thus run with a Python3 interpreter. As a result, all of KittyStore must at least be Python3 compatible. A. From barry at list.org Fri Dec 26 22:34:57 2014 From: barry at list.org (Barry Warsaw) Date: Fri, 26 Dec 2014 16:34:57 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: <8869DE25-02A6-4A22-ADC7-84A27E9BDA3D@supercoders.com.au> References: <20141222134734.698db176@anthem> <20141226174956.GN24736@carmine.pingoured.fr> <8869DE25-02A6-4A22-ADC7-84A27E9BDA3D@supercoders.com.au> Message-ID: <20141226163457.5433f510@anarchist.wooz.org> On Dec 27, 2014, at 08:21 AM, Andrew Stuart wrote: >If it is released to version 1.0 with support for Python 2 then it will need >to support Python 2 for the forseeable future. It?ll be deployed all over the >place with Python 2 - when would the Python 2 be removed ever after 1.0? >David Murray has made substantial changes to email in Py3 - supporting those >in both 2 and 3 could be an open question. A Python 3-only version 1.0 has >the longer term gain of never having to write mixed 2/3 code after 1.0. Exactly. I'm highly motivated to not release the *core* under Python 2 for the reasons you state above. Personally speaking, I've been so immersed in Python 3 for almost all my other projects (both work and play) that continuing to hack in Python 2 seems like I'm taking down my dinner with pointed sticks. ;) After finishing the port, I think it's impractical to try to have the core support both Python 2 and 3. There's *a lot* of great new stuff in the Python 3 email package, and while I'm still mostly using the legacy API, I did have to use some of the new stuff to do Subject prefix munging correctly. I expect that as time goes on, we'll need to adopt more of the new email APIs in order to fix bugs, add features, and do things more efficiently. That's not even talking about so much of the other great new stdlib stuff available in Python 3. Cheers, -Barry From barry at list.org Fri Dec 26 22:47:29 2014 From: barry at list.org (Barry Warsaw) Date: Fri, 26 Dec 2014 16:47:29 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> Message-ID: <20141226164729.4a72f134@anarchist.wooz.org> On Dec 26, 2014, at 10:32 PM, Aurelien Bompard wrote: >Right, I'm having issues too with the config file not being initalized. >I'll have a closer look. Awesome, thanks. >I'm not exclusively using the REST API though. I'm importing a couple >interfaces, mostly the archiver interface. I'm also using the custom Message >class a lot in the tests. And I'm importing a couple database types for >compatibility in my own schemas (Enum and UUID) I wonder if it would be possible to refactor some of this code into separate libraries, which would reduce the surface area for bilingual support. The config object is probably the tricky part here, but also the Message class has a lot less in it now that we don't have to worry about all that unicode stuff. Really, it's essentially just the .sender and .senders properties and some backward compatible pickle support (unnecessary atm for MM3 and probably not much needed going forward, unless the stdlib Message object in some future Python grows internal attributes that break unpickling - I'll certainly test that against what will be Python 3.5). >But I think the main problem is the import of mailman's config object in the >class that implements the IArchiver interface. I don't believe there's >another way to get the configuration. But there could be! I took a very quick look at HK (but not KittyStore) and it doesn't look like you need much. What if I added a REST API to access the core's system configuration settings? They would be read-only, and I would only start with the minimum of what HK needs. I think it's probably dangerous in the long term to expect the core's internal config API to remain stable, so accessing that through REST would provide the guarantees you need, in addition to decoupling HK from the core. >And now that I think of it, the archiver interface will be imported by >Mailman core, and will thus run with a Python3 interpreter. As a result, all >of KittyStore must at least be Python3 compatible. Yep, I missed that. Is it possible to feed HK messages out-of-process? E.g. via the command line? If so, then we write HK's IArchiver implementation to make those out of process calls, and we can even pull that bit into the core. I would be happy to help with that. Cheers, -Barry From aurelien at bompard.org Fri Dec 26 23:02:14 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Fri, 26 Dec 2014 23:02:14 +0100 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141226164729.4a72f134@anarchist.wooz.org> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <20141226164729.4a72f134@anarchist.wooz.org> Message-ID: > But there could be! I took a very quick look at HK (but not KittyStore) > and > it doesn't look like you need much. What if I added a REST API to access > the > core's system configuration settings? > That would work. > Yep, I missed that. Is it possible to feed HK messages out-of-process? > E.g. via the command line? Not now. Would an LMTP server be appropriate? If so, could Mailman's internal LMTP server be split off into a Py2-compatible library? What protocol would be best suited in your opinion? A. From barry at list.org Sat Dec 27 00:05:20 2014 From: barry at list.org (Barry Warsaw) Date: Fri, 26 Dec 2014 18:05:20 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <20141226164729.4a72f134@anarchist.wooz.org> Message-ID: <20141226180520.2640a307@anarchist.wooz.org> On Dec 26, 2014, at 11:02 PM, Aurelien Bompard wrote: >> But there could be! I took a very quick look at HK (but not KittyStore) >> and it doesn't look like you need much. What if I added a REST API to >> access the core's system configuration settings? > >That would work. Cool, let me see what I can whip up. >> Yep, I missed that. Is it possible to feed HK messages out-of-process? >> E.g. via the command line? > >Not now. Would an LMTP server be appropriate? If so, could Mailman's internal >LMTP server be split off into a Py2-compatible library? What protocol would >be best suited in your opinion? LMTP might be a very interesting way to do it. The core's LMTP server is actually a relatively thin layer over the stdlib's smtpd library, and not much really had to be changed in the Py3 port (IIRC, the error status messages changed from bytes to strs, but that should be relatively easy to straddle via the six package). The relevant API is IArchiver.archive_message() which takes an mlist and a msg, since that's the API that sends the message to the archiver. .permalink() would be implemented in the core. It looks like HK/KS just needs the fqdn listname of the destination list, plus the message. The latter would be available over LMTP of course, and while the list name could usually be dug out of the message headers, it would probably be best to provide it in a special Mailman header. The alternative of course is adding a REST API to HK, but that seems a little heavier weight for what would probably be just one API (for now anyway). What do you think? Cheers, -Barry From stephen at xemacs.org Sat Dec 27 04:42:33 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 27 Dec 2014 12:42:33 +0900 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> Message-ID: <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> Aurelien Bompard writes: > Barry writes: > > One of the advantages of accessing the core through the REST API > > is that it doesn't matter what clients like HyperKitty and > > Postorious are written in. > I'm not exclusively using the REST API though. I'm importing a > couple interfaces, mostly the archiver interface. This was a design mistake, I think. An IArchiver really needs to be in core for two reasons: first, it needs to generate a permalink and attach it to the message as distributed. Second, it needs to associate that permalink with the message so the "real" archiver process will do set that link correctly. For a production-quality archiver, that's *all* it should do. The archiver itself should be a separate process, receiving the message and permalink data by IPC. @Barry: maybe rename IArchiver to IPermalinker? ;-) Remember, Mailman core is going to be distributing Internet mail. Except in the case where 100.00% of the users are on one host, that's going to be the bottleneck on message processing. Archiving simply does not need to be fast. The archiver can implement LMTP even, although that would be overkill if we didn't already have an LMTP server. The simplest approach would be to simply put the Archive-Permalink header in the message and stream it to the archiver which would parse it out. > I'm also using the custom Message class a lot in the tests. Can't avoid that with Python 2, I guess, but using Message will be *so* much less painful with Python 3. > But I think the main problem is the import of mailman's config > object in the class that implements the IArchiver interface. I > don't believe there's another way to get the configuration. If you need that configuration (which, come to think of it, you probably do, at least parts of it), you could have a private protocol for communicating it as metadata (a message header or as metadata in the stream). > And now that I think of it, the archiver interface will be imported > by Mailman core, and will thus run with a Python3 interpreter. As a > result, all of KittyStore must at least be Python3 compatible. In the long run (ie, when nobody who's anybody uses Python 2 at all) I think everybody would be happier if you refactor to keep KittyStore at arm's length from Mailman core. From stephen at xemacs.org Sat Dec 27 04:48:43 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 27 Dec 2014 12:48:43 +0900 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141226164729.4a72f134@anarchist.wooz.org> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <20141226164729.4a72f134@anarchist.wooz.org> Message-ID: <871tnl4usk.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > But there could be! I took a very quick look at HK (but not > KittyStore) and it doesn't look like you need much. What if I > added a REST API to access the core's system configuration > settings? They would be read-only, and I would only start with the > minimum of what HK needs. Let's please take some care to think about this. ISTM this is like the Nth time we've bumped into "you need to be core to access config data". > I think it's probably dangerous in the long term to expect the > core's internal config API to remain stable, so accessing that > through REST would provide the guarantees you need, in addition to > decoupling HK from the core.h This *may* be good enough (what with the guarantees the REST interface provides), but I think this really needs to take a generic approach, and some care should be taken to confirm that it is generic. It should eventually be TOOWTDI (ie, Postorius, mailman.client, and the new CLI should all go this way). From barry at python.org Sat Dec 27 05:08:20 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 26 Dec 2014 23:08:20 -0500 Subject: [Mailman-Developers] Python 3 References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <20141226164729.4a72f134@anarchist.wooz.org> Message-ID: <20141226230820.58ad4948@anarchist.wooz.org> On Dec 26, 2014, at 11:02 PM, Aurelien Bompard wrote: >> But there could be! I took a very quick look at HK (but not KittyStore) >> and it doesn't look like you need much. What if I added a REST API to >> access the core's system configuration settings? > >That would work. It turned out to be almost ridiculously easy, so that probably means it's a good idea . Quoting from the documentation: ==================== System configuration ==================== The entire system configuration is available through the REST API. You can get a list of all defined sections. >>> dump_json('http://localhost:9001/3.0/system/configuration') http_etag: ... sections: ['antispam', 'archiver.mail_archive', 'archiver.master', ... You can also get all the values for a particular section. >>> dump_json('http://localhost:9001/3.0/system/configuration/mailman') default_language: en email_commands_max_lines: 10 filtered_messages_are_preservable: no http_etag: ... layout: testing noreply_address: noreply pending_request_life: 3d post_hook: pre_hook: sender_headers: from from_ reply-to sender site_owner: noreply at example.com Dotted section names work too, for example, to get the French language settings section. >>> dump_json('http://localhost:9001/3.0/system/configuration/language.fr') charset: iso-8859-1 description: French enabled: yes http_etag: ... -----end quote----- Of course, these configuration resources are read-only. Also, I am deprecating the `//system` resource path; for backward compatibility this will still return the system versions, but you'll notice that the self_link now points to `/system/versions` which is the new canonical resource path. I'll remove the old path when the API version gets rev'd. This makes a good point though: I'm not outright discouraging applications from using core as a library, and in fact that was part of the point of making everything importable via the `mailman` package. But I'm far from ready to make any guarantees of stability for the internal API, including interfaces. The REST API however I *am* making guarantees about, as evidenced by using an API version at the root of the resource tree. Note that configuration values are verbatim strings. You might need to convert the value to some typed data, e.g. [mailman]pending_request_lifetime has a value of '3d' which is shorthand for a timedelta. Clients must convert those data types themselves. Similarly for ints, booleans, etc. Multiline or whitespace separated strings values must be .split() by the client. But you *do* have access to all configuration settings now, although only on the py3 branch atm. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at list.org Sat Dec 27 05:10:26 2014 From: barry at list.org (Barry Warsaw) Date: Fri, 26 Dec 2014 23:10:26 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: <871tnl4usk.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <20141226164729.4a72f134@anarchist.wooz.org> <871tnl4usk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141226231026.6b9a2ec3@anarchist.wooz.org> On Dec 27, 2014, at 12:48 PM, Stephen J. Turnbull wrote: >Let's please take some care to think about this. ISTM this is like >the Nth time we've bumped into "you need to be core to access config >data". No more! See my other follow up. And there's no restriction on values. Sometimes generic is easier than specific (shouldn't that be a Zen of Python? :). Hopefully Aurelien can confirm that this will be enough for HK, but of course Postorious can use it now too. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Sat Dec 27 05:18:46 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 26 Dec 2014 23:18:46 -0500 Subject: [Mailman-Developers] Python 3 References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141226231846.17eb0ab8@anarchist.wooz.org> On Dec 27, 2014, at 12:42 PM, Stephen J. Turnbull wrote: >Remember, Mailman core is going to be distributing Internet mail. >Except in the case where 100.00% of the users are on one host, that's >going to be the bottleneck on message processing. Archiving simply >does not need to be fast. The archiver can implement LMTP even, >although that would be overkill if we didn't already have an LMTP >server. The simplest approach would be to simply put the >Archive-Permalink header in the message and stream it to the archiver >which would parse it out. Remember too that in MM3, messages only get fed to the registered IArchiver interfaces by a separate archive runner. So they aren't a bottleneck for delivery to the user, but on heavily trafficked sites, they can potential consume a lot of resources if the archiver is local and relatively inefficient. I tend to agree that a good design for any archiver is to be able to accept messages over an IPC channel. A site may for example want to run the core on one system and HK on another system (e.g. separate VMs perhaps). This would only really be possible if the core can feed HK messages over a configurable IPC. As I mentioned, I think LMTP *could* work, but REST (inside HK) could work too. Aurelien, what do you think? > > I'm also using the custom Message class a lot in the tests. > >Can't avoid that with Python 2, I guess, but using Message will be >*so* much less painful with Python 3. See my other follow up. I don't think there's much in the py3 branche's Message class that is going to be super helpful to HK. Even the core *could* use the standard email.message.Message class with a couple of utility functions, but it's just a little easier to add a few properties and methods in the subclasses. >In the long run (ie, when nobody who's anybody uses Python 2 at all) I >think everybody would be happier if you refactor to keep KittyStore at >arm's length from Mailman core. Agreed, with of course the caveat that we'll need a thin HK IArchiver implementation in the core to generate the permalink and communicate with HK over IPC. Generally we want the permalink to be able to be generated without direction communication with the archiver (see the motivation for X-Message-ID-Hash), but if the core *has* to talk to HK to generate the permalink, then I don't think an LMTP channel will work. In that case REST or some homegrown protocol may be the answer. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From stephen at xemacs.org Sat Dec 27 06:12:43 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 27 Dec 2014 14:12:43 +0900 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141226231026.6b9a2ec3@anarchist.wooz.org> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <20141226164729.4a72f134@anarchist.wooz.org> <871tnl4usk.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231026.6b9a2ec3@anarchist.wooz.org> Message-ID: <87zja93cc4.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > No more! See my other follow up. And there's no restriction on > values. Sometimes generic is easier than specific (shouldn't that > be a Zen of Python? :). Uh-oh. Looks like somebody is using EggNog programming style! Or are you just high on life? ;-) From stephen at xemacs.org Sat Dec 27 07:57:56 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 27 Dec 2014 15:57:56 +0900 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141226231846.17eb0ab8@anarchist.wooz.org> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231846.17eb0ab8@anarchist.wooz.org> Message-ID: <87y4pt37gr.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > Remember too that in MM3, messages only get fed to the registered > IArchiver interfaces by a separate archive runner. So they aren't > a bottleneck for delivery to the user, but on heavily trafficked > sites, they can potential consume a lot of resources if the > archiver is local and relatively inefficient. I'm talking about total load on the server host, not load on the Mailman subsystem. So I don't think the Mailman-to-archive function will consume many resources compared to delivery to subscribers if there are any remote users at all. A local archiver communicates at CPU-to-disk speed basically once or maybe twice as I understand it. The MTA resources for queuing alone will exceed and probably overwhelm this. Then there are the multiple Mailman queues, etc, etc. Of course the *other* side of the archiver (the client access to the message store) can be extremely resource consuming. I'm just saying that in the grand scheme of message distribution (including to the archiver), the efficiency of a local archiver is not going to be a bottleneck. > >In the long run (ie, when nobody who's anybody uses Python 2 at > >all) I think everybody would be happier if you refactor to keep > >KittyStore at arm's length from Mailman core. > > Agreed, with of course the caveat that we'll need a thin HK > IArchiver implementation in the core to generate the permalink and > communicate with HK over IPC. Generally we want the permalink to > be able to be generated without direction communication with the > archiver (see the motivation for X-Message-ID-Hash), By the way, I would say to adopt modern IETF practice here and drop the "X-" (in practice collisions are rare while the annoyance of fixing platforms to use the standardized name is frequent), and include the algorithm in the name. Eg, Message-ID-MD5 or Hashed-Message-ID-MD5. Or we could use the List-* namespace. We should do this while we still can. :-) If you want I can try to write an RFC to make it official. > but if the core *has* to talk to HK to generate the permalink, I personally don't think that is a good idea, but see below. > then I don't think an LMTP channel will work. The only reason I can think of is that you want to check that the permalink isn't already occupied (that's the only thing HyperKitty proper knows that can't be computed the same way in the IArchiver as in HyperKitty proper AFAICS), and that can be implemented as follows Mailman> LHLO mailman-host HyperKitty> 250 OK Mailman> MAIL FROM Mailman at mailman-host HyperKitty> 250 OK Mailman> RCPT TO @archiver-host HyperKitty> 553 Permalink already occupied Mailman> RCPT TO @archiver-host HyperKitty> 250 OK Mailman> DATA HyperKitty> 354 Go for it! and so on. I don't think this even violates the spirit of the LMTP protocol, but it certainly conforms to the letter as long as permalink variable parts are valid email localparts. (One could quibble about which 5xx response to give. AFAICS only "551 user not local" is out.) My own preference is for a permalink that can be computed from the originator header data (author, recipients, date, message ID, subject) by anyone with access to the message, and that means you need the archive server to be able to deal gracefully with collisions. (In practice message IDs are not perfect UUIDs, although they're very close, and some messages don't have them or have different ones assigned by mediating hosts at arrival at multiple recipients.) Steve From flo.fuchs at gmail.com Sat Dec 27 08:41:56 2014 From: flo.fuchs at gmail.com (Florian Fuchs) Date: Sat, 27 Dec 2014 08:41:56 +0100 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141226231846.17eb0ab8@anarchist.wooz.org> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231846.17eb0ab8@anarchist.wooz.org> Message-ID: <549E62C4.2020602@gmail.com> Am 27.12.2014 um 05:18 schrieb Barry Warsaw: > I tend to agree that a good design for any archiver is to be able to accept > messages over an IPC channel. A site may for example want to run the core on > one system and HK on another system (e.g. separate VMs perhaps). This would > only really be possible if the core can feed HK messages over a configurable > IPC. As I mentioned, I think LMTP *could* work, but REST (inside HK) could > work too. Aurelien, what do you think? How about a third option, a generic "pub/sub" or "event" archiver, implemented in py3? It would meet all the above criteria (archivers on other systems, no dependency on py3 -- or Python for that matter). We could start supporting one backend (zeromq for instance) and maybe add more later. Flo From flo.fuchs at gmail.com Sat Dec 27 13:16:51 2014 From: flo.fuchs at gmail.com (Florian Fuchs) Date: Sat, 27 Dec 2014 13:16:51 +0100 Subject: [Mailman-Developers] MM3 REST API testing [was: Python 3] In-Reply-To: References: <20141222134734.698db176@anthem> Message-ID: <549EA333.6000607@gmail.com> Am 26.12.2014 um 16:38 schrieb Aurelien Bompard: > And mailman.client must be ported too, since it does import stuff from > mailman for testing purposes. I think Postorious is safe, but that must be > verified (I believe it imports the testing framework from mailman.client > and thus must be ported too). I think, this is worth another thread. So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed. The first solution for that was to start the core (out of process) with a temporary configuration and on a different port before running the tests and tearing it down afterwards. That way mailman.client/Postorius would be tested with a real MM core underneath it. Compatibility with the API was guaranteed, but it was painfully slow and also a little unstable. It's not fun when every typo is punished with a 10 or 20s penalty! That's why I was extremely happy when Aur?lien came up with a much better idea a while ago. His solution made use of the webtest package, which made it possible to test the MM3 API wsgi app without the HTTP overhead or the need to start/stop the main MM process. Much faster, more stable, a real delight compared to the situation we had before. Unfortunately, this now means direct "from mailman" imports which will break with incompatible Python versions. Too bad, we didn't think about this (I, for one, was aware of the fact that Barry plans to do a monolingual port at some point in the future. So I could have thought about that earlier. :-/ meh...). Of course, in order to make our tests work with a py3 MM core, we could go back to the slow and no-fun way of testing that we had before. But maybe there's better way... Any ideas? Cheers Flo From aurelien at bompard.org Sun Dec 28 13:51:15 2014 From: aurelien at bompard.org (Aurelien Bompard) Date: Sun, 28 Dec 2014 13:51:15 +0100 Subject: [Mailman-Developers] Python 3 In-Reply-To: <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: > This was a design mistake, I think. Yeah, I see that now. But at that time, if I had known that I had two more years (and counting) before MM3 was released, I'd probably have made different choices. I hope I'm not making the same mistake again. > As I mentioned, I think LMTP *could* work, but REST (inside HK) could > work too. Aurelien, what do you think? I'd go with REST, it seems more flexible and we already have nice libraries for it. A. From tanstaafl at libertytrek.org Sun Dec 28 15:23:36 2014 From: tanstaafl at libertytrek.org (Tanstaafl) Date: Sun, 28 Dec 2014 09:23:36 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141226162557.576e549b@anarchist.wooz.org> References: <20141222134734.698db176@anthem> <20141226174956.GN24736@carmine.pingoured.fr> <20141226162557.576e549b@anarchist.wooz.org> Message-ID: <54A01268.1070805@libertytrek.org> On 12/26/2014 4:25 PM, Barry Warsaw wrote: > On Dec 26, 2014, at 08:48 PM, Geoff Shang wrote: >> FWIW, Debian Jessy (aka Testing/Frozen?) has Python 3.4.2. > Yep, Jessie will have 3.4, and Ubuntu has had it since Trusty Tahr (14.04 > LTS). I don't know about other distros. First, Jessie is not even released yet, wheezy is the current stable, and still has a long lifecycle remaining. Second, I (for one, but there are many other debian admins maintaining servers who feel the same) will not be upgrading my wheezy install to jessie due to the ongoing systemd debacle. Locking them out would be a mistake, imo. From barry at list.org Sun Dec 28 23:51:08 2014 From: barry at list.org (Barry Warsaw) Date: Sun, 28 Dec 2014 17:51:08 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: <87y4pt37gr.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231846.17eb0ab8@anarchist.wooz.org> <87y4pt37gr.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141228175108.1d905288@anthem> On Dec 27, 2014, at 03:57 PM, Stephen J. Turnbull wrote: >By the way, I would say to adopt modern IETF practice here and drop >the "X-" (in practice collisions are rare while the annoyance of >fixing platforms to use the standardized name is frequent), and >include the algorithm in the name. Eg, Message-ID-MD5 or >Hashed-Message-ID-MD5. Or we could use the List-* namespace. > >We should do this while we still can. :-) If you want I can try to >write an RFC to make it official. I like the idea of putting this information in a List-* header, and I'll take you up on the RFC offer. Are you thinking about trying to push this through the IETF to make it official? The spec currently lives on the wiki: http://wiki.list.org/display/DEV/Stable+URLs MM3 and HK should both be implementing this now, and I think mail-archive.com does too. If we change the header name, I'd want to keep X-Message-ID-Hash for the MM3 final release, but deprecate it. I.e. MM3 would write both headers. As for what the List-* header would be, well, if you wanted to include the algorithm name, to be completely accurate it would have to be something like List-Base32-Encoded-SHA1-Hash-Of-The-Message-ID. Yuck ;) The value of this header both serves to uniquely identify the message in a more regular format, and to serve as the final path component in the Archived-At (RFC 5064) header. So the following names come to mind: List-Message-ID List-Archive-ID List-Archived-At-ID suggestions welcome. >The only reason I can think of is that you want to check that the permalink >isn't already occupied (that's the only thing HyperKitty proper knows that >can't be computed the same way in the IArchiver as in HyperKitty proper >AFAICS) Right. However, when this was discussed several years ago, the mail-archive.com guys did some extensive data analysis on their vast collection of email. You'd have to go spelunking in the -developers archives for details, but I recall that the collision rate was so small as to be effectively negligible, even more so if you ignore spam. And if the X-Message-ID-Hash collides, then the Message-ID will collide, and it's likely that any archiver would drop the message anyway. >My own preference is for a permalink that can be computed from the >originator header data (author, recipients, date, message ID, subject) >by anyone with access to the message, and that means you need the >archive server to be able to deal gracefully with collisions. (In >practice message IDs are not perfect UUIDs, although they're very >close, and some messages don't have them or have different ones >assigned by mediating hosts at arrival at multiple recipients.) Right, we hash (pun intended :) all this out years ago. We can ignore collisions, and we can do the entire calculation on the server side, using Message-ID as the sole input. I think the only issue that's worth reopening is the name of the header. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at list.org Mon Dec 29 00:23:31 2014 From: barry at list.org (Barry Warsaw) Date: Sun, 28 Dec 2014 18:23:31 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: <54A01268.1070805@libertytrek.org> References: <20141222134734.698db176@anthem> <20141226174956.GN24736@carmine.pingoured.fr> <20141226162557.576e549b@anarchist.wooz.org> <54A01268.1070805@libertytrek.org> Message-ID: <20141228182331.1ab37b55@anthem> On Dec 28, 2014, at 09:23 AM, Tanstaafl wrote: >Second, I (for one, but there are many other debian admins maintaining >servers who feel the same) will not be upgrading my wheezy install to >jessie due to the ongoing systemd debacle. I *definitely* don't want to infect this list with a debate about systemd, but as I understand it, while systemd will be the default in Jessie, it will still be possibly to run it with other init systems. Sticking to the issue at hand: whether we should try to support older Python 3 versions in order to support older Linux distro versions. The mechanics of setting up automated tests for those two versions is tricky, but I took a look at the test suite for the py3 branch against Python 3.3 and 3.2. Python 3.3 is no problem; the test suite passes fully. This doesn't help for Wheezy though. Python 3.2 is more problematic. There are quite a few things that aren't supported, some of which would be easy to fix (e.g. residual uses of u'' strings), some which could be compat'd around (e.g. missing unittest.mock library and contextlib.ExitStack classes), and some which are probably blockers. The biggest immediate problem seems to be that Falcon doesn't support Python 3.2. Its minimum requirement is 3.3. But here's the thing: let's say you want to run MM3 on Wheezy, or some other Linux distro that doesn't have Python 3.4. It's unlikely that MM3 will be backported into those distro's backports channels, so you'll have to install it from some alternative location, like from source, a PPA, or an unofficial channel. The same probably goes for the newer versions of some MM3 dependencies. If you're going to install MM3 that way anyway, is it that much more effort to install Python 3.4 that way too? Cheers, -Barry From barry at list.org Mon Dec 29 00:26:48 2014 From: barry at list.org (Barry Warsaw) Date: Sun, 28 Dec 2014 18:26:48 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141228182648.76d40d96@anthem> On Dec 28, 2014, at 01:51 PM, Aurelien Bompard wrote: >> As I mentioned, I think LMTP *could* work, but REST (inside HK) could >> work too. Aurelien, what do you think? > >I'd go with REST, it seems more flexible and we already have nice libraries >for it. REST will be very nice, and I think with something like Falcon, pretty easy to implement. You probably don't need the object-based (i.e. restish-like) traversal stuff, but if so, I'd be happy to split that out into a separately library that HK could use (and keep it bilingual). Keep it simple, but think through the resource tree and leave open some opportunities for expanding the API later, both for use by MM3 and perhaps other scripting applications. Cheers, -Barry From barry at list.org Mon Dec 29 00:27:40 2014 From: barry at list.org (Barry Warsaw) Date: Sun, 28 Dec 2014 18:27:40 -0500 Subject: [Mailman-Developers] Python 3 In-Reply-To: <549E62C4.2020602@gmail.com> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231846.17eb0ab8@anarchist.wooz.org> <549E62C4.2020602@gmail.com> Message-ID: <20141228182740.00e6613e@anthem> On Dec 27, 2014, at 08:41 AM, Florian Fuchs wrote: >How about a third option, a generic "pub/sub" or "event" archiver, >implemented in py3? It would meet all the above criteria (archivers on other >systems, no dependency on py3 -- or Python for that matter). > >We could start supporting one backend (zeromq for instance) and maybe >add more later. This would be pretty cool. Would you be interested in writing an IArchiver implementation for that? Cheers, -Barry From stephen at xemacs.org Mon Dec 29 02:13:16 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 29 Dec 2014 10:13:16 +0900 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141228175108.1d905288@anthem> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231846.17eb0ab8@anarchist.wooz.org> <87y4pt37gr.fsf@uwakimon.sk.tsukuba.ac.jp> <20141228175108.1d905288@anthem> Message-ID: <87ppb32r83.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > I like the idea of putting this information in a List-* header, and > I'll take you up on the RFC offer. OK. > Are you thinking about trying to push this through the IETF to make > it official? Yes. It will depend on how much resistance I get, but having it already implemented and used in Mailman will certainly help. On the other hand, there may be resistance on the basis that RFC 5064 already does everything that is "really" needed. > The spec currently lives on the wiki: > > http://wiki.list.org/display/DEV/Stable+URLs Yes, I'm a little bit familiar with that spec. :-) > If we change the header name, I'd want to keep X-Message-ID-Hash > for the MM3 final release, but deprecate it. I.e. MM3 would write > both headers. I'll ask some of the IETF guys what they think about that. But if you put it in a public release, you're screwing the same kind of people Tanstaafl was talking about. Beta testers (and I mean beta testers, ie, people who have put the code in production even though it's not considered a public release) have signed up for this kind of annoyance. Random ancient Debian sysadmins haven't. Of course we don't want to abuse our beta testers if we can avoid it, but I think if we don't want to maintain dual headers indefinitely, the public release is the time to get rid of the X- version. > As for what the List-* header would be, well, if you wanted to > include the algorithm name, to be completely accurate it would have > to be something like > List-Base32-Encoded-SHA1-Hash-Of-The-Message-ID. Yuck ;) We'd have to think somewhat carefully about how strong a hash we want to use if we don't specify algorithm in the field name. I'm not particularly concerned with how many bytes the header takes up. Future users can just deal with the implied BASE32 vs. BASE85 or whatever. However, if somebody thinks they need a stronger hash than we chose, we'll have interoperability problems for people who receive the message off-list. > The value of this header both serves to uniquely identify the > message in a more regular format, and to serve as the final path > component in the Archived-At (RFC 5064) header. So the following > names come to mind: > > List-Message-ID > List-Archive-ID > List-Archived-At-ID > > suggestions welcome. The last two are too easily confused with Archived-At. > Right. However, when this was discussed several years ago, the > mail-archive.com guys did some extensive data analysis on their > vast collection of email. You'd have to go spelunking in the > -developers archives for details, but I recall that the collision > rate was so small as to be effectively negligible, Yes. The problem is that there are people out there with MUAs that provide bogus Message-IDs (Kyle Jones's VM used to do that), and for those people all messages after the first get dropped. Note that if the server does indeed ignore the possibility of collisions on Message-ID, then there is no need (AFAICS) for the "thin" IArchiver to communicate with the archiver proper. I don't see how it hurts to provide for the possibility of an archiver that does check content. > Right, we hash (pun intended :) all this out years ago. We can > ignore collisions, and we can do the entire calculation on the > server side, using Message-ID as the sole input. I think the only > issue that's worth reopening is the name of the header. Well, that's true for *us*. The folks at the IETF don't have a habit of leaving well enough alone, though. ;-) From barry at list.org Mon Dec 29 19:02:06 2014 From: barry at list.org (Barry Warsaw) Date: Mon, 29 Dec 2014 13:02:06 -0500 Subject: [Mailman-Developers] X-Message-ID-Hash header (was Re: Python 3) In-Reply-To: <87ppb32r83.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231846.17eb0ab8@anarchist.wooz.org> <87y4pt37gr.fsf@uwakimon.sk.tsukuba.ac.jp> <20141228175108.1d905288@anthem> <87ppb32r83.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141229130206.18c3e335@anthem> On Dec 29, 2014, at 10:13 AM, Stephen J. Turnbull wrote: > > If we change the header name, I'd want to keep X-Message-ID-Hash > > for the MM3 final release, but deprecate it. I.e. MM3 would write > > both headers. > >I'll ask some of the IETF guys what they think about that. But if you >put it in a public release, you're screwing the same kind of people >Tanstaafl was talking about. Beta testers (and I mean beta testers, >ie, people who have put the code in production even though it's not >considered a public release) have signed up for this kind of >annoyance. Random ancient Debian sysadmins haven't. > >Of course we don't want to abuse our beta testers if we can avoid it, >but I think if we don't want to maintain dual headers indefinitely, >the public release is the time to get rid of the X- version. I'd be willing to drop it if we can get agreement on the new header, and get buy-in from at least HK (abompard) and the mail-archive.com folks. AFAIK, they are the only two "clients" of the header atm. I'm not sure if the Jeffs are still reading this list, so I've CC'd them directly. Jeffs: we are considering changing the X-Message-Hash-ID header name, at least dropping the X- prefix and possibly renaming the header. > > As for what the List-* header would be, well, if you wanted to > > include the algorithm name, to be completely accurate it would have > > to be something like > > List-Base32-Encoded-SHA1-Hash-Of-The-Message-ID. Yuck ;) > >We'd have to think somewhat carefully about how strong a hash we want to use >if we don't specify algorithm in the field name. I'm not particularly >concerned with how many bytes the header takes up. Future users can just >deal with the implied BASE32 vs. BASE85 or whatever. However, if somebody >thinks they need a stronger hash than we chose, we'll have interoperability >problems for people who receive the message off-list. Base 32 is a good trade-off between compactness and readability. > > The value of this header both serves to uniquely identify the > > message in a more regular format, and to serve as the final path > > component in the Archived-At (RFC 5064) header. So the following > > names come to mind: > > > > List-Message-ID > > List-Archive-ID > > List-Archived-At-ID > > > > suggestions welcome. > >The last two are too easily confused with Archived-At. Suggestions welcome. :) > > Right. However, when this was discussed several years ago, the > > mail-archive.com guys did some extensive data analysis on their > > vast collection of email. You'd have to go spelunking in the > > -developers archives for details, but I recall that the collision > > rate was so small as to be effectively negligible, > >Yes. The problem is that there are people out there with MUAs that >provide bogus Message-IDs (Kyle Jones's VM used to do that), and for >those people all messages after the first get dropped. As you know, I have limited tolerance for broken MUAs. Gosh, do people still use VM? :) >Note that if the server does indeed ignore the possibility of >collisions on Message-ID, then there is no need (AFAICS) for the >"thin" IArchiver to communicate with the archiver proper. Right. MM3 does not current reject messages with duplicate Message-IDs, but I think it should. I had a branch in flight that implemented this, but it caused some failures I wasn't able to resolve, and the branch bitrotted. >I don't see how it hurts to provide for the possibility of an archiver that >does check content. > > > Right, we hash (pun intended :) all this out years ago. We can > > ignore collisions, and we can do the entire calculation on the > > server side, using Message-ID as the sole input. I think the only > > issue that's worth reopening is the name of the header. > >Well, that's true for *us*. The folks at the IETF don't have a habit >of leaving well enough alone, though. ;-) Right, so let's do what *we* think is right, right now, and let the committee take 10 years to define a standard. ;) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Mon Dec 29 19:28:35 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 29 Dec 2014 13:28:35 -0500 Subject: [Mailman-Developers] Python 3 References: <20141222134734.698db176@anthem> Message-ID: <20141229132835.1d09e957@anthem> Here's a crazy, probably stupid idea. I want to do one more release before the end of the year, probably tomorrow. What if I do A/B releases for 3.0b5? A would be current snapshot of trunk, i.e. MM3 running on Python 2.7. B would be current snapshot of the py3 branch, i.e. MM3 running on Python 3.4. I will hold of on the merge until 2015. The advantage of this is that folks would get a chance to download the tarballs of their choice and give it a try. I'd encourage people to try the Python 3 (i.e. B) release. I think people are more engaged with an actual release tarball than with bzr branches, so this could provide some good feedback. I think right now the two branches are functionally equivalent. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From stephen at xemacs.org Tue Dec 30 14:33:03 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 30 Dec 2014 22:33:03 +0900 Subject: [Mailman-Developers] X-Message-ID-Hash header (was Re: Python 3) In-Reply-To: <20141229130206.18c3e335@anthem> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231846.17eb0ab8@anarchist.wooz.org> <87y4pt37gr.fsf@uwakimon.sk.tsukuba.ac.jp> <20141228175108.1d905288@anthem> <87ppb32r83.fsf@uwakimon.sk.tsukuba.ac.jp> <20141229130206.18c3e335@anthem> Message-ID: <87oaql2rg0.fsf@uwakimon.sk.tsukuba.ac.jp> No Jeff-relevant discussion here, I think, so I'm not going to spam. Barry Warsaw writes: > >The last two are too easily confused with Archived-At. > > Suggestions welcome. :) When I have one, of course. But it's worth ruling out non-starters quickly, if possible. > >Yes. The problem is that there are people out there with MUAs > >that provide bogus Message-IDs (Kyle Jones's VM used to do that), > >and for those people all messages after the first get dropped. > > As you know, I have limited tolerance for broken MUAs. As I also know, you don't usually impose your opinions on third parties with a different point of view. And the first mission of mail-related applications is to make sure mail gets to where it's suppose to go. Or do you want your name cursed in the same breath with AOL and "Yahoo!"? :-) > Gosh, do people still use VM? :) Sure. The main difference between VM "virtual folders" and Gmail "labels" is that virtual folders actually do what labels are advertised to do. :-) > Right. MM3 does not current reject messages with duplicate > Message-IDs, but I think it should. That sounds like a mess to me. For one thing, do you mean "reject" (and the sender gets a bounce) or "discard" (silently)? Neither of those sounds like a good thing to me. I'd much rather that it reject messages with duplicate content and different Message-IDs. ;-) > >Well, that's true for *us*. The folks at the IETF don't have a > >habit of leaving well enough alone, though. ;-) > > Right, so let's do what *we* think is right, right now, and let the > committee take 10 years to define a standard. ;) Hey, that's *my* ten years you're offering there! From stephen at xemacs.org Tue Dec 30 14:39:03 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 30 Dec 2014 22:39:03 +0900 Subject: [Mailman-Developers] Python 3 In-Reply-To: <20141229132835.1d09e957@anthem> References: <20141222134734.698db176@anthem> <20141229132835.1d09e957@anthem> Message-ID: <87mw652r60.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > What if I do A/B releases for 3.0b5? Up to you; I have no objection. But I'm only going to be looking at the Python 3 branch. I've already ported my (work internal) Django site to Python 3 (which was mostly trivial, I had more trouble with the Django upgrade itself). From barry at list.org Tue Dec 30 20:39:38 2014 From: barry at list.org (Barry Warsaw) Date: Tue, 30 Dec 2014 14:39:38 -0500 Subject: [Mailman-Developers] RELEASED: GNU Mailman (core) 3.0b5 Message-ID: <20141230143938.48b96570@marathon> Hello friends of Mailman, and Happy New Year! You can roll that stone To the top of the hill Drag your ball and chain behind you Once again, it's time for the traditional "avoid the copyright year bump" release. I'm happy to announce the fifth beta release of Mailman 3.0 core, code named "Carve Away The Stone". We're really quite close now, but this release is a little different. Those of you who follow the proceedings on the mailman-developers mailing list will note that I have ported the core engine to Python 3.4. While it is my intent to release Mailman 3.0 core[*] final as a Python 3 application, some of the details are still be hashed out on the mailing list, and in the code base. For this reason, I am releasing two "flavors" of 3.0b5: * "A" release, which remains on Python 2.7 * "B" release, which is only compatible with Python 3.4 The A and B releases are functionally equivalent. There may be different bugs in them, but they both implement the exact same feature set, with the only difference being the version of Python they are compatible with. Some time early in 2015, I will be merging the Python 3 branch back into trunk, and dropping Python 2 support. I would encourage you to download the "B" release and try it out. If you do try both, please submit bugs for any functional differences you encounter. You can download GNU Mailman 3.0b5 core (both "A" and "B" releases) from the Python Cheeseshop: https://pypi.python.org/pypi/mailman/3.0.0b5 The documentation is available online at: http://pythonhosted.org/mailman Mailman 3.0 is released under the terms of the GNU General Public License version 3 or later. Detailed changes in 3.0b5 are available here: http://tinyurl.com/p9fpgrh Bugs can be reported here: https://bugs.launchpad.net/mailman Special thanks go to Abhilash Raj and Aur?lien Bompard, who implemented the conversion of the ORM layer from Storm to SQLAlchemy and Alembic. Thanks also to Kurt Griffiths for his excellent Falcon Framework package, which now replaces restish as our WSGI-compatible REST layer. Both of these changes were critical in allowing me to port the core to Python 3. Enjoy, and Happy New Year. -Barry [*] Postorious and HyperKitty will port to Python 3 on their own time schedule, and may remain Python 2 applications for the final release of the GNU Mailman Suite. mailman.client, the standalone REST client will be a bilingual library, supporting both Python 2 and Python 3. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From jeff at jab.org Tue Dec 30 23:39:12 2014 From: jeff at jab.org (Jeff Breidenbach) Date: Tue, 30 Dec 2014 14:39:12 -0800 Subject: [Mailman-Developers] X-Message-ID-Hash header (was Re: Python 3) In-Reply-To: <20141229130206.18c3e335@anthem> References: <20141222134734.698db176@anthem> <20141226161128.408d9ead@anarchist.wooz.org> <8738814v2u.fsf@uwakimon.sk.tsukuba.ac.jp> <20141226231846.17eb0ab8@anarchist.wooz.org> <87y4pt37gr.fsf@uwakimon.sk.tsukuba.ac.jp> <20141228175108.1d905288@anthem> <87ppb32r83.fsf@uwakimon.sk.tsukuba.ac.jp> <20141229130206.18c3e335@anthem> Message-ID: Thank you for the CC, it is appreciated. We have no problem with changing, or even completely dropping X-Message-ID-Hash. Mail-archive.com doesn't look at it. Instead, we parse Message-Id and do calculations from there. By the way, direct use of message-id seems to be growing in popularity. For example, I constantly see links to Debian's msgid-search. I think that trend is going to continue, boosted by the fact that it is easy to embed a link to a long URL in HTML mail. Jeff