From paul at boddie.org.uk Wed Jan 8 18:35:02 2014 From: paul at boddie.org.uk (Paul Boddie) Date: Wed, 8 Jan 2014 18:35:02 +0100 Subject: [Mailman-Developers] PGP-signed message verification using the email module (and in Mailman) Message-ID: <201401081835.03112.paul@boddie.org.uk> Hello, Sorry if this is too off-topic for the Mailman list and should really be directed to the email-sig list, but as I noticed that one of the GSOC projects touched upon message signing and verification, I thought that there might be more expertise on hand amongst this list's members, and there is a remote possibility that some of my difficulties can inform that work, too. I've been looking into using the Python email module for signing and verifying messages according to RFC 3156, but one difficulty I encountered was where a message with the following form is received and parsed: Content-Type: multipart/signed ... -> Content-Type: ... ... This is the content being signed (together with the part's headers). -> Content-Type: application/pgp-signature ... -----BEGIN PGP SIGNATURE----- ... -----END PGP SIGNATURE----- Obviously, the email module permits the inspection of the message, and the apparent solution in verifying the message is to obtain the signed content along with the signature (using the get_payload method on the top-level multipart container) and then to pass them both to gpg for verification. Something like this... content, signature = message.get_payload()[:2] We might also choose to decode the signature, I imagine, since it may employ a transfer encoding: signature = signature.get_payload(decode=True) However, in an almost accidental discovery involving Python 2.5 and 2.7 being used for the respective signing and verification operations (and in reverse), what I found was that the email module would parse the signed content part but then, upon being asked to provide the string representation of the part, it would reformat the headers and thus cause the verification to fail. So, where the email module in Python 2.5 likes to wrap headers using tab character indents, the module in Python 2.7 prefers to use a space for indentation instead. This means that the module reformats data upon being asked to provide a string representation of it rather than reporting exactly what it received. The behaviour of the module is fairly understandable - no-one wants to keep the original data around in addition to having a more manageable parsed form of it - but is there a convenient way of preserving the form of the original message for future use? It seems that asking for the individual headers will yield the original formatting, and that the _headers attribute on Message instances provides the original headers in order, but I wonder if I'm missing a simple method to get things as they were received and not as the particular version of the module wants to format them. Of course, RFC 3156 warns about the pitfalls of encoding the part that is to be signed, and I have been tempted to use a conservative approach where signed content is wrapped up inside an opaque message part with headers that are unlikely to be rewritten, but if I can manage to work with the email module rather than around it, I can hopefully steer clear of such inelegant measures. Does anyone have any opinions or experience that they would like to share on the matter? Paul From dkg at fifthhorseman.net Wed Jan 8 19:21:12 2014 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Wed, 08 Jan 2014 13:21:12 -0500 Subject: [Mailman-Developers] PGP-signed message verification using the email module (and in Mailman) In-Reply-To: <201401081835.03112.paul@boddie.org.uk> References: <201401081835.03112.paul@boddie.org.uk> Message-ID: <52CD9718.7060202@fifthhorseman.net> On 01/08/2014 12:35 PM, Paul Boddie wrote: > Of course, RFC 3156 warns about the pitfalls of encoding the part that is to > be signed, It doesn't just warn about the pitfalls. it states that: Multipart/signed and multipart/encrypted are to be treated by agents as opaque, meaning that the data is not to be altered in any way [2], [7]. where [2] and [7] map roughly to: [2] https://tools.ietf.org/html/rfc1847#section-2.1 which reads: Security Considerations: [multipart/signed parts] Must be treated as opaque while in transit and [7] https://tools.ietf.org/html/rfc2480#section-4 which reads: [email gateways] MUST provide the ability to tunnel multipart/signed and multipart/encrypted objects as monolithic entities if there is any chance whatsoever that MIME capabilities exist on the non-MIME side of the gateway. No changes to content of the multipart are permitted, even when the content is itself a composite MIME object. so if python's email module really does mangle this part, it cannot be used within RFC-2480-compliant mail gateways. This is a bug in python's email module, and it needs to be fixed. Have you reported it to the python email module? Thanks for raising the issue, --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1027 bytes Desc: OpenPGP digital signature URL: From stephen at xemacs.org Thu Jan 9 01:12:57 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 09 Jan 2014 09:12:57 +0900 Subject: [Mailman-Developers] PGP-signed message verification using the email module (and in Mailman) In-Reply-To: <52CD9718.7060202@fifthhorseman.net> References: <201401081835.03112.paul@boddie.org.uk> <52CD9718.7060202@fifthhorseman.net> Message-ID: <874n5egghi.fsf@uwakimon.sk.tsukuba.ac.jp> Daniel Kahn Gillmor writes: > so if python's email module really does mangle this part, it cannot be > used within RFC-2480-compliant mail gateways. This is a bug in python's > email module, and it needs to be fixed. Have you reported it to the > python email module? I believe the problem is known, and in any case the problem of header munging in general is known. I don't have a ticket number off-hand though. From paul at boddie.org.uk Thu Jan 9 01:49:14 2014 From: paul at boddie.org.uk (Paul Boddie) Date: Thu, 9 Jan 2014 01:49:14 +0100 Subject: [Mailman-Developers] PGP-signed message verification using the email module (and in Mailman) In-Reply-To: <52CD9718.7060202@fifthhorseman.net> References: <201401081835.03112.paul@boddie.org.uk> <52CD9718.7060202@fifthhorseman.net> Message-ID: <201401090149.14962.paul@boddie.org.uk> On Wednesday 8. January 2014 19.21.12 Daniel Kahn Gillmor wrote: > On 01/08/2014 12:35 PM, Paul Boddie wrote: > > Of course, RFC 3156 warns about the pitfalls of encoding the part that is > > to be signed, > > It doesn't just warn about the pitfalls. it states that: > > Multipart/signed and multipart/encrypted are to be treated by agents > as opaque, meaning that the data is not to be altered in any way [2], > [7]. > > where [2] and [7] map roughly to: > > [2] https://tools.ietf.org/html/rfc1847#section-2.1 > > which reads: > > Security Considerations: [multipart/signed parts] Must be treated as > opaque while in transit I'd mostly assumed this because it obviously wouldn't do to just have the different parts modified at random. Thanks for making this clear, though! > and > > [7] https://tools.ietf.org/html/rfc2480#section-4 > > which reads: > > [email gateways] > MUST provide the ability to tunnel multipart/signed and > multipart/encrypted objects as monolithic entities if there is > any chance whatsoever that MIME capabilities exist on the > non-MIME side of the gateway. No changes to content of the > multipart are permitted, even when the content is itself a > composite MIME object. > > so if python's email module really does mangle this part, it cannot be > used within RFC-2480-compliant mail gateways. This is a bug in python's > email module, and it needs to be fixed. Have you reported it to the > python email module? Well, I reserve the right to be wrong about this, but it is certainly the case that calling as_string on a Message causes the message to be formatted anew. Whether it is sensible that I use as_string at all is a reasonable thing to question - I steadily make new discoveries about the email API as time passes - but in effect I'd like to extract the content and signature parts and then pass them to gpg. I'm not using the gnupg module that I think the GSOC work uses, partly because I started out with my own wrapper, but the issue of extracting the content part as it was originally sent - and signed - is the critical factor here and probably outside the scope of the gnupg module anyway. There's also the matter of whether any gateway would parse and serialise messages in the way I am attempting, but in principle I think that anyone using the email module to do so would need to do things the same way unless there's another way I'm not aware of. Again, I'd only be too happy for someone to tell me I'm doing things wrong. ;-) I just searched for bugs reported about this and found the following: http://bugs.python.org/issue1974 (covers the change from tab to space indents) http://bugs.python.org/issue1372770 (covers issues of header folding) http://bugs.python.org/issue11492 (also covers header folding) http://bugs.python.org/issue1440472 (actually mentions a lack of idempotency) I think the last one probably answers my question, but I'll look at it again tomorrow. This may mean that I have to write my own message serialiser, of course. Thanks for looking at this! Paul From terri at zone12.com Thu Jan 9 08:28:57 2014 From: terri at zone12.com (Terri Oda) Date: Wed, 08 Jan 2014 23:28:57 -0800 Subject: [Mailman-Developers] GSoC 2014 ideas list Message-ID: <52CE4FB9.3060500@zone12.com> I've had prospective Google Summer of Code students emailing me since, uh, September or so... so I guess it's time to talk ideas! I've set up a wiki page, as usual: http://wiki.list.org/display/DEV/Google+Summer+of+Code+2014 But let's start with here: what small projects would you like to see us sponsor this year? I think we'll need to be more selective about the final list, but let's start with some brainstorming! From nicolas at karageuzian.com Sat Jan 11 10:54:57 2014 From: nicolas at karageuzian.com (Nicolas Karageuzian) Date: Sat, 11 Jan 2014 10:54:57 +0100 Subject: [Mailman-Developers] GSoC 2014 ideas list In-Reply-To: <52CE4FB9.3060500@zone12.com> References: <52CE4FB9.3060500@zone12.com> Message-ID: <52D114F1.1090502@karageuzian.com> Hi, I've the goal to make hyperkitty modular and then allows to install plugins for various uses, just like in other main forum platforms. This will allow to focus hyperkitty on archiving, and offer a simple entry point for extending it with out-of-archiving scope (the vote, category and tag features are candidates to be plugins, but many other useful plugins could appear, connecting archive to a variety of tools for an integrated community experience). The branch is quite advanced (@nka11, plugin-api) but still experimental and the very strict minimum is done (no docs, no unit tests). GSoC 2014 may be an opportunity to really kick it off : - achieve plugin API specification and documentation - release hyperkitty-extendable and its basic plugin set Regards Le 09/01/14 08:28, Terri Oda a ?crit : > I've had prospective Google Summer of Code students emailing me since, > uh, September or so... so I guess it's time to talk ideas! > > I've set up a wiki page, as usual: > > http://wiki.list.org/display/DEV/Google+Summer+of+Code+2014 > > But let's start with here: what small projects would you like to see us > sponsor this year? I think we'll need to be more selective about the > final list, but let's start with some brainstorming! > > > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > https://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: > http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: > https://mail.python.org/mailman/options/mailman-developers/nicolas%40karageuzian.com > > > Security Policy: http://wiki.list.org/x/QIA9 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 551 bytes Desc: OpenPGP digital signature URL: From paul at boddie.org.uk Sat Jan 11 19:06:41 2014 From: paul at boddie.org.uk (Paul Boddie) Date: Sat, 11 Jan 2014 19:06:41 +0100 Subject: [Mailman-Developers] PGP-signed message verification using the email module (and in Mailman) In-Reply-To: <201401090149.14962.paul@boddie.org.uk> References: <201401081835.03112.paul@boddie.org.uk> <52CD9718.7060202@fifthhorseman.net> <201401090149.14962.paul@boddie.org.uk> Message-ID: <201401111906.41615.paul@boddie.org.uk> On Thursday 9. January 2014 01.49.14 Paul Boddie wrote: > > I just searched for bugs reported about this and found the following: > > http://bugs.python.org/issue1974 (covers the change from tab to space > indents) http://bugs.python.org/issue1372770 (covers issues of header > folding) http://bugs.python.org/issue11492 (also covers header folding) > http://bugs.python.org/issue1440472 (actually mentions a lack of > idempotency) > > I think the last one probably answers my question, but I'll look at it > again tomorrow. This may mean that I have to write my own message > serialiser, of course. Following up to myself, here's what I decided to do for now: out = StringIO() generator = Generator(out, False, 0) # disable reformatting measures generator.flatten(message) return out.getvalue() Problems may apparently remain with superfluous whitespace found around header keys in any parsed messages, but the above seems to prevent the blatant rewriting I experienced with the Message.as_string method, doing so by overriding the Generator defaults. Paul From tom.browder at gmail.com Wed Jan 15 12:56:13 2014 From: tom.browder at gmail.com (Tom Browder) Date: Wed, 15 Jan 2014 05:56:13 -0600 Subject: [Mailman-Developers] Migration script from MM2 to MM3 In-Reply-To: <20130921114810.3b4648c6@anarchist> References: <20130921114810.3b4648c6@anarchist> Message-ID: On Sat, Sep 21, 2013 at 10:48 AM, Barry Warsaw wrote: > On Sep 20, 2013, at 12:46 PM, Aurelien Bompard wrote: > >>What do you think? Bad timing? Any existing work I'm missing? I am in the process of moving to a new server and I have several MM 2 mailing lists working well with virtual hosts (for my high school and college classmates). The lists are of manageable size so I can afford to do a fair amount of manual munging if need be and I want to convert to MM 3 on the new server. Also, nothing is absolutely critical archive-wise so I can be daring in the conversion. And, too, most users rely on MS Outlook and aren't real familiar with e-mail list usage, so I think I have an ideal situation for taking notes on a real-world set up. Are the steps needed listed somewhere? From djkevincr1989 at gmail.com Thu Jan 30 12:40:49 2014 From: djkevincr1989 at gmail.com (Kevin Ratnasekera) Date: Thu, 30 Jan 2014 17:10:49 +0530 Subject: [Mailman-Developers] Mailman web UI Error Message-ID: Hi, I am Kevin Ratnasekera, Undergraduate from University of Moratuwa Sri Lanka. I am currently developing a Cloud connector for mailman 3 as part of my internship project. For this I am using the REST api for mailman-3.0.0b3. I am currently following this tutorial http://wiki.list.org/display/DEV/A+5+minute+guide+to+get+the+Mailman+web+UI+running and I am getting this error while creating a new domain, user , list etc. NameError at /postorius/users/new/ global name 'utils' is not defined Request Method: POST Request URL: http://127.0.0.1:8000/postorius/users/new/ Django Version: 1.4 Exception Type: NameError Exception Value: global name 'utils' is not defined Exception Location: /home/kevin/postorius/src/postorius/models.py in create, line 88 Python Executable: /usr/bin/python Python Version: 2.7.3 Traceback: File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response 111. response = callback(request, *callback_args, **callback_kwargs) File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py" in _wrapped_view 20. return view_func(request, *args, **kwargs) File "/home/kevin/postorius/src/postorius/views/user.py" in user_new 261. user.save() File "/home/kevin/postorius/src/postorius/models.py" in save 151. self.objects.create(**self.kwargs) File "/home/kevin/postorius/src/postorius/models.py" in create 88. method = getattr(utils.get_client(), 'create_' + self.resource_name) Exception Type: NameError at /postorius/users/new/ Exception Value: global name 'utils' is not defined Any comments?? Regards Kevin From flo.fuchs at gmail.com Thu Jan 30 22:05:47 2014 From: flo.fuchs at gmail.com (Florian Fuchs) Date: Thu, 30 Jan 2014 22:05:47 +0100 Subject: [Mailman-Developers] Mailman web UI Error In-Reply-To: References: Message-ID: <52EABEAB.7060202@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/30/2014 12:40 PM, Kevin Ratnasekera wrote: > Hi, I am Kevin Ratnasekera, Undergraduate from University of > Moratuwa Sri Hi! > Lanka. I am currently developing a Cloud connector for mailman 3 > as part of my internship project. For this I am using the REST api > for mailman-3.0.0b3. I am currently following this tutorial > http://wiki.list.org/display/DEV/A+5+minute+guide+to+get+the+Mailman+web+UI+running > > and I am getting this error while creating a new domain, user , list etc. > > NameError at /postorius/users/new/ global name 'utils' is not > defined Unfortunately you ran into an error that just got fixed yesterday. Updating your postorius branch should fix that (bzr pull). If you're especially interested in using Mailman3's REST API, I'd also recommend taking a closer look at the Python bindings (mailman.client on launchpad - docs are in the repo's src/mailmanclient/docs directory). It's a Python wrapper which makes accessing the REST API pretty easy. Cheers Florian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS6r6nAAoJEEceGbPdavl7ObsH+QHD48xlDqEVCw/qk88OCvNM oMeJJeuJM7qournaRofCjNp8Z3JWUlltXGIBMd4iKPsIjmUdNQPTwXaxUeRJb3vD XeVw26ThLzNbjocxcM2HtgOKFwJrPcX6Ov/H0JLsNIwA83V9fClxbEuJ60LNNQ7r DFJLiqwsozJbOz1khBqjyyg/xbIhcTIko9Yo+jJyqzgEdNhv0gUBYtsJN/xOJZVB YD2MwqlFt7UVKrQIpjyZ1ceZz6PZaZnYgAxgKY2DjApChqbbA5lFnhgtcKtYwAZN CqEWJ2juB71dGmBSsG8f7ThrLmERKYCICft8iOPnRLhV/xjtWgoP8j7E+SiR6E0= =yTb3 -----END PGP SIGNATURE-----