From danci_emanuel at yahoo.com  Sun Jul  1 22:59:05 2012
From: danci_emanuel at yahoo.com (Danci Emanuel)
Date: Sun, 1 Jul 2012 13:59:05 -0700 (PDT)
Subject: [Mailman-Developers] smtp error - [Errno 111] Connection
	refused - when trying to install MM 3.0
In-Reply-To: <4FECFF6C.40006@msapiro.net>
References: <1340820439.84606.YahooMailNeo@web110410.mail.gq1.yahoo.com>
	<4FEB8695.1000009@msapiro.net>
	<1340929777.19879.YahooMailNeo@web110402.mail.gq1.yahoo.com>
	<4FECF981.9000902@msapiro.net>
	<1340930873.40897.YahooMailNeo@web110414.mail.gq1.yahoo.com>
	<4FECFF6C.40006@msapiro.net>
Message-ID: <1341176345.99760.YahooMailNeo@web110408.mail.gq1.yahoo.com>

Hi Mark!

Thank you very much for the solution! Indeed, that was the cause generating the error.
I have commented out some lines from /etc/hosts?and now?everything is working smoothly.?

Thanks again!
?
Emanuel?


________________________________
 From: Mark Sapiro <mark at msapiro.net>
To: Danci Emanuel <danci_emanuel at yahoo.com> 
Cc: "mailman-developers at python.org" <mailman-developers at python.org> 
Sent: Friday, June 29, 2012 4:05 AM
Subject: Re: [Mailman-Developers] smtp error - [Errno 111] Connection refused - when trying to install MM 3.0
 
On 6/28/2012 5:47 PM, Danci Emanuel wrote:
> 
> Exactly what happens with the 9025 one:
> 
> Trying ::1...
> telnet: Unable to connect to remote host: Connection refused
> 
> - executed as root.


So it has nothing to do with Mailman.

It looks like what's in /etc /hosts is an IPv6 address for localhost

You should have something similar to

127.0.0.1? ? ?  localhost.localdomain localhost
::1? ? ? ? ? ?  localhost6.localdomain6 localhost6

-- 
Mark Sapiro <mark at msapiro.net>? ? ? ? The highway is for gamblers,
San Francisco Bay Area, California? ? better use your sense - B. Dylan

From danci_emanuel at yahoo.com  Sun Jul  1 23:54:23 2012
From: danci_emanuel at yahoo.com (Danci Emanuel)
Date: Sun, 1 Jul 2012 14:54:23 -0700 (PDT)
Subject: [Mailman-Developers] Adding a new list attribute
Message-ID: <1341179663.11828.YahooMailNeo@web110416.mail.gq1.yahoo.com>

Hi again!

As I said in an earlier post, I am a GSoC student, working for the Systers organization, on integrating the?
dynamic sublists feature?(?http://systers.org/systers-dev/doku.php/good_to_know?s%5B%5D=postgresql?)
with Mailman 3.0, and I am facing a small problem.

What I wanna do is to adapt the UI and the back-end so they contain the options
needed in order to?enable/disable this feature for a list (at creation or after it has already
been created).

I want to get this right, so it would work as good as possible, so this is what I have done
and the error I get:

Firstly, in the class "ListNew" from "forms.py" I added a new ChoiceField, named
"dlists_enabled"?(the naming and the text from the labels are temporary?until I get
the feature to completely work)?

dlists_enabled = forms.ChoiceField(
? ? widget = forms.RadioSelect(),
? ? label = _("Enable dlists for this list?"),
? ? error_messages = {
? ? ? ? ? ? 'required': _("Please choose an option."),?
? ? ? ? },
? ? ? ? required = True,
? ? ? ? choices = (
? ? ? ? (True, _("Enable dlists for this list")),
? ? ? ? (False, _("Do not enable dlists for this list"))
? ? ? ? ))
?
This is working properly because if I print the result of "form.cleaned_data['dlists_enabled']" from
"views.py", in a file, it correctly prints the option chosen by the list owner (True or False).

I also added to "views.py" in the "list_new" function the following line (In the try block where?
list_setting are saved):

list_settings["dlists_enabled"] = form.cleaned_data['dlists_enabled']


Secondly, I tried to modify the list attributes by adding the following:?

dlists_enabled = Attribute("""Is the Dynamic sublist feature enabled?""") - "IMailingList" interface?(/interfaces/mailinglist.py)
dlists_enabled = Bool() - ?"MailingList" class (/model/mailinglist.py)

dlists_enabled = GetterSetter(as_boolean) - to the ATTRIBUTES dict from /rest/configuration.py

mlist.dlists_enabled = True - in the "apply" method from "DefaultStyle" class (/styles/default.py)


Obviously I did something wrong, or I did not add all the necessary changes because when I try to create
a new lists that has the dlists feature enabled, I get the HTTP Error 400:?

HTTP Error 400: Unknown attribute: dlists_enabled


Thanks in advance for your feedback!

Emanuel

From barry at python.org  Mon Jul  2 17:45:01 2012
From: barry at python.org (Barry Warsaw)
Date: Mon, 2 Jul 2012 11:45:01 -0400
Subject: [Mailman-Developers] Questions in regard to the database
	operations
References: <1340625106.63640.YahooMailNeo@web110401.mail.gq1.yahoo.com>
	<20120625094258.174b431f@limelight.wooz.org>
	<1340636530.41762.YahooMailNeo@web110410.mail.gq1.yahoo.com>
	<20120625143951.7ebdc02a@limelight.wooz.org>
	<1340663433.84997.YahooMailNeo@web110407.mail.gq1.yahoo.com>
	<20120625205431.3f261d99@resist.wooz.org>
	<1340810361.58129.YahooMailNeo@web110407.mail.gq1.yahoo.com>
	<20120627184334.21c52849@limelight.wooz.org>
	<1340925438.39523.YahooMailNeo@web110412.mail.gq1.yahoo.com>
Message-ID: <20120702114501.01132eb2@resist.forbidden.wifi-zones.com>

On Jun 28, 2012, at 04:17 PM, Danci Emanuel wrote:

>Yes, in terms of how the back-end works, it answers it. But, I am interested
>also in the following aspect. Lets say that a users subscribes to a list with
>his/her preferred? address. Afterwards, how can that user, who does not have
>access to the back end? of the mailing list, add other email addresses from
>which he/she can send emails to the list? This is related strictly how a user
>would do it, not necessarily how the code works in this case.

There are two ways that I see.  The first would be to use Postorius (i.e. the
web ui) to log in and then add a new email address to your account.  Postorius
would then take care of making the right calls to the REST API, and initiating
the core to do the email validation.  Because you logged in, Postorius would
know your user id so it would know what user to link your new address to.

The other way to do this would be through the email API.  This is not yet
implemented because we haven't really worked out the exact protocol for adding
a new address to an existing account.  You probably have to send the "add
email" command from an address that's already known and validated to Mailman.
Then, you could probably send a password in cleartext :( or gpg sign your
message to prove you own the account, and thus the user id to add the address
to.  Alternatively, we would have to send a confirmation message to both
addresses, and only if we get positive responses to both would be validate the
new address and link it to the original address.

-Barry

[1] We probably need to define a site-wide email command alias.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120702/e1e8efd8/attachment.pgp>

From barry at python.org  Mon Jul  2 17:55:15 2012
From: barry at python.org (Barry Warsaw)
Date: Mon, 2 Jul 2012 11:55:15 -0400
Subject: [Mailman-Developers] Adding a new list attribute
References: <1341179663.11828.YahooMailNeo@web110416.mail.gq1.yahoo.com>
Message-ID: <20120702115515.7603d9a1@resist.forbidden.wifi-zones.com>

On Jul 01, 2012, at 02:54 PM, Danci Emanuel wrote:

>Secondly, I tried to modify the list attributes by adding the following:?
>
>dlists_enabled = Attribute("""Is the Dynamic sublist feature enabled?""") -
>"IMailingList" interface?(/interfaces/mailinglist.py) dlists_enabled = Bool()
>- ?"MailingList" class (/model/mailinglist.py)

If you want this value to persist, it must also be added to the SQL tables,
e.g. sqlite.sql and/or postgres.sql.  This will make your schema incompatible
with upstream Mailman, and there will be a better way to extend the default
schema in the future, but for now, this should be enough to get you going
again.

Of course, it's not necessary to store your data in the primary database.
Once you've got a REST API for your stuff, you can really map this to any
backend storage you want.

>dlists_enabled = GetterSetter(as_boolean) - to the ATTRIBUTES dict from
>/rest/configuration.py
>
>mlist.dlists_enabled = True - in the "apply" method from "DefaultStyle" class
>(/styles/default.py)
>
>Obviously I did something wrong, or I did not add all the necessary changes
>because when I try to create a new lists that has the dlists feature enabled,
>I get the HTTP Error 400:?
>
>HTTP Error 400: Unknown attribute: dlists_enabled

That error gets thrown on line 233 of configuration.py and it only happens if
the attribute name isn't found in ATTRIBUTES.  Probably the best way to debug
this is to set a break point in get_configuration() and step through it when
self._attribute == 'dlist_enabled'.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120702/eec08770/attachment.pgp>

From danci_emanuel at yahoo.com  Mon Jul  2 22:39:53 2012
From: danci_emanuel at yahoo.com (Danci Emanuel)
Date: Mon, 2 Jul 2012 13:39:53 -0700 (PDT)
Subject: [Mailman-Developers] Questions in regard to the database
	operations
Message-ID: <1341261593.45701.YahooMailNeo@web110401.mail.gq1.yahoo.com>



Hi Barry!

I was thinking about these two ways too, but I did not know how much of these?
options has been implemented so far. In regard to the Postorius?
option, has this already been implemented (is there a box where the user can insert?
additional email addresses)? I am asking this because so far I have
worked with the rudimentary version (the test server set up using the 5-minute guide)
and I did not get the chance yet to check that too.?

Thanks,
Emanuel

From danci_emanuel at yahoo.com  Mon Jul  2 23:01:17 2012
From: danci_emanuel at yahoo.com (Danci Emanuel)
Date: Mon, 2 Jul 2012 14:01:17 -0700 (PDT)
Subject: [Mailman-Developers] Adding a new list attribute
In-Reply-To: <1341261925.35914.YahooMailNeo@web110408.mail.gq1.yahoo.com>
References: <1341261925.35914.YahooMailNeo@web110408.mail.gq1.yahoo.com>
Message-ID: <1341262877.30773.YahooMailNeo@web110409.mail.gq1.yahoo.com>



Indeed, I added these values to the tables, so now they should be fine.

Still, now I am facing another problem. Although I suspected that in the
get_configuration() is the problem, now I cannot debug that because?
now I get a new error. After adding the dlists_enabled field to the tables
and restaring mailman, when I try to access the Postorius server on?
localhost:8000 I get the HTTP Error 500:?

HTTP Error 500: A server error occurred.  Please contact the administrator.
and is what shows up in the terminal when I try to access Postorious:

root at zalman:/usr/local/mm3/mailman# Traceback (most recent call last):
? File "/usr/lib/python2.6/wsgiref/handlers.py", line 93, in run
? ? self.result = application(self.environ, self.start_response)
? File "/usr/local/mm3/mailman/src/mailman/database/transaction.py", line 60, in wrapper
? ? rtn = function(*args, **kws)
? File "/usr/local/mm3/mailman/src/mailman/rest/wsgiapp.py", line 59, in __call__
? ? environ, start_response)
? File "/usr/local/mm3/mailman/eggs/restish-0.12.1-py2.6.egg/restish/app.py", line 18, in __call__
? ? response = self.get_response(request, resource_or_response)
? File "/usr/local/mm3/mailman/eggs/restish-0.12.1-py2.6.egg/restish/app.py", line 65, in get_response
? ? resource_or_response = resource_or_response(request)
? File "/usr/local/mm3/mailman/eggs/restish-0.12.1-py2.6.egg/restish/resource.py", line 212, in __call__
? ? return _dispatch(request, match, lambda r: callable(self, r))
? File "/usr/local/mm3/mailman/eggs/restish-0.12.1-py2.6.egg/restish/resource.py", line 243, in _dispatch
? ? response = func(request)
? File "/usr/local/mm3/mailman/eggs/restish-0.12.1-py2.6.egg/restish/resource.py", line 212, in <lambda>
? ? return _dispatch(request, match, lambda r: callable(self, r))
? File "/usr/local/mm3/mailman/src/mailman/rest/lists.py", line 202, in collection
? ? resource = self._make_collection(request)
? File "/usr/local/mm3/mailman/src/mailman/rest/helpers.py", line 140, in _make_collection
? ? collection = self._get_collection(request)
? File "/usr/local/mm3/mailman/src/mailman/rest/lists.py", line 119, in _get_collection
? ? return list(getUtility(IListManager))
? File "/usr/local/mm3/mailman/src/mailman/model/listmanager.py", line 90, in __iter__
? ? for mlist in store.find(MailingList):
? File "/usr/local/mm3/mailman/eggs/storm-0.19-py2.6-linux-i686.egg/storm/store.py", line 978, in __iter__
? ? result = self._store._connection.execute(self._get_select())
? File "/usr/local/mm3/mailman/eggs/storm-0.19-py2.6-linux-i686.egg/storm/database.py", line 238, in execute
? ? raw_cursor = self.raw_execute(statement, params)
? File "/usr/local/mm3/mailman/eggs/storm-0.19-py2.6-linux-i686.egg/storm/databases/sqlite.py", line 154, in raw_execute
? ? return Connection.raw_execute(self, statement, params)
? File "/usr/local/mm3/mailman/eggs/storm-0.19-py2.6-linux-i686.egg/storm/database.py", line 322, in raw_execute
? ? self._check_disconnect(raw_cursor.execute, *args)
? File "/usr/local/mm3/mailman/eggs/storm-0.19-py2.6-linux-i686.egg/storm/database.py", line 371, in _check_disconnect
? ? return function(*args, **kwargs)
OperationalError: no such column: mailinglist.dlists_enabled


Indeed, is a pretty intuitive error but still, I did not manage to find the proper solution.?

Thanks for your help and sorry for bugging you with so many questions, but I really
want to make this work as it should. :)

Thanks again,
Emanuel

________________________________
 From: Danci Emanuel <danci_emanuel at yahoo.com>
To: "danci_emanuel at yahoo.com" <danci_emanuel at yahoo.com> 
Sent: Monday, July 2, 2012 11:45 PM
Subject: asd
 

On Jul 01, 2012, at 02:54 PM, Danci Emanuel wrote: >Secondly, I tried to modify the list attributes by adding the following:? >>dlists_enabled = Attribute("""Is the Dynamic sublist feature enabled?""") - >"IMailingList" interface?(/interfaces/mailinglist.py) dlists_enabled = Bool() >- ?"MailingList" class (/model/mailinglist.py)  If you want this value to persist, it must also be added to the SQL tables,
e.g. sqlite.sql and/or postgres.sql.  This will make your schema incompatible
with upstream Mailman, and there will be a better way to extend the default
schema in the future, but for now, this should be enough to get you going
again. Of course, it's not necessary to store your data in the primary database.
Once you've got a REST API for your stuff, you can really map this to any
backend storage you want. >dlists_enabled = GetterSetter(as_boolean) - to the ATTRIBUTES dict from >/rest/configuration.py >>mlist.dlists_enabled = True - in the "apply" method from "DefaultStyle" class >(/styles/default.py) >>Obviously I did something wrong, or I did not add all the necessary changes >because when I try to create a new lists that has the dlists feature enabled, >I get the HTTP Error 400:? >>HTTP Error 400: Unknown attribute: dlists_enabled  That error gets thrown on line 233 of configuration.py and it only happens if
the attribute name isn't found in ATTRIBUTES.  Probably the best way to debug
this is to set a break point in get_configuration() and step through it when
self._attribute == 'dlist_enabled'. Cheers,
-Barry
?
Emanuel DANCI

From alexander at sulfrian.net  Tue Jul  3 14:55:11 2012
From: alexander at sulfrian.net (Alexander Sulfrian)
Date: Tue, 03 Jul 2012 14:55:11 +0200
Subject: [Mailman-Developers] Twisted NNTP
Message-ID: <87pq8dknrk.wl%alexander@sulfrian.net>

Hi,

I working for gsoc creating NNTP access to the mailing list archives
for Mailman 3. Therefor I tried Twisted NNTP, during the last days. I
noticed, that it is not standard conform in any way. For example: it
missing required commands and returns wrong error codes on unknown sub
commands.

I will try to fix it. But while I modify it, I could port it to
asyncore/async_chat. Would it be preferred? anyncore is bundled with
core python and so the new implementation would be runnable without
depending on Twisted. But if patching the Twisted implementation,
other projects using it could gain profit, too.

So what is your position?

Thanks,
Alex 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120703/2937249c/attachment.pgp>

From barry at list.org  Tue Jul  3 22:30:55 2012
From: barry at list.org (Barry Warsaw)
Date: Tue, 3 Jul 2012 16:30:55 -0400
Subject: [Mailman-Developers] Twisted NNTP
In-Reply-To: <87pq8dknrk.wl%alexander@sulfrian.net>
References: <87pq8dknrk.wl%alexander@sulfrian.net>
Message-ID: <20120703163055.6e42381f@limelight.wooz.org>

On Jul 03, 2012, at 02:55 PM, Alexander Sulfrian wrote:

>I working for gsoc creating NNTP access to the mailing list archives
>for Mailman 3. Therefor I tried Twisted NNTP, during the last days. I
>noticed, that it is not standard conform in any way. For example: it
>missing required commands and returns wrong error codes on unknown sub
>commands.
>
>I will try to fix it. But while I modify it, I could port it to
>asyncore/async_chat. Would it be preferred? anyncore is bundled with
>core python and so the new implementation would be runnable without
>depending on Twisted. But if patching the Twisted implementation,
>other projects using it could gain profit, too.

I guess one question is whether it's more work to fix Twisted's NNTP protocol
support or bootstrap the whole thing from scratch async.  I don't know how
involved NNTP is, so a careful reading of RFC 6048 would be in order first.
Last time I tried it (admittedly many years ago), Twisted was good enough for
a couple of NNTP clients I tried.

Does the NNTP server part of it have to be fully conformant for you to show
progress on the Mailman integration work?  IOW, maybe it's good enough that
you don't need to be side-tracked right now on fixing Twisted's support.
Maybe if nntplib, Thunderbird (and selfishly, Claws) can read the archives via
NNTP, that would be enough for now.  If you go that route, definitely file
bugs with Twisted for any protocol defects you find.

In a similar way, if you think implementing RFC 6048 on top of async will not
distract you too much, then don't worry about getting that perfect.  Get it
good enough for a few clients to do the common things, and then work on the
Mailman integration parts.  If you're still interested, you can come back to
the NNTP bits and refine them.  (An nntpd.py would be an interesting
subproject, and probably mostly standalone from Mailman.)

How much can you get done without committing to either Twisted or a new
async-based NNTP server?  Maybe there would be a lot of commonality between
the two backends, so that you could work on the code in such a way that it
wouldn't be difficult to switch between them.

Hope that helps,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120703/e9a923e1/attachment.pgp>

From pidgeot18 at gmail.com  Tue Jul  3 22:48:43 2012
From: pidgeot18 at gmail.com (Joshua Cranmer)
Date: Tue, 03 Jul 2012 16:48:43 -0400
Subject: [Mailman-Developers] Twisted NNTP
In-Reply-To: <20120703163055.6e42381f@limelight.wooz.org>
References: <87pq8dknrk.wl%alexander@sulfrian.net>
	<20120703163055.6e42381f@limelight.wooz.org>
Message-ID: <4FF35AAB.9060201@gmail.com>

On 7/3/2012 4:30 PM, Barry Warsaw wrote:
> I guess one question is whether it's more work to fix Twisted's NNTP protocol
> support or bootstrap the whole thing from scratch async.  I don't know how
> involved NNTP is, so a careful reading of RFC 6048 would be in order first.
> Last time I tried it (admittedly many years ago), Twisted was good enough for
> a couple of NNTP clients I tried.
You mean RFC 3977--RFC 6048 only extends the LIST command.

In practice, you'd need to implement ARTICLE, HEAD, BODY, STAT, LIST, 
NEXT, LAST, and QUIT to be compatible with all clients (the necessary 
parts for readers for RFC 977). (X)OVER and the other LIST commands 
would be extremely nice, as would, to a lesser degree, (X)HDR, XPAT, and 
CAPABILITIES. Essentially, only those commands that need the READER and 
LIST capabilities under RFC 3977 (and their 2980 counterparts). AUTHINFO 
would be a good idea if you want to support password-protected archives.

-- 
Joshua Cranmer
News submodule owner
DXR coauthor


From terri at zone12.com  Tue Jul  3 23:23:35 2012
From: terri at zone12.com (Terri Oda)
Date: Tue, 03 Jul 2012 15:23:35 -0600
Subject: [Mailman-Developers] Questions in regard to the database
	operations
In-Reply-To: <1341261593.45701.YahooMailNeo@web110401.mail.gq1.yahoo.com>
References: <1341261593.45701.YahooMailNeo@web110401.mail.gq1.yahoo.com>
Message-ID: <4FF362D7.9070606@zone12.com>

On 07/02/2012 02:39 PM, Danci Emanuel wrote:
>
> In regard to the Postorius
> option, has this already been implemented (is there a box where the user can insert
> additional email addresses)? I am asking this because so far I have
> worked with the rudimentary version (the test server set up using the 5-minute guide)
> and I did not get the chance yet to check that too.

I don't believe it's fully implemented in Postorius yet (unless 
Florian's done it?) but it is on my list of things to do.  I can't 
remember if the UI framework is there (it might be) but I'm pretty sure 
the hooks to update the actual records are not.

If you need it sooner rather than later, I should be able to take a look 
at it this weekend.

  Terri

From alexander at sulfrian.net  Wed Jul  4 00:59:20 2012
From: alexander at sulfrian.net (Alexander Sulfrian)
Date: Wed, 04 Jul 2012 00:59:20 +0200
Subject: [Mailman-Developers] Twisted NNTP
In-Reply-To: <4FF35AAB.9060201@gmail.com>
References: <87pq8dknrk.wl%alexander@sulfrian.net>
	<20120703163055.6e42381f@limelight.wooz.org>
	<4FF35AAB.9060201@gmail.com>
Message-ID: <87mx3glad3.wl%alexander@sulfrian.net>

At Tue, 03 Jul 2012 16:48:43 -0400,
Joshua Cranmer wrote:
> 
> On 7/3/2012 4:30 PM, Barry Warsaw wrote:
> > I guess one question is whether it's more work to fix Twisted's NNTP protocol
> > support or bootstrap the whole thing from scratch async.  I don't know how
> > involved NNTP is, so a careful reading of RFC 6048 would be in order first.
> > Last time I tried it (admittedly many years ago), Twisted was good enough for
> > a couple of NNTP clients I tried.
> You mean RFC 3977--RFC 6048 only extends the LIST command.
> 
> In practice, you'd need to implement ARTICLE, HEAD, BODY, STAT, LIST, 
> NEXT, LAST, and QUIT to be compatible with all clients (the necessary 
> parts for readers for RFC 977). (X)OVER and the other LIST commands 
> would be extremely nice, as would, to a lesser degree, (X)HDR, XPAT, and 
> CAPABILITIES. Essentially, only those commands that need the READER and 
> LIST capabilities under RFC 3977 (and their 2980 counterparts). AUTHINFO 
> would be a good idea if you want to support password-protected archives.

RFC 3977 explicit requires the CAPABILITIES command to work for
conformance. It should not be hard to implement it, so in my opinion
it should be listed as required.

@TOPIC: I will try to create a async_chat wrapper for the NNTP
protocol. I think that this should not be very hard. The only
difficulty would be the parsing of the command with multiple lines as
argument.

Alex
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120704/4af0ba06/attachment.pgp>

From jeff at jab.org  Wed Jul  4 01:32:25 2012
From: jeff at jab.org (Jeff Breidenbach)
Date: Tue, 3 Jul 2012 16:32:25 -0700
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
Message-ID: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>

Hello mailman-developers,

I suspect that there are folks who would like to see what Mailman 3
and particularly Postorius without going through the effort of
software installation. Jeff Marshall was kind enough to set up a test
instance that people can look at and play with. This is a temporary
setup so don't use for anything real. Also try not to spam anyone as
we'll take it down at the first sign of trouble. I hope this helps
with the development process.

Cheers,
Jeff

===

http://mailman3.jab.org:8000/

username: public-admin
pw: tryitout

From barry at list.org  Wed Jul  4 01:57:06 2012
From: barry at list.org (Barry Warsaw)
Date: Tue, 3 Jul 2012 19:57:06 -0400
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
Message-ID: <20120703195706.0e9e8067@limelight.wooz.org>

On Jul 03, 2012, at 04:32 PM, Jeff Breidenbach wrote:

>I suspect that there are folks who would like to see what Mailman 3
>and particularly Postorius without going through the effort of
>software installation. Jeff Marshall was kind enough to set up a test
>instance that people can look at and play with. This is a temporary
>setup so don't use for anything real. Also try not to spam anyone as
>we'll take it down at the first sign of trouble. I hope this helps
>with the development process.

Fantastic, thanks Jeffs!

BTW, you can prevent spam by enabling devmode.  In the core's mailman.cfg
file, add this:

[devmode]
enabled: yes
# Season to taste.  All mail gets sent here.
recipient: devnull at example.com

And of course, restart.

Cheers,
-Barry

From p at state-of-mind.de  Wed Jul  4 06:39:21 2012
From: p at state-of-mind.de (Patrick Ben Koetter)
Date: Wed, 4 Jul 2012 06:39:21 +0200
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
Message-ID: <20120704043920.GA9924@state-of-mind.de>

* Jeff Breidenbach <jeff at jab.org>:
> Hello mailman-developers,
> 
> I suspect that there are folks who would like to see what Mailman 3
> and particularly Postorius without going through the effort of
> software installation. Jeff Marshall was kind enough to set up a test
> instance that people can look at and play with. This is a temporary
> setup so don't use for anything real. Also try not to spam anyone as
> we'll take it down at the first sign of trouble. I hope this helps
> with the development process.
> 
> Cheers,
> Jeff
> 
> ===
> 
> http://mailman3.jab.org:8000/

[Errno 111] Connection refused


-- 
state of mind ()

http://www.state-of-mind.de

Franziskanerstra?e 15      Telefon +49 89 3090 4664
81669 M?nchen              Telefax +49 89 3090 4666

Amtsgericht M?nchen        Partnerschaftsregister PR 563


From jeff at jab.org  Wed Jul  4 07:15:36 2012
From: jeff at jab.org (Jeff Breidenbach)
Date: Tue, 3 Jul 2012 22:15:36 -0700
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <20120704043920.GA9924@state-of-mind.de>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
	<20120704043920.GA9924@state-of-mind.de>
Message-ID: <CAHjiUbq3O0Wv9MJDjS33U6wmLNVyfLzsYFddidDP=aFBM-Tvgg@mail.gmail.com>

That sure didn't take very long...

From stephen at xemacs.org  Wed Jul  4 10:05:19 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 04 Jul 2012 17:05:19 +0900
Subject: [Mailman-Developers] Twisted NNTP
In-Reply-To: <87mx3glad3.wl%alexander@sulfrian.net>
References: <87pq8dknrk.wl%alexander@sulfrian.net>
	<20120703163055.6e42381f@limelight.wooz.org>
	<4FF35AAB.9060201@gmail.com> <87mx3glad3.wl%alexander@sulfrian.net>
Message-ID: <87r4ss6jeo.fsf@uwakimon.sk.tsukuba.ac.jp>

Alexander Sulfrian writes:

 > RFC 3977 explicit requires the CAPABILITIES command to work for
 > conformance. It should not be hard to implement it, so in my opinion
 > it should be listed as required.

Sure, but please remember that *you* have a deadline.  When you write
up requirements, make sure you distinguish between "what Mailman
should have to be a good citizen" and "what Alex promises for GSoC".

I *guess* that what others are talking about is the latter.  IMO
Barry's pragmatic test (a quickie driver around nntplib and TBird
should be able to read articles) is what you should be aiming at
first.

Steve

From f at state-of-mind.de  Wed Jul  4 10:34:17 2012
From: f at state-of-mind.de (Florian Fuchs)
Date: Wed, 04 Jul 2012 10:34:17 +0200
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
Message-ID: <4FF40009.5030802@state-of-mind.de>

Hi Jeff,

thank you, that's great!

The "Connection refused" error is caused by Postorius failing to connect
to Mailman's rest api. Can you check if Mailman is currently running?

Cheers,
Florian

Am 04.07.12 01:32, schrieb Jeff Breidenbach:
> Hello mailman-developers,
> 
> I suspect that there are folks who would like to see what Mailman 3
> and particularly Postorius without going through the effort of
> software installation. Jeff Marshall was kind enough to set up a test
> instance that people can look at and play with. This is a temporary
> setup so don't use for anything real. Also try not to spam anyone as
> we'll take it down at the first sign of trouble. I hope this helps
> with the development process.
> 
> Cheers,
> Jeff
> 
> ===
> 
> http://mailman3.jab.org:8000/
> 
> username: public-admin
> pw: tryitout
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind.de
> 
> Security Policy: http://wiki.list.org/x/QIA9
> 


From barry at list.org  Wed Jul  4 16:42:04 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 4 Jul 2012 10:42:04 -0400
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <CAHjiUbq3O0Wv9MJDjS33U6wmLNVyfLzsYFddidDP=aFBM-Tvgg@mail.gmail.com>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
	<20120704043920.GA9924@state-of-mind.de>
	<CAHjiUbq3O0Wv9MJDjS33U6wmLNVyfLzsYFddidDP=aFBM-Tvgg@mail.gmail.com>
Message-ID: <20120704104204.4e0d77f6@limelight.wooz.org>

On Jul 03, 2012, at 10:15 PM, Jeff Breidenbach wrote:

>That sure didn't take very long...

If the core crashed, please take a look at the error log and file bugs!

Thanks,
-Barry

From stephen at xemacs.org  Thu Jul  5 02:24:33 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 05 Jul 2012 09:24:33 +0900
Subject: [Mailman-Developers] URGENT: Google Summer of Code status
 report and code due
In-Reply-To: <20120704110306.58c241ad@limelight.wooz.org>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
Message-ID: <87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>

Note: CC'ing Mailman Developers.  I think this was private, so not
trimming.

At the bottom there's a discussion of the newsgroup naming scheme,
which is an important bikeshed to paint.  If you're not interested in
a bunch of "me too" comments, skipp right on down there!

Barry Warsaw writes:
 > Hi Alex.
 > 
 > On Jun 28, 2012, at 02:00 AM, Alexander Sulfrian wrote:
 > 
 > >Currently I am using twisted-news-11.0.0, but that just because it is
 > >the current stable version in gentoo. Debian contains twisted-10.1.0
 > >in stable and twisted-12.0.0 in testing and unstable. I would use
 > >12.0.0 if you have no objections. Additional to that, I would like to
 > >continue to use the package manager version. This has the advantage
 > >that requisites are installed automatically.
 > 
 > I think this is fine.  We've got 11.1.0 in Ubuntu 12.04 and 12.0.0
 > in Quantal.  Debian Wheezy (now frozen) has 12.0.0.  I can easily
 > spin up Debian or Gentoo instances (it's been *years* since I ran
 > Gentoo).  Go with whichever of those will give you the easiest path
 > to success and we should be able to reproduce your environment.

+1.  I can also do both Debian and Gentoo easily.  I would say stick
to Gentoo for now.

 > >I already looked into the code examples, that comes with
 > >twisted-news. As far as I could see, twisted-news is manly an
 > >reactor-based server, that is doing some string parsing to handle
 > >the protocol. There are some sample back ends that use pickle
 > >files, the twisted dirdbm or the twisted DB-API. But for the
 > >project it should be necessary to write a custom back end.
 > 
 > I expected a new backend might be necessary.  Again, since this
 > isn't the most important part of the project, do only what's
 > necessary to show off the Mailman integration work.

+1

 > >My first intention would be to implement it like a runner. The
 > >lmtp runner is also an daemon. It could be implemented in a
 > >similar way. But the nntp archive daemon is not a runner in the
 > >first place. So it maybe would be better to put it under the
 > >archiving module. It could be implemented like a runner. It should
 > >be possible to configure it via the configuration file like the
 > >other modules, that are available. But I do not know where it is
 > >preferred to put it.
 > 
 > Daemon/server control should be implemented as a runner,

+1.  Unless some showstopper crops up, I think all Mailman daemons
should be controlled via runners.

 > but it can be pretty simple.  The IRunner interface really just
 > needs to implement run() and stop() and even forking a subprocess
 > should be fairly easy to do.
 > 
 > You have to worry about two things.  First, you need a server to
 > vend the messages through NNTP.  Second, you need some way to get
 > new messages into the twisted-news (or whatever) backend when they
 > show up on the mailing list.  This is the part that you'll need an
 > IArchiver implementation for.  Note that the only IArchiver method
 > you *must* implement is archive_message().  You'll have to figure
 > out whether you need the twisted-news server running or not for
 > that.

IMO, make that *three* things.  It ought to be possible to fire up the
NNTP runner on an existing archive, and vend messages.  I suppose this
means a separate database for news properties like the message to
message number mapping, and perhaps for newsgroup names.

Newsgroup names are an issue here.  It seems to me that (if not
gateway'd to Usenet) they should be something like (pseudo-code)

    "mailman." + join(reverse(split(list-id,".")),".")

Eg, this list would be "mailman.org.python.mailman-developers".  I
know that's considered ugly-out-the-wazoo, but these need to be UUIDs
(consider mirrors), and mailman at python.org should not be in the same
subtree as mailman at python.net (ie,

    mailman.net.python.mailman    vs  mailman.org.python.mailman

not

    mailman.python.net.mailman    vs  mailman.python.org.mailman

The top-level maybe shouldn't be "mailman", but rather something like
"list-archive".

Another issue with newsgroup names is that some lists *are* registered
in the news hierarchy, so provision for such aliases should be allowed.

Let the bikeshed-painting begin!

 > >The new code should be integrated into the existent test set. I know
 > >that a successful test is not sufficient, but it is a good
 > >start. Additionally to that, we should define some acceptance tests.
 > 
 > +1

I was going to write something here, so remind me.  (Must run off to
torture would-be grad students.)

From iane at sussex.ac.uk  Thu Jul  5 12:37:15 2012
From: iane at sussex.ac.uk (Ian Eiloart)
Date: Thu, 5 Jul 2012 10:37:15 +0000
Subject: [Mailman-Developers] URGENT: Google Summer of Code status
 report and code due
In-Reply-To: <87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <7D454704-F482-4771-B358-80849833119B@sussex.ac.uk>

On 5 Jul 2012, at 01:24, Stephen J. Turnbull wrote:
> 
> 
> Eg, this list would be "mailman.org.python.mailman-developers".  I
> know that's considered ugly-out-the-wazoo, but these need to be UUIDs
> (consider mirrors), and mailman at python.org should not be in the same
> subtree as mailman at python.net (ie,
> 
>    mailman.net.python.mailman    vs  mailman.org.python.mailman
> 
> not
> 
>    mailman.python.net.mailman    vs  mailman.python.org.mailman
> 
> The top-level maybe shouldn't be "mailman", but rather something like
> "list-archive".
> 
> Another issue with newsgroup names is that some lists *are* registered
> in the news hierarchy, so provision for such aliases should be allowed.
> 
> Let the bikeshed-painting begin!

OK. Where do these two email addresses sit?

foo at bar.example.com
foo.bar at example.com

They're distinct addresses, but they clash in this naming scheme because of the collapse of (@,.) to (.).

If '@' can't be mapped to itself, or another unused character, then it needs to be mapped to a rarely used string, or it needs to be mapped to a configurable string. I suppose the default could be '.' if the string were configurable.


-- 
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148


From stephen at xemacs.org  Thu Jul  5 14:46:34 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 05 Jul 2012 21:46:34 +0900
Subject: [Mailman-Developers] URGENT: Google Summer of Code status
 report and code due
In-Reply-To: <7D454704-F482-4771-B358-80849833119B@sussex.ac.uk>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
	<7D454704-F482-4771-B358-80849833119B@sussex.ac.uk>
Message-ID: <87ipe274ut.fsf@uwakimon.sk.tsukuba.ac.jp>

Ian Eiloart writes:

 > OK. Where do these two email addresses sit?

Addresses aren't relevant.  I proposed using List-Ids, which have to
be unique (RFC 2919).  If an administrator specifies List-Ids that
collide, that's not our problem.  (The author of RFC 2919 was aware of
similar problems, though not this particular one AFAICS.)

Such an administrator will be able to work around it using whatever
aliasing mechanism we develop for lists that are gatewayed to and from
Usenet.


From barry at list.org  Thu Jul  5 17:05:20 2012
From: barry at list.org (Barry Warsaw)
Date: Thu, 5 Jul 2012 11:05:20 -0400
Subject: [Mailman-Developers] URGENT: Google Summer of Code status
 report and code due
In-Reply-To: <87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20120705110520.44870b62@limelight.wooz.org>

On Jul 05, 2012, at 09:24 AM, Stephen J. Turnbull wrote:

>IMO, make that *three* things.  It ought to be possible to fire up the
>NNTP runner on an existing archive, and vend messages.

+1

>I suppose this means a separate database for news properties like the message
>to message number mapping, and perhaps for newsgroup names.
>
>Newsgroup names are an issue here.  It seems to me that (if not
>gateway'd to Usenet) they should be something like (pseudo-code)
>
>    "mailman." + join(reverse(split(list-id,".")),".")
>
>Eg, this list would be "mailman.org.python.mailman-developers".  I
>know that's considered ugly-out-the-wazoo, but these need to be UUIDs
>(consider mirrors), and mailman at python.org should not be in the same
>subtree as mailman at python.net (ie,
>
>    mailman.net.python.mailman    vs  mailman.org.python.mailman
>
>not
>
>    mailman.python.net.mailman    vs  mailman.python.org.mailman
>
>The top-level maybe shouldn't be "mailman", but rather something like
>"list-archive".

Why is the prefix needed at all, especially since you qualified this as "not
gatewayed to Usenet"?  If all the messages are local to the server, there
should be no collisions on reverse list-id newsgroup names.  I've always kind
of felt that Gmane's 'gmane.' prefix was a bit superfluous.  I'm sure there's
a good reason for it (probably buried in some Gmane FAQ).

+1 for the reversed list-ids.

>Another issue with newsgroup names is that some lists *are* registered
>in the news hierarchy, so provision for such aliases should be allowed.

Maybe that's the reason for the prefix?

>Let the bikeshed-painting begin!

blue, no yellow! aaarrrggg!

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120705/87fca6d9/attachment.pgp>

From terri at zone12.com  Thu Jul  5 17:40:11 2012
From: terri at zone12.com (Terri Oda)
Date: Thu, 05 Jul 2012 09:40:11 -0600
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <4FF40009.5030802@state-of-mind.de>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
	<4FF40009.5030802@state-of-mind.de>
Message-ID: <4FF5B55B.1030404@zone12.com>

How hard is it for us to make a nicer, more helpful error page for this? 
  I'm guessing it's going to be a common problem when more people start 
running the two together.

(I know, I know, I should know the answer and be able to write the 
solution myself, but my django-fu isn't quite up to speed yet, so I'm 
asking first.)

  Terri

PS - Thanks again to the Jeffs!

On 07/04/2012 02:34 AM, Florian Fuchs wrote:
> Hi Jeff,
>
> thank you, that's great!
>
> The "Connection refused" error is caused by Postorius failing to connect
> to Mailman's rest api. Can you check if Mailman is currently running?
>
> Cheers,
> Florian
>
> Am 04.07.12 01:32, schrieb Jeff Breidenbach:
>> Hello mailman-developers,
>>
>> I suspect that there are folks who would like to see what Mailman 3
>> and particularly Postorius without going through the effort of
>> software installation. Jeff Marshall was kind enough to set up a test
>> instance that people can look at and play with. This is a temporary
>> setup so don't use for anything real. Also try not to spam anyone as
>> we'll take it down at the first sign of trouble. I hope this helps
>> with the development process.
>>
>> Cheers,
>> Jeff
>>
>> ===
>>
>> http://mailman3.jab.org:8000/
>>
>> username: public-admin
>> pw: tryitout
>> _______________________________________________
>> Mailman-Developers mailing list
>> Mailman-Developers at python.org
>> http://mail.python.org/mailman/listinfo/mailman-developers
>> Mailman FAQ: http://wiki.list.org/x/AgA3
>> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
>> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind.de
>>
>> Security Policy: http://wiki.list.org/x/QIA9
>>
>
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/terri%40zone12.com
>
> Security Policy: http://wiki.list.org/x/QIA9
>


From f at state-of-mind.de  Thu Jul  5 18:14:20 2012
From: f at state-of-mind.de (Florian Fuchs)
Date: Thu, 05 Jul 2012 18:14:20 +0200
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <4FF5B55B.1030404@zone12.com>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
	<4FF40009.5030802@state-of-mind.de> <4FF5B55B.1030404@zone12.com>
Message-ID: <4FF5BD5C.20707@state-of-mind.de>

Hi,

Am 05.07.12 17:40, schrieb Terri Oda:
> How hard is it for us to make a nicer, more helpful error page for this?

Shouldn't be too hard. I just filed a bug report for this:
https://bugs.launchpad.net/postorius/+bug/1021364

In a non-debug mode environment a 500 error page would be returned -
which is only slightly nicer than what we are seeing now...

>  I'm guessing it's going to be a common problem when more people start
> running the two together.
> 
> (I know, I know, I should know the answer and be able to write the
> solution myself, but my django-fu isn't quite up to speed yet, so I'm
> asking first.)

This should be changed inside the Manager classes in models.py. If the
error is catched and MailmanApiError is raised instead, the views will
render an error page with a "REST API not found" message.

Cheers
Florian



> 
>  Terri
> 
> PS - Thanks again to the Jeffs!
> 
> On 07/04/2012 02:34 AM, Florian Fuchs wrote:
>> Hi Jeff,
>>
>> thank you, that's great!
>>
>> The "Connection refused" error is caused by Postorius failing to connect
>> to Mailman's rest api. Can you check if Mailman is currently running?
>>
>> Cheers,
>> Florian
>>
>> Am 04.07.12 01:32, schrieb Jeff Breidenbach:
>>> Hello mailman-developers,
>>>
>>> I suspect that there are folks who would like to see what Mailman 3
>>> and particularly Postorius without going through the effort of
>>> software installation. Jeff Marshall was kind enough to set up a test
>>> instance that people can look at and play with. This is a temporary
>>> setup so don't use for anything real. Also try not to spam anyone as
>>> we'll take it down at the first sign of trouble. I hope this helps
>>> with the development process.
>>>
>>> Cheers,
>>> Jeff
>>>
>>> ===
>>>
>>> http://mailman3.jab.org:8000/
>>>
>>> username: public-admin
>>> pw: tryitout
>>> _______________________________________________
>>> Mailman-Developers mailing list
>>> Mailman-Developers at python.org
>>> http://mail.python.org/mailman/listinfo/mailman-developers
>>> Mailman FAQ: http://wiki.list.org/x/AgA3
>>> Searchable Archives:
>>> http://www.mail-archive.com/mailman-developers%40python.org/
>>> Unsubscribe:
>>> http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind.de
>>>
>>>
>>> Security Policy: http://wiki.list.org/x/QIA9
>>>
>>
>> _______________________________________________
>> Mailman-Developers mailing list
>> Mailman-Developers at python.org
>> http://mail.python.org/mailman/listinfo/mailman-developers
>> Mailman FAQ: http://wiki.list.org/x/AgA3
>> Searchable Archives:
>> http://www.mail-archive.com/mailman-developers%40python.org/
>> Unsubscribe:
>> http://mail.python.org/mailman/options/mailman-developers/terri%40zone12.com
>>
>>
>> Security Policy: http://wiki.list.org/x/QIA9
>>
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives:
> http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe:
> http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind.de
> 
> 
> Security Policy: http://wiki.list.org/x/QIA9


From richard at NFSNet.org  Thu Jul  5 20:58:11 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Thu, 5 Jul 2012 13:58:11 -0500
Subject: [Mailman-Developers] [Bug 1020683] Feature Request: Make
	regular_*_lists and *_these_nonmembers recursive (and aware
	of each other).
In-Reply-To: <20120705175313.20355.63604.malone@chaenomeles.canonical.com>
References: <20120703181818.4927.56021.malonedeb@soybean.canonical.com>
	<20120705175313.20355.63604.malone@chaenomeles.canonical.com>
Message-ID: <865949D8-F83F-4FB2-B7DA-363560456FC6@NFSNet.org>

Robert,

Although I recognize your goal, it isn't as easy as simply providing transitive membership.

First, I presume that you are willing to restrict yourself to the case where lists listA, listB, and listC are served by the same instance of MM.
On the distribution side, that restriction certainly is not necessary for list inclusion, but, in general, there is no mechanism for listA to learn the members of listC unless they are served by a common server instance.

Even so, the issue is whether, or not, everyone who receives listB should be allowed to submit to listA (without moderation). Or perhaps you would only want those who can make unmoderated submissions to listB to have that transitive permission.

I can visualize both use cases.

Richard

On Jul 5, 2012, at 12:53 PM, Robert Arlt Jr. wrote:

> I think making it recursive would be sufficient.
> 
> However, it isn't really that I want *_these_nonmembers to be recursive
> to *_these_nonmembers as I may want different sets of lists to be able
> to send to a particular list.  I guess it might be better if I just
> explained what I really want, which is the ability to add @listC to the
> members of listB such that listA, which has accept_these_nonmembers set
> to @listB, can be sent to unmoderated by members of listC.  In
> otherwords making the @listname format usable as a list member such that
> *_these_nonmembers and sibling lists can transverse them.
> 
> Neither of the solutions you have talked about are exactly that, but
> could be used instead to obtain the same goal.  Basically what I am
> trying to do is minimize the number of lists that would have to be
> changed if a new lists is created that should have rights to send to a
> number of lists.
> 
> Perhaps I should have split this into two feature requests, the
> recursion request is by far more of a useful feature.  The other feature
> would add on only a minimal amount of capability.
> 
> -- 
> You received this bug notification because you are a member of Mailman
> Coders, which is subscribed to GNU Mailman.
> https://bugs.launchpad.net/bugs/1020683
> 
> Title:
>  Feature Request: Make regular_*_lists and *_these_nonmembers recursive
>  (and aware of each other).
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mailman/+bug/1020683/+subscriptions
> _______________________________________________
> Mailman-coders mailing list
> Mailman-coders at python.org
> http://mail.python.org/mailman/listinfo/mailman-coders


From barry at list.org  Thu Jul  5 21:22:39 2012
From: barry at list.org (Barry Warsaw)
Date: Thu, 5 Jul 2012 15:22:39 -0400
Subject: [Mailman-Developers] [Bug 1020683] Feature Request: Make
 regular_*_lists and *_these_nonmembers recursive (and aware of each other).
In-Reply-To: <865949D8-F83F-4FB2-B7DA-363560456FC6@NFSNet.org>
References: <20120703181818.4927.56021.malonedeb@soybean.canonical.com>
	<20120705175313.20355.63604.malone@chaenomeles.canonical.com>
	<865949D8-F83F-4FB2-B7DA-363560456FC6@NFSNet.org>
Message-ID: <20120705152239.2d6180ca@limelight.wooz.org>

On Jul 05, 2012, at 01:58 PM, Richard Wackerbarth wrote:

>First, I presume that you are willing to restrict yourself to the case where
>lists listA, listB, and listC are served by the same instance of MM.  On the
>distribution side, that restriction certainly is not necessary for list
>inclusion, but, in general, there is no mechanism for listA to learn the
>members of listC unless they are served by a common server instance.
>
>Even so, the issue is whether, or not, everyone who receives listB should be
>allowed to submit to listA (without moderation). Or perhaps you would only
>want those who can make unmoderated submissions to listB to have that
>transitive permission.

I'll note that with Mailman 3 and clever implementation of the IRoster
interface, various riffs on this should be doable.  Most implementations of
IRoster are not physical objects persisted in the database, but instead
"virtual" objects such as queries.  There's no reason I can think of that an
IRoster implementation couldn't perform cross-list queries, or even for that
matter cross-server queries, assuming the necessary communication channels
were available.

I've not actually tried to implement anything like this, but I can somewhat
visualize how this would work.  I invite you to experiment with this.

Cheers,
-Barry


From richard at NFSNet.org  Thu Jul  5 22:33:29 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Thu, 5 Jul 2012 15:33:29 -0500
Subject: [Mailman-Developers] [Bug 1020683] Feature Request: Make
	regular_*_lists and *_these_nonmembers recursive (and aware
	of each other).
In-Reply-To: <20120705194822.29278.93580.malone@soybean.canonical.com>
References: <20120703181818.4927.56021.malonedeb@soybean.canonical.com>
	<20120705194822.29278.93580.malone@soybean.canonical.com>
Message-ID: <E27A8762-932D-4B67-9CE6-B06DB0AF6046@NFSNet.org>


On Jul 5, 2012, at 2:48 PM, Robert Arlt Jr. wrote:
> I can as well.  In my view of the issue I believe that allowing everyone
> on listB to submit is the better option as I can easily make a list
> composed of those allowed to send unmoderated to listB, call it listB2
> and add listB2 to listB then use listB2 as the list allowed to send to
> listA (all of this assuming recursion is implemented).

But that would be confusing to the members of listB. They would have to realize that they are not really a member of listB, but that they are actually a member of listB2.

So, if you are going to do that, and perhaps for any inclusion that is transparent on message delivery, an attempt to change delivery preferences on listA should take the user to the listB2 preferences.

However, if the delivery of listA is done by publishing a copy of the message on listB, then, for the purpose of preferences, the listB user should not be considered a member of listA.

From stephen at xemacs.org  Fri Jul  6 07:36:56 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 06 Jul 2012 14:36:56 +0900
Subject: [Mailman-Developers] URGENT: Google Summer of Code status
 report and code due
In-Reply-To: <20120705110520.44870b62@limelight.wooz.org>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20120705110520.44870b62@limelight.wooz.org>
Message-ID: <87d34978nb.fsf@uwakimon.sk.tsukuba.ac.jp>

I don't think Terri needs a cc, if she's not on mm-d.

Barry Warsaw writes:

 > >The top-level maybe shouldn't be "mailman", but rather something like
 > >"list-archive".
 > 
 > Why is the prefix needed at all, especially since you qualified this as "not
 > gatewayed to Usenet"?  If all the messages are local to the server, there
 > should be no collisions on reverse list-id newsgroup names.

It's not clear to me how the MUA would know that.  Many MUAs (well, at
least one, and you know which one I'm talking about!  Reindeer and
mooses and Gnus, oh my!) can handle multiple feeds simultaneously.
Suppose you have multiple archives (mirrors!) which happen to have the
same list archived?  How does the MUA de-dupe lists (and/or merge
them, although I don't know if any MUAs can do that) if we don't have
a unique public name?

 > >Another issue with newsgroup names is that some lists *are* registered
 > >in the news hierarchy, so provision for such aliases should be allowed.
 > 
 > Maybe that's the reason for the prefix?

Part of it, yes.

From barry at list.org  Fri Jul  6 15:08:01 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 6 Jul 2012 09:08:01 -0400
Subject: [Mailman-Developers] URGENT: Google Summer of Code status
 report and code due
In-Reply-To: <87d34978nb.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20120705110520.44870b62@limelight.wooz.org>
	<87d34978nb.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20120706090801.6bd56fb8@limelight.wooz.org>

On Jul 06, 2012, at 02:36 PM, Stephen J. Turnbull wrote:

>It's not clear to me how the MUA would know that.  Many MUAs (well, at
>least one, and you know which one I'm talking about!  Reindeer and
>mooses and Gnus, oh my!) can handle multiple feeds simultaneously.
>Suppose you have multiple archives (mirrors!) which happen to have the
>same list archived?  How does the MUA de-dupe lists (and/or merge
>them, although I don't know if any MUAs can do that) if we don't have
>a unique public name?
>
> > >Another issue with newsgroup names is that some lists *are* registered
> > >in the news hierarchy, so provision for such aliases should be allowed.
> > 
> > Maybe that's the reason for the prefix?
>
>Part of it, yes.

Should the prefix be configurable then?

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120706/8a544319/attachment.pgp>

From stephen at xemacs.org  Fri Jul  6 19:01:09 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 07 Jul 2012 02:01:09 +0900
Subject: [Mailman-Developers] URGENT: Google Summer of Code status
 report and code due
In-Reply-To: <20120706090801.6bd56fb8@limelight.wooz.org>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20120705110520.44870b62@limelight.wooz.org>
	<87d34978nb.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20120706090801.6bd56fb8@limelight.wooz.org>
Message-ID: <87vci06cyy.fsf@uwakimon.sk.tsukuba.ac.jp>

Barry Warsaw writes:

 > Should the prefix [for the "archived news hierarchy"] be
 > configurable then?

Yes, but once we figure out what it "should" be, admins should be
strongly advised not to change the prefix if they allow mirrors.


From marshman at gmail.com  Fri Jul  6 20:20:55 2012
From: marshman at gmail.com (Jeff Marshall)
Date: Fri, 6 Jul 2012 11:20:55 -0700
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <CAHjiUbo9h-0mzS+A26PgXK7PUjqmnXhk+j6dB=wyjgf5tLVGUQ@mail.gmail.com>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
	<4FF40009.5030802@state-of-mind.de> <4FF5B55B.1030404@zone12.com>
	<4FF5BD5C.20707@state-of-mind.de>
	<CAHjiUbo9h-0mzS+A26PgXK7PUjqmnXhk+j6dB=wyjgf5tLVGUQ@mail.gmail.com>
Message-ID: <CAKHXeC9F0pRUTQcZe4+gx+UFDVSJi9cj4=QE-L__am_yq8+cEA@mail.gmail.com>

First, I restarted mailman3 and the error at
http://mailman3.jab.org:8000/persisted.  Then I killed and restarted
postorius and the error cleared up.
 The postorius instance is accessible again.



Postorius output showed (I don't see a postorius output log file anywhere
so I don't know if I can pull up history farther back?):

[04/Jul/2012 06:05:00] "GET / HTTP/1.1" 500 93610
[04/Jul/2012 08:59:05] "GET / HTTP/1.1" 500 93655
[04/Jul/2012 09:53:42] "GET / HTTP/1.0" 500 93553
[04/Jul/2012 09:56:39] "GET / HTTP/1.1" 500 93838
[04/Jul/2012 10:12:32] "GET / HTTP/1.1" 500 93775
[04/Jul/2012 10:25:23] "GET / HTTP/1.1" 500 93655
[04/Jul/2012 12:25:56] "GET / HTTP/1.1" 500 93583
[04/Jul/2012 13:24:33] "GET / HTTP/1.1" 500 93443
[04/Jul/2012 13:26:42] "GET / HTTP/1.1" 500 93443
[04/Jul/2012 21:50:56] "GET /static/postorius/defaultimages/favicon.ico
HTTP/1.1" 404 1803
[05/Jul/2012 05:39:37] "GET / HTTP/1.1" 500 93882
Traceback (most recent call last):
  File
"/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
line 284, in run
    self.finish_response()
  File
"/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
line 324, in finish_response
    self.write(data)
  File
"/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
line 420, in write
    self._write(data)
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('134.95.19.9', 26446)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 284, in
_handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File
"/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
line 570, in __init__
    BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
  File "/usr/lib/python2.7/SocketServer.py", line 641, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 694, in finish
    self.wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
[05/Jul/2012 06:14:25] "GET / HTTP/1.1" 500 94198
[05/Jul/2012 10:39:53] "GET / HTTP/1.1" 500 93559
[05/Jul/2012 10:51:25] "GET / HTTP/1.1" 500 93882
[05/Jul/2012 11:30:01] "GET / HTTP/1.1" 500 93538
[05/Jul/2012 11:54:47] "GET / HTTP/1.1" 500 93655
[05/Jul/2012 12:09:25] "GET / HTTP/1.1" 500 94323



And the mailman3 var/log/mailman.log looked like this throughout:

Jul 04 01:17:37 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org HTTP/1.1" 200 307
Jul 04 01:17:37 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org/config HTTP/1.1" 200 1647
Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
jablist at mailman3.jab.org HTTP/1.1" 200 295
Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
jablist at mailman3.jab.org HTTP/1.1" 200 295
Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
jablist at mailman3.jab.org/config HTTP/1.1" 200 1611
Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
public-test at mailman3.jab.org HTTP/1.1" 200 311
Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
public-test at mailman3.jab.org HTTP/1.1" 200 311
Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
public-test at mailman3.jab.org/config HTTP/1.1" 200 1634
Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org HTTP/1.1" 200 307
Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org HTTP/1.1" 200 307
Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org/config HTTP/1.1" 200 1647
Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org HTTP/1.1" 200 307
Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org HTTP/1.1" 200 307
Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org/held HTTP/1.1" 200 90
Jul 04 01:17:58 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org HTTP/1.1" 200 307
Jul 04 01:17:58 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
third-list at mailman3.jab.org HTTP/1.1" 200 307
Jul 06 18:08:19 2012 (10046) Master watcher caught SIGTERM.  Exiting.
Jul 06 18:08:20 2012 (10052) IncomingRunner runner caught SIGTERM.
 Stopping.
Jul 06 18:08:20 2012 (10053) DigestRunner runner caught SIGTERM.  Stopping.
Jul 06 18:08:20 2012 (10051) NewsRunner runner caught SIGTERM.  Stopping.
Jul 06 18:08:20 2012 (10053) DigestRunner runner exiting.
Jul 06 18:08:20 2012 (10051) NewsRunner runner exiting.

That's me stopping mailman on Jul 06 18:08.  Nothing in the mailman3 log
showing any errors anywhere.

Please let me know if there is other data I can pull out for you.

Thanks

Jeff

From barry at list.org  Fri Jul  6 21:26:40 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 6 Jul 2012 15:26:40 -0400
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <CAKHXeC9F0pRUTQcZe4+gx+UFDVSJi9cj4=QE-L__am_yq8+cEA@mail.gmail.com>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
	<4FF40009.5030802@state-of-mind.de> <4FF5B55B.1030404@zone12.com>
	<4FF5BD5C.20707@state-of-mind.de>
	<CAHjiUbo9h-0mzS+A26PgXK7PUjqmnXhk+j6dB=wyjgf5tLVGUQ@mail.gmail.com>
	<CAKHXeC9F0pRUTQcZe4+gx+UFDVSJi9cj4=QE-L__am_yq8+cEA@mail.gmail.com>
Message-ID: <20120706152640.386c77d6@resist.wooz.org>

On Jul 06, 2012, at 11:20 AM, Jeff Marshall wrote:

>First, I restarted mailman3 and the error at
>http://mailman3.jab.org:8000/persisted.  Then I killed and restarted
>postorius and the error cleared up.
> The postorius instance is accessible again.

Thanks Jeff; so it does look like a bug in Postorius and not the core.

-Barry

From barry at list.org  Sat Jul  7 00:39:48 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 6 Jul 2012 18:39:48 -0400
Subject: [Mailman-Developers] Adding a new list attribute
In-Reply-To: <1341262877.30773.YahooMailNeo@web110409.mail.gq1.yahoo.com>
References: <1341261925.35914.YahooMailNeo@web110408.mail.gq1.yahoo.com>
	<1341262877.30773.YahooMailNeo@web110409.mail.gq1.yahoo.com>
Message-ID: <20120706183948.7627fa2f@limelight.wooz.org>

On Jul 02, 2012, at 02:01 PM, Danci Emanuel wrote:

>Indeed, I added these values to the tables, so now they should be fine.
>
>Still, now I am facing another problem. Although I suspected that in the
>get_configuration() is the problem, now I cannot debug that because?
>now I get a new error. After adding the dlists_enabled field to the tables
>and restaring mailman, when I try to access the Postorius server on?
>localhost:8000 I get the HTTP Error 500:?
>
>HTTP Error 500: A server error occurred.  Please contact the administrator.
>and is what shows up in the terminal when I try to access Postorious:

>? File "/usr/local/mm3/mailman/eggs/storm-0.19-py2.6-linux-i686.egg/storm/database.py", line 371, in _check_disconnect
>? ? return function(*args, **kwargs)
>OperationalError: no such column: mailinglist.dlists_enabled
>
>
>Indeed, is a pretty intuitive error but still, I did not manage to find the
>proper solution.?

Did your database already exist before you added the code?  If so, then the
existing table in the database will not have this column.  You can test this
by wiping out your mailman.sql file and starting over.  If it still crashes,
then, well, it's Something Else ;).

>Thanks for your help and sorry for bugging you with so many questions, but I
>really want to make this work as it should. :)

No worries.  Helping with this kind of thing is what this mailing list is all
about!

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120706/2390165b/attachment.pgp>

From barry at list.org  Sat Jul  7 04:00:10 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 6 Jul 2012 22:00:10 -0400
Subject: [Mailman-Developers] Bug 971013 - schema migrations BLOCKED
Message-ID: <20120706220010.4603b21e@limelight.wooz.org>

I've been working on and off for quite some time on bug 971013

https://bugs.launchpad.net/mailman/+bug/971013

This bug is about providing orderly schema migrations.  I now admit that I'm
blocked and I'm looking for suggestions.

As you know, I made a promise that after beta 1, we'd only change the schema
if we can provide automatic migrations.  Or IOW, I won't make incompatible
schema changes that break existing mm3 installations.

We had some discussion on this list a while ago about mechanisms to do this,
and trunk has some rudimentary support for it.  In the meantime, I found out
about a nice tool called Alembic

http://alembic.readthedocs.org/en/latest/index.html

which provides a better framework for doing migrations.  Even though it's
primarily geared toward SQLAlchemy, I've had some off-line discussions with
Alembic's author Michael Bayer about how we could utilize Alembic without
pulling in all of SQLAlchemy[1].  I experimented with a branch and it looked
promising, but that's not what's blocking progress on this.

What's blocking progress is SQLite, as described in this FAQ:

http://sqlite.org/faq.html#q11

tl;dr: SQLite has a very limited ALTER TABLE command, and specifically does
not support column renaming or deleting.  This really sucks for implementing
any kind of schema migration, regardless of the framework.  The little example
shown in the FAQ doesn't really scale well when you have lots of relationships
between the tables, as is the case for the `mailinglist` table as currently
defined.  Here's where I am blocked.

There are a couple of options, none of which I like very much.

 - I could break the beta1 schema migration promise.  I'd just make the schema
   changes and all your existing systems would break (all three of them
   <wink>).  As unfortunate as that is, going this route is really a cop out
   because as soon as 3.0 final is out, we can almost guarantee there will be
   changes in 3.1 so we'll have to face this issue eventually.

 - We could drop support for SQLite.  The other database we (semi-)officially
   support is Postgres (still waiting on a MySQL branch), which has a
   sufficiently powerful ALTER TABLE for our needs.  Some people would claim
   that SQLite isn't really a viable database anyway, since it's prone to
   deadlocks in multithreaded/processing applications such as mm3.  While
   true, especially for large, busy sites, I think SQLite can work for smaller
   sites, and it does make installation much easier since SQLite support comes
   with Python.  It certainly makes testing much easier.

 - As has been suggested before, we could convert the mailinglist table to
   key/value pairs, which is actually how the pendings table works.  This
   might be a good idea *anyway* to better support extensions, but of course
   I'd have to break the beta1 promise (just this once <wink>) to make this
   change, and it wouldn't help for any other table.  I think it would also
   make queries trickier, and heck if we go that route why even use a
   relational database?

 - We could try to only add columns in SQLite and try to smack the migration
   scripts into only copying data from the old columns to the new ones.  This
   might actually be the most feasible option, but it does leave column
   detritus in the SQLite database.  Who cares though, right?  There may be
   other problems with this approach I'm not yet aware of.

I'm sure there are things I've overlooked, so I welcome your comments and
feedback.  I'd love to break the logjam on this bug so that we can get moving
again toward the 3.0 final release.

Cheers,
-Barry

FTR, the two existing WIP branches:

https://code.launchpad.net/~barry/mailman/bug-971013
https://code.launchpad.net/~barry/mailman/alembic

[1] Though, if Storm never gets ported to Python 3, I might just re-evaluate
the decision to use it as the ORM after mm3.0.  SQLAlchemy supports Python 3.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120706/62f951e0/attachment-0001.pgp>

From dave at aasv.org  Sat Jul  7 04:30:48 2012
From: dave at aasv.org (David H. Brown)
Date: Fri, 06 Jul 2012 22:30:48 -0400
Subject: [Mailman-Developers] Bug 971013 - schema migrations BLOCKED
In-Reply-To: <20120706220010.4603b21e@limelight.wooz.org>
References: <20120706220010.4603b21e@limelight.wooz.org>
Message-ID: <9053981a-6b70-4378-97b1-6aee66048949@email.android.com>

It would be slow and consume disk, but couldn?t you copy all the data to a temporary table, delete the entire sqlite table with the old column names, create a new table with the correct column names, copy the data back, and delete the temporary table? Not like you're doing this every hour.
--
David Brown, dave at aasv.org (via mobile)

Barry Warsaw <barry at list.org> wrote:

I've been working on and off for quite some time on bug 971013

https://bugs.launchpad.net/mailman/+bug/971013

This bug is about providing orderly schema migrations. I now admit that I'm
blocked and I'm looking for suggestions.

As you know, I made a promise that after beta 1, we'd only change the schema
if we can provide automatic migrations. Or IOW, I won't make incompatible
schema changes that break existing mm3 installations.

We had some discussion on this list a while ago about mechanisms to do this,
and trunk has some rudimentary support for it. In the meantime, I found out
about a nice tool called Alembic

http://alembic.readthedocs.org/en/latest/index.html

which provides a better framework for doing migrations. Even though it's
primarily geared toward SQLAlchemy, I've had some off-line discussions with
Alembic's author Michael Bayer about how we could utilize Alembic without
pulling in all of SQLAlchemy[1]. I experimented with a branch and it looked
promising, but that's not what's blocking progress on this.

What's blocking progress is SQLite, as described in this FAQ:

http://sqlite.org/faq.html#q11

tl;dr: SQLite has a very limited ALTER TABLE command, and specifically does
not support column renaming or deleting. This really sucks for implementing
any kind of schema migration, regardless of the framework. The little example
shown in the FAQ doesn't really scale well when you have lots of relationships
between the tables, as is the case for the `mailinglist` table as currently
defined. Here's where I am blocked.

There are a couple of options, none of which I like very much.

- I could break the beta1 schema migration promise. I'd just make the schema
changes and all your existing systems would break (all three of them
<wink>). As unfortunate as that is, going this route is really a cop out
because as soon as 3.0 final is out, we can almost guarantee there will be
changes in 3.1 so we'll have to face this issue eventually.

- We could drop support for SQLite. The other database we (semi-)officially
support is Postgres (still waiting on a MySQL branch), which has a
sufficiently powerful ALTER TABLE for our needs. Some people would claim
that SQLite isn't really a viable database anyway, since it's prone to
deadlocks in multithreaded/processing applications such as mm3. While
true, especially for large, busy sites, I think SQLite can work for smaller
sites, and it does make installation much easier since SQLite support comes
with Python. It certainly makes testing much easier.

- As has been suggested before, we could convert the mailinglist table to
key/value pairs, which is actually how the pendings table works. This
might be a good idea *anyway* to better support extensions, but of course
I'd have to break the beta1 promise (just this once <wink>) to make this
change, and it wouldn't help for any other table. I think it would also
make queries trickier, and heck if we go that route why even use a
relational database?

- We could try to only add columns in SQLite and try to smack the migration
scripts into only copying data from the old columns to the new ones. This
might actually be the most feasible option, but it does leave column
detritus in the SQLite database. Who cares though, right? There may be
other problems with this approach I'm not yet aware of.

I'm sure there are things I've overlooked, so I welcome your comments and
feedback. I'd love to break the logjam on this bug so that we can get moving
again toward the 3.0 final release.

Cheers,
-Barry

FTR, the two existing WIP branches:

https://code.launchpad.net/~barry/mailman/bug-971013
https://code.launchpad.net/~barry/mailman/alembic

[1] Though, if Storm never gets ported to Python 3, I might just re-evaluate
the decision to use it as the ORM after mm3.0. SQLAlchemy supports Python 3.
_____________________________________________

Mailman-Developers mailing list
Mailman-Developers at python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/dave%40aasv.org

Security Policy: http://wiki.list.org/x/QIA9


From stephen at xemacs.org  Sat Jul  7 08:23:59 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 07 Jul 2012 15:23:59 +0900
Subject: [Mailman-Developers]  Bug 971013 - schema migrations BLOCKED
In-Reply-To: <20120706220010.4603b21e@limelight.wooz.org>
References: <20120706220010.4603b21e@limelight.wooz.org>
Message-ID: <87sjd45bsw.fsf@uwakimon.sk.tsukuba.ac.jp>

Barry Warsaw writes:

 > What's blocking progress is SQLite, [which] has a very limited
 > ALTER TABLE command, and specifically does not support column
 > renaming or deleting.  This really sucks for implementing any kind
 > of schema migration, regardless of the framework.

So it's the renaming that causes problems, and the issue with renaming
(ie, why SQLite doesn't support it) is that not only do you change the
name of the column, but you need to fix up all references in
relations, is that it?

I still don't see why this is conceptually hard.  Since you're working
with schema changes, you have a list of renamings oldname -> newname,
and row data in the tables whose values do not depend on variable
names.  Then you create a new database (ie, empty tables) with the new
schema, you extend the list of renames with trivial renames (oldname
-> oldname), and you copy the rows in each table from (*oldnames) ->
(*newnames).  What am I missing?

Is there an implicit requirement that this be done quickly? in minimum
space? in place?  Aren't all of those YAGNIs (as David points out more
briefly)?  Except of course for you, since you are testing the
migrations. ;-)  In fact, I would be tempted to say "keep the old
database around, in case you need to reverse migrate" (and for you, it
would allow post-mortems and in general cross-checking of the
correctness of migrations since you have both old and new databases).

 >  - We could try to only add columns in SQLite and try to smack the migration
 >    scripts into only copying data from the old columns to the new ones.  This
 >    might actually be the most feasible option, but it does leave column
 >    detritus in the SQLite database.  Who cares though, right?  There may be
 >    other problems with this approach I'm not yet aware of.

Two problems I can foresee.  First, name collisions: it's quite
possible you'll add a new column with somewhat different semantics
(and therefore different values) and want to give it the name attached
to a previously deleted column.  Easy to workaround if you're doing
things by hand, but an enthusiasm-sapper for you (wearing your DB
developer hat) because I suspect the only thing an auto-migrate script
can do is fatal here.

Second, if the deleted column is involved in relationships, these
relationships will go unmaintained in the new schema.  I don't know if
this would result in an actual bug, but it's unclean.



From sophron at latthi.com  Sun Jul  8 01:03:22 2012
From: sophron at latthi.com (George Chatzisofroniou)
Date: Sun, 8 Jul 2012 02:03:22 +0300
Subject: [Mailman-Developers] [GSoC 2012] Metrics
In-Reply-To: <20120519125944.696eea9e@resist.wooz.org>
References: <CACeRBzmGUUvVNUHfR5MxTV-nAOjeJWOvdhm-awfjv48A=Z1TpQ@mail.gmail.com>
	<20120505162628.4fed030f@limelight.wooz.org>
	<CACeRBz=X8gSpmvLkLg1F60to7_kYkQo4_JR=TQSgHr-n38pbrA@mail.gmail.com>
	<CACeRBzkJhYe1Z-V+TRQrY9ALB5=c4dwUU2HJdPeuz7K-fNxK2A@mail.gmail.com>
	<CACeRBz=rFjHrotrOxCaVY7GWrUML9NP6NhUsnxWpfM6s6=Rp=A@mail.gmail.com>
	<87zk94pekc.fsf@uwakimon.sk.tsukuba.ac.jp>
	<C9F0DD7B-B963-455C-B9B6-9F28C68E3BF3@NFSNet.org>
	<20120519125944.696eea9e@resist.wooz.org>
Message-ID: <20120707230322.GB17663@gmail.com>

Hi everyone,

Here's my new report [1] just before the midterm evaluation.

---
? am before the midterm evaluation and i?m having fun coding and
learning.

The project has reached the first version and achieves most of the
targets set. Specifically:

Metrics store is ready and it is designed in a way to make it easy to
add new metrics.  Every metric is either an event counter (eg posts
sent) or a measured level (eg number of total subscribers) and it is
stored in the database representing an interval of a granularity.
Special methods that retrieve counts for the specified interval or tally
the events with their effects (like a message sent or a new
subscription) are also completed.

Most of the graph tags are completed. In order to make it easy for the
designer to output the metrics he wishes, i have created a language and
its parser.  For example, if the designer wishes to output the data of
monthly number of posts for the last two years, he can easily extract
these counts as follows: ? {% EXTRACT AS graph1 %} posts MONTHLY FOR 2
YEARS {% ENDEXTRACT %} ?. The whole syntax of tag?s language is defined
by a BNF Language description (you may find an earlier version of this
language in a previous post [2]).

Usually, the designer will add a template specification for the graph
tag and the returned values will be placed in this context, but is is
also possible to customize the way this context is presented or create
his own template (For example, he can easily change the date format or
publish the metrics through a table), since the ?EXTRACT? tag actually
returns a complex data structure that can be used in any manner.

However, some templates are already created for the designer, with the
most significant to be the one that outputs the graphs. I used jqplot
library [3] (minimal and fast rendered graphs) to render the graphs. So, in
the previous example, to build the graph from the extracted data, the
designer simply has to add the following to his template: ? {% include
?MM3/line_graph.js? with dataset=graph1 title=?List Activity? %}?, after
loading the appropriate plotting libraries (there are ready templates
that can be included to do that).

For both metrics store and graph tags, i have already created a number
of tests. But i still need to add even more.

There are some things that have not need be done yet and i?ll work on
them in the following days. I need to complete the implementation of the
language (there are some rules that my parser does not handle yet).  I
also want to create a template for bar graphs as well, design a
coalescor that will maintain the number of entries in the database at a
reasonable level and finish my generator that will generate the metrics
from scratch (a very first version is done). Last, i need to use
IArchiver to make a real connection with MM3 core (i currently use a
simulator for testing purposes).
---

[1]: http://sophron.latthi.com/gsoc-mailman/?p=70
[2]: http://sophron.latthi.com/gsoc-mailman/?p=62
[3]: http://www.jqplot.com/


-- 
George Chatzisofroniou
sophron.latthi.com

From syst3m.w0rm at gmail.com  Sun Jul  8 01:52:25 2012
From: syst3m.w0rm at gmail.com (Aamir Khan)
Date: Sat, 7 Jul 2012 19:52:25 -0400
Subject: [Mailman-Developers] [GSoC 2012] HyperKitty
Message-ID: <CAOb12VUde56K6S00-H0urJTSm2Cw246PQG7DcotJTB7J81DMRQ@mail.gmail.com>

Hi,

It has been some time since the last update about HyperKitty. I have
deployed[1] the latest code at mm3test.fedoraproject.org for everyone to
take a look. Please file any bugs/missing features here[2]

HyperKitty offer features like upvoting/downvoting of messages, adding tags
(to be implemented soon!) to threads, user profiles and other user specific
features. We require users to be logged in before using any of these
features. So, the first task for my GSoC project was to implement login
procedure for hyperkitty. It uses django-social-auth application [3] to
implement login functionality. Currently, there are four ways of logging
into hyperkitty test server namely, Google openid, yahoo openid, browserid,
and registering the user on the site itself. I have not enabled other ways
of logging in (e.g, twitter) but it should be fairly easy to enable those.

I have written a blog post about the essential features an archiver should
posses[4] in my opinion. The second most important thing for me is the
ability to promote good content posted on the list. An archiver should be
intelligent enough to suppress the "spamy" content and bring good content
on the surface. For this we have implemented a feature in which a user can
upvote/downvote a message. The users accessing a particular thread will be
able to sort the messages based on votes of other users.

I have also implemented the basic user profiles in which you can see your
personal info and the messages you rated. You will be able to edit the user
profiles soon.

I have also started to write unit tests [5].


Best regards,
Aamir


[1]
http://blog.aamirkhan.co.in/post/26723715431/deploying-hyperkitty-using-apache-wsgi
[2] https://fedorahosted.org/hyperkitty/
[3] https://github.com/omab/django-social-auth
[4] http://blog.aamirkhan.co.in/post/26555164201/hyperkitty-modern-archiver
[5] http://blog.aamirkhan.co.in/post/26723628968/unit-testing-hyperkitty

From danci_emanuel at yahoo.com  Mon Jul  9 20:32:06 2012
From: danci_emanuel at yahoo.com (Danci Emanuel)
Date: Mon, 9 Jul 2012 11:32:06 -0700 (PDT)
Subject: [Mailman-Developers] Adding a new list attribute
In-Reply-To: <20120706183948.7627fa2f@limelight.wooz.org>
References: <1341261925.35914.YahooMailNeo@web110408.mail.gq1.yahoo.com>
	<1341262877.30773.YahooMailNeo@web110409.mail.gq1.yahoo.com>
	<20120706183948.7627fa2f@limelight.wooz.org>
Message-ID: <1341858726.70567.YahooMailNeo@web110409.mail.gq1.yahoo.com>



After all I managed to fix this one also. The problem was that mailman was?
trying to access ?/var/data/mailman.db, which did not contain
the dlists_enabled column, so it was seeing it as a "corrupt" file.
I just deleted it, restarted mailman and it`s working great!

Thanks!
Emanuel?


________________________________
 From: Barry Warsaw <barry at list.org>
To: Danci Emanuel <danci_emanuel at yahoo.com> 
Cc: "mailman-developers at python.org" <mailman-developers at python.org> 
Sent: Saturday, July 7, 2012 1:39 AM
Subject: Re: [Mailman-Developers] Adding a new list attribute
 
On Jul 02, 2012, at 02:01 PM, Danci Emanuel wrote:

>Indeed, I added these values to the tables, so now they should be fine.
>
>Still, now I am facing another problem. Although I suspected that in the
>get_configuration() is the problem, now I cannot debug that because?
>now I get a new error. After adding the dlists_enabled field to the tables
>and restaring mailman, when I try to access the Postorius server on?
>localhost:8000 I get the HTTP Error 500:?
>
>HTTP Error 500: A server error occurred.? Please contact the administrator.
>and is what shows up in the terminal when I try to access Postorious:

>? File "/usr/local/mm3/mailman/eggs/storm-0.19-py2.6-linux-i686.egg/storm/database.py", line 371, in _check_disconnect
>? ? return function(*args, **kwargs)
>OperationalError: no such column: mailinglist.dlists_enabled
>
>
>Indeed, is a pretty intuitive error but still, I did not manage to find the
>proper solution.?

Did your database already exist before you added the code?? If so, then the
existing table in the database will not have this column.? You can test this
by wiping out your mailman.sql file and starting over.? If it still crashes,
then, well, it's Something Else ;).

>Thanks for your help and sorry for bugging you with so many questions, but I
>really want to make this work as it should. :)

No worries.? Helping with this kind of thing is what this mailing list is all
about!

-Barry

From stephen at xemacs.org  Tue Jul 10 02:54:12 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 10 Jul 2012 09:54:12 +0900
Subject: [Mailman-Developers] Adding a new list attribute
In-Reply-To: <1341858726.70567.YahooMailNeo@web110409.mail.gq1.yahoo.com>
References: <1341261925.35914.YahooMailNeo@web110408.mail.gq1.yahoo.com>
	<1341262877.30773.YahooMailNeo@web110409.mail.gq1.yahoo.com>
	<20120706183948.7627fa2f@limelight.wooz.org>
	<1341858726.70567.YahooMailNeo@web110409.mail.gq1.yahoo.com>
Message-ID: <87hatg5tcb.fsf@uwakimon.sk.tsukuba.ac.jp>

Danci Emanuel writes:

 > After all I managed to fix this one also. The problem was that mailman was?
 > trying to access ?/var/data/mailman.db, which did not contain
 > the dlists_enabled column, so it was seeing it as a "corrupt" file.
 > I just deleted it, restarted mailman and it`s working great!

I don't think it's your GSoC responsibility to *fix* anything here
(ask your mentor, though), but it would be kindly appreciated if you
would document that this error is confusing (ie, file an RFE for a
better error message).

Thanks!

From alexander at sulfrian.net  Tue Jul 10 20:37:02 2012
From: alexander at sulfrian.net (Alexander Sulfrian)
Date: Tue, 10 Jul 2012 20:37:02 +0200
Subject: [Mailman-Developers] GSoC midterm evaluations start Monday July
	9th!
In-Reply-To: <4FF72373.2000801@zone12.com>
References: <4FF72373.2000801@zone12.com>
Message-ID: <87sjczjwdt.wl%alexander@sulfrian.net>

Hi,

sorry for delaying my report until now, but I just wanted to finish
some basic parts. So now here comes my summary what I have done during
the past weeks:

After the first steps of checking out a copy of Mailman 3 and Twisted,
building it and executing the test suites, I started with a install of
Mailman 3, to get a public test system. My Mailman 3 instance is
public available [1] and is working after first tests. Additional to
that, I created a blog post describing the setup and the difficulties
I encountered during the installation[2].

After that I read some main parts of the NNTP RFC (RFC3977[3]) and
identified major problems with the twisted nntp module. Apart from the
missing documentation there are some problems, that were discussed
recently on the mailing list. So I decided to write my own
implementation of the NNTP protocol based on asyncore. I recently
pushed a basic runner, protocol parser and handler to my personal
mailman branch[4]. I also created some basic unit tests, that could be
executed for example with "bin/test -m mailman.archiving.nntp". 

I also reviewed the posts about different archiving backends from the
mailing list. I think that KittyStore as proposed, should be
sufficient to access the stored mails by the NNTP daemon. But the
daemon needs to keep additional information in its on database. It
should be possible to use the database of the mailman backend for that
purpose.

The next steps in the next few days, would be the identification and
basic implementation if the required NNTP commands. Basic browsing of
available lists should be possible until tomorrow. The required
configuration options in the backend, the required REST API changes
and the required changes in Postorius would be the next step
afterwards.

Thanks,
Alex

1: http://mailman3.animux.de/
2: http://blog.animux.de/mailman-3-installation.html
3: http://tools.ietf.org/html/rfc3977
4: https://code.launchpad.net/~alexander-sd4c/mailman/mailman
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120710/8e151e45/attachment.pgp>

From f at state-of-mind.de  Tue Jul 10 23:09:02 2012
From: f at state-of-mind.de (Florian Fuchs)
Date: Tue, 10 Jul 2012 23:09:02 +0200
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <CAKHXeC9F0pRUTQcZe4+gx+UFDVSJi9cj4=QE-L__am_yq8+cEA@mail.gmail.com>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
	<4FF40009.5030802@state-of-mind.de>
	<4FF5B55B.1030404@zone12.com> <4FF5BD5C.20707@state-of-mind.de>
	<CAHjiUbo9h-0mzS+A26PgXK7PUjqmnXhk+j6dB=wyjgf5tLVGUQ@mail.gmail.com>
	<CAKHXeC9F0pRUTQcZe4+gx+UFDVSJi9cj4=QE-L__am_yq8+cEA@mail.gmail.com>
Message-ID: <4FFC99EE.2020901@state-of-mind.de>

Hi Jeff,

I am not completely sure, but both the Traceback and the fact that it
worked again after the restart seem to point in direction of the the
django devserver. I'd love to help you setting up postorius with
Apache/mod_wsgi if you like. That would be much more stable than the
built-in server. Also, you'd have much more control over logging etc.

Just email me if you'd like to do that.

Florian



Am 06.07.12 20:20, schrieb Jeff Marshall:
> First, I restarted mailman3 and the error at
> http://mailman3.jab.org:8000/persisted.  Then I killed and restarted
> postorius and the error cleared up.
>  The postorius instance is accessible again.
> 
> 
> 
> Postorius output showed (I don't see a postorius output log file anywhere
> so I don't know if I can pull up history farther back?):
> 
> [04/Jul/2012 06:05:00] "GET / HTTP/1.1" 500 93610
> [04/Jul/2012 08:59:05] "GET / HTTP/1.1" 500 93655
> [04/Jul/2012 09:53:42] "GET / HTTP/1.0" 500 93553
> [04/Jul/2012 09:56:39] "GET / HTTP/1.1" 500 93838
> [04/Jul/2012 10:12:32] "GET / HTTP/1.1" 500 93775
> [04/Jul/2012 10:25:23] "GET / HTTP/1.1" 500 93655
> [04/Jul/2012 12:25:56] "GET / HTTP/1.1" 500 93583
> [04/Jul/2012 13:24:33] "GET / HTTP/1.1" 500 93443
> [04/Jul/2012 13:26:42] "GET / HTTP/1.1" 500 93443
> [04/Jul/2012 21:50:56] "GET /static/postorius/defaultimages/favicon.ico
> HTTP/1.1" 404 1803
> [05/Jul/2012 05:39:37] "GET / HTTP/1.1" 500 93882
> Traceback (most recent call last):
>   File
> "/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
> line 284, in run
>     self.finish_response()
>   File
> "/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
> line 324, in finish_response
>     self.write(data)
>   File
> "/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
> line 420, in write
>     self._write(data)
>   File "/usr/lib/python2.7/socket.py", line 324, in write
>     self.flush()
>   File "/usr/lib/python2.7/socket.py", line 303, in flush
>     self._sock.sendall(view[write_offset:write_offset+buffer_size])
> error: [Errno 32] Broken pipe
> ----------------------------------------
> Exception happened during processing of request from ('134.95.19.9', 26446)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/SocketServer.py", line 284, in
> _handle_request_noblock
>     self.process_request(request, client_address)
>   File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
>     self.finish_request(request, client_address)
>   File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
>     self.RequestHandlerClass(request, client_address, self)
>   File
> "/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
> line 570, in __init__
>     BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
>   File "/usr/lib/python2.7/SocketServer.py", line 641, in __init__
>     self.finish()
>   File "/usr/lib/python2.7/SocketServer.py", line 694, in finish
>     self.wfile.flush()
>   File "/usr/lib/python2.7/socket.py", line 303, in flush
>     self._sock.sendall(view[write_offset:write_offset+buffer_size])
> error: [Errno 32] Broken pipe
> ----------------------------------------
> [05/Jul/2012 06:14:25] "GET / HTTP/1.1" 500 94198
> [05/Jul/2012 10:39:53] "GET / HTTP/1.1" 500 93559
> [05/Jul/2012 10:51:25] "GET / HTTP/1.1" 500 93882
> [05/Jul/2012 11:30:01] "GET / HTTP/1.1" 500 93538
> [05/Jul/2012 11:54:47] "GET / HTTP/1.1" 500 93655
> [05/Jul/2012 12:09:25] "GET / HTTP/1.1" 500 94323
> 
> 
> 
> And the mailman3 var/log/mailman.log looked like this throughout:
> 
> Jul 04 01:17:37 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org HTTP/1.1" 200 307
> Jul 04 01:17:37 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org/config HTTP/1.1" 200 1647
> Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> jablist at mailman3.jab.org HTTP/1.1" 200 295
> Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> jablist at mailman3.jab.org HTTP/1.1" 200 295
> Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> jablist at mailman3.jab.org/config HTTP/1.1" 200 1611
> Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> public-test at mailman3.jab.org HTTP/1.1" 200 311
> Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> public-test at mailman3.jab.org HTTP/1.1" 200 311
> Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> public-test at mailman3.jab.org/config HTTP/1.1" 200 1634
> Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org HTTP/1.1" 200 307
> Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org HTTP/1.1" 200 307
> Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org/config HTTP/1.1" 200 1647
> Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org HTTP/1.1" 200 307
> Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org HTTP/1.1" 200 307
> Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org/held HTTP/1.1" 200 90
> Jul 04 01:17:58 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org HTTP/1.1" 200 307
> Jul 04 01:17:58 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> third-list at mailman3.jab.org HTTP/1.1" 200 307
> Jul 06 18:08:19 2012 (10046) Master watcher caught SIGTERM.  Exiting.
> Jul 06 18:08:20 2012 (10052) IncomingRunner runner caught SIGTERM.
>  Stopping.
> Jul 06 18:08:20 2012 (10053) DigestRunner runner caught SIGTERM.  Stopping.
> Jul 06 18:08:20 2012 (10051) NewsRunner runner caught SIGTERM.  Stopping.
> Jul 06 18:08:20 2012 (10053) DigestRunner runner exiting.
> Jul 06 18:08:20 2012 (10051) NewsRunner runner exiting.
> 
> That's me stopping mailman on Jul 06 18:08.  Nothing in the mailman3 log
> showing any errors anywhere.
> 
> Please let me know if there is other data I can pull out for you.
> 
> Thanks
> 
> Jeff
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind.de
> 
> Security Policy: http://wiki.list.org/x/QIA9
> 


From richard at NFSNet.org  Tue Jul 10 23:44:33 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Tue, 10 Jul 2012 16:44:33 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
Message-ID: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>

I am encountering a problem with login on Postorius. There is no mechanism to keep the MM user database synchronized to the one which django creates. If you use BrowserID to log in, or if you are otherwise in the django user database, this is currently causing a access error for any user that was not previously defined in the MM database when attempting to access the MM interface.

Further, in implementing extensions to the login function, having the password (but not the other user profile information, including alternate credential specifications) on a separate system greatly complicates the handling of login.

It would greatly simplify things if all of the user information, including profile, and whatever information the "social media" folks want to keep, were stored in one place.

If we are not going to have a single database, to which all components of the system have equal access, then I suggest that we treat the user information as a separate realm from both the UI and the "core" and require both of them to access that information through the same REST interface.







From stephen at xemacs.org  Wed Jul 11 04:14:38 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 11 Jul 2012 11:14:38 +0900
Subject: [Mailman-Developers]  Login / User Identification Issues in MM3
In-Reply-To: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
Message-ID: <87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > I am encountering a problem with login on Postorius. There is no
 > mechanism to keep the MM user database synchronized to the one
 > which django creates.

Since Postorius is the facility by which admins will expect to access
the user database, I have to consider that a bug in Postorius.


From marshman at gmail.com  Wed Jul 11 05:31:36 2012
From: marshman at gmail.com (Jeff Marshall)
Date: Tue, 10 Jul 2012 20:31:36 -0700
Subject: [Mailman-Developers] mailman3 instance for preview + exploration
In-Reply-To: <4FFC99EE.2020901@state-of-mind.de>
References: <CAHjiUbqNbo_zOkhP2Q+q+p8QXdBDgrJ=nc78Tihiw5vOaADe1w@mail.gmail.com>
	<4FF40009.5030802@state-of-mind.de> <4FF5B55B.1030404@zone12.com>
	<4FF5BD5C.20707@state-of-mind.de>
	<CAHjiUbo9h-0mzS+A26PgXK7PUjqmnXhk+j6dB=wyjgf5tLVGUQ@mail.gmail.com>
	<CAKHXeC9F0pRUTQcZe4+gx+UFDVSJi9cj4=QE-L__am_yq8+cEA@mail.gmail.com>
	<4FFC99EE.2020901@state-of-mind.de>
Message-ID: <CAKHXeC_5moPuHaC_g-ZKFn5LEtR=spSOTDCYfdhkS-nFc4+6MA@mail.gmail.com>

Sure thing.  Do you have instructions written up you can send around?  If
it requires more hand-holding then feel free to email me directly and we
can IM or I can give you a login to that server.

Thanks

Jeff


On Tue, Jul 10, 2012 at 2:09 PM, Florian Fuchs <f at state-of-mind.de> wrote:

> Hi Jeff,
>
> I am not completely sure, but both the Traceback and the fact that it
> worked again after the restart seem to point in direction of the the
> django devserver. I'd love to help you setting up postorius with
> Apache/mod_wsgi if you like. That would be much more stable than the
> built-in server. Also, you'd have much more control over logging etc.
>
> Just email me if you'd like to do that.
>
> Florian
>
>
>
> Am 06.07.12 20:20, schrieb Jeff Marshall:
> > First, I restarted mailman3 and the error at
> > http://mailman3.jab.org:8000/persisted.  Then I killed and restarted
> > postorius and the error cleared up.
> >  The postorius instance is accessible again.
> >
> >
> >
> > Postorius output showed (I don't see a postorius output log file anywhere
> > so I don't know if I can pull up history farther back?):
> >
> > [04/Jul/2012 06:05:00] "GET / HTTP/1.1" 500 93610
> > [04/Jul/2012 08:59:05] "GET / HTTP/1.1" 500 93655
> > [04/Jul/2012 09:53:42] "GET / HTTP/1.0" 500 93553
> > [04/Jul/2012 09:56:39] "GET / HTTP/1.1" 500 93838
> > [04/Jul/2012 10:12:32] "GET / HTTP/1.1" 500 93775
> > [04/Jul/2012 10:25:23] "GET / HTTP/1.1" 500 93655
> > [04/Jul/2012 12:25:56] "GET / HTTP/1.1" 500 93583
> > [04/Jul/2012 13:24:33] "GET / HTTP/1.1" 500 93443
> > [04/Jul/2012 13:26:42] "GET / HTTP/1.1" 500 93443
> > [04/Jul/2012 21:50:56] "GET /static/postorius/defaultimages/favicon.ico
> > HTTP/1.1" 404 1803
> > [05/Jul/2012 05:39:37] "GET / HTTP/1.1" 500 93882
> > Traceback (most recent call last):
> >   File
> > "/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
> > line 284, in run
> >     self.finish_response()
> >   File
> > "/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
> > line 324, in finish_response
> >     self.write(data)
> >   File
> > "/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
> > line 420, in write
> >     self._write(data)
> >   File "/usr/lib/python2.7/socket.py", line 324, in write
> >     self.flush()
> >   File "/usr/lib/python2.7/socket.py", line 303, in flush
> >     self._sock.sendall(view[write_offset:write_offset+buffer_size])
> > error: [Errno 32] Broken pipe
> > ----------------------------------------
> > Exception happened during processing of request from ('134.95.19.9',
> 26446)
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/SocketServer.py", line 284, in
> > _handle_request_noblock
> >     self.process_request(request, client_address)
> >   File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
> >     self.finish_request(request, client_address)
> >   File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
> >     self.RequestHandlerClass(request, client_address, self)
> >   File
> > "/usr/local/lib/python2.7/dist-packages/django/core/servers/basehttp.py",
> > line 570, in __init__
> >     BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
> >   File "/usr/lib/python2.7/SocketServer.py", line 641, in __init__
> >     self.finish()
> >   File "/usr/lib/python2.7/SocketServer.py", line 694, in finish
> >     self.wfile.flush()
> >   File "/usr/lib/python2.7/socket.py", line 303, in flush
> >     self._sock.sendall(view[write_offset:write_offset+buffer_size])
> > error: [Errno 32] Broken pipe
> > ----------------------------------------
> > [05/Jul/2012 06:14:25] "GET / HTTP/1.1" 500 94198
> > [05/Jul/2012 10:39:53] "GET / HTTP/1.1" 500 93559
> > [05/Jul/2012 10:51:25] "GET / HTTP/1.1" 500 93882
> > [05/Jul/2012 11:30:01] "GET / HTTP/1.1" 500 93538
> > [05/Jul/2012 11:54:47] "GET / HTTP/1.1" 500 93655
> > [05/Jul/2012 12:09:25] "GET / HTTP/1.1" 500 94323
> >
> >
> >
> > And the mailman3 var/log/mailman.log looked like this throughout:
> >
> > Jul 04 01:17:37 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org HTTP/1.1" 200 307
> > Jul 04 01:17:37 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org/config HTTP/1.1" 200 1647
> > Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > jablist at mailman3.jab.org HTTP/1.1" 200 295
> > Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > jablist at mailman3.jab.org HTTP/1.1" 200 295
> > Jul 04 01:17:42 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > jablist at mailman3.jab.org/config HTTP/1.1" 200 1611
> > Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > public-test at mailman3.jab.org HTTP/1.1" 200 311
> > Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > public-test at mailman3.jab.org HTTP/1.1" 200 311
> > Jul 04 01:17:48 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > public-test at mailman3.jab.org/config HTTP/1.1" 200 1634
> > Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org HTTP/1.1" 200 307
> > Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org HTTP/1.1" 200 307
> > Jul 04 01:17:52 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org/config HTTP/1.1" 200 1647
> > Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org HTTP/1.1" 200 307
> > Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org HTTP/1.1" 200 307
> > Jul 04 01:17:55 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org/held HTTP/1.1" 200 90
> > Jul 04 01:17:58 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org HTTP/1.1" 200 307
> > Jul 04 01:17:58 2012 (10058) mailman3.jab.org - - "GET /3.0/lists/
> > third-list at mailman3.jab.org HTTP/1.1" 200 307
> > Jul 06 18:08:19 2012 (10046) Master watcher caught SIGTERM.  Exiting.
> > Jul 06 18:08:20 2012 (10052) IncomingRunner runner caught SIGTERM.
> >  Stopping.
> > Jul 06 18:08:20 2012 (10053) DigestRunner runner caught SIGTERM.
>  Stopping.
> > Jul 06 18:08:20 2012 (10051) NewsRunner runner caught SIGTERM.  Stopping.
> > Jul 06 18:08:20 2012 (10053) DigestRunner runner exiting.
> > Jul 06 18:08:20 2012 (10051) NewsRunner runner exiting.
> >
> > That's me stopping mailman on Jul 06 18:08.  Nothing in the mailman3 log
> > showing any errors anywhere.
> >
> > Please let me know if there is other data I can pull out for you.
> >
> > Thanks
> >
> > Jeff
> > _______________________________________________
> > Mailman-Developers mailing list
> > Mailman-Developers at python.org
> > http://mail.python.org/mailman/listinfo/mailman-developers
> > Mailman FAQ: http://wiki.list.org/x/AgA3
> > Searchable Archives:
> http://www.mail-archive.com/mailman-developers%40python.org/
> > Unsubscribe:
> http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind.de
> >
> > Security Policy: http://wiki.list.org/x/QIA9
> >
>
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives:
> http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe:
> http://mail.python.org/mailman/options/mailman-developers/marshman%40gmail.com
>
> Security Policy: http://wiki.list.org/x/QIA9
>

From richard at NFSNet.org  Wed Jul 11 05:51:30 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Tue, 10 Jul 2012 22:51:30 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>

Yes, it is a bug in Postorius. However, that does not negate the fact that the present design, by forcing a split database, makes it difficult to accomplish the desired behavior.

As far as I am concerned, if Postorius is corrected to handle this situation robustly, it will be in spite of, and not with the help of, the design being forced upon it.

Since by far the most complicated, and most used, logins will be made from the web interface, it is much more logical to allow Postorius to be responsible for all of the user profile information and have the core, on those rare occasions where it requires it, obtain password confirmation from that source.

Richard

On Jul 10, 2012, at 9:14 PM, Stephen J. Turnbull wrote:

> Richard Wackerbarth writes:
> 
>> I am encountering a problem with login on Postorius. There is no
>> mechanism to keep the MM user database synchronized to the one
>> which django creates.
> 
> Since Postorius is the facility by which admins will expect to access
> the user database, I have to consider that a bug in Postorius.
> 


From stephen at xemacs.org  Wed Jul 11 07:12:30 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 11 Jul 2012 14:12:30 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
Message-ID: <878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > Since by far the most complicated, and most used, logins will be
 > made from the web interface, it is much more logical to allow
 > Postorius to be responsible for all of the user profile information
 > and have the core, on those rare occasions where it requires it,
 > obtain password confirmation from that source.

>From the point of view of Postorius implementers, no question about
it.<wink/>

But isn't that going to take us a long way down the road where we
anoint Postorius the one-and-only admin interface?  If that really
needs to be, OK, but I don't much like it.  Among other things, it
will make the design and detailed UI of Postorius a focus of
discussion for everybody concerned with Mailman 3.  And it makes the
option to "build one to throw away" much more difficult -- the design
decisions already made, and will be made in the near future, will
probably live as long as Pipermail has (and Pipermail will continue
for several more years, at least!)

If that doesn't scare you....<wink/>


From terri at zone12.com  Wed Jul 11 08:55:25 2012
From: terri at zone12.com (Terri Oda)
Date: Wed, 11 Jul 2012 00:55:25 -0600
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4FFD235D.80104@zone12.com>

On 12-07-10 11:12 PM, Stephen J. Turnbull wrote:
> But isn't that going to take us a long way down the road where we
> anoint Postorius the one-and-only admin interface?  If that really
> needs to be, OK, but I don't much like it.  Among other things, it
> will make the design and detailed UI of Postorius a focus of
> discussion for everybody concerned with Mailman 3.  And it makes the
> option to "build one to throw away" much more difficult -- the design
> decisions already made, and will be made in the near future, will
> probably live as long as Pipermail has (and Pipermail will continue
> for several more years, at least!)

I think it may be possible that the core authentication stuff can be 
pushed into REST without tying us to postorius forever, but I haven't 
got it quite set in my head how that will go yet.

* Right now, Postorious can do logins based on email/password pairs in 
REST.

* We'd like to do BrowserId, which only needs the email (and we're 
trusting the browser to do the authentication) so that shouldn't be a 
problem.  BrowserID was not completely implemented when I last played in 
there... unless someone else has finished the hookup, please do not 
assume that it's fully working and feel free to file bugs so what's not 
working is clearly indicated somewhere other than my head. ;)  Right 
now, it generates a login, but has no useful interaction with REST settings.

*  We'd also like to do openid, which means we need to somehow associate 
an openid token with an email address.

So right now, postorius needs email address, username (for direct 
authentication), and potentially a list of openid or other tokens.  
That's a small enough list that we may be able to justify making mailman 
core aware of a small token list (or a single openid token?), or we can 
let postorius handle that and have core only understand "I am the owner 
of this email address -- let me see the associated settings of me."  I 
think my preference would be to have mailman understand more than 
email/password authentication, because I think it'll make things easier 
and not have us duplicating data in hyperkitty etc, though.

The messy part, IMO, is what to do with the non-authentication user 
data. I'm guessing we'll probably want some sort of theme preference 
data (possibly shared between postorius/hyperkitty/others?).  Not sure 
what else.  That stuff... really doesn't have much place in core, but 
probably will need to be shared between several web components... do we 
have a second rest server for user data?



From richard at NFSNet.org  Wed Jul 11 12:39:02 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Wed, 11 Jul 2012 05:39:02 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>

Stephen,

I'm not advocating that Postorious be THE keeper of the user information. But it would be a good candidate for the job.

What I am advocating is that the "core" message handler NOT be the keeper of ONLY PART of it.

I am perfectly happy to have the user info handled by an independent stand-alone module which is willing to take responsibility for ALL of the user profile info. It should provide a REST interface that allows other modules the ability perform authentications, manipulate the profile, etc.

Another acceptable methodology would be to store all of the data in a real relational database and allow each of the modules direct access to the database engine.

Richard

On Jul 11, 2012, at 12:12 AM, Stephen J. Turnbull wrote:

> Richard Wackerbarth writes:
> 
>> Since by far the most complicated, and most used, logins will be
>> made from the web interface, it is much more logical to allow
>> Postorius to be responsible for all of the user profile information
>> and have the core, on those rare occasions where it requires it,
>> obtain password confirmation from that source.
> 
> From the point of view of Postorius implementers, no question about
> it.<wink/>
> 
> But isn't that going to take us a long way down the road where we
> anoint Postorius the one-and-only admin interface?  If that really
> needs to be, OK, but I don't much like it.  Among other things, it
> will make the design and detailed UI of Postorius a focus of
> discussion for everybody concerned with Mailman 3.  And it makes the
> option to "build one to throw away" much more difficult -- the design
> decisions already made, and will be made in the near future, will
> probably live as long as Pipermail has (and Pipermail will continue
> for several more years, at least!)
> 
> If that doesn't scare you....<wink/>
> 


From stephen at xemacs.org  Wed Jul 11 15:14:54 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 11 Jul 2012 22:14:54 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
Message-ID: <87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > What I am advocating is that the "core" message handler NOT be the
 > keeper of ONLY PART of it.

What I'm advocating (mildly, because somebody else is going to have to
do the work) is that the core be the keeper of ALL of it.  The core is
not just a "message handler".  It is also a database, containing both
list information and subscriber information.  Since a minimum of
subscriber information is absolutely essential to the core job, all of
it may as well be in there.  In some configurations we will want the
subscribers to be authenticated, so we may as well keep all such
information in the core's database.

Steve

From richard at NFSNet.org  Wed Jul 11 15:34:25 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Wed, 11 Jul 2012 08:34:25 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>

On Jul 11, 2012, at 8:14 AM, Stephen J. Turnbull wrote:
> Richard Wackerbarth writes:
> 
>> What I am advocating is that the "core" message handler NOT be the
>> keeper of ONLY PART of it.
> 
> What I'm advocating (mildly, because somebody else is going to have to
> do the work) is that the core be the keeper of ALL of it.  The core is
> not just a "message handler".  It is also a database, containing both
> list information and subscriber information.  

OK, so we agree that ALL of the information SHOULD be stored in one place.
That means that this database will need a lot more information, such as access control specifications, etc.
Further, it needs to be extensible so that various users can add whatever customizations and extensions they need.

And each of those functions will need supporting views, etc.

Pretty soon, you will find that what you need approaches something that already exists -- a relational database.
Rather than "reinventing the wheel", we should just use an already existing database system and make all of the data directly accessible.

> Since a minimum of subscriber information is absolutely essential to the core job, all of
> it may as well be in there.

This does not follow logically.  Since only a minimum of information is essential to the core job, it may well be more appropriate for it to get that information from another source as needed.

> In some configurations we will want the subscribers to be authenticated, so we may as well keep all such
> information in the core's database.
> 
> Steve

Applying your previous argument, I could equally say "since the web user needs to be authenticated, we may as well keep all such information in the webUI's database"

Richard

From richard at NFSNet.org  Wed Jul 11 16:27:23 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Wed, 11 Jul 2012 09:27:23 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <4FFD235D.80104@zone12.com>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
Message-ID: <592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>

On Jul 11, 2012, at 1:55 AM, Terri Oda wrote:
> *  We'd also like to do openid, which means we need to somehow associate an openid token with an email address.
> 
> So right now, postorius needs email address, username (for direct authentication), and potentially a list of openid or other tokens.  That's a small enough list that we may be able to justify making mailman core aware of a small token list (or a single openid token?), or we can let postorius handle that and have core only understand "I am the owner of this email address -- let me see the associated settings of me."  I think my preference would be to have mailman understand more than email/password authentication, because I think it'll make things easier and not have us duplicating data in hyperkitty etc, though.

> The messy part, IMO, is what to do with the non-authentication user data. I'm guessing we'll probably want some sort of theme preference data (possibly shared between postorius/hyperkitty/others?).  Not sure what else.  That stuff... really doesn't have much place in core, but probably will need to be shared between several web components... do we have a second rest server for user data?

If we are going to use REST interfaces to tie components together, then, in the design, we should do a few things differently.

First, we should define all of the data storage in terms of the REST access points to the data. (And not, as presently done, the other way around)
Next, we should access all of the REST interface URLs indirectly so that functionality can be moved around simply by changing a single reference definition.  This is the kind of scheme that django uses with its {% url %} tag. (like DNS vs IP addresses)
Finally, we should "black box encapsulate" the access to the data, requiring that EVERY module utilize this common interface, and only this interface, to the data.

From f at state-of-mind.de  Wed Jul 11 17:07:33 2012
From: f at state-of-mind.de (Florian Fuchs)
Date: Wed, 11 Jul 2012 17:07:33 +0200
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
Message-ID: <4FFD96B5.8050107@state-of-mind.de>

Hi there,

Am 11.07.12 15:34, schrieb Richard Wackerbarth:
> On Jul 11, 2012, at 8:14 AM, Stephen J. Turnbull wrote:
>> Richard Wackerbarth writes:
>>
>>> What I am advocating is that the "core" message handler NOT be the
>>> keeper of ONLY PART of it.
>>
>> What I'm advocating (mildly, because somebody else is going to have to
>> do the work) is that the core be the keeper of ALL of it.  The core is
>> not just a "message handler".  It is also a database, containing both
>> list information and subscriber information.  

If we're only talking authentication data, I agree.

But I also agree with Terri that there might be a good amount of user
data used by Postorius, Hyperkitty or any other web ui/client that just
doesn't have anything to do with mailman's core tasks. And I don't see
why something like "preferred ui theme" or profile-related stuff like
"irc nick" should be stored in the core db.

Isn't it very common that applications combine information from
different sources (databases, webservices,...) in one place (with or
without caching them locally)? I don't see anything unusual in the
concept of having some mailman-related user data managed by the mailman
core and other kinds of data handled by the
database/file-structure/key-value-store/web-service(s) that a web
application is using.

If Postorius and HyperKitty decide to share some information in one
place, because the projects are so closely related, that's of course a
fine idea. But I wouldn't try to cram everything into the core db just
for the sake of having it all in one place.

Florian


> 
> OK, so we agree that ALL of the information SHOULD be stored in one place.
> That means that this database will need a lot more information, such as access control specifications, etc.
> Further, it needs to be extensible so that various users can add whatever customizations and extensions they need.
> 
> And each of those functions will need supporting views, etc.
> 
> Pretty soon, you will find that what you need approaches something that already exists -- a relational database.
> Rather than "reinventing the wheel", we should just use an already existing database system and make all of the data directly accessible.
> 
>> Since a minimum of subscriber information is absolutely essential to the core job, all of
>> it may as well be in there.
> 
> This does not follow logically.  Since only a minimum of information is essential to the core job, it may well be more appropriate for it to get that information from another source as needed.
> 
>> In some configurations we will want the subscribers to be authenticated, so we may as well keep all such
>> information in the core's database.
>>
>> Steve
> 
> Applying your previous argument, I could equally say "since the web user needs to be authenticated, we may as well keep all such information in the webUI's database"
> 
> Richard
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind.de
> 
> Security Policy: http://wiki.list.org/x/QIA9
> 


From richard at NFSNet.org  Wed Jul 11 18:46:53 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Wed, 11 Jul 2012 11:46:53 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <4FFD96B5.8050107@state-of-mind.de>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<4FFD96B5.8050107@state-of-mind.de>
Message-ID: <0685777A-91BD-4A50-AEC1-05D874989D16@NFSNet.org>

The problem in splitting data is that links between the various related entries need to be maintained.
This means having the ability to go from one entry to the related ones, and back again.
When the data is stored in multiple tables, foreign keys provide the link. When both tables reside in the same database, joins can be performed, etc. However, when the tables are in separate databases, at a minimum, each database needs to "know about" (store the inverse key) the other entries. They also need to be able to assure that the corresponding entries in the foreign tables get created, and likely require that they be unique. Because of thread locking considerations, this may not be so easy to accomplish.

On Jul 11, 2012, at 10:07 AM, Florian Fuchs wrote:

> Hi there,
> 
> Am 11.07.12 15:34, schrieb Richard Wackerbarth:
>> On Jul 11, 2012, at 8:14 AM, Stephen J. Turnbull wrote:
>>> Richard Wackerbarth writes:
>>> 
>>>> What I am advocating is that the "core" message handler NOT be the
>>>> keeper of ONLY PART of it.
>>> 
>>> What I'm advocating (mildly, because somebody else is going to have to
>>> do the work) is that the core be the keeper of ALL of it.  The core is
>>> not just a "message handler".  It is also a database, containing both
>>> list information and subscriber information.  
> 
> If we're only talking authentication data, I agree.
> 
> But I also agree with Terri that there might be a good amount of user
> data used by Postorius, Hyperkitty or any other web ui/client that just
> doesn't have anything to do with mailman's core tasks. And I don't see
> why something like "preferred ui theme" or profile-related stuff like
> "irc nick" should be stored in the core db.
> 
> Isn't it very common that applications combine information from
> different sources (databases, webservices,...) in one place (with or
> without caching them locally)? I don't see anything unusual in the
> concept of having some mailman-related user data managed by the mailman
> core and other kinds of data handled by the
> database/file-structure/key-value-store/web-service(s) that a web
> application is using.
> 
> If Postorius and HyperKitty decide to share some information in one
> place, because the projects are so closely related, that's of course a
> fine idea. But I wouldn't try to cram everything into the core db just
> for the sake of having it all in one place.
> 
> Florian
> 
> 
>> 
>> OK, so we agree that ALL of the information SHOULD be stored in one place.
>> That means that this database will need a lot more information, such as access control specifications, etc.
>> Further, it needs to be extensible so that various users can add whatever customizations and extensions they need.
>> 
>> And each of those functions will need supporting views, etc.
>> 
>> Pretty soon, you will find that what you need approaches something that already exists -- a relational database.
>> Rather than "reinventing the wheel", we should just use an already existing database system and make all of the data directly accessible.
>> 
>>> Since a minimum of subscriber information is absolutely essential to the core job, all of
>>> it may as well be in there.
>> 
>> This does not follow logically.  Since only a minimum of information is essential to the core job, it may well be more appropriate for it to get that information from another source as needed.
>> 
>>> In some configurations we will want the subscribers to be authenticated, so we may as well keep all such
>>> information in the core's database.
>>> 
>>> Steve
>> 
>> Applying your previous argument, I could equally say "since the web user needs to be authenticated, we may as well keep all such information in the webUI's database"
>> 
>> Richard
>> _______________________________________________
>> Mailman-Developers mailing list
>> Mailman-Developers at python.org
>> http://mail.python.org/mailman/listinfo/mailman-developers
>> Mailman FAQ: http://wiki.list.org/x/AgA3
>> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
>> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/f%40state-of-mind.de
>> 
>> Security Policy: http://wiki.list.org/x/QIA9
>> 
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/richard%40nfsnet.org
> 
> Security Policy: http://wiki.list.org/x/QIA9


From stephen at xemacs.org  Wed Jul 11 19:50:09 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 12 Jul 2012 02:50:09 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <4FFD96B5.8050107@state-of-mind.de>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<4FFD96B5.8050107@state-of-mind.de>
Message-ID: <874npe5gry.fsf@uwakimon.sk.tsukuba.ac.jp>

Florian Fuchs writes:

 > But I also agree with Terri that there might be a good amount of user
 > data used by Postorius, Hyperkitty or any other web ui/client that just
 > doesn't have anything to do with mailman's core tasks. And I don't see
 > why something like "preferred ui theme" or profile-related stuff like
 > "irc nick" should be stored in the core db.

Because there may be multiple clients wanting to access that data.  

 > Isn't it very common that applications combine information from
 > different sources (databases, webservices,...) in one place (with or
 > without caching them locally)?

Yes, in fact it is a common RFE ("I'd like to get general information
about our organization members from our organization's membership
database and combine it with Mailman-specific information.")

Nevertheless, it would be preferable if that is wrapped in an API that
makes it look like it's all coming form one place, and which manages
the linkages so that data is not store redundantly in different
places, or inaccessible to certain clients.  Even UI theme, which you
might think would be very specific to each app, is likely to be
unified site-wide.  At least, site designers will probably want that
to be possible.

I just don't think there's all that much data which nobody will ever
find a use for sharing it cross-app.  So we may as well provide APIs
for that sharing from the start.

 > I don't see anything unusual in the concept of having some
 > mailman-related user data managed by the mailman core and other
 > kinds of data handled by the
 > database/file-structure/key-value-store/web-service(s) that a web
 > application is using.

Common, yes, but the cross-DB linkage problems that arise in that
architecture are both predictable and already being observed.

 > If Postorius and HyperKitty decide to share some information in one
 > place, because the projects are so closely related, that's of course a
 > fine idea. But I wouldn't try to cram everything into the core db just
 > for the sake of having it all in one place.

I don't see any "cramming" here.  The point of using an RDBMS (or a
derivative such as an ORM or Django itself) is to give the the
database a natural structure in multiple tables or objects, while
accessing it through a common API.

If the core wants to delegate some DB maintenance to another module or
an external app such as an organizational member DB, that's OK, but I
think that there should be a central point of contact for all of the
data for a given object such as a user.

Steve

From stephen at xemacs.org  Wed Jul 11 19:50:40 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 12 Jul 2012 02:50:40 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
Message-ID: <87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > Pretty soon, you will find that what you need approaches something
 > that already exists -- a relational database.  Rather than
 > "reinventing the wheel", we should just use an already existing
 > database system and make all of the data directly accessible.

We're already doing that, with ORMs overlying the RDBMS.

 > Since only a minimum of information is essential to the core job,
 > it may well be more appropriate for it to get that information from
 > another source as needed.

True, but we've already agreed that the user information should be
kept in one place, based on your experience.

 > Applying your previous argument, I could equally say "since the web
 > user needs to be authenticated, we may as well keep all such
 > information in the webUI's database"

It's not the same argument.  A mailing list needs a message
distribution agent; it doesn't *need* a webUI.

There may be implementation reasons why it's better to handle database
requirements in the webUI or a new daemon, but nobody's given any yet.


From richard at NFSNet.org  Wed Jul 11 21:22:55 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Wed, 11 Jul 2012 14:22:55 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>

On Jul 11, 2012, at 12:50 PM, Stephen J. Turnbull wrote:

> Richard Wackerbarth writes:
> 
>> Pretty soon, you will find that what you need approaches something
>> that already exists -- a relational database.  Rather than
>> "reinventing the wheel", we should just use an already existing
>> database system and make all of the data directly accessible.
> 
> We're already doing that, with ORMs overlying the RDBMS.

No! Although you are making available (some/most/all) of the data values, you are not making available the ability to make arbitrary SQL-type queries to view the data.

>> Since only a minimum of information is essential to the core job,
>> it may well be more appropriate for it to get that information from
>> another source as needed.
> 
> True, but we've already agreed that the user information should be
> kept in one place, based on your experience.
> 
>> Applying your previous argument, I could equally say "since the web
>> user needs to be authenticated, we may as well keep all such
>> information in the webUI's database"
> 
> It's not the same argument.  A mailing list needs a message
> distribution agent; it doesn't *need* a webUI.

In this day and age, try selling that one! Only those of us, and that especially includes me, who were around "way back when", before http even existed, know any other way. :)

Additionally, look at what other MM developers are doing. For example, Hyperkitty is not oriented to retrieving archived messages by email. I think we MUST assume that there will be a web interface. If someone wants to have a mailing list without one, it should be easy enough to "stub off" the limited amount of user information that is needed. But I think that this use case will be the exception rather than the rule.

> There may be implementation reasons why it's better to handle database
> requirements in the webUI or a new daemon, but nobody's given any yet.


As far as the complexity of access is concerned, the mail handler probably has the lowest requirements. The present architecture would have to be extended significantly to provide for appropriate handling of ALL of the data. That includes a much richer query capability.

Presently, the message handling is integrally tied to the database implementation. Customization extensions will intrude into parts of the system which they should not affect.

As far as I am concerned, those are more than adequate reasons.

I view your argument as the message handler claiming "I'm special! Everyone else has do do things my way. I get special privileges."  -- IMHO, the tail is wagging the dog.

Let's split shared data storage from the message handler, and from any "admin-by-mail" component, treating each of them as separate logical components. Provide each of them EXACTLY the same accesses as those provided to the WebUI, Message Archiver, etc.


From stephen at xemacs.org  Wed Jul 11 22:29:59 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 12 Jul 2012 05:29:59 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
Message-ID: <87vchu3ut4.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > No! Although you are making available (some/most/all) of the data
 > values, you are not making available the ability to make arbitrary
 > SQL-type queries to view the data.

AFAIK the plan is to do that via the REST interface.  I don't see why
they need to be "arbitrary SQL-type" queries; you need to be a bit
more explicit about that.

 > > It's not the same argument.  A mailing list needs a message
 > > distribution agent; it doesn't *need* a webUI.
 > 
 > In this day and age, try selling that one! Only those of us, and
 > that especially includes me, who were around "way back when",
 > before http even existed, know any other way. :)

So what?  The fact that a webUI is very convenient is not the point.
The point is that the message distribution agent is mission-critical;
if it goes down you are well-and-truly screwed.  If the web UI goes
down, it might not even be noticed for weeks.

 > I think we MUST assume that there will be a web interface.

Of course there will be a web interface.  As you point out, we
couldn't even give away the product without it.  But there might be
half a dozen different ones, too.

 > As far as the complexity of access is concerned, the mail handler
 > probably has the lowest requirements. The present architecture
 > would have to be extended significantly to provide for appropriate
 > handling of ALL of the data. That includes a much richer query
 > capability.

So what?  This extension needs to be done *somewhere*; you aren't
going to be able to avoid it by throwing it out of the core.  What you
are going to be able to do by throwing it out of the core is ensure
that each non-core module will do it differently and incompatibly.  In
fact, as you point out, they already are different and incompatible.
I don't see any reason for that to change unless the DB API is
centralized somewhere.

I see no reason for that somewhere to be anywhere but the core.  The
core is the only component that we *know* *has* to be there, and there
will be only one implementation of it.  The various UIs have
substitutes, and one reason for splitting them out was to make it
possible to have multiple implementations (and you've been at pains to
point that out).

 > Presently, the message handling is integrally tied to the database
 > implementation. Customization extensions will intrude into parts of
 > the system which they should not affect.

Why?  Parts of the system that don't need them just ignore them.
Where's the problem?

 > I view your argument as the message handler claiming "I'm special!

It is.  First, it is mission-critical; nothing else is.  Second, it
does need to ensure good performance, which I doubt is true of other
components.  Whether that justifies bypassing the REST API or
whatever, I don't know.

 > Everyone else has do do things my way. I get special privileges."

Well, you've misunderstood me, then.  My intention is that the message
handler use the same APIs available to everybody else, except that
other applications might need to use the REST interface where the core
might have more direct access.

 > -- IMHO, the tail is wagging the dog.

Call it "claim" if you like, but the message handler is the dog.

 > Let's split shared data storage from the message handler,

I don't have a problem with that as a matter of design, but for
distribution it will be bundled with the message handler.  That's not
necessarily true of other components.

Steve

From richard at NFSNet.org  Thu Jul 12 00:04:32 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Wed, 11 Jul 2012 17:04:32 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87vchu3ut4.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
	<87vchu3ut4.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <E08184B7-4B5B-4021-AE5F-9F9DF07AEDAA@NFSNet.org>

On Jul 11, 2012, at 3:29 PM, Stephen J. Turnbull wrote:

> Richard Wackerbarth writes:
> 
>> No! Although you are making available (some/most/all) of the data
>> values, you are not making available the ability to make arbitrary
>> SQL-type queries to view the data.
> 
> AFAIK the plan is to do that via the REST interface.  I don't see why
> they need to be "arbitrary SQL-type" queries; you need to be a bit
> more explicit about that.

As an example, suppose that I want to have an "intelligent" ToDo indicator.
As a minimum, I need to be able to get from the data store a list of MLs that have pending requests AND for which I am authorized to do that work.  Typically, this would be some kind of join.
Or the "social media" guys might want to add some "respected author" value and incorporate that into the way that messages get displayed. That data doesn't even exist in the "stock" distribution.
I don't pretend to know just what our users will want to add. But they should be allowed to write an SQL-type description of their needs and they shouldn't "muck" with the inner workings of the message handling schema to do so.

> The point is that the message distribution agent is mission-critical;
> if it goes down you are well-and-truly screwed.  If the web UI goes
> down, it might not even be noticed for weeks.

I don't buy that.  If you advertise a subscribe URL, or any other function,
that is just as much a "mission critical" component as any other.

I don't see user passwords providing much direct use in the mail distribution system.
They might be critical to a list that requires moderation. But, even there, I suspect that the moderators are likely to utilize the web interface.

>> I think we MUST assume that there will be a web interface.
> 
> Of course there will be a web interface.  As you point out, we
> couldn't even give away the product without it.  But there might be
> half a dozen different ones, too.
> 
>> As far as the complexity of access is concerned, the mail handler
>> probably has the lowest requirements. The present architecture
>> would have to be extended significantly to provide for appropriate
>> handling of ALL of the data. That includes a much richer query
>> capability.
> 
> So what?  This extension needs to be done *somewhere*; you aren't
> going to be able to avoid it by throwing it out of the core.

No, but I will "compartmentalize" it.

>  What you are going to be able to do by throwing it out of the core is ensure
> that each non-core module will do it differently and incompatibly.

No, I am suggesting that either you implement the functionality by specifying that some particular structure be set in a standard database (and provide a reference implementation of doing so) or that you specify REST interfaces that implement the appropriate functions and REQUIRE that all components manipulate that data ONLY through those interfaces.

The REST interface is not a single entity, but a collection of components that inter-operate. Each of them is "mission critical".

Further, "each non-core module will do it differently and incompatibly" is a red herring. There MUST be a SPECIFICATION of the interface and EVERY implementation MUST meet those REQUIREMENTS. What ever else it does will not affect any other part of the system.

>  In fact, as you point out, they already are different and incompatible.

That is because you have not followed the principles and allow "someone else" to provide that service.

> I don't see any reason for that to change unless the DB API is
> centralized somewhere.
> 
> I see no reason for that somewhere to be anywhere but the core.  The
> core is the only component that we *know* *has* to be there, and there
> will be only one implementation of it.  The various UIs have
> substitutes, and one reason for splitting them out was to make it
> possible to have multiple implementations (and you've been at pains to
> point that out).

>> Presently, the message handling is integrally tied to the database
>> implementation. Customization extensions will intrude into parts of
>> the system which they should not affect.
> 
> Why?  Parts of the system that don't need them just ignore them.
> Where's the problem?

Spaghetti. In any system of complexity, there will be parts that could care less about the details of most other parts.
design modularity practices separate those parts and provide limited access between them.

>> I view your argument as the message handler claiming "I'm special!
> 
> It is.  First, it is mission-critical; nothing else is.

And the underlying RDBMS, the MTA, etc. are not?

>  Second, it
> does need to ensure good performance, which I doubt is true of other
> components.  Whether that justifies bypassing the REST API or
> whatever, I don't know.

The question is whether particular data is local or shared. If I, as one of the data consumers, don't need access to some of the data, there is no reason to attempt to expose it via REST. Therefore, even if "maintain ..... (certain data)" is a part of the specification, how it is maintained is an internal detail.

>> Everyone else has do do things my way. I get special privileges."
> 
> Well, you've misunderstood me, then.  My intention is that the message
> handler use the same APIs available to everybody else, except that
> other applications might need to use the REST interface where the core
> might have more direct access.

This is my objection. IF some particular data is exposed, then it should be maintained by one handler, without back doors.
If that handler is local, then the interface need not serialize the data and transmit it, but the access should be isomorphic to doing so.

>> -- IMHO, the tail is wagging the dog.
> 
> Call it "claim" if you like, but the message handler is the dog.

With respect to messages, yes. But not with respect to credentials.

Credentials should be kept in a separate box. And that box should be kept where ever it best fits in the overall data flow. From a design perspective, it should be easy to place it anywhere the installer desires.

>> Let's split shared data storage from the message handler,
> 
> I don't have a problem with that as a matter of design, but for
> distribution it will be bundled with the message handler.  That's not
> necessarily true of other components.

For distribution, a reference implementation of EVERY interface should be included.

And substituting a different implementation should be a simple as excluding the distribution version and dropping in the alternate.

> Steve


From barry at list.org  Thu Jul 12 04:05:43 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:05:43 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
Message-ID: <20120711220543.747177c4@limelight.wooz.org>

Thanks for starting this discussion.  Since the thread's already long, I'm
just going to answer randomly with my own thoughts.

One thing I have a real problem with is defining the database query layer as
the interface between components.  To me that just unacceptably ties us to a
specific database, and/or a specific protocol.  For example, I do not want to
*require* Postgres in order to run Mailman, or to integrate *a* web ui with
the core.  I just think that as convenient as that might seem today, it will
lock us into a system design we're going to regret somewhere down the road.

So let's say for the moment that we agree that all the user data should live
in one place.  I don't have a problem with that conceptually, and I actually
don't care whether that's part of the core or in a separate component.  The
other problem I have is extending the core's data model to include things it
doesn't care about.  When you realize all of that has to be documented and
tested, that just seems like it's adding lots of extra baggage to the core.

For example, today you might want Twitter and Facebook ids in that database.
Five years ago maybe you also wanted an AIM id in there.  Do you today?  Will
you still want Google+ ids in there, or BrowserIDs, or OpenIDs five years from
now?  Yet, if it's part of the core's data model, we have to support it, test
it, document it, go through deprecation cycles, etc. etc.

One of the important design decisions I made was using Zope interfaces to
formally define the touch points between the different components of the
system.  This isn't just for the fun of it; instead, it gives us great
implementation flexibility.

For example, if you need to know what email addresses a user has registered,
you access that through the IUser interface.  Rosters are another great
example of where you access things through the IRoster interface and nothing
else.  Nothing except the implementation of that even cares that they are
implemented as queries and don't exist as *real* objects in the system.  They
can return whatever they want, as long as they conform to the IUser or
whatever interface.

This all might lead to inefficiencies, but I don't think that matters right
now.  It probably will some day, but let's worry about that if and when we
need to.  What we care about now is the *flexibility* and the *stability* of
the system.

For the sake of argument, let's say that all the user information should be
stored in Postorius.  What kind of changes would be needed in the core to keep
its view of the user world in sync with Postorius's view of the world.  No
matter how you slice it, you are going to have two separate processes that
need to be kept in sync.

You actually could, as I think Richard advocates, just expose the SQL queries
to both processes.  You would in theory have to only re-implement a handful of
interfaces to keep the rest of the system humming.  IOW, when the IUserManager
needs to look up a user by their email address, instead of running a query
against the local SQLite database, you would run it against the Postorius
database.  But - and here's the key thing - you would *still* return some
object that implements the IUser interface.  If you do that, you've localized
the changes you have to make to the core and everything else Just Works
(again, in theory ;).

One of the things I've tried to do, with unknown success because nobody's
tried it, is to keep in mind three broad slices of data: the user data, the
list data, and the message data.  So for example, an IMember associates an
IAddress with an IMailingList.  The standard implementation of that doesn't
use a foreign key between the `member` table and the `mailinglist` table.
Instead it uses the fqdn_listname, i.e. a string.  What that *should* mean is
that you could move the user data anywhere you want and not have to also move
the list data and message store data.

There *should* be enough hooks in the system already for a system
administrator to say "I want to use Postorius, so I must enable the Postorius
IUserManager implementation".  For global objects like this, we use the Zope
Component Architecture (ZCA), so in a Postorius-owns-the-world scenario, what
has to happen is that

    usermanager = getUtility(IUserManager)

must return the PostoriusUserManager instance and not the SQLite based
UserManager instance.  Once you've done that, you have to change *nothing*
else in the system because everything talks to that object through the
interface, and as long as that keeps its contract, the rest of the system
should, again Just Work.

I have no idea whether the above will be easy or not, since nobody's tried
it.  But the system design should allow you to do it this way, and I would be
very open to the right hooks, fixes, and extensions to make this possible.
I hope you can see how this approach lets someone run Mailman in many
different configurations, from a core-only system, to Postorius, to a system
where all the user database is in ZODB or already locked up in a proprietary
closed database.

There is another approach of course, which may end up being simpler, if more
brittle.  You could just try to keep the two databases in sync.  It doesn't
matter too much which is the master, you just have to decide.  This is
essentially how Launchpad's integration with Mailman 2 works.  Launchpad is
the master database and whenever something in that database chances that could
affect Mailman, that information is communicated to the Mailman server.  The
details are mostly unimportant, and yes, it does work.  It's been brittle in
the past, but now with enough logging, monitoring, and fail-safes it works
great.

How would you keep these two in sync?  First, if something changes in the
core, the idea is that an event is triggered.  Other components of the system
watch for those events and react to the ones they care about.  For example,
let's say a user changes their password via email command.  Once the core acts
on that change, it will trigger a PasswordChangeEvent which has a reference to
the user effecting that change.  If Postorius was the master database for
passwords, we'd have to add a little event subscriber which, when it got a
PasswordChangeEvent, then talked to Postorius to make that change.  Or maybe
it updated the shared user data component, or made the appropriate SQL UPDATE
call.  The key thing again, is that the core just fires the
PasswordChangeEvent, and other things react to it.

Alternatively, let's say a user changes their password through the web ui.  I
think this case is already covered, because the way to keep that in sync with
the core is to make the appropriate REST call, probably PATCHing the user's
password.

Very likely we don't have enough events defined to cover all the actions that
the core must take (e.g. through email commands).  But events are easy to add
and again, I'm not opposed to adding any events which make the integration
easier.

It's also likely that the REST API is incomplete for every bit of information
Postorius wants to get into the core or out of the core.  Again, it's easy to
extend the REST API, so let's fill in what's missing.

I hope this lays out the basic design constraints that I want to follow.
Maybe it sparks some thoughts about different possibilities.

Cheers,
-Barry

From barry at list.org  Thu Jul 12 04:09:54 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:09:54 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
Message-ID: <20120711220954.042db281@limelight.wooz.org>

On Jul 10, 2012, at 10:51 PM, Richard Wackerbarth wrote:

>Yes, it is a bug in Postorius. However, that does not negate the fact that
>the present design, by forcing a split database, makes it difficult to
>accomplish the desired behavior.

Why exactly is it difficult?  I'm not trolling, I really want to know.  Is it
because there's too much to keep in sync?  Or is it because the REST API calls
to read or write the necessary information is missing?  Or is it just a pain
to chase down everywhere in Postorius that things can change and need to be
pushed to Mailman?

The core has machinery to push or pull information, or for information to be
pushed to or pulled from it.  Let's use those and expand them where
necessary.  Or please help me understand why those can't possibly work, or at
least why they are not working right now.

Since this thread started with logins, walk me through what Postorius has to
do to log a user in when the password is in the core.  Where are you stuck?

-Barry

From barry at list.org  Thu Jul 12 04:14:00 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:14:00 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20120711221400.654773e3@limelight.wooz.org>

On Jul 11, 2012, at 02:12 PM, Stephen J. Turnbull wrote:

>But isn't that going to take us a long way down the road where we
>anoint Postorius the one-and-only admin interface?  If that really
>needs to be, OK, but I don't much like it.

I don't either; it's unacceptable.

Many, maybe most, sites will be fine just using the default Postorius, but
plenty of sites will not.  They will want to make mailing list administration,
or user subscriptions, fit in with their own site design.  Think of the
code-hosting example where you join teams and that automatically puts you on
that team's mailing list.  Maybe through the code-hosting's regular web-ui you
can manage your subscriptions, but maybe they make all the other decisions for
you, in order to keep the web-ui really really simple.  They don't want
Postorius, they just want the core.  We must keep that as a possibility.

Cheers,
-Barry

From barry at list.org  Thu Jul 12 04:16:42 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:16:42 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
Message-ID: <20120711221642.1a770ab8@limelight.wooz.org>

On Jul 11, 2012, at 05:39 AM, Richard Wackerbarth wrote:

>I am perfectly happy to have the user info handled by an independent
>stand-alone module which is willing to take responsibility for ALL of the
>user profile info. It should provide a REST interface that allows other
>modules the ability perform authentications, manipulate the profile, etc.
>
>Another acceptable methodology would be to store all of the data in a real
>relational database and allow each of the modules direct access to the
>database engine.

I think I've outlined how the core would be extended so that either of the
above implementation approaches could work.

Just for the record, as long as it's done in a modular way (i.e. so that
people who don't want Postorius can ignore it), I have no problem shipping
well tested components that make integrating the core with Postorius easier.
It just can't be the only way to do it.

Cheers,
-Barry

From barry at list.org  Thu Jul 12 04:21:11 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:21:11 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20120711222111.24a5e7bb@limelight.wooz.org>

On Jul 12, 2012, at 02:50 AM, Stephen J. Turnbull wrote:

>It's not the same argument.  A mailing list needs a message
>distribution agent; it doesn't *need* a webUI.

+1

Or I might rephrase that as: it doesn't need *a* webUI :).

Postorius rocks, and it is going to be a fantastic piece of the default
Mailman story.  It will in fact be the most visible piece, so it has to rock.
Think of all the thousands of average-Joe mailing lists out there.  They will
be ecstatic to see the improvement over the current circa-1996 webUI.  It's
not even funny how much better Postorius will be.

We just have to keep the design open enough that the webUI can be changed or
discarded and the mailing list system will continue to function.

-Barry


From barry at list.org  Thu Jul 12 04:27:43 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:27:43 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
Message-ID: <20120711222743.12bcebff@limelight.wooz.org>

On Jul 11, 2012, at 02:22 PM, Richard Wackerbarth wrote:

>No! Although you are making available (some/most/all) of the data values, you
>are not making available the ability to make arbitrary SQL-type queries to
>view the data.

Which frankly, I don't think we should do, in the core.  The core is not a
generalized database engine.

If this is absolutely required for the design of Postorius, then in my mind
there has to be a separate database component, and an implementation layer on
the bottom of the core which talks to that component.

>In this day and age, try selling that one! Only those of us, and that
>especially includes me, who were around "way back when", before http even
>existed, know any other way. :)

Don't forget too that there are use cases where you interact with the system
solely by email commands.  Sure it might feel antiquated, but people who want
to run their systems this way are passionate about it.  This is something else
that we must keep possible.

>Presently, the message handling is integrally tied to the database
>implementation. Customization extensions will intrude into parts of the
>system which they should not affect.

Then *that's* a bug, but I think it's a bug of lack-of-implementation rather
than design, as I've described in previous messages.

>As far as I am concerned, those are more than adequate reasons.
>
>I view your argument as the message handler claiming "I'm special! Everyone
>else has do do things my way. I get special privileges."  -- IMHO, the tail
>is wagging the dog.

Well, I don't think that's the case, but of course message handling is kind of
the whole point, isn't it?  I mean, without that, what are we building? :)

-Barry

From alexander at sulfrian.net  Thu Jul 12 04:42:28 2012
From: alexander at sulfrian.net (Alexander Sulfrian)
Date: Thu, 12 Jul 2012 04:42:28 +0200
Subject: [Mailman-Developers] URGENT: Google Summer of Code status
	report and code due
In-Reply-To: <87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <87r4shitt7.wl%alexander@sulfrian.net>

aAt Thu, 05 Jul 2012 09:24:33 +0900,
Stephen J. Turnbull wrote:
> [...]
> Newsgroup names are an issue here.  It seems to me that (if not
> gateway'd to Usenet) they should be something like (pseudo-code)
> 
>     "mailman." + join(reverse(split(list-id,".")),".")
> 
> Eg, this list would be "mailman.org.python.mailman-developers".  I
> know that's considered ugly-out-the-wazoo, but these need to be UUIDs
> (consider mirrors), and mailman at python.org should not be in the same
> subtree as mailman at python.net (ie,
> 
>     mailman.net.python.mailman    vs  mailman.org.python.mailman
> 
> not
> 
>     mailman.python.net.mailman    vs  mailman.python.org.mailman
> 
> The top-level maybe shouldn't be "mailman", but rather something like
> "list-archive".

Currently I am thinking about the naming scheme. I think that we
should not split and reverse the list. So currently I implemented
something like that (pseudo-code):

    join(reverse(split(mail_host, '.')), '.') + ',' + list_name


If the list_name would be also reversed, it could lead to some
surprising subtree clashing. For example web2.0 would be in the same
subtree like something1.0 (people sometimes use strange list
names...). Even with the current implementation the group names are
ugly. Maybe we should eliminate the dots from the list names by
default and only allow separate groups with the alias mechanism?


Alex
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120712/7a65b08d/attachment.pgp>

From barry at list.org  Thu Jul 12 04:45:53 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:45:53 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <E08184B7-4B5B-4021-AE5F-9F9DF07AEDAA@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
	<87vchu3ut4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E08184B7-4B5B-4021-AE5F-9F9DF07AEDAA@NFSNet.org>
Message-ID: <20120711224553.740ba6ce@limelight.wooz.org>

On Jul 11, 2012, at 05:04 PM, Richard Wackerbarth wrote:

>As an example, suppose that I want to have an "intelligent" ToDo indicator.
>As a minimum, I need to be able to get from the data store a list of MLs that
>have pending requests AND for which I am authorized to do that work.
>Typically, this would be some kind of join.

This should be hidden behind a REST API.

>Or the "social media" guys might want to add some "respected author" value
>and incorporate that into the way that messages get displayed. That data
>doesn't even exist in the "stock" distribution.

Nor probably, should it.

>I don't pretend to know just what our users will want to add. But they should
>be allowed to write an SQL-type description of their needs and they shouldn't
>"muck" with the inner workings of the message handling schema to do so.

Then, for sites that want that feature, the data should live outside the
core.  As I hope I've explained, that could mean a separate database component
with an alternative IUser (et al) implementation to make the external
queries.  Or it could mean a REST API to push relevant changes in the separate
database component back into the core.

>> The point is that the message distribution agent is mission-critical;
>> if it goes down you are well-and-truly screwed.  If the web UI goes
>> down, it might not even be noticed for weeks.
>
>I don't buy that.  If you advertise a subscribe URL, or any other function,
>that is just as much a "mission critical" component as any other.

But maybe you don't have such a subscribe URL.  I've described several use
cases where such a thing isn't needed.  It must be possible to support both
situations.

>I don't see user passwords providing much direct use in the mail distribution
>system.  They might be critical to a list that requires moderation. But, even
>there, I suspect that the moderators are likely to utilize the web interface.

Users do make changes to their subscriptions, or moderate their mailing lists
through email commands today.  Of course, even that is optional.

>No, I am suggesting that either you implement the functionality by specifying
>that some particular structure be set in a standard database (and provide a
>reference implementation of doing so) or that you specify REST interfaces
>that implement the appropriate functions and REQUIRE that all components
>manipulate that data ONLY through those interfaces.
>
>The REST interface is not a single entity, but a collection of components
>that inter-operate. Each of them is "mission critical".

Agreed.  But the core's REST API should *only* reflect the data model that is
critical to its operations.

In fact, the core already calls out to a REST API to retrieve some
information.  Did you know that the header and footer added to messages is a
resource described by a URL?  Mailman does a GET on that URL with a defined
format to retrieve the appropriate decoration, which it then caches for a
little while.  That's the core making a REST call for data living in an
external system[*].

>Further, "each non-core module will do it differently and incompatibly" is a
>red herring. There MUST be a SPECIFICATION of the interface and EVERY
>implementation MUST meet those REQUIREMENTS. What ever else it does will not
>affect any other part of the system.

Right.  But I would strongly object to that specification in any way being
tied to SQL or a specific database implementation.  HTTP is the language of
the web and REST is a not-perfect but good enough convention built on HTTP.

Cheers,
-Barry

[*] Possibly.  The data may live on an internal-to-the-core resource, which
are defined as mailman:// schemes.  The core doesn't care where the resource
lives, it just does a GET which by definition is REST.

From barry at list.org  Thu Jul 12 04:48:51 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:48:51 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <874npe5gry.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<4FFD96B5.8050107@state-of-mind.de>
	<874npe5gry.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20120711224851.454eea44@limelight.wooz.org>

On Jul 12, 2012, at 02:50 AM, Stephen J. Turnbull wrote:

>Nevertheless, it would be preferable if that is wrapped in an API that
>makes it look like it's all coming form one place, and which manages
>the linkages so that data is not store redundantly in different
>places, or inaccessible to certain clients.

Then it has to be outside the core, and the core must be extended to access
that external information, using one of the many mechanisms I've outlined.

-Barry

From barry at list.org  Thu Jul 12 04:53:55 2012
From: barry at list.org (Barry Warsaw)
Date: Wed, 11 Jul 2012 22:53:55 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
Message-ID: <20120711225355.15988470@limelight.wooz.org>

On Jul 11, 2012, at 09:27 AM, Richard Wackerbarth wrote:

>First, we should define all of the data storage in terms of the REST access
>points to the data. (And not, as presently done, the other way around) Next,
>we should access all of the REST interface URLs indirectly so that
>functionality can be moved around simply by changing a single reference
>definition.  This is the kind of scheme that django uses with its {% url %}
>tag. (like DNS vs IP addresses) Finally, we should "black box encapsulate"
>the access to the data, requiring that EVERY module utilize this common
>interface, and only this interface, to the data.

If there were a separate user database that exposed all sorts of data via
REST, including stuff the core doesn't care about, then all the things I
described before about re-implementing various Zope interfaces in the core
holds true.  They would just do REST calls instead of database queries.

Both implementations in fact can live side-by-side, and choosing one or the
other is something you do when you install and configure the system.

-Barry

From richard at NFSNet.org  Thu Jul 12 07:25:34 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Thu, 12 Jul 2012 00:25:34 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <20120711220543.747177c4@limelight.wooz.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<20120711220543.747177c4@limelight.wooz.org>
Message-ID: <51A64575-9148-452F-A8C6-697E6291A816@NFSNet.org>

On Jul 11, 2012, at 9:05 PM, Barry Warsaw wrote:

> Thanks for starting this discussion.  Since the thread's already long, I'm
> just going to answer randomly with my own thoughts.

And thanks for your reply. I just spotted something in it that makes me feel that rather than fundamental disagreement, we may simply be saying things differently.  I'll dig through all of the rest of your remarks in detail at an appropriate interval.

> One thing I have a real problem with is defining the database query layer as the interface between components.

Note that I expressed the preference that the interface be defined in terms of the REST access.

As I say in another post, it is not the exact expression of the interface that is important, but that all of the descriptions are isomorphic and that no component accesses shared data by "private" methods.

> There *should* be enough hooks in the system already for a system

> administrator to say "I want to use Postorius, so I must enable the Postorius
> IUserManager implementation".

And, by saying this, you  imply that the storage of passwords is no longer the responsibility of "core".

You have delegated it to IUserManager. And the default Postorius compatible IUserManager could store those passwords where ever it wishes. In particular, it can store them in the Postorius User database, along with all of the other User profile information that the site wants to keep.
> because everything talks to that object through the interface, and as long as that keeps its contract, the rest of the system should, again Just Work.

> I have no idea whether the above will be easy or not, since nobody's tried
> it.  But the system design should allow you to do it this way, and I would be
> very open to the right hooks, fixes, and extensions to make this possible.
> I hope you can see how this approach lets someone run Mailman in many
> different configurations, from a core-only system, to Postorius, to a system
> where all the user database is in ZODB or already locked up in a proprietary
> closed database.
> 
> There is another approach of course, which may end up being simpler, if more
> brittle.  You could just try to keep the two databases in sync.  It doesn't
> matter too much which is the master, you just have to decide.

Yes, but from my experience, only with great care. As you note that arrangement is brittle. I would prefer to avoid it.

> Once the core acts on that change, it will trigger a PasswordChangeEvent which has a reference to
> the user effecting that change.  If Postorius was the master database for
> passwords, we'd have to add a little event subscriber which, when it got a
> PasswordChangeEvent, then talked to Postorius to make that change.  Or maybe
> it updated the shared user data component, or made the appropriate SQL UPDATE
> call.  The key thing again, is that the core just fires the
> PasswordChangeEvent, and other things react to it.
> 
> Alternatively, let's say a user changes their password through the web ui.  I
> think this case is already covered, because the way to keep that in sync with
> the core is to make the appropriate REST call, probably PATCHing the user's
> password.

I think that this is the wrong approach to the extent that "core", and here, by "core" I think you mean the "admin-by-mail" component, does anything other than call the same REST interface that the web UI would use.

The implementation of that interface item could reside anywhere in the system and it would manipulate the database through the IUserManager.

- - - -
> So for example, an IMember associates an
> IAddress with an IMailingList.  The standard implementation of that doesn't
> use a foreign key between the `member` table and the `mailinglist` table.
> Instead it uses the fqdn_listname, i.e. a string.

I don't agree. Your string which contains the fqdn_listname IS a foreign key to the mailinglist table. It may not be the primary key in some installations, but it is one-to-one with that key and could be used directly as the primary key.
- - - -

From stephen at xemacs.org  Thu Jul 12 09:05:52 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 12 Jul 2012 16:05:52 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <E08184B7-4B5B-4021-AE5F-9F9DF07AEDAA@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
	<87vchu3ut4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E08184B7-4B5B-4021-AE5F-9F9DF07AEDAA@NFSNet.org>
Message-ID: <87txxd4fxr.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > As an example, suppose that I want to have an "intelligent" ToDo
 > indicator.  As a minimum, I need to be able to get from the data
 > store a list of MLs that have pending requests AND for which I am
 > authorized to do that work.  Typically, this would be some kind of
 > join.

OK.  But in my head, Python is a dynamic language, and we should be
able to use the ORM to dynamically revise the DB schema, and access
such complexly specified data.

 > I don't pretend to know just what our users will want to add. But
 > they should be allowed to write an SQL-type description of their
 > needs and they shouldn't "muck" with the inner workings of the
 > message handling schema to do so.

So by "SQL-type" do you mean they *must* have access to the RDBMS so
they can actually write SQL, or just that the provided interface needs
to allow queries with logical operators?  The "-type" suggests you
mean the latter.

But you've also suggested the former.  I don't like the idea of direct
access to the underlying database, because there isn't necessarily
going to be just one, and it may be that Mailman needs certain kinds
of access to component DBs (eg, updating email addresses) but the
organization would like to have access controls on them based on
another component database (authorized admins, say).  Also, we're not
in a position to require that all databases be kept in, say,
Postgresql.  They may not even be RDBMSes (LDAP member databases,
sendmail alias files).  So we need a layer of abstraction.

 > > The point is that the message distribution agent is
 > > mission-critical; if it goes down you are well-and-truly screwed.
 > > If the web UI goes down, it might not even be noticed for weeks.
 > 
 > I don't buy that.  If you advertise a subscribe URL, or any other
 > function, that is just as much a "mission critical" component as
 > any other.

We'll have to agree to disagree.

 > I don't see user passwords providing much direct use in the mail
 > distribution system.

I don't understand what you're thinking.  You started this thread with
the observation that various components are keeping data in different
places, and that this data is often redundant but not synced or
inaccessible.  To me this suggests a design principle: a single
conceptual database managed by a core component (i.e., one that is
present in every Mailman 3 system).

The implementation of that database may very well include multiple
database systems (eg, the organization's LDAP directory, a Postgresql
database for the tables related to list configurations, and an MTA
alias file for the list addresses).  However, these need to be managed
via a single common API, and the data must not be private to any
non-core component.

The fact that some data are not useful to all components seems to me
to be a red herring.  The point of a DBMS in general is that you can
flexibly access only the data you need for the job at hand, in a form
optimized for the job at hand.

 > > So what?  This extension needs to be done *somewhere*; you aren't
 > > going to be able to avoid it by throwing it out of the core.
 > 
 > No, but I will "compartmentalize" it.

You mean "as a single entity in the distribution of core components",
or "as per-component entities containing what each component needs"?

 > No, I am suggesting that either you implement the functionality by
 > specifying that some particular structure be set in a standard
 > database (and provide a reference implementation of doing so) or

I think that's a non-starter.  We are not in a position to specify
that there even *be* a standard database backing our API, unless we're
willing to push the redundancy/inaccessibility problems to the next
higher level by copying databases from organizational sources
*outside* of Mailman *into* Mailman-only databases.  I consider that
unacceptable; use of external databases for subscriber lists is a
high-frequency RFE, and it would be *way* higher if it weren't for the
extremely high quality of MM-U participants, most of whom check the
FAQ/tracker and notice that there already is an RFE on file.  AIUI,
Barry does too.

 > that you specify REST interfaces that implement the appropriate
 > functions and REQUIRE that all components manipulate that data ONLY
 > through those interfaces.
 > 
 > The REST interface is not a single entity, but a collection of
 > components that inter-operate.

This makes no sense to me.  I see the architecture as

          +--------------+             +-------+
          |   Message    |             |       |
          | Distribution |  . . . . .  | WebUI |
          +--------------+             +-------+
                 \            |            /
                  \           |           /
                   \          |          /
                  +-----------------------+
                  |       REST API        |
                  +-----------------------+
                    /???? |         \
                   /          |          \
                  /           |           \
          +------------+             +------------+
          | Subscriber |  . . . . .  |   Social   |
          |    List    |             | Networking |
          |            |             |    Data    |
          +------------+             +------------+

where the "MD" component may perceive a member in terms of only
subscriber data (i.e., something on the order of (FullName, Email,
BounceCount)), while the "WUI" component might be interested in
something like (Avatar, FullName, Email, IsATroll).  (Of course the
lower ellipsis also include a site config DB and a list config DB.)

To my mind a Pythonic base REST API would return MemberObjects with
appropriate properties, and the properties would be turned into DB
queries on access.

For performance-critical cases there would be a separate .query()
method on MemberObjects that would look up a vector of attributes in
one DB query.  Also a .select() method on the MailmanDB object which
would return a list of MemberObjects with specified properties,
optionally as a (MemberObject, *values_of_requested_properties) tuple
or dict.

 > Further, "each non-core module will do it differently and
 > incompatibly" is a red herring. There MUST be a SPECIFICATION of
 > the interface and EVERY implementation MUST meet those
 > REQUIREMENTS. What ever else it does will not affect any other part
 > of the system.

Have you ever told a baby to stop sucking their thumb, and use the
pacifier?  You have to pull the thumb out to get the point across.  In
the same way, there's going to have to be one implementation, and that
implementation will be distributed with the core.  Otherwise there
WILL be a SPECIFICATION of the interface and EVERY implementation WILL
meet those REQUIREMENTS (except where the implementer finds it
inconvenient), and we're back where we started.

 > That is because you have not followed the principles and allow
 > "someone else" to provide that service.

True.  (I wish you'd stop using "you" in this kind of statement; it
isn't true, I didn't code any of this.  And it doesn't matter who
did.)  Announcing principles isn't going to help enough, though.
Python operates on the basis of "consenting adults" and can't force
anybody write a program in a particular way.  Unless the API is
actually provided in *every* Mailman 3 distribution, and is well-
enough designed to be TOOWTDI, implementers will work around it.

 > >> I view your argument as the message handler claiming "I'm special!
 > > 
 > > It is.  First, it is mission-critical; nothing else is.
 > 
 > And the underlying RDBMS, the MTA, etc. are not?

This confounds levels of architecture.

 > This is my objection. IF some particular data is exposed, then it
 > should be maintained by one handler, without back doors.  If that
 > handler is local, then the interface need not serialize the data
 > and transmit it, but the access should be isomorphic to doing so.

That's not an objection, that's a somewhat more precise restatement of
what I wrote.

 > Credentials should be kept in a separate box. And that box should
 > be kept where ever it best fits in the overall data flow.

Precisely.  Since databases will be needed by all components when
present, they should be kept in or with a component that will always
be present.  That's what "core" means.

 > From a design perspective, it should be easy to place it anywhere
 > the installer desires.

No.  That exposes an implementation detail.  As far as installers are
concerned, the database *is* the API.  Where it is located is none of
their business.

There will need to be a little leakage here, because admins will want
to link the Mailman DB to existing organizational DBs.  So the
possibility of specifying an existing external database needs to be
considered.  But this is only slightly more than the amount of
information required to configure Mailman's own PostgreSQL or MySQL
database, and these are not going to be "placed" by a Mailman admin,
but rather configured and accessed from a provided installation
(whether by the user organization, or by an OS distro).  So I don't
see a need to make a big distinction here, except that the "own"
database will have a schema designed for Mailman, but an external
database will need some kind of "adapter" to match schema.

 > For distribution, a reference implementation of EVERY interface
 > should be included.

I don't see how that's possible in your design, since you propose to
allow components to implement their own databases.

 > And substituting a different implementation should be a simple as
 > excluding the distribution version and dropping in the alternate.

Sure, but is there a reason why this might be difficult?  ISTM that
Python's orientation to duck-typing will make this happen naturally.
(I don't mean to ignore the possibility of problems, but if you have
something specific in mind we can be careful to avoid that in the
design process.)

From stephen at xemacs.org  Thu Jul 12 10:44:37 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 12 Jul 2012 17:44:37 +0900
Subject: [Mailman-Developers] URGENT: Google Summer of Code
	status	report and code due
In-Reply-To: <87r4shitt7.wl%alexander@sulfrian.net>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87r4shitt7.wl%alexander@sulfrian.net>
Message-ID: <87sjcx4bd6.fsf@uwakimon.sk.tsukuba.ac.jp>

Alexander Sulfrian writes:

 > If the list_name would be also reversed, it could lead to some
 > surprising subtree clashing. For example web2.0 would be in the same
 > subtree like something1.0 (people sometimes use strange list
 > names...).

I agree that list_name should *not* be reversed; it is an atom.

This "atomicity" is a problem.  We have three different namespaces and
syntaxes to deal with here: RFC 5322 email addresses, RFC 2919
List-Ids, and RFC 5536.  In RFC 5322, there's a special class, the
"dotted-atom", which may be used in the mailbox component of an
address (and thus denotes an atomic resource).  But not in RFC 5536,
where dots aren't allowed in newsgroup name components.  I think this
is a problem for post-GSoC, though.

 > Even with the current implementation the group names are
 > ugly.

I would expect that MUA presentations will deal with this.  For
example, exploiting the hierarchy, the dots could appear as
breadcrumbs:

    mailman > org > python > mailman-developers

    MAILMAN-DEVELOPERS

    [summary lines]

    [current message header info such as author, subject, date]

    [current message body]

 > Maybe we should eliminate the dots from the list names by default
 > and only allow separate groups with the alias mechanism?

Quite possibly, but don't worry about it for the purposes of GSoC I
think.  The worst that would happen is that a few, relatively unusual
lists would be inaccessible.  But I think dealing with this requires
some thought, so let's not get committed to a hasty design.  Document
that dotted names may show strange behavior (including being
inaccessible), and move on for now.



From alexander at sulfrian.net  Thu Jul 12 12:39:14 2012
From: alexander at sulfrian.net (Alexander Sulfrian)
Date: Thu, 12 Jul 2012 12:39:14 +0200
Subject: [Mailman-Developers] URGENT: Google Summer of Code
	status	report and code due
In-Reply-To: <87sjcx4bd6.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <4FCD219A.4050203@zone12.com>
	<87aa0g690p.wl%alexander@sulfrian.net>
	<87pq8m91ms.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87ipec8fvr.wl%alexander@sulfrian.net>
	<20120704110306.58c241ad@limelight.wooz.org>
	<87k3yj6on2.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87r4shitt7.wl%alexander@sulfrian.net>
	<87sjcx4bd6.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <87obnli7ql.wl%alexander@sulfrian.net>

At Thu, 12 Jul 2012 17:44:37 +0900,
Stephen J. Turnbull wrote:
> Alexander Sulfrian writes:
> 
>  > If the list_name would be also reversed, it could lead to some
>  > surprising subtree clashing. For example web2.0 would be in the same
>  > subtree like something1.0 (people sometimes use strange list
>  > names...).
> 
> I agree that list_name should *not* be reversed; it is an atom.
> 
> This "atomicity" is a problem.  We have three different namespaces and
> syntaxes to deal with here: RFC 5322 email addresses, RFC 2919
> List-Ids, and RFC 5536.  In RFC 5322, there's a special class, the
> "dotted-atom", which may be used in the mailbox component of an
> address (and thus denotes an atomic resource).  But not in RFC 5536,
> where dots aren't allowed in newsgroup name components.  I think this
> is a problem for post-GSoC, though.
> 
>  > Even with the current implementation the group names are
>  > ugly.
> 
> I would expect that MUA presentations will deal with this.  For
> example, exploiting the hierarchy, the dots could appear as
> breadcrumbs:
> 
>     mailman > org > python > mailman-developers
> 
>     MAILMAN-DEVELOPERS
> 
>     [summary lines]
> 
>     [current message header info such as author, subject, date]
> 
>     [current message body]

Yeah, it is currently working this way. The ugly names, I refered to,
occur for example with "web2.0 at example.com":

      com > example > web2 > 0

Thunderbird is even more ugly. It shorten the name in the overview to
display only the first letter of each subtree:

        c.e.w.0

But as you said, I will leave it for now in this state and keep in
mind, that we should find a better solution in future.
 
>  > Maybe we should eliminate the dots from the list names by default
>  > and only allow separate groups with the alias mechanism?
> 
> Quite possibly, but don't worry about it for the purposes of GSoC I
> think.  The worst that would happen is that a few, relatively unusual
> lists would be inaccessible.  But I think dealing with this requires
> some thought, so let's not get committed to a hasty design.  Document
> that dotted names may show strange behavior (including being
> inaccessible), and move on for now.

Despite of having a unusual name all lists should be accessible. The
current implementation should not lead to inaccessible groups. So I
think it is acceptable for now.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120712/203dbc00/attachment.pgp>

From barry at list.org  Thu Jul 12 17:07:51 2012
From: barry at list.org (Barry Warsaw)
Date: Thu, 12 Jul 2012 11:07:51 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <51A64575-9148-452F-A8C6-697E6291A816@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<20120711220543.747177c4@limelight.wooz.org>
	<51A64575-9148-452F-A8C6-697E6291A816@NFSNet.org>
Message-ID: <20120712110751.6a82f84e@limelight.wooz.org>

On Jul 12, 2012, at 12:25 AM, Richard Wackerbarth wrote:

>On Jul 11, 2012, at 9:05 PM, Barry Warsaw wrote:

>> There *should* be enough hooks in the system already for a system
>
>> administrator to say "I want to use Postorius, so I must enable the
>> Postorius IUserManager implementation".
>
>And, by saying this, you imply that the storage of passwords is no longer the
>responsibility of "core".
>
>You have delegated it to IUserManager. And the default Postorius compatible
>IUserManager could store those passwords where ever it wishes. In particular,
>it can store them in the Postorius User database, along with all of the other
>User profile information that the site wants to keep.  > because everything
>talks to that object through the interface, and as long as that keeps its
>contract, the rest of the system should, again Just Work.

Note that when I say IUserManager above I'm talking specifically about the
zope.interface that is used in the core.  You're talking about a conceptual
independent component called a "user manager".  The former is a coding
specification that contains only the things the core is interested in, it
wouldn't be used as an inter-system definition.  The latter would be specified
by its REST API (let's say).  Then the core would have an implementation of
IUserManager which maps its data and functionality to REST calls against this
external component.

>> I have no idea whether the above will be easy or not, since nobody's tried
>> it.  But the system design should allow you to do it this way, and I would
>> be very open to the right hooks, fixes, and extensions to make this
>> possible.  I hope you can see how this approach lets someone run Mailman in
>> many different configurations, from a core-only system, to Postorius, to a
>> system where all the user database is in ZODB or already locked up in a
>> proprietary closed database.
>> 
>> There is another approach of course, which may end up being simpler, if more
>> brittle.  You could just try to keep the two databases in sync.  It doesn't
>> matter too much which is the master, you just have to decide.
>
>Yes, but from my experience, only with great care. As you note that
>arrangement is brittle. I would prefer to avoid it.

It definitely means data kept in two places, and you're right that great care
has to be taken.  What if the core is down for maintenance when a user changes
her password through Postorius?  These are the kinds of things that the
Launchpad integration work addressed, but that was in a highly controlled
environment, with lots of operational assurances.  In our case, we can't just
hand-wave that brittleness away.

>I think that this is the wrong approach to the extent that "core", and here,
>by "core" I think you mean the "admin-by-mail" component, does anything other
>than call the same REST interface that the web UI would use.

It could, though internally, that would be hidden behind the implementation of
IUserManager, IUser, and associated objects.  We're saying the same thing but
in a different way.  The advantage of hiding this behind the zope.interfaces
is that we can have multiple alternative implementations of those interfaces,
so that whether it's REST or a SQL call doesn't matter to most of the core.

>> So for example, an IMember associates an
>> IAddress with an IMailingList.  The standard implementation of that doesn't
>> use a foreign key between the `member` table and the `mailinglist` table.
>> Instead it uses the fqdn_listname, i.e. a string.
>
>I don't agree. Your string which contains the fqdn_listname IS a foreign key
>to the mailinglist table. It may not be the primary key in some
>installations, but it is one-to-one with that key and could be used directly
>as the primary key.

The point is that it's not a foreign key constraint in the members table.
Doing that would, I think, prevent you from putting the mailinglist table in
one database and the member table in another.  It makes the associated queries
less efficient, but provides for a useful measure of flexibility.

Cheers,
-Barry

P.S. FWIW, I think this separate "user database" component with REST calls in
the IUserManager (et al) implementation would not actually be difficult to
mock up.  I don't have the time to do the coding, but would be happy to
discuss details if someone wanted to take a crack at it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120712/5d95bb2a/attachment.pgp>

From franck at peachymango.org  Wed Jul 11 06:45:28 2012
From: franck at peachymango.org (Franck Martin)
Date: Tue, 10 Jul 2012 23:45:28 -0500 (CDT)
Subject: [Mailman-Developers] mailing list to work with ADSP and DMARC
In-Reply-To: <1279196768.90372.1341981179483.JavaMail.root@peachymango.org>
Message-ID: <1020347342.90522.1341981928849.JavaMail.root@peachymango.org>

I did a branch on the 2.1 series to test rewriting the From: header to be able to make emails to comply with DMARC and ADSP 

You can find the branch here: 

https://code.launchpad.net/~mlm-author/mailman/2.1-author 

and the diff with mailman main branch here: 
http://bazaar.launchpad.net/~mlm-author/mailman/2.1-author/revision/1340?start_revid=1340&remember=1338&compare_revid=1338 

I have also a test mailing list running at: 
http://lists.peachymango.org/mailman/listinfo/mlm-auth 

The best to see how it works is to subscribe, send an email, and see how the email is distributed. I guess you need to receive an email to see that... I'll reply... 

Several ways to solve this problem were explored with mailman: 
http://wiki.list.org/display/DEV/DKIM 

I took a 7th option which is to rewrite the From: without using the + and put the information about the sender in the Reply-To: 

This may not be optimum but it works. 

Overall this solution is one out of 3 ways to get mailing lists running with DMARC (see DMARC FAQ) 
1) don't break DKIM 
2) use AOR 
3) rewrite the from 

There is another one which is to whitelist emails coming from mailing lists.... 

Also note that while I'm involved in the DMARC group, I do this code as a personal project. 

Now send comments, and be nice with me :P 


From richard at NFSNet.org  Thu Jul 12 18:39:58 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Thu, 12 Jul 2012 11:39:58 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <20120712110751.6a82f84e@limelight.wooz.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<20120711220543.747177c4@limelight.wooz.org>
	<51A64575-9148-452F-A8C6-697E6291A816@NFSNet.org>
	<20120712110751.6a82f84e@limelight.wooz.org>
Message-ID: <03E21F46-A297-478F-941E-C8D701881735@NFSNet.org>

In many respects I think that we are in agreement.

If I understand you, we both envision a distributed database whereby any of the components might actually provide the data storage for some specific class of information. For example, without specifying where in the system it will reside, there is a "UserManager" that maintains information about the users. It would also be responsible for registering ALL user instances and providing lists of users that meet certain constraints. In addition, it provides an access mechanism for properties associated with particular users.

However, I think that we differ in terms of the public interface specifications.

The specification that you suggest produces a particular python object.

The specification that I suggest contains only those characteristics which are exposed on the REST interface.
It does not specify the exact format in which those characteristics are presented, but only what characteristics are available and what operations can be performed. In this form, no component external to the UserManager is permitted to rely on any aspect of the representation which falls outside of the available public characteristics.

The place where I am unclear of your intentions concerns this python object. In theory, it may have characteristics that are not exposed to the REST interface. Do you permit your functional modules, (the message handler or the administration interface, for example) to rely on any characteristics that are not properly exposed by the REST interface. If so, why?

On Jul 12, 2012, at 10:07 AM, Barry Warsaw wrote:
>> I think that this is the wrong approach to the extent that "core", and here,
>> by "core" I think you mean the "admin-by-mail" component, does anything other
>> than call the same REST interface that the web UI would use.
> 
> It could, though internally, that would be hidden behind the implementation of
> IUserManager, IUser, and associated objects. We're saying the same thing but
> in a different way.  The advantage of hiding this behind the zope.interfaces
> is that we can have multiple alternative implementations of those interfaces,
> so that whether it's REST or a SQL call doesn't matter to most of the core.

>>> So for example, an IMember associates an
>>> IAddress with an IMailingList.  The standard implementation of that doesn't
>>> use a foreign key between the `member` table and the `mailinglist` table.
>>> Instead it uses the fqdn_listname, i.e. a string.
>> 
>> I don't agree. Your string which contains the fqdn_listname IS a foreign key
>> to the mailinglist table. It may not be the primary key in some
>> installations, but it is one-to-one with that key and could be used directly
>> as the primary key.
> 
> The point is that it's not a foreign key constraint in the members table.

OK, I think we are in agreement WRT this concept. You seem to have taken "foreign key" to imply a specific implementation of the concept to which I was referring, namely some key which designates a unique instance in the referenced table.

> Doing that would, I think, prevent you from putting the mailinglist table in
> one database and the member table in another.  It makes the associated queries
> less efficient, but provides for a useful measure of flexibility.

Were this not Python, but C++, I would suggest that we consider offering multiple signatures for the exposed functionality. For example, it the function were to deliver a list of MLs to which an Address is subscribed, the address might be provided as a string (the email address in canonical form), the pk of the entry in the underlying Address database, or as an object of the Address class.

Although the same could be done in python through introspection, or similar techniques, I don't know to what extent doing so would introduce inefficiencies.

> P.S. FWIW, I think this separate "user database" component with REST calls in
> the IUserManager (et al) implementation would not actually be difficult to
> mock up.  I don't have the time to do the coding, but would be happy to
> discuss details if someone wanted to take a crack at it.

I definitely will take a crack at it since I suspect that doing so would allow Postorius to have all of its user information handled in one place and still provide the "admin by mail" to work with its restricted expectations.


From richard at NFSNet.org  Thu Jul 12 18:51:12 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Thu, 12 Jul 2012 11:51:12 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87txxd4fxr.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
	<87vchu3ut4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E08184B7-4B5B-4021-AE5F-9F9DF07AEDAA@NFSNet.org>
	<87txxd4fxr.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <083FECEB-7062-4725-9768-0C66A6D6D912@NFSNet.org>

... taken out of sequence ...
>> That is because you have not followed the principles and allow
>> "someone else" to provide that service.
> 
> True.  (I wish you'd stop using "you" in this kind of statement; it
> isn't true, I didn't code any of this.  And it doesn't matter who

I apologize for my terminology. Rather than using the second person, I should use the third. However,
no where do I mean "you" to reflect on you personally. I tend to use personal pronouns as a short form representing "the argument/proposal/design that, in the context of the message, is being described by a particular person", as distinguished from one which another person has described/advocated. Similarly, if I were to use "Barry" in that context, I would be referring to an idea that he has described. I do not know, necessarily, if any methodology is the preferred approach of the individual describing it.

There seems to be two fundamental design strategies being discussed.
One of them has a monolithic data store and the other has a distributed store.
Barry has expressed some reservations about overloading a monolithic data store with data extraneous to the fundamental mission of message handling.

I have expressed concern in requiring any implementation to maintain related data in a split format.
I recognize that there will be cases where this is necessary (for example the Launchpad case as described by Barry in another message). But, as he notes, such implementations tend to be "brittle". Especially where there are multiple components which can alter the data.  But, unless it is a constraint external to MM, I do not believe that such a restriction should be introduced.

There is also an issue of what the term "core" means. Perhaps you have been referring to a distribution package. I have been referring to one component of such a package, in particular that component which interacts with the MTAs and redistributes messages. I consider the processing of administrative messages to be a separate component. And I consider the storage of configuration information to be yet another component. In my view, each component extends only as far as its parts interact with the same private data representation.

On Jul 12, 2012, at 2:05 AM, Stephen J. Turnbull wrote:

> Richard Wackerbarth writes:
>> I don't pretend to know just what our users will want to add. But
>> they should be allowed to write an SQL-type description of their
>> needs and they shouldn't "muck" with the inner workings of the
>> message handling schema to do so.
> 
> So by "SQL-type" do you mean they *must* have access to the RDBMS so
> they can actually write SQL, or just that the provided interface needs
> to allow queries with logical operators?  The "-type" suggests you
> mean the latter.

I do mean the latter. But, if the real underlying database is a RDBMS, then, within the "black box", these queries probably should be implemented by translating them to real SQL queries and passing those to the RDBMS.
There has been a lot of work, by many far more qualified that we, to handle such details within the RDBMS. We should not attempt to reinvent their wheel.

> But you've also suggested the former.

I have suggested it as an alternative implementation. I do so only because that strategy exposes a powerful resource and avoids the burden of adding a new mechanism in order to meet the requirement for customized extensions and access that need to interact efficiently with the data that MM needs to have maintained.

>  I don't like the idea of direct
> access to the underlying database, because there isn't necessarily
> going to be just one, and it may be that Mailman needs certain kinds
> of access to component DBs (eg, updating email addresses) but the
> organization would like to have access controls on them based on
> another component database (authorized admins, say).  Also, we're not
> in a position to require that all databases be kept in, say,
> Postgresql.  They may not even be RDBMSes (LDAP member databases,
> sendmail alias files).  So we need a layer of abstraction.

I agree that we need the abstraction.

>> I don't see user passwords providing much direct use in the mail
>> distribution system.
> 
> I don't understand what you're thinking.

First, we seem to have a different conceptual model of MM.
I view that which is being called "core", not as a single entity, but a collection of components, most of which are critical to the operation of the system.
Among those components, I distinguish message routing and distribution, configuration storage, and processing of administrative messages.

The first is what I have been calling the message handling. It interacts with the MTAs, maintains queues of partially completed work, etc.

The second is critical in that it provides the customization information which causes each mailing list to be distinctive.
It can be further subdivided into structural configuration (the location of various interfaces, the parameters defining the lists, etc.), rosters of subscribers, and subscription preferences.

The third component implements the processing of messages which are designed to alter the state of the configuration storage and/or the state of messages queued in the message handler. I do not consider this element "critical" in that the messages which it will process can be queued and handled later, or the component can be omitted entirely in a system that utilizes a webUI as an alternative access to handle those administrative functions.


> You started this thread with
> the observation that various components are keeping data in different
> places, and that this data is often redundant but not synced or
> inaccessible.  To me this suggests a design principle: a single
> conceptual database managed by a core component (i.e., one that is
> present in every Mailman 3 system).

Yes, that is how I started the thread. However, you misinterpret the requirement for a monolithic database.
Certainly a monolithic database would be one way to accomplish DRY storage of the data, but it can also be accomplished in a distributed manner. What I am suggesting is that in a distributed system, no component of the system has the right to demand that it have the exclusive right to be the keeper of certain shared data. But, further, 
that any component taking on that responsibility should also be responsible for the storage of any related items.

> The implementation of that database may very well include multiple
> database systems (eg, the organization's LDAP directory, a Postgresql
> database for the tables related to list configurations, and an MTA
> alias file for the list addresses).
Agreed.

>  However, these need to be managed
> via a single common API, and the data must not be private to any
> non-core component.
I would agree only if you drop the "non-core". Each component may have "private" data. But that data cannot include any data that needs to be exposed by the API.

> The fact that some data are not useful to all components seems to me
> to be a red herring.  The point of a DBMS in general is that you can
> flexibly access only the data you need for the job at hand, in a form
> optimized for the job at hand.
This is the reason that utilizing, and exposing, the database engine is an attractive way to implement the storage.

>>> So what?  This extension needs to be done *somewhere*; you aren't
>>> going to be able to avoid it by throwing it out of the core.
>> 
>> No, but I will "compartmentalize" it.
> 
> You mean "as a single entity in the distribution of core components",
> or "as per-component entities containing what each component needs"?
> 
>> No, I am suggesting that either you implement the functionality by
>> specifying that some particular structure be set in a standard
>> database (and provide a reference implementation of doing so) or
> 
> I think that's a non-starter.  We are not in a position to specify
> that there even *be* a standard database backing our API, unless we're
> willing to push the redundancy/inaccessibility problems to the next
> higher level by copying databases from organizational sources
> *outside* of Mailman *into* Mailman-only databases.  I consider that
> unacceptable; use of external databases for subscriber lists is a
> high-frequency RFE, and it would be *way* higher if it weren't for the
> extremely high quality of MM-U participants, most of whom check the
> FAQ/tracker and notice that there already is an RFE on file.  AIUI,
> Barry does too.

I agree. I consider the ability to store "rosters" and/or user information in databases which are not 
under the control of MM is something that I would make a design requirement. But, going along with that, use of such external storage also negates MM's responsibility to provide management for that data.

>> Further, "each non-core module will do it differently and
>> incompatibly" is a red herring. There MUST be a SPECIFICATION of
>> the interface and EVERY implementation MUST meet those
>> REQUIREMENTS. What ever else it does will not affect any other part
>> of the system.
> 
> Have you ever told a baby to stop sucking their thumb, and use the
> pacifier?  You have to pull the thumb out to get the point across.  In
> the same way, there's going to have to be one implementation, and that
> implementation will be distributed with the core.  Otherwise there
> WILL be a SPECIFICATION of the interface and EVERY implementation WILL
> meet those REQUIREMENTS (except where the implementer finds it
> inconvenient), and we're back where we started.

There is going to have to be one REFERENCE implementation and that implementation will be sufficient to get a minimal system operational.  But, because that implementation will not meet the operational needs of most users, there will be alternate implementations. You cannot stop that. You can only hope that those implementations will meet the specification.



From stephen at xemacs.org  Thu Jul 12 20:31:51 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 13 Jul 2012 03:31:51 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <083FECEB-7062-4725-9768-0C66A6D6D912@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<567B02B3-485F-4C28-82A3-010B3D9AF56D@NFSNet.org>
	<87629u5tip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<D70B62B5-7D40-47F2-BCE8-6F908C6C3428@NFSNet.org>
	<87394y5gr3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<A87E6B64-953E-416A-BF57-2E2BD108D886@NFSNet.org>
	<87vchu3ut4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E08184B7-4B5B-4021-AE5F-9F9DF07AEDAA@NFSNet.org>
	<87txxd4fxr.fsf@uwakimon.sk.tsukuba.ac.jp>
	<083FECEB-7062-4725-9768-0C66A6D6D912@NFSNet.org>
Message-ID: <87obnk4yqw.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > There seems to be two fundamental design strategies being discussed.
 > One of them has a monolithic data store and the other has a
 > distributed store.
 > Barry has expressed some reservations about overloading a
 > monolithic data store with data extraneous to the fundamental
 > mission of message handling.
 > 
 > I have expressed concern in requiring any implementation to
 > maintain related data in a split format.
 > I recognize that there will be cases where this is necessary (for
 > example the Launchpad case as described by Barry in another
 > message). But, as he notes, such implementations tend to be
 > "brittle". Especially where there are multiple components which can
 > alter the data.  But, unless it is a constraint external to MM, I
 > do not believe that such a restriction should be introduced.

I don't think that anybody is considering requiring a monolithic
store, in the sense of putting everything into a single backend DBMS,
because we all agree that it should be possible to take member lists
from an external database and augment them with Mailman-specific
properties that we may not be permitted to store in the external
database.

(Aside: I don't think we should assume that external databases are
necessarily read-only.  For example, I can imagine informal
organizations that would allow Mailman to add new subscribers to the
member directory, or sales organizations that would allow people to
subscribe to product announcement lists and automatically add them the
to CRM database.)

What I propose is a requirement is that any data added to any
databases used by Mailman be accessible via standard Python
introspection techniques.  (In principle and by default, that is;
Mailman already hides some data from some interfaces, such as the
member list.)  For example, if we use the "user as Python object"
model, then the introspection method would simply be the 'dir'
function.  Other possibilities would be to have components register
mutators and accessors for "their" data.

 > There is also an issue of what the term "core" means. Perhaps you
 > have been referring to a distribution package. I have been
 > referring to one component of such a package, in particular that
 > component which interacts with the MTAs and redistributes
 > messages.

I find that highly unintuitive.  The core is the set of functions that
are essential.  The "distribution package" description is an
heuristic.  I.e., "these are the functions that would completely stop
the show if you installed Mailman and discovered any one of them was
not present."

 > I consider the processing of administrative messages to be a
 > separate component. And I consider the storage of configuration
 > information to be yet another component. In my view, each component
 > extends only as far as its parts interact with the same private
 > data representation.

I don't think that's a useful definition, to be honest.  On the one
hand, most functions have local variables, but surely that doesn't
make them components by themselves.  On the other, pretty much
everything in Mailman interacts with mailing lists in one way or
another, but surely none of us thinks of Mailman as a one-component
application.

I think of "component" as a concept that belongs to the art of
programming, and not having a technical definition.  A component is
any body of code and content that is a convenient unit of creation,
maintenance, and administration.  Of course issues of coherence and
coupling will help determine what is "convenient", but I don't think
they're sufficient in themselves.

 > I do mean the latter. But, if the real underlying database is a
 > RDBMS, then, within the "black box", these queries probably should
 > be implemented by translating them to real SQL queries and passing
 > those to the RDBMS.

Sure.  But this is more likely if we have a good ORM (which is a more
Pythonic way of thinking about things) as an interface to the RDBMS,
and all of that is wrapped in a convenient powerful API that allows
the programmer to delegate data persistence to some component of
Mailman.

 > First, we seem to have a different conceptual model of MM.
 > I view that which is being called "core", not as a single entity,
 > but a collection of components, most of which are critical to the
 > operation of the system.

That's not what you said above; above you restrict it to the message
routing and distribution component.  I believe that is the definition
you have been pretty consistently using throughout the thread.  No?

Anyway, I find this one very close to my own thinking.

 > > You started this thread with the observation that various
 > > components are keeping data in different places, and that this
 > > data is often redundant but not synced or inaccessible.  To me
 > > this suggests a design principle: a single conceptual database
 > > managed by a core component (i.e., one that is present in every
 > > Mailman 3 system).
 > 
 > Yes, that is how I started the thread. However, you misinterpret
 > the requirement for a monolithic database.

I think you're misinterpreting my words, actually, though I'm open to
correction by a third party.  By a "single conceptual database", I
mean that there is a single API for accessing persistent Mailman data,
and that you don't have to specify a connection to a database to
access data.  The implementation knows where all the data is stored,
whether that happens to be a single humongous ZODB, or an heterogeous
array of LDAP, SQL, and flatfile data stores.

 > Certainly a monolithic database would be one way to accomplish DRY
 > storage of the data, but it can also be accomplished in a
 > distributed manner. What I am suggesting is that in a distributed
 > system, no component of the system has the right to demand that it
 > have the exclusive right to be the keeper of certain shared
 > data. But, further, that any component taking on that
 > responsibility should also be responsible for the storage of any
 > related items.

I question whether the pain of having an (explicitly) distributed
system is worth the gain.  As you've explained it here, I see it as
setting us up for a situation where each component (including
components that are substitutes performing the same conceptual
function) will make their own decisions about what to store and where,
and what is private and what is public, so that components will
continually need to negotiate with each other over who have authority
and responsibility for certain data.

>From the responses of several people in this thread, I strongly
suspect that most implementers will decide that most of the data they
use is not interesting to other modules and make it private, rather
than spend the effort needed to generalize.  So I think the costs will
be higher and the amount of shared data lower than for a system where
one component is responsible for all connections to databases.

 > I would agree only if you drop the "non-core". Each component may
 > have "private" data. But that data cannot include any data that
 > needs to be exposed by the API.

And how do we know what "needs" to be exposed?  We don't.

I'm sure we can make a killer MLM with a distributed database and each
component storing private data.  What I don't think we can do is make
an MLM that's capable of killing web fora and Usenet, too that way.  I
think it's worth the extra effort to keep things general.

From richard at NFSNet.org  Thu Jul 12 20:42:01 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Thu, 12 Jul 2012 13:42:01 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <20120711225355.15988470@limelight.wooz.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
Message-ID: <F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>


On Jul 11, 2012, at 9:53 PM, Barry Warsaw wrote:

> On Jul 11, 2012, at 09:27 AM, Richard Wackerbarth wrote:
> 
>> First, we should define all of the data storage in terms of the REST access
>> points to the data. (And not, as presently done, the other way around) Next,
>> we should access all of the REST interface URLs indirectly so that
>> functionality can be moved around simply by changing a single reference
>> definition.  This is the kind of scheme that django uses with its {% url %}
>> tag. (like DNS vs IP addresses) Finally, we should "black box encapsulate"
>> the access to the data, requiring that EVERY module utilize this common
>> interface, and only this interface, to the data.
> 
> If there were a separate user database that exposed all sorts of data via
> REST, including stuff the core doesn't care about, then all the things I
> described before about re-implementing various Zope interfaces in the core
> holds true.  They would just do REST calls instead of database queries.

Agreed.

> Both implementations in fact can live side-by-side, and choosing one or the
> other is something you do when you install and configure the system.

I'm not sure that I agree with attempting to maintain two implementations "side-by-side"

"separate, but equal" comes to mind.   And we all know how that worked.

My preference remains to use the REST interface to define the object contract.
By that, I mean that every action performed on the object is isomorphic to a REST interaction.

To do otherwise creates an inequality between those accessing the object.

> -Barry


From barry at list.org  Thu Jul 12 23:38:11 2012
From: barry at list.org (Barry Warsaw)
Date: Thu, 12 Jul 2012 17:38:11 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <03E21F46-A297-478F-941E-C8D701881735@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<20120711220543.747177c4@limelight.wooz.org>
	<51A64575-9148-452F-A8C6-697E6291A816@NFSNet.org>
	<20120712110751.6a82f84e@limelight.wooz.org>
	<03E21F46-A297-478F-941E-C8D701881735@NFSNet.org>
Message-ID: <20120712173811.178dbcf1@resist.wooz.org>

On Jul 12, 2012, at 11:39 AM, Richard Wackerbarth wrote:

>If I understand you, we both envision a distributed database whereby any of
>the components might actually provide the data storage for some specific
>class of information. For example, without specifying where in the system it
>will reside, there is a "UserManager" that maintains information about the
>users. It would also be responsible for registering ALL user instances and
>providing lists of users that meet certain constraints. In addition, it
>provides an access mechanism for properties associated with particular users.
>
>However, I think that we differ in terms of the public interface
>specifications.

Actually, I wouldn't say that IUserManager is a public interface
specification.  It's an internal implementation specification meaning, it's
exactly the API - no more, no less - that the core requires from the "user
manager service".

>The specification that you suggest produces a particular python object.
>
>The specification that I suggest contains only those characteristics which
>are exposed on the REST interface.  It does not specify the exact format in
>which those characteristics are presented, but only what characteristics are
>available and what operations can be performed. In this form, no component
>external to the UserManager is permitted to rely on any aspect of the
>representation which falls outside of the available public characteristics.

Sure, and if this independent user manager service existed, then the
IUserManager implementation inside the core would have to map the attributes
and actions it requires to this external service via REST calls.

>The place where I am unclear of your intentions concerns this python
>object. In theory, it may have characteristics that are not exposed to the
>REST interface. Do you permit your functional modules, (the message handler
>or the administration interface, for example) to rely on any characteristics
>that are not properly exposed by the REST interface. If so, why?

The core's current REST API has grown organically, and has been filled out
only to the extent that it provides information other services (like
Postorius) have needed.  We've never yet needed, let alone figured out, what
the REST API for the "user manager service" would look like.  I guess that's
what we're doing here in this thread! :)

Some of the IUserManager API has been implicitly exposed on the REST API,
e.g. creating users, searching for users, and accessing properties of those
individual users.  But the IUserManager API hasn't yet been fully necessary.

I don't see any reason why it couldn't be mapped to a REST API; it's just a
lack of need, and nobody's written it yet.  The question is whether
IUserManager is actually the right API for a REST interface - in general I
don't think you need a 1-to-1 mapping of internal model or APIs to REST.

What would a specification of a RESTful user manager look like?

>OK, I think we are in agreement WRT this concept. You seem to have taken
>"foreign key" to imply a specific implementation of the concept to which I
>was referring, namely some key which designates a unique instance in the
>referenced table.

Yep, because I was talking about the implementation of the relationships in
the core's current table layout.

>> Doing that would, I think, prevent you from putting the mailinglist table
>> in one database and the member table in another.  It makes the associated
>> queries less efficient, but provides for a useful measure of flexibility.
>
>Were this not Python, but C++, I would suggest that we consider offering
>multiple signatures for the exposed functionality. For example, it the
>function were to deliver a list of MLs to which an Address is subscribed, the
>address might be provided as a string (the email address in canonical form),
>the pk of the entry in the underlying Address database, or as an object of
>the Address class.

What I described above though is at the database/sql layer, below any language
mappings.  If Mailman's database maintained the mailinglist table, but the
user table lived in some external Postgres databases, you couldn't use foreign
key constraints to express the relationship of those two in the member table.
We've just planned ahead and made the relationship between members and the
mailing lists based on the fqdn_listname of the mailing list[*].

>Although the same could be done in python through introspection, or similar
>techniques, I don't know to what extent doing so would introduce
>inefficiencies.
>
>> P.S. FWIW, I think this separate "user database" component with REST calls
>> in the IUserManager (et al) implementation would not actually be difficult
>> to mock up.  I don't have the time to do the coding, but would be happy to
>> discuss details if someone wanted to take a crack at it.
>
>I definitely will take a crack at it since I suspect that doing so would
>allow Postorius to have all of its user information handled in one place and
>still provide the "admin by mail" to work with its restricted expectations.

Excellent!  What I'd recommend is starting by looking at the following
zope.interfaces in the core, and thinking about how the "user manager service"
(or whatever it's called[**]) could provide the necessary functionality.  Of
course this service will probably be a superset of what the core needs, which
is fine, since the core will just ignore the parts of the REST resource tree
it doesn't care about.  This service also doesn't have to directly express the
zope.interfaces, just that we have to be able to implement the methods and
properties defined in these interfaces using REST calls.

Probably a good start would be:

 * IAddress
 * IMember
 * IRegistrar
 * IRoster
 * ISubscriptionService
 * IUser
 * IUserManager

Cheers,
-Barry

[*] Of course, now that I think about it, this will probably cause us problems
later when we support list renaming.  The relationship should probably be
between the List-ID and the user, since the former won't change even if the
mailing list is renamed.  Sigh.

[**] Hey Florian, continuing on a theme, maybe "Vicious".  Get it? :)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120712/03cefa38/attachment.pgp>

From barry at list.org  Thu Jul 12 23:41:53 2012
From: barry at list.org (Barry Warsaw)
Date: Thu, 12 Jul 2012 17:41:53 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
Message-ID: <20120712174153.5783316c@limelight.wooz.org>

On Jul 12, 2012, at 01:42 PM, Richard Wackerbarth wrote:

>On Jul 11, 2012, at 9:53 PM, Barry Warsaw wrote:
>
>> Both implementations in fact can live side-by-side, and choosing one or the
>> other is something you do when you install and configure the system.
>
>I'm not sure that I agree with attempting to maintain two implementations
>"side-by-side"
>
>"separate, but equal" comes to mind.  And we all know how that worked.

The question is whether we want to support running the core without having
this user service available, i.e. in standalone mode.  Are we going to require
that this service be running in order to run the core?  I think we shouldn't
be so strict.

I'm not too concerned about multiple implementations getting out of sync.  I
think we'll be able to devise sufficient tests to ensure that, as far as the
core is concerned, both implementations are providing all the necessary
information.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120712/4dacf334/attachment.pgp>

From stephen at xemacs.org  Fri Jul 13 04:57:21 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 13 Jul 2012 11:57:21 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <20120712173811.178dbcf1@resist.wooz.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<20120711220543.747177c4@limelight.wooz.org>
	<51A64575-9148-452F-A8C6-697E6291A816@NFSNet.org>
	<20120712110751.6a82f84e@limelight.wooz.org>
	<03E21F46-A297-478F-941E-C8D701881735@NFSNet.org>
	<20120712173811.178dbcf1@resist.wooz.org>
Message-ID: <87liio4bce.fsf@uwakimon.sk.tsukuba.ac.jp>

Barry Warsaw writes:

 > I don't see any reason why it couldn't be mapped to a REST API; it's just a
 > lack of need, and nobody's written it yet.  The question is whether
 > IUserManager is actually the right API for a REST interface - in general I
 > don't think you need a 1-to-1 mapping of internal model or APIs to REST.

My suspicion is that in particular (ie, when addressing a specific set
of requirements) you don't need a 1-1 map of internal APIs to REST.
But if you want World Domination (and you have expressed that ambition
on occasion), you need a system that grows new complex functionality
organically.  That, I think, means that we want various parts of the
system to cooperatively build a single database that is offered to all
components.

If you want to say, OK, I'm not really serious about that, that's fine
by me: we *need* a killer MLM, and Mailman 2 isn't it (any more).  But
if you want to try for World Domination, I think the database (ie, the
"social network engine") is the one component that's going to make or
break that enterprise.

 > We've just planned ahead and made the relationship between members
 > and the mailing lists based on the fqdn_listname of the mailing
 > list[*].

This should be the List-Id.  RFC 2919 essentially says that the
mapping of "mailing lists" (whatever *they* "really" are) to List-Ids
is 1-1 by definition.  I think that in practice this is probably going
to work well, in the sense that people who thoughtlessly initialize a
new List-Id when they migrate domains probably don't think (much)
about the fact that it's a new list, while people who take the care to
keep the same List-Id surely do care about the list's identity.

 > zope.interfaces, just that we have to be able to implement the
 > methods and properties defined in these interfaces using REST
 > calls.
 > 
 > Probably a good start would be:
 > 
 >  * IAddress
 >  * IMember
 >  * IRegistrar
 >  * IRoster
 >  * ISubscriptionService
 >  * IUser
 >  * IUserManager

This list kind of worries me -- it's very specific and fragmented.  Is
there a higher-level way of doing this mapping?

 > [**] Hey Florian, continuing on a theme, maybe "Vicious".  Get it? :)

I see ....


From stephen at xemacs.org  Fri Jul 13 05:03:53 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 13 Jul 2012 12:03:53 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <20120712174153.5783316c@limelight.wooz.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
Message-ID: <87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>

Barry Warsaw writes:

 > The question is whether we want to support running the core without
 > having this user service available, i.e. in standalone mode.  Are
 > we going to require that this service be running in order to run
 > the core?  I think we shouldn't be so strict.

But *some* user service needs to be available.  This is to my mind a
core component of Mailman in any case.  I don't see why having a
single interface (with perhaps a REST "binding" for IPC and a direct
zope.interfaces binding for within-process use) and a default
implementation of that service as a core component (but separate from
the post routing and delivery and admin routing and handling services)
would be very costly in implementation and maintenance, but it ensures
the flexibility to deal with unknown future requirements everybody
says is important.

From richard at NFSNet.org  Fri Jul 13 18:56:05 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Fri, 13 Jul 2012 11:56:05 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>

Stephen,

From this posting to which I am replying and other recent ones from Barry, etc. I conclude that it is highly likely that the three of us have somewhat similar conceptual models of an extensible MM system. However, because of the terms that we have chosen in our descriptions, we have failed to communicate. As a result, we have misinterpreted the others' descriptions.

For example, in earlier postings, I used the term "component". Rather than the characteristics that Stephen seemed to apply to the term, my intended concept is one which, in the text quoted, I would now call a "service" -- a "user service" for information related to the persons participating in the lists, a "post routing and delivery service" that interfaces with the MTA, an "admin handling service" for changing the state of the stored choices, etc. (My exact grouping of functionality may not be correct. But that is a separate issue.)

Now, when it comes to interfaces, in that context, I have been referring to "service interfaces".
Barry has acknowledged that the (IUser, et al.) zope interfaces are, collectively, an implementation-level interface which, presumedly, provides capability sufficient to implement the portion of the service-level interfaces that are required by the "core" modules provided. They may also provide some additional functionality that someone anticipated to be of use by other services which might wish to interact with these interfaced objects.

Similarly, I now believe that Stephen's use of "core database" was more a reference to the union of all of these service-level interfaces rather than that of the interpretation which I had placed on the term, namely a "persistent store running in conjunction with the post routing service"

Now, hopefully, we can agree that it is necessary to have a "user service". Further it is reasonable to attempt to have this service distinct from the "post routing service".
The question is then "How will these services interface to each other?"

Stephen says:
> ... having a single interface (with perhaps a REST "binding" for IPC and a direct
> zope.interfaces binding for within-process use)

I agree with the "single interface" aspect of it, particularly if that interface is viewed at the functional level rather than at an implementation level. However, I feel that there should be a complete REST binding for that functional interface. This interface can then be implemented by a zope.interface, and/or in any other format appropriate for the implementing code. However, the implementation interface must not provide any inter-service capability that is not reflected in the "interface". Intra-service capabilities are quite useful, particularly when they provide alternate functional signatures which accept local proxies for interacting objects.

Further, as an inter-service design constraint, it should be assumed that the implementation of any service might be "remote" and accessible only by way of the REST interface to it. In addition it should be assumed that the implementation of that service will be in an arbitrary programming language other than python. Designs which violate either of these assumptions should not be accepted because they reduce the ductility of the interface.

The implementation interfaces provide only a proxy for the conceptual objects defined in the service interface and any action on those proxies needs to be conveyed to the conceptual object by means of an access specified in the system interface.

Richard


On Jul 12, 2012, at 10:03 PM, Stephen J. Turnbull wrote:

> Barry Warsaw writes:
> 
>> The question is whether we want to support running the core without
>> having this user service available, i.e. in standalone mode.  Are
>> we going to require that this service be running in order to run
>> the core?  I think we shouldn't be so strict.
> 
> But *some* user service needs to be available.  This is to my mind a
> core component of Mailman in any case.  I don't see why having a
> single interface (with perhaps a REST "binding" for IPC and a direct
> zope.interfaces binding for within-process use) and a default
> implementation of that service as a core component (but separate from
> the post routing and delivery and admin routing and handling services)
> would be very costly in implementation and maintenance, but it ensures
> the flexibility to deal with unknown future requirements everybody
> says is important.


From barry at list.org  Fri Jul 13 20:37:59 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 13 Jul 2012 14:37:59 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87liio4bce.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<20120711220543.747177c4@limelight.wooz.org>
	<51A64575-9148-452F-A8C6-697E6291A816@NFSNet.org>
	<20120712110751.6a82f84e@limelight.wooz.org>
	<03E21F46-A297-478F-941E-C8D701881735@NFSNet.org>
	<20120712173811.178dbcf1@resist.wooz.org>
	<87liio4bce.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20120713143759.34a74024@limelight.wooz.org>

On Jul 13, 2012, at 11:57 AM, Stephen J. Turnbull wrote:

>Barry Warsaw writes:
>
> > I don't see any reason why it couldn't be mapped to a REST API; it's just a
> > lack of need, and nobody's written it yet.  The question is whether
> > IUserManager is actually the right API for a REST interface - in general I
> > don't think you need a 1-to-1 mapping of internal model or APIs to REST.
>
>My suspicion is that in particular (ie, when addressing a specific set
>of requirements) you don't need a 1-1 map of internal APIs to REST.

I agree.

>But if you want World Domination (and you have expressed that ambition on
>occasion), you need a system that grows new complex functionality
>organically.  That, I think, means that we want various parts of the system
>to cooperatively build a single database that is offered to all components.
>
>If you want to say, OK, I'm not really serious about that, that's fine by me:
>we *need* a killer MLM, and Mailman 2 isn't it (any more).  But if you want
>to try for World Domination, I think the database (ie, the "social network
>engine") is the one component that's going to make or break that enterprise.

I don't necessarily disagree with any of that, but that doesn't necessarily
mean that the API exposed by the core needs to provide that database.  OTOH, a
separate user database component certainly could expose that stuff, and where
the two systems overlap in their support for the uber-model, certainly the
resource locations and semantics should be as close as possible.

As an example, the core currently exposes just a few attributes at the user
resource in REST: user_id, created_on, password (optional), and display_name
(optional).  These live at <baseurl>/users/<userid>.  If the separate user
database also kept track of things like Facebook and Twitter ids, these would
also be available at the same location (different base-url of course) and the
returned JSON would just contain additional entries for that extra data.

> > We've just planned ahead and made the relationship between members
> > and the mailing lists based on the fqdn_listname of the mailing
> > list[*].
>
>This should be the List-Id.  RFC 2919 essentially says that the
>mapping of "mailing lists" (whatever *they* "really" are) to List-Ids
>is 1-1 by definition.  I think that in practice this is probably going
>to work well, in the sense that people who thoughtlessly initialize a
>new List-Id when they migrate domains probably don't think (much)
>about the fact that it's a new list, while people who take the care to
>keep the same List-Id surely do care about the list's identity.

LP: #1024509

> > zope.interfaces, just that we have to be able to implement the
> > methods and properties defined in these interfaces using REST
> > calls.
> > 
> > Probably a good start would be:
> > 
> >  * IAddress
> >  * IMember
> >  * IRegistrar
> >  * IRoster
> >  * ISubscriptionService
> >  * IUser
> >  * IUserManager
>
>This list kind of worries me -- it's very specific and fragmented.  Is
>there a higher-level way of doing this mapping?

There's not unfortunately.

> > [**] Hey Florian, continuing on a theme, maybe "Vicious".  Get it? :)
>
>I see ....

I think we'll have to start calling the core engine "Lee", since as we all
know, it should be the center of attention. :)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120713/3db7a8e4/attachment.pgp>

From barry at list.org  Fri Jul 13 20:43:34 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 13 Jul 2012 14:43:34 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20120713144334.4b861a87@limelight.wooz.org>

On Jul 13, 2012, at 12:03 PM, Stephen J. Turnbull wrote:

>Barry Warsaw writes:
>
> > The question is whether we want to support running the core without
> > having this user service available, i.e. in standalone mode.  Are
> > we going to require that this service be running in order to run
> > the core?  I think we shouldn't be so strict.
>
>But *some* user service needs to be available.

Agreed, but my question is whether the limited user manager service that the
core currently provides is enough for standalone deployments where all the
fancy stuff about Facebook ids and reputations doesn't matter.  I have no
problem with providing a more limited set of functionality in that case
(i.e. "just the messages, ma'am"[*]).

Cheers,
-Barry

[*] w/apologies to Jack Webb/Joe Friday, who never actually uttered that
paraphrase exactly.  http://www.snopes.com/radiotv/tv/dragnet.asp
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120713/411a4f96/attachment.pgp>

From barry at list.org  Fri Jul 13 21:02:24 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 13 Jul 2012 15:02:24 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
Message-ID: <20120713150224.44dbd1b3@limelight.wooz.org>

On Jul 13, 2012, at 11:56 AM, Richard Wackerbarth wrote:

>Now, hopefully, we can agree that it is necessary to have a "user
>service". Further it is reasonable to attempt to have this service distinct
>from the "post routing service".  The question is then "How will these
>services interface to each other?"

I think we're definitely converging on a common language for all the moving
parts.

>I agree with the "single interface" aspect of it, particularly if that
>interface is viewed at the functional level rather than at an implementation
>level. However, I feel that there should be a complete REST binding for that
>functional interface.

WADL is, afaik, the definition language for RESTful web services:

http://en.wikipedia.org/wiki/Web_Application_Description_Language

It is an XML-based format.

Some REST implementation libraries provide a way to generate the WADL from the
code, but frankly the ones I've used have enough downsides to far outweigh
automatic WADL generation.  I'm not aware of anything that takes WADL as the
input and creates the REST implementation for you, so all things considered, I
think WADL is fairly useless to us.

I would be happy enough with human readable <wink> documentation that
described this user service in terms of resource locations and contents.
Although it's been years since I read it, Leonard Richardson's definitive
O'Reilly book RESTful Web Services talks about how this description can even
(or maybe *should*) be HTML, with links to the actual resources being
described.

Or thought about it a different way, mm3's REST API only supports JSON
representation for resources.  You could implement a parallel representation
using HTML that included these descriptions.  I bet you could even do this in
Sphinx in a very human-friendly format, with "base-url" templating so that the
resource locations are resolvable wherever the user service is deployed.

Note that this service can have varying levels of compatibility.  E.g. we
could say that level 1 is required, and define this as the minimal service
required by the core.  Things like Facebook id would not be in level 1.
Higher levels would provide additional information that could be used to
enhance the user experience.  I have no idea whether more than two levels is
necessary, but I do think there needs to be some discussion of extensibility.

>This interface can then be implemented by a zope.interface, and/or in any
>other format appropriate for the implementing code. However, the
>implementation interface must not provide any inter-service capability that
>is not reflected in the "interface". Intra-service capabilities are quite
>useful, particularly when they provide alternate functional signatures which
>accept local proxies for interacting objects.
>
>Further, as an inter-service design constraint, it should be assumed that the
>implementation of any service might be "remote" and accessible only by way of
>the REST interface to it. In addition it should be assumed that the
>implementation of that service will be in an arbitrary programming language
>other than python. Designs which violate either of these assumptions should
>not be accepted because they reduce the ductility of the interface.
>
>The implementation interfaces provide only a proxy for the conceptual objects
>defined in the service interface and any action on those proxies needs to be
>conveyed to the conceptual object by means of an access specified in the
>system interface.

I think the use of the term "proxy" is informative here.

Let's say the core only cares about level 1 user service compliance.  Maybe
that's implemented as an external (to the core) service.  In this case, the
implementation of interfaces like IUser and IUserManager would simply be
proxies to this service.  The service itself doesn't care about these
zope.interfaces, it only cares about providing a level 1 compatible user
service via REST (probably with a JSON representation).

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120713/57b542c3/attachment-0001.pgp>

From richard at NFSNet.org  Sat Jul 14 00:43:00 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Fri, 13 Jul 2012 17:43:00 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <20120713150224.44dbd1b3@limelight.wooz.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
Message-ID: <6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>


On Jul 13, 2012, at 2:02 PM, Barry Warsaw wrote:

> On Jul 13, 2012, at 11:56 AM, Richard Wackerbarth wrote:
> 
>> Now, hopefully, we can agree that it is necessary to have a "user
>> service". Further it is reasonable to attempt to have this service distinct
>> from the "post routing service".  The question is then "How will these
>> services interface to each other?"
> 
> I think we're definitely converging on a common language for all the moving
> parts.

That, in itself, is significant. 

>> I agree with the "single interface" aspect of it, particularly if that
>> interface is viewed at the functional level rather than at an implementation
>> level. However, I feel that there should be a complete REST binding for that
>> functional interface.
> 
> WADL is, afaik, the definition language for RESTful web services:

From your description, I, also, question what we would gain by attempting to use WADL as our formalization language.

> I would be happy enough with human readable <wink> documentation that
> described this user service in terms of resource locations and contents.

I wouldn't even go that far. The service does not need to be DESCRIBED in terms of locations at all.

Certainly, somewhere in the implementation, we would have to provide that CONFIGURATION information, but from the perspective of a consumer of the "user service", my interface is akin to (expressed as meta-language, not code)

services.user_service.change_password(user_identifier, new_password, ... )

It does not matter where, or how, the user_service processes that information except to the extent that I, and the another services, expect it to persist and can later enquire

password_is_valid = services.user_service.verify_password(user_identifier, submitted_password, ... )

> Although it's been years since I read it, Leonard Richardson's definitive
> O'Reilly book RESTful Web Services talks about how this description can even
> (or maybe *should*) be HTML, with links to the actual resources being
> described.

This is a useful technique for publishing documentation where the service is being exposed to "outside" consumers.
For example, I may learn that there is a RESTful stock quotation service at http:// .....  I know nothing more about it.
Since I will be using http to access the service, it is convenient to publish usage documentation along with data.

However, in a controlled environment, documenting the interface "python style" will, IMHO will be just as useful.

> Or thought about it a different way, mm3's REST API only supports JSON
> representation for resources.  You could implement a parallel representation
> using HTML that included these descriptions.  I bet you could even do this in
> Sphinx in a very human-friendly format, with "base-url" templating so that the
> resource locations are resolvable wherever the user service is deployed.

I'm not opposed to having the documentation published in this, or any other, manner.
I would only hope that doing so can be done following the DRY principle.

> Note that this service can have varying levels of compatibility.  E.g. we
> could say that level 1 is required, and define this as the minimal service
> required by the core.  Things like Facebook id would not be in level 1.
> Higher levels would provide additional information that could be used to
> enhance the user experience.  I have no idea whether more than two levels is
> necessary, but I do think there needs to be some discussion of extensibility.

I have some ideas about ways by which we might even make the interface "auto-configurable".
-- like passing a query of requirements (or reading a list of capabilities)
For example, through the services manager, "ask" it a webUI service is available. If it is, ask the webUI service for the URLs that go into a welcome message.  (Or the alternative, always treat things as if there is a webUI. If the installer has not configured one, then a default stub would return null strings for those requests.

>> This interface can then be implemented by a zope.interface, and/or in any
>> other format appropriate for the implementing code. However, the
>> implementation interface must not provide any inter-service capability that
>> is not reflected in the "interface". Intra-service capabilities are quite
>> useful, particularly when they provide alternate functional signatures which
>> accept local proxies for interacting objects.
>> 
>> Further, as an inter-service design constraint, it should be assumed that the
>> implementation of any service might be "remote" and accessible only by way of
>> the REST interface to it. In addition it should be assumed that the
>> implementation of that service will be in an arbitrary programming language
>> other than python. Designs which violate either of these assumptions should
>> not be accepted because they reduce the ductility of the interface.
>> 
>> The implementation interfaces provide only a proxy for the conceptual objects
>> defined in the service interface and any action on those proxies needs to be
>> conveyed to the conceptual object by means of an access specified in the
>> system interface.
> 
> I think the use of the term "proxy" is informative here.

> Let's say the core only cares about level 1 user service compliance.  Maybe
> that's implemented as an external (to the core) service.  In this case, the
> implementation of interfaces like IUser and IUserManager would simply be
> proxies to this service.  The service itself doesn't care about these
> zope.interfaces, it only cares about providing a level 1 compatible user
> service via REST (probably with a JSON representation).

Yes, but it goes farther than that. Even if the user service is local, in the (core), IUser is still a proxy for the user service. However, for those calls, rather than serializing the data, converting internal object identifiers to external ones, etc. the information is just copied into the corresponding internal structure that the user service builds as it translates a request for REST services.



From barry at list.org  Sat Jul 14 02:02:04 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 13 Jul 2012 20:02:04 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
Message-ID: <20120713200204.6828684c@limelight.wooz.org>

On Jul 13, 2012, at 05:43 PM, Richard Wackerbarth wrote:

>> I would be happy enough with human readable <wink> documentation that
>> described this user service in terms of resource locations and contents.
>
>I wouldn't even go that far. The service does not need to be DESCRIBED in
>terms of locations at all.
>
>Certainly, somewhere in the implementation, we would have to provide that
>CONFIGURATION information, but from the perspective of a consumer of the
>"user service", my interface is akin to (expressed as meta-language, not
>code)
>
>services.user_service.change_password(user_identifier, new_password, ... )
>
>It does not matter where, or how, the user_service processes that information
>except to the extent that I, and the another services, expect it to persist
>and can later enquire
>
>password_is_valid = services.user_service.verify_password(user_identifier,
>submitted_password, ... )

Actually, I think it's essential that we describe the service in terms of
resources and locations.  The pseudo-code above is an example of a language
binding, but isn't enough because the implementer of that binding would not
know where to point the proxy, or how to do the HTTP calls, etc.  In fact, the
pseudo-code would look significantly different depending on what
implementation language is used (the above looks Pythonic, but what about
Ruby, or JavaScript?).

If we're using REST then the description of the service must be in terms of
the API that REST exposes, e.g. URLs and JSON (for example).  Then authors of
the language bindings can expose them to client code in whatever way makes
sense.

Take a look at the work done in Postorius for accessing the core's REST API.
There's a `mailman.client` package which is essentially this Python language
binding to the REST API.  Then the Django code can call the mailman.client, or
it could do straight up HTTP itself, or the JavaScript on AJAX-y dynamic pages
could do their own calls, etc.

>> Although it's been years since I read it, Leonard Richardson's definitive
>> O'Reilly book RESTful Web Services talks about how this description can even
>> (or maybe *should*) be HTML, with links to the actual resources being
>> described.
>
>This is a useful technique for publishing documentation where the service is
>being exposed to "outside" consumers.  For example, I may learn that there is
>a RESTful stock quotation service at http:// .....  I know nothing more about
>it.  Since I will be using http to access the service, it is convenient to
>publish usage documentation along with data.
>
>However, in a controlled environment, documenting the interface "python
>style" will, IMHO will be just as useful.

No, we really need the resource locations and definitions because consumers of
the user manager will not just be Python code.  And even if it were, that
Python code has to be translated to dozens or scores of HTTP calls, and those
cannot all be configuration options.  The base-url to the service will be
configurable, yes.

>I have some ideas about ways by which we might even make the interface
>"auto-configurable".  -- like passing a query of requirements (or reading a
>list of capabilities) For example, through the services manager, "ask" it a
>webUI service is available. If it is, ask the webUI service for the URLs that
>go into a welcome message.  (Or the alternative, always treat things as if
>there is a webUI. If the installer has not configured one, then a default
>stub would return null strings for those requests.

I think you're right that the level of compatibility can be auto-configured.
The alternative of course is just EAFP (easier to ask forgiveness,
i.e. prepare to catch any errors or handle any missing data).

>Yes, but it goes farther than that. Even if the user service is local, in the
>(core), IUser is still a proxy for the user service. However, for those
>calls, rather than serializing the data, converting internal object
>identifiers to external ones, etc. the information is just copied into the
>corresponding internal structure that the user service builds as it
>translates a request for REST services.

Kind of.  Note that IUser doesn't describe the service, so really the use of
that zope.interface, or how the calls are proxied or not is purely an
implementation detail of the consumer of the service.  I wouldn't even expect
all consumers to use IUser, or even Python.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120713/0278b64f/attachment.pgp>

From richard at NFSNet.org  Sat Jul 14 02:34:19 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Fri, 13 Jul 2012 19:34:19 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <20120713200204.6828684c@limelight.wooz.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
Message-ID: <8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>

I think that you have missed a level of abstraction.

The service, per se, does not REQUIRE a REST interface in order to operate.
It only requires an implementation (in any language) and mutually agreed intra-service bindings in a mutually acceptable language.

Because we have chosen a RESTful mechanism as a communications medium for services hosted on separate processors, we will need a REST interface specification. But, that REST interface is only one implementation of the service interface. The REST particulars are not really a part of the service.

Including the locations, etc. are details of a REST IMPLEMENTATION of the service interface in the same way that Python classes and methods would be used to define a python interface. Or C++ class headers, etc.

Richard

On Jul 13, 2012, at 7:02 PM, Barry Warsaw wrote:

> On Jul 13, 2012, at 05:43 PM, Richard Wackerbarth wrote:
>> I wouldn't even go that far. The service does not need to be DESCRIBED in
>> terms of locations at all.
>> 
>> Certainly, somewhere in the implementation, we would have to provide that
>> CONFIGURATION information, but from the perspective of a consumer of the
>> "user service", my interface is akin to (expressed as meta-language, not
>> code)
>> 
>> services.user_service.change_password(user_identifier, new_password, ... )
>> 
>> It does not matter where, or how, the user_service processes that information
>> except to the extent that I, and the another services, expect it to persist
>> and can later enquire
>> 
>> password_is_valid = services.user_service.verify_password(user_identifier,
>> submitted_password, ... )
> 
> Actually, I think it's essential that we describe the service in terms of
> resources and locations.  The pseudo-code above is an example of a language
> binding, but isn't enough because the implementer of that binding would not
> know where to point the proxy, or how to do the HTTP calls, etc.  In fact, the
> pseudo-code would look significantly different depending on what
> implementation language is used (the above looks Pythonic, but what about
> Ruby, or JavaScript?).
> 
> If we're using REST then the description of the service must be in terms of
> the API that REST exposes, e.g. URLs and JSON (for example).  Then authors of
> the language bindings can expose them to client code in whatever way makes
> sense.


From stephen at xemacs.org  Sat Jul 14 07:00:59 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 14 Jul 2012 14:00:59 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
	<8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
Message-ID: <87bojj3pis.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > I think that you have missed a level of abstraction.

Why do we need it?  AFAICS, at this point we have a bunch of services
that need to be specified somehow.  We *need* a RESTful interface for
some functions because they make the most sense if accessed remotely,
and there doesn't seem to be a good reason to go beyond HTTP and
specify individual transport protocols for these remote services.

Given that, and that REST is a pretty severe restriction, it seems
likely to me that if we actually need to go beyond REST for
performance reasons (eg), that can easily enough be done later.  We
may as well specify using REST now, and have everything target that
protocol until we need something more powerful or efficient.



From richard at NFSNet.org  Sat Jul 14 14:33:00 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Sat, 14 Jul 2012 07:33:00 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <87bojj3pis.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
	<8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
	<87bojj3pis.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <156F1AC3-873D-4651-BEA2-116953D2AA18@NFSNet.org>

We need it because we need to maintain that separation of services even for service characteristics that do not need to be exposed to an actual REST implementation. It is this conceptual separation that keeps the "post routing service" separable from the "admin-by-mail service" and the "mailing list archive service".

Remember that, as Barry has acknowledged, the IXxx interfaces are an implementation-level interface.
If we don't maintain the separation at the "service level", interactions between the services which happen to share a common implementation interface MIGHT end up bypassing the service level by using capabilities which are available in the implementation level, but not in the service level. If they do so, it will not be possible to provide an alternate implementation of one of those services without also having to provide the alternate for the other service.

At present, there is  only an implementation level specification of the interface between the "user service" and the other services presently in "core". In order to create an alternate implementation of the "user service", it will be necessary to unravel these services and assure that they are not interacting via any method which is not reflected in the service level.

We should not be required to repeat this in the future. Therefore, we need to have a service level interface for ALL inter-service interactions, even if we do not require the RESTful implementation of those interactions at this time.


On Jul 14, 2012, at 12:00 AM, Stephen J. Turnbull wrote:

> Richard Wackerbarth writes:
> 
>> I think that you have missed a level of abstraction.
> 
> Why do we need it?  AFAICS, at this point we have a bunch of services
> that need to be specified somehow.  We *need* a RESTful interface for
> some functions because they make the most sense if accessed remotely,
> and there doesn't seem to be a good reason to go beyond HTTP and
> specify individual transport protocols for these remote services.

Perhaps you still fail to understand what I mean by a service-level interface.

The service-level interface describes the functional interaction between objects and controllers.
It deals with conceptual objects and operations. It is not language or protocol specific.

ALL transport protocols are below this level.

Consider what is involved when someone wants to change a password:

The storage of the passwords and the later verification of them is handled by the "user service".
(That is a statement is a part of the service-level specification.)

There is another service, which is processing the request (hands waived about how we get to that point in the data flow, but it is in a service other than the user service). How does it accomplish its task?

For discussion, I will call this service the "admin service". After doing whatever it needs to do to decide that it is ready,

The (admin) service presents a "user identifier" and the "new password" to the "user service" in a "request to change password".
This is another service level specification.

Note that there is no mention of REST interface, or python objects, etc.

It is an implementation detail as to the format in which the "user identifier" is presented. It might be a string with the user's Id, a python object of class IUser, etc. In fact, the implementation MAY provide multiple signatures for this action. The important thing is that the ADMIN service DOES NOT "change the password field in the user object".

"change the password field in the user object" might be an internal method within the user service, but it is not visible to the admin object.

> Given that, and that REST is a pretty severe restriction, it seems
> likely to me that if we actually need to go beyond REST for
> performance reasons (eg), that can easily enough be done later.  We
> may as well specify using REST now, and have everything target that
> protocol until we need something more powerful or efficient.

Conceptually, I agree. All along, I have been advocating that the service level interface be organized in a manner which would allow it to be implemented in a RESTful manner.

I suspect that we will find some inefficiencies that would result if every operation were actually carried out in that manner and, as a result, in a particular implementation, we will group some services and allow them to bypass the strict requirement of interacting with each other only through the methods exposed in the service-level interface. However, in doing so, we must acknowledge that any replacement for one service of that implementation will have to replace the entire service group.

Richard


From alexander at sulfrian.net  Sun Jul 15 00:26:12 2012
From: alexander at sulfrian.net (Alexander Sulfrian)
Date: Sun, 15 Jul 2012 00:26:12 +0200
Subject: [Mailman-Developers] Exposing not advertised lists via nntp archiver
Message-ID: <87obnidlob.wl%alexander@sulfrian.net>

Hi,

I have a conceptional problem:

There are mailinglists with a pubic archive but that not be advertised
by the server. The nntp archiver now have to supply a list of all
available archives. If this list does not contain a mailing list, in
most news clients you do not have a chance to subscribe to that group and
read the posts.

How to solve that? The only possibility I could think of uses the
filter mechanism of the available newsgroups. The clients could supply
a regexp like filter. I would implement a mechanism, that shows
unadvertized lists with an archive only if the supplied filter is
equal to the newsgroup name. Any other idea?


Alex
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120715/3d858f5e/attachment.pgp>

From pidgeot18 at gmail.com  Sun Jul 15 04:34:48 2012
From: pidgeot18 at gmail.com (Joshua Cranmer)
Date: Sat, 14 Jul 2012 22:34:48 -0400
Subject: [Mailman-Developers] Exposing not advertised lists via nntp
	archiver
In-Reply-To: <87obnidlob.wl%alexander@sulfrian.net>
References: <87obnidlob.wl%alexander@sulfrian.net>
Message-ID: <50022C48.3060500@gmail.com>

On 7/14/2012 6:26 PM, Alexander Sulfrian wrote:
> Hi,
>
> I have a conceptional problem:
>
> There are mailinglists with a pubic archive but that not be advertised
> by the server. The nntp archiver now have to supply a list of all
> available archives. If this list does not contain a mailing list, in
> most news clients you do not have a chance to subscribe to that group and
> read the posts.
>
> How to solve that? The only possibility I could think of uses the
> filter mechanism of the available newsgroups. The clients could supply
> a regexp like filter. I would implement a mechanism, that shows
> unadvertized lists with an archive only if the supplied filter is
> equal to the newsgroup name. Any other idea?

You could enable the newsgroups only if a user has authenticated.

-- 
Joshua Cranmer
News submodule owner
DXR coauthor


From stephen at xemacs.org  Sun Jul 15 07:35:43 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 15 Jul 2012 14:35:43 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <156F1AC3-873D-4651-BEA2-116953D2AA18@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
	<8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
	<87bojj3pis.fsf@uwakimon.sk.tsukuba.ac.jp>
	<156F1AC3-873D-4651-BEA2-116953D2AA18@NFSNet.org>
Message-ID: <877gu54mds.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > We should not be required to repeat this in the future. Therefore,
 > we need to have a service level interface for ALL inter-service
 > interactions, even if we do not require the RESTful implementation
 > of those interactions at this time.

I guess I think of "components" as "interfaces with implementations
*potentially independent* of other components", and would argue that
these could (and will) be implemented as separate processes, even as
remote processes.  These require RESTful implementations AIUI.

I would certainly expect that authentication services, the user
database, the mailflow handlers, owner/site webUI, and user webUI
would be (conceptually) five separate components, although Postorius
might provide both owner/site UI and user UI.

Intracomponent APIs might be defined as zope.interfaces, and even
"more internal" APIs might be simply Python classes, etc.  But it
seems to me requiring RESTful interfaces for all intercomponent
communication is the way to go.

From stephen at xemacs.org  Sun Jul 15 08:04:40 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 15 Jul 2012 15:04:40 +0900
Subject: [Mailman-Developers] Exposing not advertised lists via
	nntp	archiver
In-Reply-To: <50022C48.3060500@gmail.com>
References: <87obnidlob.wl%alexander@sulfrian.net> <50022C48.3060500@gmail.com>
Message-ID: <87629p4l1j.fsf@uwakimon.sk.tsukuba.ac.jp>

Joshua Cranmer writes:

 > > How to solve that? The only possibility I could think of uses the
 > > filter mechanism of the available newsgroups. The clients could supply
 > > a regexp like filter. I would implement a mechanism, that shows
 > > unadvertized lists with an archive only if the supplied filter is

-> unadvertized public lists

 > > equal to the newsgroup name. Any other idea?

I think that's the closest match to the current behavior of Pipermail
archives.  But that is implementation-specific.  Is that desirable
behavior?  I think it would be a good idea to discuss the
specification with the HyperKitty folks, and on the Mailman Users list
as well.

 > You could enable the newsgroups only if a user has authenticated.

That doesn't correspond well to Pipermail behavior (where users
authenticate to a specific private list), and therefore would be
surprising.  That's not to say it's a bad idea, just that I think this
is one case where we should at least discuss it with a broad group of
users and admins, and not just developers.

Steve

From sophron at latthi.com  Mon Jul 16 00:49:01 2012
From: sophron at latthi.com (George Chatzisofroniou)
Date: Mon, 16 Jul 2012 01:49:01 +0300
Subject: [Mailman-Developers] [GSoC 2012] Metrics
In-Reply-To: <20120707230322.GB17663@gmail.com>
References: <CACeRBzmGUUvVNUHfR5MxTV-nAOjeJWOvdhm-awfjv48A=Z1TpQ@mail.gmail.com>
	<20120505162628.4fed030f@limelight.wooz.org>
	<CACeRBz=X8gSpmvLkLg1F60to7_kYkQo4_JR=TQSgHr-n38pbrA@mail.gmail.com>
	<CACeRBzkJhYe1Z-V+TRQrY9ALB5=c4dwUU2HJdPeuz7K-fNxK2A@mail.gmail.com>
	<CACeRBz=rFjHrotrOxCaVY7GWrUML9NP6NhUsnxWpfM6s6=Rp=A@mail.gmail.com>
	<87zk94pekc.fsf@uwakimon.sk.tsukuba.ac.jp>
	<C9F0DD7B-B963-455C-B9B6-9F28C68E3BF3@NFSNet.org>
	<20120519125944.696eea9e@resist.wooz.org>
	<20120707230322.GB17663@gmail.com>
Message-ID: <CACeRBz=JEV=3ajLBHHP0ZkQfFQBQrDZY2GdN52jWFrKLhJuWRQ@mail.gmail.com>

Hello,

Here's my new report. I've finally published my code and created some
samples for all of you. Youhou! :) Any feedback is very welcome.

----
I?m happy to report that the first version of the software is now
public. I have set up the app with some graph samples here [1], and
hosted the code in Launchpad [2].

Above each graph, there is the code snippet that is being used to
query the database. There are many syntax rules (most of them are
already working) that allow a rich vocabulary for the expression of
queries. You may see the BNF language description in the section ?How
to use them? in the documentation [3].

The JS library that renders the graphs is jqplot. It?s not difficult
(even for someone with minor experience to Django) to use another
plotting library. The data structure returned by the extraction tag is
very flexible and can be used in any manner. See the section
?Configuring the output? in the documentation [3].

My next step will be to implement the coalescor which (optionally)
merges database entries in order to limit the growth of the number of
entries stored in the database

Thanks for reading and testing. I would highly appreciate any comments.

[1]: http://sophron.latthi.com/metrics/
[2]: https://code.launchpad.net/~sophron/mm3-metrics/trunk
[3]: http://bazaar.launchpad.net/~sophron/mm3-metrics/trunk/view/head:/MM3_metrics_display/templatetags/__init__.py

----

-- 
George Chatzisofroniou
sophron.latthi.com

From richard at NFSNet.org  Mon Jul 16 13:10:38 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Mon, 16 Jul 2012 06:10:38 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <877gu54mds.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
	<8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
	<87bojj3pis.fsf@uwakimon.sk.tsukuba.ac.jp>
	<156F1AC3-873D-4651-BEA2-116953D2AA18@NFSNet.org>
	<877gu54mds.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <36310E5D-33B3-49F4-BF68-F16339CC984B@NFSNet.org>


On Jul 15, 2012, at 12:35 AM, Stephen J. Turnbull wrote:

> Richard Wackerbarth writes:
> 
>> We should not be required to repeat this in the future. Therefore,
>> we need to have a service level interface for ALL inter-service
>> interactions, even if we do not require the RESTful implementation
>> of those interactions at this time.
> 
> I guess I think of "components" as "interfaces with implementations
> *potentially independent* of other components", and would argue that
> these could (and will) be implemented as separate processes, even as
> remote processes.  These require RESTful implementations AIUI.

I think that "require" is too strong. What is REQUIRED is AN implementation that supports separate processes.
Don't mis-understand, I fully expect that the implementation which we develop will be a RESTful one. However, I could, just as well, develop one that used, for example, SNMP. A specification provides REQUIREMENTS. Since there is no fundamental requirement that the inter-service communication be RESTful, that I object to describing the service-level interface in terms of a RESTful implementation. The service level specification should be more general.

> I would certainly expect that authentication services, the user
> database, the mailflow handlers, owner/site webUI, and user webUI
> would be (conceptually) five separate components, although Postorius
> might provide both owner/site UI and user UI.

Even here, I would start by describing the "site admin UI" and the "user UI" as separate services.
From there, we might describe a "service group" wherein a group of services are implemented in a manner intended to run on a single host. Within this service group, we could allow intra-group communications that bypass the inter-service interface. However, if we do so, we need to be careful that such communications are only alternate implementations for efficiency and do not represent fundamental communications that could not be handled strictly through the inter-service interface. 

> Intracomponent APIs might be defined as zope.interfaces, and even
> "more internal" APIs might be simply Python classes, etc.  But it
> seems to me requiring RESTful interfaces for all intercomponent
> communication is the way to go.

As I indicated above, I consider RESTful interfaces to be only one implementation of a conceptual interface.

Richard


From barry at list.org  Tue Jul 17 18:07:44 2012
From: barry at list.org (Barry Warsaw)
Date: Tue, 17 Jul 2012 12:07:44 -0400
Subject: [Mailman-Developers] Exposing not advertised lists via nntp
 archiver
In-Reply-To: <87obnidlob.wl%alexander@sulfrian.net>
References: <87obnidlob.wl%alexander@sulfrian.net>
Message-ID: <20120717120744.49da78d1@limelight.wooz.org>

On Jul 15, 2012, at 12:26 AM, Alexander Sulfrian wrote:

>There are mailinglists with a pubic archive but that not be advertised
>by the server. The nntp archiver now have to supply a list of all
>available archives. If this list does not contain a mailing list, in
>most news clients you do not have a chance to subscribe to that group and
>read the posts.
>
>How to solve that? The only possibility I could think of uses the
>filter mechanism of the available newsgroups. The clients could supply
>a regexp like filter. I would implement a mechanism, that shows
>unadvertized lists with an archive only if the supplied filter is
>equal to the newsgroup name. Any other idea?

'Advertised' lists are kind of a funny middle ground between private and
public.  They are public lists but don't show up on the overview page.

One way to think of this in a mm3 world is that advertisement is purely a
function of the web ui.  For all intents and purposes then, this flag has no
impact on public vs. private, so it can be ignored.  If it's a public list,
then a user should be able to subscribe to the newsgroup, even if it's
unadvertised.  If the list admin really wants to prevent unauthorized access
to the newsgroup, then she needs to make it a private list (in which case,
authenticated users with the proper permission should be able to read the
list).

At least, that's my current thinking, but I'm certainly open to other
opinions.   I almost wish we could just get rid of the advertised flag.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120717/4c0df764/attachment.pgp>

From richard at NFSNet.org  Tue Jul 17 20:34:46 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Tue, 17 Jul 2012 13:34:46 -0500
Subject: [Mailman-Developers] Exposing not advertised lists via nntp
	archiver
In-Reply-To: <20120717120744.49da78d1@limelight.wooz.org>
References: <87obnidlob.wl%alexander@sulfrian.net>
	<20120717120744.49da78d1@limelight.wooz.org>
Message-ID: <C51CC6A2-21BF-4DD6-9221-85DE46168052@NFSNet.org>

On Jul 17, 2012, at 11:07 AM, Barry Warsaw wrote:

> On Jul 15, 2012, at 12:26 AM, Alexander Sulfrian wrote:
> 
>> There are mailinglists with a pubic archive but that not be advertised
>> by the server.
> 'Advertised' lists are kind of a funny middle ground between private and
> public.  They are public lists but don't show up on the overview page.
I assume that you mean "Unadvertised".

> One way to think of this in a mm3 world is that advertisement is purely a
> function of the web ui.

And the equivalent "admin-by-mail" UI.

> For all intents and purposes then, this flag has no impact on public vs. private, so it can be ignored.

But there probably should be a per-list configuration option for gateway-ing a list to the news.
There may well be lists for which I am willing to advertise the existence without wanting their content to be spread far and wide.

> At least, that's my current thinking, but I'm certainly open to other
> opinions.   I almost wish we could just get rid of the advertised flag.
> 
> -Barry

Richard


From barry at list.org  Tue Jul 17 20:46:32 2012
From: barry at list.org (Barry Warsaw)
Date: Tue, 17 Jul 2012 14:46:32 -0400
Subject: [Mailman-Developers] Exposing not advertised lists via nntp
 archiver
In-Reply-To: <C51CC6A2-21BF-4DD6-9221-85DE46168052@NFSNet.org>
References: <87obnidlob.wl%alexander@sulfrian.net>
	<20120717120744.49da78d1@limelight.wooz.org>
	<C51CC6A2-21BF-4DD6-9221-85DE46168052@NFSNet.org>
Message-ID: <20120717144632.420d0c6f@limelight.wooz.org>

On Jul 17, 2012, at 01:34 PM, Richard Wackerbarth wrote:

>But there probably should be a per-list configuration option for gateway-ing
>a list to the news.  There may well be lists for which I am willing to
>advertise the existence without wanting their content to be spread far and
>wide.

Agreed.  Some thoughts.

One is whether the mailing list is gatewayed to "Usenet" (really any external
NNTP server).  We already have settings for this.

In an mm3 world where we can have multiple system-wide configured and enabled
archivers, it probably makes sense to allow individual lists to opt-in or -out
of specific available archivers.  We've talked about this before, and it's
something I'm generally in favor of allowing.  The complication is that the
system may want to force individual lists to enable certain archivers.
E.g. if the site administrators want to keep a local maildir of all messages
to all mailing lists.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120717/3c8bf5ce/attachment.pgp>

From richard at NFSNet.org  Tue Jul 17 22:52:15 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Tue, 17 Jul 2012 15:52:15 -0500
Subject: [Mailman-Developers] Exposing not advertised lists via nntp
	archiver
In-Reply-To: <20120717144632.420d0c6f@limelight.wooz.org>
References: <87obnidlob.wl%alexander@sulfrian.net>
	<20120717120744.49da78d1@limelight.wooz.org>
	<C51CC6A2-21BF-4DD6-9221-85DE46168052@NFSNet.org>
	<20120717144632.420d0c6f@limelight.wooz.org>
Message-ID: <F5D21D9C-D86E-452D-9E81-EC71ED10E025@NFSNet.org>

On Jul 17, 2012, at 1:46 PM, Barry Warsaw wrote:
> On Jul 17, 2012, at 01:34 PM, Richard Wackerbarth wrote:
> 
>> But there probably should be a per-list configuration option for gateway-ing
>> a list to the news.  There may well be lists for which I am willing to
>> advertise the existence without wanting their content to be spread far and
>> wide.
> 
> Agreed.  Some thoughts.
> 
> One is whether the mailing list is gatewayed to "Usenet" (really any external
> NNTP server).  We already have settings for this.

Then that should suffice for Alex's purposes. He should not have to "discover" that the gateway is active. If he gets a message on the archive service interface, he sends it out. If it doesn't appear, it wasn't meant to be.

> In an mm3 world where we can have multiple system-wide configured and enabled
> archivers, it probably makes sense to allow individual lists to opt-in or -out
> of specific available archivers.

Yes, this also allows George to collect metrics about messages without having the underlying messages archived. Or it also allows the messages to be archived without being counted by the metrics. It's all up to how our users choose to configure things.

>  We've talked about this before, and it's something I'm generally in favor of allowing.

> The complication is that the system may want to force individual lists to enable certain archivers.
> E.g. if the site administrators want to keep a local maildir of all messages to all mailing lists.

This is a more general question concerning just how the configuration service enforces site policies.

On an implementation level, it gets mapped so that, to each of the various message handling services, each list has its own full set of configuration parameters. In other words, a message handler does not care whether "switch x" is set because it is set specifically for the current list or because it is set for all of the lists within some collection of lists.

Richard

From barry at list.org  Wed Jul 18 01:38:17 2012
From: barry at list.org (Barry Warsaw)
Date: Tue, 17 Jul 2012 19:38:17 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
	<8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
Message-ID: <20120717193817.21c3e81f@resist.wooz.org>

On Jul 13, 2012, at 07:34 PM, Richard Wackerbarth wrote:

>I think that you have missed a level of abstraction.

I guess it's "turtles all the way down" :)

>The service, per se, does not REQUIRE a REST interface in order to operate.
>It only requires an implementation (in any language) and mutually agreed
>intra-service bindings in a mutually acceptable language.

I think my vision of the architectural organization is very similar to
Stephen's, and we only need the abstractions at a particular level to describe
how other components at that same level will interoperate.

Within the core engine, which is all one self contained Python application
(logically, if not actually, a single address space), then I think
zope.interfaces and the Zope Component Architecture are as abstract as we need
to get.  If a component within the core, say the bounce detector, needs
information about mailing lists, then it uses the ZCA abstractions to get
Python objects (which may contain concrete implementation, or simply be
proxies) which provide the services promised by those interfaces.  I don't see
any value in getting any more abstract than that, within the core, at least
not right now.

As we talk about separating out components into separate processes, possibly
residing on different machines, then the ZCA isn't an appropriate
abstraction.  We're really talking about IPC here, and while there is no end
to IPC protocols, it seems to me that HTTP/REST is a good one to choose,
because it's built on fundamental, well-supported protocols, is easy to
understand, document, and implement, and is discoverable.

I'll note though that there's no reason why other IPC protocols couldn't be
added later.  An important implementation principle with the mm3 core, based
on my experiences with its violation in mm2, is that the REST implementation
contains no functional logic.  All it does is translate input from HTTP into
Python objects on the way in, and Python objects to HTTP on the way out.  This
principle is replicated by the shell scripts and the email commands, which
actually, are alternative parallel forms of IPC into the core.  So if someone
wanted to add XMLRPC, CORBA, or whatever new protocol all the kids are using
next year, it can be done probably with the same ease at which REST support
was added.

Also, these abstraction layers are not mutually exclusive.  Internally, the
core could use the ZCA abstraction to get a proxy to a user object which
communicates to the user manager over LDAP and the core wouldn't blink an eye.

>Because we have chosen a RESTful mechanism as a communications medium for
>services hosted on separate processors, we will need a REST interface
>specification. But, that REST interface is only one implementation of the
>service interface. The REST particulars are not really a part of the service.

I see what you're saying, and this 1000ft abstraction of the service may be
useful in the sense that it's documentation for anyone else wanting to
implement said service using a different IPC mechanism, but right now where
the project is, this is a bit too abstract to be meaningful *to me*.  I have
no problem with this kind of service description being defined before the user
manager service is implemented, but it still has to be made more concrete one
level down into an IPC protocol definition before it can be useful.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120717/108aa580/attachment-0001.pgp>

From richard at NFSNet.org  Thu Jul 19 00:53:57 2012
From: richard at NFSNet.org (Richard Wackerbarth)
Date: Wed, 18 Jul 2012 17:53:57 -0500
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <20120717193817.21c3e81f@resist.wooz.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
	<8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
	<20120717193817.21c3e81f@resist.wooz.org>
Message-ID: <4D39BFBF-7A42-40CB-928B-6E184BE49755@NFSNet.org>

On Jul 17, 2012, at 6:38 PM, Barry Warsaw wrote:
> On Jul 13, 2012, at 07:34 PM, Richard Wackerbarth wrote:
>> But, that REST interface is only one implementation of the service interface.
>> The REST particulars are not really a part of the service.
> 
> I see what you're saying, and this 1000ft abstraction of the service may be
> useful in the sense that it's documentation for anyone else wanting to
> implement said service using a different IPC mechanism, but right now where
> the project is, this is a bit too abstract to be meaningful *to me*.  I have
> no problem with this kind of service description being defined before the user
> manager service is implemented, but it still has to be made more concrete one
> level down into an IPC protocol definition before it can be useful.

I think that you are finally beginning to understand the perspective that I am trying to provide.

Yes, to implement anything, we will need the concrete descriptions at the protocol level.
For example, we will need some definition of the services implemented as a RESTful interface.
Also, at least for the message handler, because you have chosen to implement it that way in Python, an object description such as ZCA for the implementation interface will be required.

However, and this is the reason that I object to the bypassing of the service level description, we also need to assure that each of these implementations respect the requirement that ALL inter-service interactions be representable strictly in terms of service-level interfaces. If you define only the implementation interface, then you have no standard upon which to judge adherence to any standard that does not REQUIRE ALL implementations to exactly duplicate all aspects of that particular implementation.

Specifically, one interface may provide multiple implementations which accomplish the same system service. (Many ways to skin the cat) Alternate implementations are required to provide just one of them.

From barry at list.org  Thu Jul 19 20:54:40 2012
From: barry at list.org (Barry Warsaw)
Date: Thu, 19 Jul 2012 14:54:40 -0400
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <4D39BFBF-7A42-40CB-928B-6E184BE49755@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
	<8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
	<20120717193817.21c3e81f@resist.wooz.org>
	<4D39BFBF-7A42-40CB-928B-6E184BE49755@NFSNet.org>
Message-ID: <20120719145440.0d57b755@resist.wooz.org>

On Jul 18, 2012, at 05:53 PM, Richard Wackerbarth wrote:

>However, and this is the reason that I object to the bypassing of the service
>level description, we also need to assure that each of these implementations
>respect the requirement that ALL inter-service interactions be representable
>strictly in terms of service-level interfaces. If you define only the
>implementation interface, then you have no standard upon which to judge
>adherence to any standard that does not REQUIRE ALL implementations to
>exactly duplicate all aspects of that particular implementation.
>
>Specifically, one interface may provide multiple implementations which
>accomplish the same system service. (Many ways to skin the cat) Alternate
>implementations are required to provide just one of them.

Things are getting a bit too abstract for me.  Why don't you take a crack at
this service level description for the user manager so we can have something
concrete to further the discussion?  You can post it here or in the wiki.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120719/6bcd450c/attachment.pgp>

From stephen at xemacs.org  Thu Jul 19 21:30:38 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 20 Jul 2012 04:30:38 +0900
Subject: [Mailman-Developers] Login / User Identification Issues in MM3
In-Reply-To: <4D39BFBF-7A42-40CB-928B-6E184BE49755@NFSNet.org>
References: <06828AA4-5691-4CB3-8D79-AB9786C71DEE@NFSNet.org>
	<87d34359ip.fsf@uwakimon.sk.tsukuba.ac.jp>
	<DE48831E-4F6C-45B3-B4B3-568EF675687E@NFSNet.org>
	<878veq6fup.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4FFD235D.80104@zone12.com>
	<592CBC7D-5BD7-4667-A003-337B491BE1BE@NFSNet.org>
	<20120711225355.15988470@limelight.wooz.org>
	<F516CA6E-C0F3-4215-8F0D-33A79AD3AA3A@NFSNet.org>
	<20120712174153.5783316c@limelight.wooz.org>
	<87k3y84b1i.fsf@uwakimon.sk.tsukuba.ac.jp>
	<E34E2684-18C1-473F-9E30-7FD76309A830@NFSNet.org>
	<20120713150224.44dbd1b3@limelight.wooz.org>
	<6AF49663-5700-4F01-BCDA-567E81CDC039@NFSNet.org>
	<20120713200204.6828684c@limelight.wooz.org>
	<8ABDF3B9-92D5-4B36-9FEC-3DD3FE82E39C@NFSNet.org>
	<20120717193817.21c3e81f@resist.wooz.org>
	<4D39BFBF-7A42-40CB-928B-6E184BE49755@NFSNet.org>
Message-ID: <87sjcn35wh.fsf@uwakimon.sk.tsukuba.ac.jp>

Richard Wackerbarth writes:

 > I think that you are finally beginning to understand the
 > perspective that I am trying to provide.

I don't have any trouble understanding the perspective you're trying
to provide in the abstract; it's just Software Systems Design 301.

What I have trouble seeing is why you want to go to this level of
abstraction for Mailman 3, and what implications is has for design and
implementation of Mailman services.

 > Yes, to implement anything, we will need the concrete descriptions
 > at the protocol level.

But AIUI, REST isn't a concrete description.  Wikipedia, for example,
calls it an "architectural style".  Specifically, that article defines
it as a service meeting 5 constraints (and there's an optional 6th).[1]
If your abstract service doesn't satisfy those constraints, then it
can't be implemented RESTfully, which in turn constrains implementer
designs (eg, it means changing multiple modules to implement a new
service).

So for this purpose, I see the HTTP verbs used in REST, such as GET,
as being primitives obeying certain constraints, but not as
necessarily implemented by trivial references to the concrete HTTP
implementation.  Similarly, the REST approach to addressing resources
via URLs puts some constraints on the structure of Mailman's data.
IOW, REST is a language for describing interactions among Mailman
components, not a specific implementation of such interactions.

 > Specifically, one interface may provide multiple implementations
 > which accomplish the same system service. (Many ways to skin the
 > cat) Alternate implementations are required to provide just one of
 > them.

I don't understand.  How is one component that speaks REST supposed to
communicate with another specified in terms of the ZCA?  They satisfy
different constraints.

Footnotes: 
[1]  http://en.wikipedia.org/wiki/REST



From sophron at latthi.com  Fri Jul 20 13:45:22 2012
From: sophron at latthi.com (George Chatzisofroniou)
Date: Fri, 20 Jul 2012 14:45:22 +0300
Subject: [Mailman-Developers] Implementing the IArchiver
Message-ID: <CACeRBzmPbKuthnjnNc_fQUrBUs0FHRy+-xVoERRS705ZSkO-YA@mail.gmail.com>

Hello everyone,

For my metrics app (GSoC project), I'm currently using a MM3 simulator
for testing purposes. Proceeding with the development, i think it's
time to implement a connection with a real MM3 environment.

For the current version of the app, the only information i need is who
posted what to which list on what date. So, as we have already
discussed, the IArchiver should be the origin of the interface.

But i know nothing on how to implement this. How will i add my app to
the active IA modules? Is there any code that extends the IArchiver
and will help me understand how it works? (I guess HK extends it to
inject the messages into the archiver.)

Thanks in advance,

-- 
George Chatzisofroniou
sophron.latthi.com

From barry at list.org  Fri Jul 20 16:14:37 2012
From: barry at list.org (Barry Warsaw)
Date: Fri, 20 Jul 2012 10:14:37 -0400
Subject: [Mailman-Developers] Implementing the IArchiver
In-Reply-To: <CACeRBzmPbKuthnjnNc_fQUrBUs0FHRy+-xVoERRS705ZSkO-YA@mail.gmail.com>
References: <CACeRBzmPbKuthnjnNc_fQUrBUs0FHRy+-xVoERRS705ZSkO-YA@mail.gmail.com>
Message-ID: <20120720101437.06a94b16@limelight.wooz.org>

Hi George,

On Jul 20, 2012, at 02:45 PM, George Chatzisofroniou wrote:

>For my metrics app (GSoC project), I'm currently using a MM3 simulator
>for testing purposes. Proceeding with the development, i think it's
>time to implement a connection with a real MM3 environment.
>
>For the current version of the app, the only information i need is who
>posted what to which list on what date. So, as we have already
>discussed, the IArchiver should be the origin of the interface.
>
>But i know nothing on how to implement this. How will i add my app to
>the active IA modules? Is there any code that extends the IArchiver
>and will help me understand how it works? (I guess HK extends it to
>inject the messages into the archiver.)

There isn't too much boilerplate you need to write in order to implement
IArchiver, or any interface.  You can look at the existing implementations in
src/mailman/archiving/*.py for examples, but I'll briefly outline what you
need here.

from zope.interface import implementer
from mailman.interfaces.archiver import IArchiver

@implementer(IArchiver)
class MyArchiver:
    """This is the docstring."""

    name = 'myarchiver'

    @staticmethod
    def list_url(mlist):
        """See `IArchiver`."""
        # do something

    @staticmethod
    def permalink(mlist, msg):
        """See `IArchiver`."""
        # do something

    @staticmethod
    def archive_message(mlist, message):
        """See `IArchiver`."""
        # do something


You probably only care about archive_message().  For methods list_url() and
permalink() which you don't care about, you can just `raise NotImplemented`.

That's about it!  Of course the details of archive_message() are up to you.

Cheers,
-Barry

From franck at peachymango.org  Sat Jul 21 18:49:40 2012
From: franck at peachymango.org (Franck Martin)
Date: Sat, 21 Jul 2012 11:49:40 -0500 (CDT)
Subject: [Mailman-Developers] mailing list to work with ADSP and DMARC
In-Reply-To: <1020347342.90522.1341981928849.JavaMail.root@peachymango.org>
Message-ID: <7293005.10914.1342889380380.JavaMail.franck@Francks-Mac-Pro.local>

As I have not received feedback.. then no feedback is good feedback right? 

I submitted the code to be merged into main branch (using launchpad interface). It is 2.1.x branch and I know the development happens on 3.x, but let me know the status and if I should try to do a branch in 3.x code base in a view to submit a similar patch.

Thanks.

----- Original Message -----
From: "Franck Martin" <franck at peachymango.org>
To: "mailman-developers" <mailman-developers at python.org>
Sent: Tuesday, July 10, 2012 9:45:28 PM
Subject: [Mailman-Developers] mailing list to work with ADSP and DMARC

I did a branch on the 2.1 series to test rewriting the From: header to be able to make emails to comply with DMARC and ADSP 

You can find the branch here: 

https://code.launchpad.net/~mlm-author/mailman/2.1-author 

and the diff with mailman main branch here: 
http://bazaar.launchpad.net/~mlm-author/mailman/2.1-author/revision/1340?start_revid=1340&remember=1338&compare_revid=1338 

I have also a test mailing list running at: 
http://lists.peachymango.org/mailman/listinfo/mlm-auth 

The best to see how it works is to subscribe, send an email, and see how the email is distributed. I guess you need to receive an email to see that... I'll reply... 

Several ways to solve this problem were explored with mailman: 
http://wiki.list.org/display/DEV/DKIM 

I took a 7th option which is to rewrite the From: without using the + and put the information about the sender in the Reply-To: 

This may not be optimum but it works. 

Overall this solution is one out of 3 ways to get mailing lists running with DMARC (see DMARC FAQ) 
1) don't break DKIM 
2) use AOR 
3) rewrite the from 

There is another one which is to whitelist emails coming from mailing lists.... 

Also note that while I'm involved in the DMARC group, I do this code as a personal project. 

Now send comments, and be nice with me :P 

_______________________________________________
Mailman-Developers mailing list
Mailman-Developers at python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/franck%40peachymango.org

Security Policy: http://wiki.list.org/x/QIA9

From p at state-of-mind.de  Sat Jul 21 20:43:54 2012
From: p at state-of-mind.de (Patrick Ben Koetter)
Date: Sat, 21 Jul 2012 20:43:54 +0200
Subject: [Mailman-Developers] mailing list to work with ADSP and DMARC
In-Reply-To: <7293005.10914.1342889380380.JavaMail.franck@Francks-Mac-Pro.local>
References: <1020347342.90522.1341981928849.JavaMail.root@peachymango.org>
	<7293005.10914.1342889380380.JavaMail.franck@Francks-Mac-Pro.local>
Message-ID: <20120721184354.GA25893@state-of-mind.de>

Frank,

* Franck Martin <franck at peachymango.org>:
> As I have not received feedback.. then no feedback is good feedback right? 
> 
> I submitted the code to be merged into main branch (using launchpad
> interface). It is 2.1.x branch and I know the development happens on 3.x,
> but let me know the status and if I should try to do a branch in 3.x code
> base in a view to submit a similar patch.

ADSP is considered to be broken and marked deprecated. DMARC is its successor
and on the rise. Currently DMARC is undergoing interop testing. It is said to
advance towards IETF standard sometime in autumn.

You might just be a little too early and you might work on the wrong code.

On Mailman-Developers currently most discussions deal with/most efforts are
put into MM3 development.

My bet is you will receive more feedback if you start implementing DMARC
relevant features into MM3 code. If you get Murray Kucherawy, I don't know if
he's still on the list, to contribute/share ideas on DMARC you will very
likely get a bulletproof implementation - he's one of the driving forces
behind DMARC.

p at rick



> ----- Original Message -----
> From: "Franck Martin" <franck at peachymango.org>
> To: "mailman-developers" <mailman-developers at python.org>
> Sent: Tuesday, July 10, 2012 9:45:28 PM
> Subject: [Mailman-Developers] mailing list to work with ADSP and DMARC
> 
> I did a branch on the 2.1 series to test rewriting the From: header to be able to make emails to comply with DMARC and ADSP 
> 
> You can find the branch here: 
> 
> https://code.launchpad.net/~mlm-author/mailman/2.1-author 
> 
> and the diff with mailman main branch here: 
> http://bazaar.launchpad.net/~mlm-author/mailman/2.1-author/revision/1340?start_revid=1340&remember=1338&compare_revid=1338 
> 
> I have also a test mailing list running at: 
> http://lists.peachymango.org/mailman/listinfo/mlm-auth 
> 
> The best to see how it works is to subscribe, send an email, and see how the email is distributed. I guess you need to receive an email to see that... I'll reply... 
> 
> Several ways to solve this problem were explored with mailman: 
> http://wiki.list.org/display/DEV/DKIM 
> 
> I took a 7th option which is to rewrite the From: without using the + and put the information about the sender in the Reply-To: 
> 
> This may not be optimum but it works. 
> 
> Overall this solution is one out of 3 ways to get mailing lists running with DMARC (see DMARC FAQ) 
> 1) don't break DKIM 
> 2) use AOR 
> 3) rewrite the from 
> 
> There is another one which is to whitelist emails coming from mailing lists.... 
> 
> Also note that while I'm involved in the DMARC group, I do this code as a personal project. 
> 
> Now send comments, and be nice with me :P 
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/franck%40peachymango.org
> 
> Security Policy: http://wiki.list.org/x/QIA9
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/p%40state-of-mind.de
> 
> Security Policy: http://wiki.list.org/x/QIA9

-- 
state of mind ()

http://www.state-of-mind.de

Franziskanerstra?e 15      Telefon +49 89 3090 4664
81669 M?nchen              Telefax +49 89 3090 4666

Amtsgericht M?nchen        Partnerschaftsregister PR 563


From barry at list.org  Sun Jul 22 00:17:02 2012
From: barry at list.org (Barry Warsaw)
Date: Sat, 21 Jul 2012 18:17:02 -0400
Subject: [Mailman-Developers] Bug 971013 - schema migrations BLOCKED
In-Reply-To: <20120706220010.4603b21e@limelight.wooz.org>
References: <20120706220010.4603b21e@limelight.wooz.org>
Message-ID: <20120721181702.639656c1@resist.wooz.org>

On Jul 06, 2012, at 10:00 PM, Barry Warsaw wrote:

>This bug is about providing orderly schema migrations.  I now admit that I'm
>blocked and I'm looking for suggestions.

I actually think I've finally cracked this nut.  I have it working for SQLite,
and am now testing for Postgres.  When I get that working and committed to
trunk, it would be great for others to try it on your test lists.  You
shouldn't have to do anything but stop, install the new version, and restart.

Looking forward to making progress again.
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120721/5f4a9f14/attachment.pgp>

From franck at peachymango.org  Sun Jul 22 04:21:39 2012
From: franck at peachymango.org (Franck Martin)
Date: Sat, 21 Jul 2012 21:21:39 -0500 (CDT)
Subject: [Mailman-Developers] mailing list to work with ADSP and DMARC
In-Reply-To: <20120721184354.GA25893@state-of-mind.de>
References: <1020347342.90522.1341981928849.JavaMail.root@peachymango.org>
	<7293005.10914.1342889380380.JavaMail.franck@Francks-Mac-Pro.local>
	<20120721184354.GA25893@state-of-mind.de>
Message-ID: <07858461-1052-4F5E-821C-752C50A8F12C@peachymango.org>

Thanks very much for the feedback, but note what I wrote at the end of the first post. I'm very much involved in DMARC but I do this one as a personal project.

Ok so mm3 it is, it seems.

Printed on recycled paper!

On 21/07/2012, at 11:50, Patrick Ben Koetter <p at state-of-mind.de> wrote:

> Frank,
> 
> * Franck Martin <franck at peachymango.org>:
>> As I have not received feedback.. then no feedback is good feedback right? 
>> 
>> I submitted the code to be merged into main branch (using launchpad
>> interface). It is 2.1.x branch and I know the development happens on 3.x,
>> but let me know the status and if I should try to do a branch in 3.x code
>> base in a view to submit a similar patch.
> 
> ADSP is considered to be broken and marked deprecated. DMARC is its successor
> and on the rise. Currently DMARC is undergoing interop testing. It is said to
> advance towards IETF standard sometime in autumn.
> 
> You might just be a little too early and you might work on the wrong code.
> 
> On Mailman-Developers currently most discussions deal with/most efforts are
> put into MM3 development.
> 
> My bet is you will receive more feedback if you start implementing DMARC
> relevant features into MM3 code. If you get Murray Kucherawy, I don't know if
> he's still on the list, to contribute/share ideas on DMARC you will very
> likely get a bulletproof implementation - he's one of the driving forces
> behind DMARC.
> 
> p at rick
> 
> 
> 
>> ----- Original Message -----
>> From: "Franck Martin" <franck at peachymango.org>
>> To: "mailman-developers" <mailman-developers at python.org>
>> Sent: Tuesday, July 10, 2012 9:45:28 PM
>> Subject: [Mailman-Developers] mailing list to work with ADSP and DMARC
>> 
>> I did a branch on the 2.1 series to test rewriting the From: header to be able to make emails to comply with DMARC and ADSP 
>> 
>> You can find the branch here: 
>> 
>> https://code.launchpad.net/~mlm-author/mailman/2.1-author 
>> 
>> and the diff with mailman main branch here: 
>> http://bazaar.launchpad.net/~mlm-author/mailman/2.1-author/revision/1340?start_revid=1340&remember=1338&compare_revid=1338 
>> 
>> I have also a test mailing list running at: 
>> http://lists.peachymango.org/mailman/listinfo/mlm-auth 
>> 
>> The best to see how it works is to subscribe, send an email, and see how the email is distributed. I guess you need to receive an email to see that... I'll reply... 
>> 
>> Several ways to solve this problem were explored with mailman: 
>> http://wiki.list.org/display/DEV/DKIM 
>> 
>> I took a 7th option which is to rewrite the From: without using the + and put the information about the sender in the Reply-To: 
>> 
>> This may not be optimum but it works. 
>> 
>> Overall this solution is one out of 3 ways to get mailing lists running with DMARC (see DMARC FAQ) 
>> 1) don't break DKIM 
>> 2) use AOR 
>> 3) rewrite the from 
>> 
>> There is another one which is to whitelist emails coming from mailing lists.... 
>> 
>> Also note that while I'm involved in the DMARC group, I do this code as a personal project. 
>> 
>> Now send comments, and be nice with me :P 
>> 
>> _______________________________________________
>> Mailman-Developers mailing list
>> Mailman-Developers at python.org
>> http://mail.python.org/mailman/listinfo/mailman-developers
>> Mailman FAQ: http://wiki.list.org/x/AgA3
>> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
>> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/franck%40peachymango.org
>> 
>> Security Policy: http://wiki.list.org/x/QIA9
>> _______________________________________________
>> Mailman-Developers mailing list
>> Mailman-Developers at python.org
>> http://mail.python.org/mailman/listinfo/mailman-developers
>> Mailman FAQ: http://wiki.list.org/x/AgA3
>> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
>> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/p%40state-of-mind.de
>> 
>> Security Policy: http://wiki.list.org/x/QIA9
> 
> -- 
> state of mind ()
> 
> http://www.state-of-mind.de
> 
> Franziskanerstra?e 15      Telefon +49 89 3090 4664
> 81669 M?nchen              Telefax +49 89 3090 4666
> 
> Amtsgericht M?nchen        Partnerschaftsregister PR 563
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/franck%40peachymango.org
> 
> Security Policy: http://wiki.list.org/x/QIA9

From stephen at xemacs.org  Sun Jul 22 08:16:38 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 22 Jul 2012 15:16:38 +0900
Subject: [Mailman-Developers] Bug 971013 - schema migrations BLOCKED
In-Reply-To: <20120721181702.639656c1@resist.wooz.org>
References: <20120706220010.4603b21e@limelight.wooz.org>
	<20120721181702.639656c1@resist.wooz.org>
Message-ID: <87k3xw2ud5.fsf@uwakimon.sk.tsukuba.ac.jp>

Barry Warsaw writes:

 > >This bug is about providing orderly schema migrations.  I now admit that I'm
 > >blocked and I'm looking for suggestions.
 > 
 > I actually think I've finally cracked this nut.  I have it working
 > for SQLite, and am now testing for Postgres.  When I get that
 > working and committed to trunk, it would be great for others to try
 > it on your test lists.  You shouldn't have to do anything but stop,
 > install the new version, and restart.

Ooh, I'd better hurry up and install an old version quick!<wink/>

From stephen at xemacs.org  Sun Jul 22 08:19:52 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 22 Jul 2012 15:19:52 +0900
Subject: [Mailman-Developers] mailing list to work with ADSP and DMARC
In-Reply-To: <20120721184354.GA25893@state-of-mind.de>
References: <1020347342.90522.1341981928849.JavaMail.root@peachymango.org>
	<7293005.10914.1342889380380.JavaMail.franck@Francks-Mac-Pro.local>
	<20120721184354.GA25893@state-of-mind.de>
Message-ID: <87ipdg2u7r.fsf@uwakimon.sk.tsukuba.ac.jp>

Patrick Ben Koetter writes:

 > My bet is you will receive more feedback if you start implementing DMARC
 > relevant features into MM3 code. If you get Murray Kucherawy, I don't know if
 > he's still on the list, to contribute/share ideas on DMARC you will very
 > likely get a bulletproof implementation - he's one of the driving forces
 > behind DMARC.

Indeed, Murray is the right person to talk to.  I would like to be
included in the loop, it's possible that I will be participating in
discussions as sort of a Mailman rep.  Barry will also be involved,
but if he doesn't specifically ask to be included in off-list
discussions, I'll be keeping notes and archives of any traffic and
will share with him.

Steve


From syst3m.w0rm at gmail.com  Wed Jul 25 06:43:29 2012
From: syst3m.w0rm at gmail.com (Aamir Khan)
Date: Wed, 25 Jul 2012 00:43:29 -0400
Subject: [Mailman-Developers] HyperKitty packaging!
Message-ID: <CAOb12VVvZ5duoeEZ1e+1qnQEaz9OKVTu-9a=-hwYto1aMUfL=A@mail.gmail.com>

Hey Everyone!

I have packaged HyperKitty and I have created hk-app standalone django
project which uses the hyperkitty django application. In future I hope  we
can replace both hk-app [1] and postorius_standalone [2] by a single django
project, mm_ui for administrators who want to install hyperkitty as well as
postorius on their server.

In the process of packaging hyperkitty I am facing some issues with the
code and unable to figure out the exact problem. I have updated the readme
files for both hyperkitty [3] django application and hk-app [2] django
project. It will be great if you guys can help me to resolve the issue.

Try to run "python manage.py syncdb" in hk-app and you will get
ImportError: No module named kittystore.* *I think the system path is
messed up or probably creating symlink is not the right way to do it inside
a packaged application.



[1] =>
https://code.launchpad.net/~mailman-coders/postorius/postorius_standalone
[2] => https://github.com/syst3mw0rm/hk-app
[3] => https://github.com/syst3mw0rm/hyperkitty






-- 
Aamir Khan | 3rd Year  | Computer Science & Engineering | IIT Roorkee

From syst3m.w0rm at gmail.com  Thu Jul 26 00:19:39 2012
From: syst3m.w0rm at gmail.com (Aamir Khan)
Date: Wed, 25 Jul 2012 18:19:39 -0400
Subject: [Mailman-Developers] HyperKitty packaging!
In-Reply-To: <CAOb12VVvZ5duoeEZ1e+1qnQEaz9OKVTu-9a=-hwYto1aMUfL=A@mail.gmail.com>
References: <CAOb12VVvZ5duoeEZ1e+1qnQEaz9OKVTu-9a=-hwYto1aMUfL=A@mail.gmail.com>
Message-ID: <CAOb12VVA4Qdi21aEL5+kxAH2RqsVq3=j-FOa-byJEDvn9OXB5A@mail.gmail.com>

Nevermind, the issue has been resolved.

On Wed, Jul 25, 2012 at 12:43 AM, Aamir Khan <syst3m.w0rm at gmail.com> wrote:

> Hey Everyone!
>
> I have packaged HyperKitty and I have created hk-app standalone django
> project which uses the hyperkitty django application. In future I hope  we
> can replace both hk-app [1] and postorius_standalone [2] by a single django
> project, mm_ui for administrators who want to install hyperkitty as well as
> postorius on their server.
>
> In the process of packaging hyperkitty I am facing some issues with the
> code and unable to figure out the exact problem. I have updated the readme
> files for both hyperkitty [3] django application and hk-app [2] django
> project. It will be great if you guys can help me to resolve the issue.
>
> Try to run "python manage.py syncdb" in hk-app and you will get
> ImportError: No module named kittystore.* *I think the system path is
> messed up or probably creating symlink is not the right way to do it inside
> a packaged application.
>
>
>
> [1] =>
> https://code.launchpad.net/~mailman-coders/postorius/postorius_standalone
> [2] => https://github.com/syst3mw0rm/hk-app
> [3] => https://github.com/syst3mw0rm/hyperkitty
>
>
>
>
>
>
> --
> Aamir Khan | 3rd Year  | Computer Science & Engineering | IIT Roorkee
>
>
>


-- 
Aamir Khan | 3rd Year  | Computer Science & Engineering | IIT Roorkee

From barry at python.org  Thu Jul 26 06:27:23 2012
From: barry at python.org (Barry Warsaw)
Date: Thu, 26 Jul 2012 00:27:23 -0400
Subject: [Mailman-Developers] Bug 971013 - schema migrations BLOCKED
References: <20120706220010.4603b21e@limelight.wooz.org>
	<20120721181702.639656c1@resist.wooz.org>
Message-ID: <20120726002723.1e37bc73@resist.wooz.org>

On Jul 21, 2012, at 06:17 PM, Barry Warsaw wrote:

>On Jul 06, 2012, at 10:00 PM, Barry Warsaw wrote:
>
>>This bug is about providing orderly schema migrations.  I now admit that I'm
>>blocked and I'm looking for suggestions.
>
>I actually think I've finally cracked this nut.  I have it working for SQLite,
>and am now testing for Postgres.  When I get that working and committed to
>trunk, it would be great for others to try it on your test lists.  You
>shouldn't have to do anything but stop, install the new version, and restart.

This has just landed in lp:mailman and I have it working locally for both
SQLite and PostgreSQL.

If you have beta1 deployed, please do give the upgrade a try.  Make backups
first though! :)  If you encounter bugs, please file them in Launchpad,
hopefully with a way to reproduce the problem, or at least let me know whether
you're using SQLite or PostgreSQL.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120726/649338ed/attachment.pgp>

From barry at list.org  Thu Jul 26 06:31:48 2012
From: barry at list.org (Barry Warsaw)
Date: Thu, 26 Jul 2012 00:31:48 -0400
Subject: [Mailman-Developers] Bug 971013 - schema migrations BLOCKED
In-Reply-To: <20120726002723.1e37bc73@resist.wooz.org>
References: <20120706220010.4603b21e@limelight.wooz.org>
	<20120721181702.639656c1@resist.wooz.org>
	<20120726002723.1e37bc73@resist.wooz.org>
Message-ID: <20120726003148.2074061f@resist.wooz.org>

On Jul 26, 2012, at 12:27 AM, Barry Warsaw wrote:

>If you have beta1 deployed, please do give the upgrade a try.  Make backups
>first though! :)  If you encounter bugs, please file them in Launchpad,
>hopefully with a way to reproduce the problem, or at least let me know whether
>you're using SQLite or PostgreSQL.

I should mention too that there may be additional schema changes coming for
beta 2.  I do not (necessarily) intend to make intra-release upgrades migrate
smoothly.

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120726/11e6d55a/attachment.pgp>

From subin at opensourcesolutions.co.in  Fri Jul 27 10:59:57 2012
From: subin at opensourcesolutions.co.in (subin-opensourcesolutions.co.in)
Date: Fri, 27 Jul 2012 14:29:57 +0530
Subject: [Mailman-Developers] Issue: Mailman integration with Postfix(zimbra)
Message-ID: <ad1c0fc017dccdf30501b4cc7439c48f@webmail.ucsrv01.unixclinic.net>

Hello,

NB: Sorry, my query is not related to development, but an issue in 
mailman integration with Postfix.

Scenario: I just setup mailman with Postfix(zimbra). I was able to 
create new list, subscribe to it. When I susbscribed using different 
mail addresses, I received notification mail asking to confirm the 
subscription of the list. On confirmation , I was able to receive the 
welcome mail also. Now when I send the mail to listname at domain.tld, the 
subscribed users are not able to receive the group/list mail. Please 
advice ! Thanks.

I followed the below forum integration that was suggested official by 
zimbra community:

http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html

Thanks,


From mark at msapiro.net  Fri Jul 27 20:13:01 2012
From: mark at msapiro.net (Mark Sapiro)
Date: Fri, 27 Jul 2012 11:13:01 -0700
Subject: [Mailman-Developers] Issue: Mailman integration with
	Postfix(zimbra)
In-Reply-To: <ad1c0fc017dccdf30501b4cc7439c48f@webmail.ucsrv01.unixclinic.net>
Message-ID: <PC195201207271113010078a9753d4f@MSAPIRO>

subin-opensourcesolutions.co.in wrote:
>
>Now when I send the mail to listname at domain.tld, the 
>subscribed users are not able to receive the group/list mail. Please 
>advice ! Thanks.
>
>I followed the below forum integration that was suggested official by 
>zimbra community:
>
>http://www.zimbra.com/forums/administrators/1380-solved-zimbra-mailman-howto.html


Please see the FAQs at <http://wiki.list.org/x/OIDD> and
<http://wiki.list.org/x/A4E9>.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan


From sophron at latthi.com  Mon Jul 30 16:22:54 2012
From: sophron at latthi.com (George Chatzisofroniou)
Date: Mon, 30 Jul 2012 17:22:54 +0300
Subject: [Mailman-Developers] [GSoC 2012] Metrics
In-Reply-To: <CACeRBz=JEV=3ajLBHHP0ZkQfFQBQrDZY2GdN52jWFrKLhJuWRQ@mail.gmail.com>
References: <CACeRBzmGUUvVNUHfR5MxTV-nAOjeJWOvdhm-awfjv48A=Z1TpQ@mail.gmail.com>
	<20120505162628.4fed030f@limelight.wooz.org>
	<CACeRBz=X8gSpmvLkLg1F60to7_kYkQo4_JR=TQSgHr-n38pbrA@mail.gmail.com>
	<CACeRBzkJhYe1Z-V+TRQrY9ALB5=c4dwUU2HJdPeuz7K-fNxK2A@mail.gmail.com>
	<CACeRBz=rFjHrotrOxCaVY7GWrUML9NP6NhUsnxWpfM6s6=Rp=A@mail.gmail.com>
	<87zk94pekc.fsf@uwakimon.sk.tsukuba.ac.jp>
	<C9F0DD7B-B963-455C-B9B6-9F28C68E3BF3@NFSNet.org>
	<20120519125944.696eea9e@resist.wooz.org>
	<20120707230322.GB17663@gmail.com>
	<CACeRBz=JEV=3ajLBHHP0ZkQfFQBQrDZY2GdN52jWFrKLhJuWRQ@mail.gmail.com>
Message-ID: <CACeRBznf8b24jvN=rHoKtCOqCj-R32adO-p521wQUFH-_AgFmg@mail.gmail.com>

Greetings,

Following a new report. I've finished a coalecor daemon and i wrote
some generators for KittyStore too. I'm now into establishing a real
MM3 connection.

----
The first version of coalescor is ready!

Coalescor is a custom Django admin command that is responsible for
maintaining the number of entries in the database at a reasonable
level. If the number of entries in the database is not a concern, its
use is completely optional.

There are five parameters which determine the behavior of the
coalesence: DAILY_DETAIL_RETAINED, WEEKLY_DETAIL_RETAINED,
MONTHLY_DETAIL_RETAINED and ANNUAL_DETAIL_RETAINED. Each one of them
represents the number of grains for which that level of detail is
maintained.

The daemon coalesces only the new data from its last operation date
and it was designed in a manner not to be memory intensive.

The last two days i also worked on generators. I had already created a
script that generates metrics from a mailbox. There is a KittyStore
API availabe, so it wasn?t hard to also generate metrics from SQL or
MongoDB.

----

-- 
George Chatzisofroniou
sophron.latthi.com