[Mailman-Developers] Anatomy of list ids does not keep with that of urls causes some REST end points to return 404 always

[Richard Damon]

On 3/7/15 5:27 AM, Ankush Sharma wrote:
> Hello everyone,
>
> I want to discuss a serious issue regarding mailman architecture in context
> with bug #1429366 <https://bugs.launchpad.net/mailman/+bug/1429366>.
> The *hash(#)
> *is a valid character as far as the local part of the email addresses is
> concerned as per the email RFCs. So, as the mailing list addresses are
> email addresses too, we can use # in the list names too. And, in context
> with mailman it works well. We can create a list with  list_id
> *sam#hashed.host.org
> <http://hashed.host.org>* for the address *sam#hashed at host.org
> <hashed at host.org> *. This works fine. But it makes the *list_id *to contain
> the hash character and therefore the REST endpoint for retrieving list wise
> info becomes invalid, i.e :
>
> *<api-root>/lists/sam#hashed.host.org <http://hashed.host.org>*
>
> Because in an URL the stuff after # is treated as *document starting point*
> i.e an id identifier or something of a dom element. This is not a valid
> PATH for the server. Therefore the *falcon wsgi request object* does not
> contain information of that and the *req.path* simply returns *sam* as the
> *list_id* ( http://bazaar.launchpad.net/~mailman-coders/mailman/3.0/view/head:/src/mailman/rest/wsgiapp.py#L65
> ) giving a 404 because there is no any list with list id *sam*.
> The mailman client works fine, it sends a GET to
> *<api-root>lists/sam#hashed.host.org
> <http://hashed.host.org>. *
>
> This causes the REST end points which needs *list_id* to return 404 or in
> worse we can have a list_id clash between ids* sam#XXXXX* and *sam*.
> Further more if the list_id starts with a # character then the server finds
> list_id to be *empty string* and therefore we get a KEY ERROR because
> *fqdn_listname
> *is not set too. The bug highly effects postorius too. The lists index
> template at */postorius/lists/ *cannot be rendered as it uses the former
> REST endpoint and again a 404 is given. And, until we delete this list from
> the database, we can't do anything except of getting a 404 and KEY ERROR
> each time.
> As far as security is concerned, if an another user created a public list
> using a hash character, then public list indexing would also fail.
>
> *Possible fixes:*
> 1. Put a constraint over the use of this kind of characters, but it's like
> falling back.
> 2. Do not derive list ids directly from the list addresses, use some kind
> of hashing or encoding technique to generate list_ids. Need to change a lot
> in architecture;  :( .
> 3. Use obscure techniques to retain the hash part as discussed here
> http://stackoverflow.com/questions/317760/how-to-get-url-hash-from-server-side
>   .
>
> I think this is quite a serious issue in mailman architecture; i.e
> obtaining list_ids directly from list addresses. I have been trying to fix
> this bug using some obscure techniques. But as it is related to mailman
> architecture, I thought it should be worthy to discuss on the mailing list
> first.
>
> Thanks,
> Ankush Sharma
> github.com/black-perl
>
The simple, and standard complient, solution would seem to be to define 
that special characters get URL encoded as needed (and decoded on the 
receiving end. It should be standard procedure that ANY routine building 
a URL would use URL encoding on the pieces unless it is positive that it 
is not needed.

(URL encoding is replacing the problem chars with %nn where nn is the 
hex value of the character, this now makes % a problem char so it is 
sent as part of the URL as %25)

-- 
Richard Damon