[Mailman-Developers] Fine grained subscription control, scaling, noise, relevancy, subscription rules, dynamic lists

[Andrew Stuart]

Refer to the bottom of this email for some previous quotes from @barry on this topic. I’ve also had off list discussions with Barry in which he has mentioned this same topic so it seems to have some previous thought gone into it. 

I’m wanting to raise the topic of “fine grained subscription control” (for want of a better term) for discussion. Please note these thoughts refer to discussion style lists rather than notification style lists. Please prefix every sentence here with IMO - I’m not saying I’m right about anything, just putting forward food for thought. Also, I’m no Mailman expert so if my assumptions are plain wrong please let me know.

One of the core problems of mailing lists (at least as implemented in Mailman) is that there’s not much middle ground between being very involved (i.e receive all posts to list), or being almost-not-involved (i.e. receiving daily digests or no digest).

List noise and relevancy is the main problem and it’s a big impediment to lists being more widely used.

Very engaged users might be ok with constantly deleting messages that they are not interested in (i.e. irrelevant noise). Less engaged users will be annoyed with the noise and unsubscribe, or will switch to digest notification which I suggest to you is close to unsubscription anyway because that user no longer sees emails that they might have been interested in had they known there was a message on that subject. Digest users I suggest are at best observers rather than participators.

The more active the list, the greater the noise problem, the more users will drop out of the list.  As a list gains more subscribers, eventually even the most engaged user will have had enough of the volume of messages and will drop back to digest or write some mailbox rule that moves the emails to a folder, effectively dis-engaging them from the list conversation flow. Thus Mailman discussion-style lists currently don’t scale.

Even for low volume lists, noise is a major inhibitor to usage in certain contexts.  Consider for example the CIO of a company or the manager of a division of 30 developers. There are various mailing lists being used by the various projects that they are responsible for. The leader is not participating however because the noise from those lists would be overwhelming. They would however like to be partipcating in list discussions either that they initiated, are explicitly copied into, or relate to topics (keywords) that they are interested in. In reality, I’m not convinced Mailman would often be used in contexts like this currently because the relevancy/noise/digest/unsubscribe problem is a showstopper.

The systers have recognised this problem and their solution is Dynamic sublists as described here:
http://wiki.list.org/DEV/Dynamic%20Sublists
"List subscribers can decide whether to be a part of new conversations or not. If the users decide to subscribe to new conversations, then they will receive all the messages of a new conversation unless they explicitly unsubscribe from it. If they otherwise decide to unsubscribe from new conversations,then they will receive only the first message of every new conversation unless they explicitly subscribe to it.”

I don’t think dynamic sublists are an effective solution (IMO, remember?).  Getting the root message in all conversations is still too noisy and its a clumsy mechanic to have to opt in to further discussion and my thinking is that opt-in-to-discuss will impose a barrier to engagement that will reduce user interaction - put another way - “opt in to every discussion?  too hard”. 

Possibly a solution worth considering is fine-grained subscription control, in which there is a set of SEND and DONTSEND rules at a List default level and at a user level.

SEND:
Discussions where list member email address is in to/from/cc:
Discussions where to/from/cc contains one or more of: sally at example.org, davina at example.org, *.example.com
Discussions where subject contains keyword: hyperkitty
Discussions where body contains keyword: hyperkitty

DONTSEND:
Discussions where list member email address is in to/from/cc:
Discussions where to/from/cc contains one or more of: sally at example.org, davina at example.org
Discussions where subject contains keyword: Java
Discussions where body contains keyword: ruby

The technical hurdle to making this work is that Mailman needs access to historical messages to make it work (i.e. integrating some level of aqrchive like functionality into the Mailman database). I suggest this this may not be as hard as it sounds and hey, we’ve got a database there anyway so why not use it? 

One possibility is that this could be pushed into archiving but I don’t think that actually is practical - such concepts really need to be built deep into the core. I’m not advocating archiving-in-core here because I think archiving should like outside core.  I do think however that there is value in the core having access to archive data to implement fine grained subscription control and I think it should do so on its own terms (and in its own database) rather than asking archivers to provide the needed functionality.

Fine grained subscription control whilst probably relatively involved to implement, is sure to have a huge payoff in giving users more relevance and less noise, perhaps increasing uptake of Mailman in situations where the noise and relevancy problem currently make Mailman impractical.

Thoughts anyone?

as

Quoting @barry ….

http://mailman.9.n7.nabble.com/Improving-the-archives-tt31296.html#a31340

“”"I like this for several reasons.  I've long wanted a bridge between the traditional mailing list and a forum because to me they’re related along a spectrum of emotional investment. 

What I mean is this.  For the subjects and projects I care deeply about, I join the mailing list.  I want to be intimately involved in the day-to-day collaboration that being subscribed gives me.  I care enough about that that I'm willing to put up with the pain that comes along with mailing lists, such as the overhead for subscribing, deleting topics I don't care about, the occasional spam, the overhead of going on vacation or leaving the list, etc. 

But there are even more topics or projects that I have only a fleeting interest in.  Say I find a bug in some X program, or wake up and decide to learn how to use setuptools, or find that some recent update broke my Linux server.  In all those cases, I might want to start a thread of discussion or ask a question, and be very involved in that thread for a week or two.  Then, my interest wanes, or I get my question answered, or other projects pique my interest.  Mailing lists are pretty bad at managing those kinds of fleeting involvement, but forums are quite nice.  There's usually fairly low overhead (and probably even less if OpenID and such were in widespread adoption) for joining, and when I lose interest the forum doesn't fill up my inbox.  OTOH, forums seem good for short 'instant' messages, but not so good (IMO) for free ranging, detailed discussions.  So there's a spectrum. “””