From barry at list.org Sat Jan 8 01:03:52 2011 From: barry at list.org (Barry Warsaw) Date: Fri, 7 Jan 2011 19:03:52 -0500 Subject: [Mailman-Developers] Suggestions about Mailman bounce processing In-Reply-To: References: <877heul8dk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20110107190352.12f28504@list.org> On Dec 28, 2010, at 12:54 PM, Mark Sapiro wrote: >>Note that the final decision is Marks (v2.x) or Barry's (v3)q, of >>course. But I tend to think they'll agree with me. > >FWIW, I do agree. As do I for Mailman 3, although I also suspect such behavior will be more easily pluggable for customization. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Sat Jan 8 01:07:00 2011 From: barry at list.org (Barry Warsaw) Date: Fri, 7 Jan 2011 19:07:00 -0500 Subject: [Mailman-Developers] Suggestions about Mailman bounce processing In-Reply-To: References: <201012270216480765.31F10804@mailhub.iastate.edu> <87fwtileoy.fsf@uwakimon.sk.tsukuba.ac.jp> <201012272358300000.3698C2F6@mailhub.iastate.edu> <877heul8dk.fsf@uwakimon.sk.tsukuba.ac.jp> <201012280127200656.36EA19D7@mailhub.iastate.edu> Message-ID: <20110107190700.4b6f5ab6@list.org> On Dec 29, 2010, at 12:50 AM, Thomas Hochstein wrote: >"Superticker2 (Mark)" schrieb: > >> If there's a service or utility that can verify a list of email addresses >> (prior to a mass operation), please suggest it. > > is a Perl script to >check the validity of a mail address (by trying a SMTP dialog up to >RCPT TO). FWIW, in Mailman 3, email validation is a "utility" in the ZCA[*] sense of the term, so in theory it should be fairly easily pluggable. The current algorithm is not much different than the Mailman 2 checks. -Barry [*] Zope Component Architecture -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at list.org Sat Jan 8 01:30:51 2011 From: barry at list.org (Barry Warsaw) Date: Fri, 7 Jan 2011 19:30:51 -0500 Subject: [Mailman-Developers] Bounce processing in MM3 Message-ID: <20110107193051.65fcc785@list.org> One of the last major subsystems that I need to get working in Mailman 3 is bounce processing. This is different than bounce detection, which has been successfully ported from Mailman 2, but doesn't differ in any significant way. The question I am thinking about now is what to do with a bounce once we detect one. There are a couple of interesting things in MM3 that makes it different from MM2. In MM3, users and addresses are global to the system, while membership is specific to a mailing list. This means if we register a bounce on an address, we can have that score affect the address's subscription to all relevant mailing list. We can also do things like automatically roll over to another registered and validated email address for that user, if there is one, or at least send notifications to the other address. There's also the question about how all the bounce scores are managed, and the knobs you as a list administrator can tweak to control how and when things happen based on the score. Reporting and logging are also part of the plan. Because MM3 uses a relational database underneath the hood, my plan is to have a single table that only appends new bounce events. That way, Mailman will have a permanent record of every bounce that occurred. Exactly what information we can or should put in that table is up for discussion. I do plan also to keep all bounce messages in MM3's "message store" so that postmortem debugging is easier. Because I'm just starting to think about all this, I wanted to throw this out to the list to get your feedback on things you'd like to see. What is it about MM2's bounce processing that you like? What don't you like? What MM2 bounce features can you do without? What would you like to see added? Any and all feedback is welcome. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From p at state-of-mind.de Sat Jan 8 09:55:48 2011 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Sat, 8 Jan 2011 09:55:48 +0100 Subject: [Mailman-Developers] Bounce processing in MM3 In-Reply-To: <20110107193051.65fcc785@list.org> References: <20110107193051.65fcc785@list.org> Message-ID: <20110108085548.GB1832@state-of-mind.de> * Barry Warsaw : > One of the last major subsystems that I need to get working in Mailman 3 is > bounce processing. This is different than bounce detection, which has been > successfully ported from Mailman 2, but doesn't differ in any significant > way. The question I am thinking about now is what to do with a bounce once we > detect one. > > There are a couple of interesting things in MM3 that makes it different from > MM2. In MM3, users and addresses are global to the system, while membership > is specific to a mailing list. This means if we register a bounce on an > address, we can have that score affect the address's subscription to all > relevant mailing list. We can also do things like automatically roll over to > another registered and validated email address for that user, if there is one, > or at least send notifications to the other address. Here's an "I think while I type" section... ;) Should we apply a global action to a local problem i.e. if an address in one list bounces, suspend that address in all lists? Sounds like a good feature to keep the mail system from wasting ressources, but is this a good service for the recipient? If a message cannot be delivered to a specific address is undeliverable it usually is because - the receiving mail system is down -> global action, because no list will be able to deliver a message - the recipient does not exist anymore -> global action, because no list will be able to deliver a message - the recipient never existed -> paradox: User never was able to verify list membership. Maybe she was, with another list and we just took the address for granted. We should always confirm an address for deliverablity reasons! - the recipients mailbox is over quota -> global action, because no list will be able to deliver a message - the envelope sender is being rejected -> problem! We can't tell if the specific envelope sender or the whole envelope sender domain was rejected. Action: Local. In dubio pro reo. In this case I'd leave it to all lists to learn that the particular recipient is undeliverable. > There's also the question about how all the bounce scores are managed, and the > knobs you as a list administrator can tweak to control how and when things > happen based on the score. Reporting and logging are also part of the plan. We could apply a score and disable sending in all lists if e.g. three lists detected the recipient address bounces. But what's the consequence? Once we disable one recipient, should we apply that to all recipients in that recipients domain? Looks like a great admin service, but also like a great opportunity to shoot oneself into the foot automatically. It should be a feature that must be called manually. > Because MM3 uses a relational database underneath the hood, my plan is to have > a single table that only appends new bounce events. That way, Mailman will > have a permanent record of every bounce that occurred. Exactly what > information we can or should put in that table is up for discussion. I do > plan also to keep all bounce messages in MM3's "message store" so that > postmortem debugging is easier. Send priority by reputation. Reputation deducted from deliverabilty or should I better say receivability? We could create a domains or even a hosts receivabilty record. Those with the worst record are the once we send to last, because they likely will clutter our MSAs outgoing mailqueue. On a sidenote: I'd welcome an abuse database to blacklist addresses in a global range. And I like the idea that each MM instance should offer these data as service to other installations. Site-wide. Cross-Site. > Because I'm just starting to think about all this, I wanted to throw this out > to the list to get your feedback on things you'd like to see. What is it > about MM2's bounce processing that you like? What don't you like? What MM2 > bounce features can you do without? What would you like to see added? I am not acquainted with MM2 bounce features. p at rick -- state of mind Digitale Kommunikation http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 316 bytes Desc: Digital signature URL: From stephen at xemacs.org Sun Jan 9 08:59:43 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 09 Jan 2011 16:59:43 +0900 Subject: [Mailman-Developers] Bounce processing in MM3 In-Reply-To: <20110107193051.65fcc785@list.org> References: <20110107193051.65fcc785@list.org> Message-ID: <87bp3qa4nk.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > There are a couple of interesting things in MM3 that makes it different from > MM2. In MM3, users and addresses are global to the system, while membership > is specific to a mailing list. This means if we register a bounce on an > address, we can have that score affect the address's subscription to all > relevant mailing list. That would probably upset users at ISPs that count how many of their addresses are receiving the same message, and bounce them as spam if there are too many. Needs to be an option. From adam-mailman at amyl.org.uk Sun Jan 9 16:04:02 2011 From: adam-mailman at amyl.org.uk (Adam McGreggor) Date: Sun, 9 Jan 2011 15:04:02 +0000 Subject: [Mailman-Developers] Bounce processing in MM3 In-Reply-To: <20110107193051.65fcc785@list.org> References: <20110107193051.65fcc785@list.org> Message-ID: <20110109150402.GD2565@hendricks.amyl.org.uk> On Fri, Jan 07, 2011 at 07:30:51PM -0500, Barry Warsaw wrote: > There are a couple of interesting things in MM3 that makes it different from > MM2. In MM3, users and addresses are global to the system, while membership > is specific to a mailing list. This means if we register a bounce on an > address, we can have that score affect the address's subscription to all > relevant mailing list. This makes me feel slightly uncomfortable, particularly for large (multi-site, multi-client) installations -- Hosting Co, &c. > We can also do things like automatically roll over to > another registered and validated email address for that user, if there is one, > or at least send notifications to the other address. Hum. If the auto-rollover knew "oh, it's a different MX" there might be a point. However, just trying adam at amyl.org.uk, instead of adam-mailman at amyl.org.uk, or adam+tarpit at amyl.org.uk, would not be very useful, I'd imagine. Maybe an option to specify "this is my recovery address, send bounce-notifications here, please" might be useful? (for end users). It would obviously need to spell out, quite clearly for which address it releated to, as finding an envelope-to: header seems to be tricky for users. > There's also the question about how all the bounce scores are managed, and the > knobs you as a list administrator can tweak to control how and when things > happen based on the score. Reporting and logging are also part of the plan. I'm perhaps a little cavalier in my approach; I generally let Mailman handle the bounces, so I can do something useful. About the most I delve, when I don't need to investigate "why aren't I getting mail" is a monthly report of numbers of subscribers, changes to that figure from previous month, and "reasons" why people left, pulled from subscribe.log, at the moment. > Because MM3 uses a relational database underneath the hood, my plan is to have > a single table that only appends new bounce events. That way, Mailman will > have a permanent record of every bounce that occurred. What may be useful is to supplement this with a pertinent dates table, too, something like start-date/end-date/few-words-on-problem, either controlled by Chief Goncho (aka site-admins), or maybe with something for listadmins; the case I'm thinking of may be to show that, say the LINX have had problems for a couple of months, "MTA tweak for redelivery attempts to yahoo.com made on 2010-02-04"; these would be added to a gnuplot/graph in a separate color, in my vision (maybe I've used google analytics too long, but clicking on the event for more info would be grand). Perhaps that's function creep, though. > Exactly what > information we can or should put in that table is up for discussion. I do > plan also to keep all bounce messages in MM3's "message store" so that > postmortem debugging is easier. In which case, there should definately be an option for "keep bounce messages in store for N months", and perhaps make list-specific ones available to list-admins. > Because I'm just starting to think about all this, I wanted to throw this out > to the list to get your feedback on things you'd like to see. What is it > about MM2's bounce processing that you like? It does most of the work for me; I set the global parameters, and generally, just leave Mailman to do everything for me. I might sometimes see the "been removed from the list due to bounce" mails; if those were on a grid-thing somewhere in the admin pages, I don't think I'd need/want the mails. > What don't you like? What MM2 > bounce features can you do without? What would you like to see added? > > Any and all feedback is welcome. > -Barry > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers at python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/adam-mailman%40amyl.org.uk > > Security Policy: http://wiki.list.org/x/QIA9 -- "You know it cannot have been a good night when you get into a fight with Spider-Man and two cross-dressing men" -- Mark Davies (defence lawyer, regarding 'Cage fighters picked on because they were dressed as women for a stag night') From barry at list.org Wed Jan 19 20:45:18 2011 From: barry at list.org (Barry Warsaw) Date: Wed, 19 Jan 2011 14:45:18 -0500 Subject: [Mailman-Developers] Wiki downtime Message-ID: <20110119144518.34214d79@list.org> Our hosting provider is going to upgrade our wiki instance. There may be a short amount of downtime. Apologies for any inconvenience. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From neupane4 at gmail.com Wed Jan 26 07:14:32 2011 From: neupane4 at gmail.com (narayan neupane) Date: Wed, 26 Jan 2011 11:59:32 +0545 Subject: [Mailman-Developers] Mailman Admin section! Message-ID: Hello Developer Team, I am a web developer in php. Currently i have migrated the site but the admin section was in mailman software at this moment how can i migrate the admin section on new hosting please suggest me with procedure to installed with no data lost at all. Narayan Neupane (Web Developer) From mark at msapiro.net Wed Jan 26 17:04:59 2011 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 26 Jan 2011 08:04:59 -0800 Subject: [Mailman-Developers] Mailman Admin section! In-Reply-To: Message-ID: narayan neupane wrote: > >I am a web developer in php. Currently i have migrated the site but the >admin section was in mailman software at this moment how can i migrate the >admin section on new hosting please suggest me with procedure to installed >with no data lost at all. See the FAQs at , and . -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From geoff at QuiteLikely.com Wed Jan 26 17:34:37 2011 From: geoff at QuiteLikely.com (Geoff Shang) Date: Wed, 26 Jan 2011 18:34:37 +0200 (IST) Subject: [Mailman-Developers] Mailman Admin section! In-Reply-To: References: Message-ID: On Wed, 26 Jan 2011, narayan neupane wrote: > I am a web developer in php. Currently i have migrated the site but the > admin section was in mailman software at this moment how can i migrate the > admin section on new hosting please suggest me with procedure to installed > with no data lost at all. This is really a Mailman-users question, but since you're here... http://wiki.list.org/display/DOC/How+do+I+move+a+list+to+a+different+server-Mailman+installation. Note to devs: I had trouble finding this as it's in the list admin FAQ rather than the site admin or installation FAQs. Geoff. From CNulk at scu.edu Fri Jan 28 22:58:08 2011 From: CNulk at scu.edu (C Nulk) Date: Fri, 28 Jan 2011 13:58:08 -0800 Subject: [Mailman-Developers] Adding a message id to the vette log Message-ID: <4D433BF0.2010504@scu.edu> Hello all, I am using Mailman v2.1.9 with some minor local mods on CentOS 5.5. In looking through the vette log, I noticed some of the log entries for refused postings and discarded postings do not indicate what message (via the message id) was acted upon. I would like to the message id to those entries. I found where the entries are written to in ListAdmin.py. I made a few changes to the file. Since I am barely literate with python, I thought I would post my changes to the list and ask you fine people if I did will work, or am I doomed. There changes are (in diff -u format): --- /usr/lib/mailman/Mailman/ListAdmin.py 2008-05-24 13:44:12.000000000 -0700 +++ ListAdmin.py.SCU 2011-01-28 13:26:12.000000000 -0800 @@ -340,13 +340,21 @@ fmsg.send(self) # Log the rejection if rejection: + try: + msg = readMessage(path) + except IOError, e: + if e.errno <> errno.ENOENT: raise + return LOST + msg = readMessage(path) note = '''%(listname)s: %(rejection)s posting: \tFrom: %(sender)s -\tSubject: %(subject)s''' % { +\tSubject: %(subject)s +\tMessage-id: %(messageid)''' % { 'listname' : self.internal_name(), 'rejection': rejection, 'sender' : str(sender).replace('%', '%%'), 'subject' : str(subject).replace('%', '%%'), + 'messageid': msg.get('message-id', 'n/a').replace('%', '%%'), } if comment: note += '\n\tReason: ' + comment.replace('%', '%%') Also, are there any other files I may need to change? Thanks, Chris From mark at msapiro.net Sat Jan 29 00:30:01 2011 From: mark at msapiro.net (Mark Sapiro) Date: Fri, 28 Jan 2011 15:30:01 -0800 Subject: [Mailman-Developers] Adding a message id to the vette log In-Reply-To: <4D433BF0.2010504@scu.edu> Message-ID: C Nulk wrote: > >I am using Mailman v2.1.9 with some minor local mods on CentOS 5.5. In >looking through the vette log, I noticed some of the log entries for >refused postings and discarded postings do not indicate what message >(via the message id) was acted upon. I would like to the message id to >those entries. I found where the entries are written to in >ListAdmin.py. I made a few changes to the file. Since I am barely >literate with python, I thought I would post my changes to the list and >ask you fine people if I did will work, or am I doomed. There changes >are (in diff -u format): > >--- /usr/lib/mailman/Mailman/ListAdmin.py 2008-05-24 >13:44:12.000000000 -0700 >+++ ListAdmin.py.SCU 2011-01-28 13:26:12.000000000 -0800 >@@ -340,13 +340,21 @@ > fmsg.send(self) > # Log the rejection > if rejection: >+ try: >+ msg = readMessage(path) >+ except IOError, e: >+ if e.errno <> errno.ENOENT: raise >+ return LOST >+ msg = readMessage(path) I would do this a bit differently for two reasons. First, you may already have the message, and second, the last of the above added lines is redundant (I know it's in the source above you copied, but it's redundant there too and should be removed). So I would say if rejection: if not msg: try: msg = readMessage(path) except IOError, e: if e.errno <> errno.ENOENT: raise return LOST But, this will not log anything at all in the (rare) case when the message is lost. So I would go a step further and instead of the above say if rejection: if not msg: try: msg = readMessage(path) except IOError, e: if e.errno <> errno.ENOENT: raise if msg: messageid = msg.get('message-id', 'n/a') else messageid = 'n/a' > note = '''%(listname)s: %(rejection)s posting: > \tFrom: %(sender)s >-\tSubject: %(subject)s''' % { >+\tSubject: %(subject)s >+\tMessage-id: %(messageid)''' % { > 'listname' : self.internal_name(), > 'rejection': rejection, > 'sender' : str(sender).replace('%', '%%'), > 'subject' : str(subject).replace('%', '%%'), >+ 'messageid': msg.get('message-id', 'n/a').replace('%', >'%%'), And also replace the line above with 'messageid': messageid.replace('%','%%') > } > if comment: > note += '\n\tReason: ' + comment.replace('%', '%%') > > >Also, are there any other files I may need to change? That depends. Currently, messages which are automatically rejected due to member_moderation_action, reject_these_nonmembers or generic_nonmember_action are not logged at all. If you want logging for them, look at the "except Errors.RejectMessage" clause in the _dopipeline() definition in IncomingRunner.py -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan