[Mailman-Users] Help with ONE broken list (Mailman 2.1.9) - part 2

Thu May 7 19:11:07 CEST 2015

At 3:09 PM -0700 5/5/15, Mark Sapiro wrote:
>I'm a bit confused, but I think I understand. You say this is Mailman
>2.1.9, but the above traceback says
>/usr/lib/mailman/Mailman/Gui/Privacy.py defines a list attribute
>dmarc_quarantine_moderation_action which didn't exist before 2.1.18.
>
>My best guess is something backported the DMARC mitigations into your
>Mailman 2.1.9 and also included code in Mailman/versions.py to update
>older lists, but whether or not a list gets updated by this process
>depends on the list's data_version being less than the DATA_FILE_VERSION
>defined in Mailman/Version.py.
>
>So, still guessing but, the backport incremented DATA_FILE_VERSION so
>all the existing lists got updated except the list that came from the
>2.1.14 Mailman that already had a data_version >= the new DATA_FILE_VERSION.
>
>To fix this, do the following:
>
>bin/withlist -l problem_list_name
>(responses from withlist ending with prompt)
>>>> m.data_version = 1
>>>> m.Save()
> >>>            <- control-D here to exit
>
>If I am correct, that will fix the problem, and If it is something else,
>no harm will be done.

Mark --

Thanks for the suggestion.  I did try it, and it actually did make things worse, though I've been able to get back to the previous (yet still broken) state. 

It appears to me that something within /lib/mailman/lists/problem-list/config.pck is the problem child.  I'd be happy to remove and re-create the list with good settings.  But I _must_ have the subscribers including their bounce status, digest-status, the real-names they've entered through the Mailman confirmation page, etc.etc.  Is there a way to do this in Mailman 2.1.9?  (Remembering that I have a working version of the list on a restored snapshot of the server, where everthing actually does work.)

I say this based on the results of applying the change Mark suggested, and recovering from the troubles it caused.  Sorry to include all the gory details below, and thanks to anyone who is brave enough to read to the end....

Thanks Again,
Drew Tenenholz

Before applying the change Mark suggested, I thought I'd at least capture what /Mailman.Versions.py actually says:

[root]# more /usr/lib/mailman/Mailman/Version.py

# Copyright (C) 1998-2006 by the Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
# USA.

# Mailman version
VERSION = "2.1.9"

# And as a hex number in the manner of PY_VERSION_HEX
ALPHA = 0xa
BETA  = 0xb
GAMMA = 0xc
# release candidates
RC    = GAMMA
FINAL = 0xf

MAJOR_REV = 2
MINOR_REV = 1
MICRO_REV = 9
REL_LEVEL = FINAL
# at most 15 beta releases!
REL_SERIAL = 0

HEX_VERSION = ((MAJOR_REV << 24) | (MINOR_REV << 16) | (MICRO_REV << 8) |
               (REL_LEVEL << 4)  | (REL_SERIAL << 0))

# config.pck schema version number
DATA_FILE_VERSION = 97

# qfile/*.db schema version number
QFILE_SCHEMA_VERSION = 3

# version number for the lists/<listname>/pending.db file schema
PENDING_FILE_SCHEMA_VERSION = 2

# version number for the lists/<listname>/request.db file schema
REQUESTS_FILE_SCHEMA_VERSION = 1

After applying the change to the one list, I got this:

[root]# bin/withlist -l problem-list
{enter m.data_version change here....}

Loading list promed-rus (locked)
Traceback (most recent call last):
  File "./withlist", line 297, in ?
    main()
  File "./withlist", line 275, in main
    r = do_list(listname, args, func)
  File "./withlist", line 194, in do_list
    m = MailList.MailList(listname, lock=LOCK)
  File "/usr/lib/mailman/Mailman/MailList.py", line 128, in __init__
    self.Lock()
  File "/usr/lib/mailman/Mailman/MailList.py", line 165, in Lock
    self.Load()
  File "/usr/lib/mailman/Mailman/MailList.py", line 670, in Load
    self.CheckVersion(dict)
  File "/usr/lib/mailman/Mailman/MailList.py", line 723, in CheckVersion
    Update(self, stored_state)
  File "/usr/lib/mailman/Mailman/versions.py", line 51, in Update
    UpdateOldVars(l, stored_state)
  File "/usr/lib/mailman/Mailman/versions.py", line 264, in UpdateOldVars
    elif l.data_version <= 10 and l.posters:
  File "/usr/lib/mailman/Mailman/MailList.py", line 146, in __getattr__
    raise AttributeError, name
AttributeError: posters

Furthermore, even the simple URL http://list.server.tld/mailman/listinfo threw a Python error, and bin/list_lists errored as well:

[root]# bin/list_lists
Traceback (most recent call last):
  File "./list_lists", line 122, in ?
    main()
  File "./list_lists", line 94, in main
    mlist = MailList.MailList(n, lock=0)
  File "/usr/lib/mailman/Mailman/MailList.py", line 130, in __init__
    self.Load()
  File "/usr/lib/mailman/Mailman/MailList.py", line 670, in Load
    self.CheckVersion(dict)
  File "/usr/lib/mailman/Mailman/MailList.py", line 723, in CheckVersion
    Update(self, stored_state)
  File "/usr/lib/mailman/Mailman/versions.py", line 51, in Update
    UpdateOldVars(l, stored_state)
  File "/usr/lib/mailman/Mailman/versions.py", line 264, in UpdateOldVars
    elif l.data_version <= 10 and l.posters:
  File "/usr/lib/mailman/Mailman/MailList.py", line 146, in __getattr__
    raise AttributeError, name
AttributeError: posters

So, I decided to try and get back to a working state by deleting the problem list with bin/rmlist problem-list.  THIS WORKED.

Then, I re-created the list with the same name, knowing that the archives would be re-linked, but of course, the settings would need to be imported.  What I then did was to diff the settings from the broken list (which I can get them from the restored snapshot of the server before it was updated) with the current settings for a different, completely working list.  I saw only two sections that were different, and to my surprise, the WORKING list has the dmarc references and the BROKEN list does not.  So, I inserted two sections (below) into the output of config_list -o of the BROKEN list.

# Several protocols now in wide use attempt to ensure that use of the
# domain in the author's address (ie, in the From: header field) is
# authorized by that domain.  These protocols may be incompatible with
# common list features such as footers, causing participating email
# services to bounce list traffic merely because of the address in the
# From: field.  This has resulted in members being unsubscribed despite
# being perfectly able to receive mail.
#
# The following actions are applied to all list messages when selected
# here.  To apply these actions only to messages where the domain in the
# From: header is determined to use such a protocol, see the <a
# href="?VARHELP=privacy/sender/dmarc_moderation_action">
# dmarc_moderation_action settings under Privacy options... -> Sender
# filters. Settings:
#
# No Do nothing special.  This is appropriate for anonymous lists. It is
# appropriate for dedicated announcement lists, unless the From: address
# of authorized posters might be in a domain with a DMARC or similar
# policy. It is also appropriate if you choose to use
# dmarc_moderation_action other than Accept for this list. Munge From
# This action replaces the poster's address in the From: header with the
# list's posting address and adds the poster's address to the addresses
# in the original Reply-To: header. Wrap Message Just wrap the message
# in an outer message with the From: header containing the list's
# posting address and with the original From: address added to the
# addresses in the original Reply-To: header and with Content-Type:
# message/rfc822.  This is effectively a one message MIME format digest.
#
# The transformations for anonymous_list are applied before any of these
# actions. It is not useful to apply actions other than No to an
# anonymous list, and if you do so, the result may be surprising. The
# Reply-To: header munging actions below interact with these actions as
# follows:
#  first_strip_reply_to = Yes will remove all the incoming
# Reply-To: addresses but will still add the poster's address to
# Reply-To: for all three settings of reply_goes_to_list which
# respectively will result in just the poster's address, the poster's
# address and the list posting address or the poster's address and the
# explicit reply_to_address in the outgoing Reply-To: header. If
# first_strip_reply_to = No the poster's address in the original From:
# header, if not already included in the Reply-To:, will be added to any
# existing Reply-To: address(es). These actions, whether selected here
# or via <a href="?VARHELP=privacy/sender/dmarc_moderation_action">
# dmarc_moderation_action, do not apply to messages in digests or
# archives or sent to usenet via the Mail<->News gateways. If <a
# href="?VARHELP=privacy/sender/dmarc_moderation_action">
# dmarc_moderation_action applies to this message with an action other
# than Accept, that action rather than this is applied
#
# legal values are:
#    0 = "No"
#    1 = "Munge From"
#    2 = "Wrap Message"
from_is_list = 0

and

# Munge From -- applies the <a
# href="?VARHELP=general/from_is_list">from_is_list Munge From
# transformation to these messages.
#
# Wrap Message -- applies the <a
# href="?VARHELP=general/from_is_list">from_is_list Wrap Message
# transformation to these messages.
#
# Reject -- this automatically rejects the message by sending a bounce
# notice to the post's author.  The text of the bounce notice can be <a
# href="?VARHELP=privacy/sender/dmarc_moderation_notice" >configured by
# you.
#
# Discard -- this simply discards the message, with no notice sent to
# the post's author.
#
#
# This setting takes precedence over the <a
# href="?VARHELP=general/from_is_list"> from_is_list setting if the
# message is From: an affected domain and the setting is other than
# Accept.
#
# legal values are:
#    0 = "Accept"
#    1 = "Munge From"
#    2 = "Wrap Message"
#    3 = "Reject"
#    4 = "Discard"
dmarc_moderation_action = 0

# No -- this applies dmarc_moderation_action to only those posts From: a
# domain with DMARC p=reject.  This is appropriate if you are concerned
# about bounced messages, but want to apply dmarc_moderation_action to
# as few messages as possible. Yes -- this applies
# dmarc_moderation_action to posts From: a domain with DMARC p=reject or
# p=quarantine. If a message is From: a domain with DMARC p=quarantine
# and dmarc_moderation_action is not applied (this set to No) the
# message will likely not bounce, but will be delivered to recipients'
# spam folders or other hard to find places.
#
# legal values are:
#    0 = "No"
#    1 = "Yes"
dmarc_quarantine_moderation_action = True

# Text to include in any <a
# href="?VARHELP=privacy/sender/dmarc_moderation_action" >rejection
# notice to be sent to anyone who posts to this list from a domain with
# a DMARC Reject/Quarantine Policy.
dmarc_moderation_notice = ''

I then used
bin/config_list -v -i /path-to-folder/updated_settings problem-list -v

and got a result with NO ERRORS!  (minor Hooray here!) 

When I try to access the list, the http://mailman.server.tld/mailman/admin/promed-mod/general page loads just fine and the list would seem to have the appearance of working.  But, of course, the subscriber list and their individual settings are completely missing.  (not a surprise, once I thought about it.)

So, I did a slightly rude restoration of the list members by overwriting
/var/lib/mailman/lists/broken-list/config.pck

with a copy of the same file from before the damage occurred.  I made sure mailman was stopped, and manually corrected file permissions before restarting mailman.

And, just like that, my problems have been restored, along with the subscribers.

So, it would appear that something within /lib/mailman/lists/problem-list/config.pck is the problem child.  I'd be happy to remove and re-create the list with good settings.  But I _must_ have the subscribers including their bounce status, digest-status, the real-names they've entered through the Mailman confirmation page, etc.etc.  Is there a way to do this in Mailman 2.1.9?  (Remembering that I have a working version of the list on a restored snapshot of the server, where everything actually does work.)

Thanks Again,
Drew Tenenholz