[Mailman-Users] What happened to my archive? Why isn't the archive process running?

Wed Oct 29 14:57:39 CET 2008

I'm sorry to turn up in your support forum again, Mark. I've fought hard
here to try to avoid that. The truth is it was our struggle with trying to
configure my old server for mailman back in June and July that proved to be
the straw that broke the camel's back with my old dedicated server hosting
service. In early July I asked them about getting an updated server and
software but the price they quoted was so darn high I decided to bail out on
them and ended up choosing a hosting company that provided much more server
for the $ plus the latest version of Debian Etch (rather than the 5 year old
version of RedHat I was running) for the same price I'd been paying to my
old hosting service. The BIG difference was I had to take full admin
responsibility for my server setup and configuration; but I figured what the
hell, I'm already doing 80% of that job anyway with very little support
coming from my old host. 

So, when my old server came up for renewal at the end of July, I went
month-to-month on my lease with them and bought the new server to take its
place. I struggled through server setup and the migration of all my existing
sites from the old server in August and September. At the end of September
with just 3 sites left to move I grabbed the last 3 sites, made my final
backups, and pulled the plug on the old server on the weekend before it
would have renewed for another month... jumping out the window (from what
turned out to be the 12th floor) with my final backups under my arm. :-)

During October, I struggled to get the server set up to support mailman for
multiple accounts. That entailed installing and testing it for the current
client and setting up the server to support virtual domain hosting under
Apache2, postfix and mailman so that we can eventually support mailman for
multiple domains on this server. Until you mentioned suEXEC in your message
yesterday, I'd nearly forgotten that part of our June-July nightmare. 

When I checked this morning, I found that when Apache2 was installed on this
server it's standard installation process did include suEXEC. However, under
Debian's Apache2 setup, it's easy to disable suEXEC and restart Apache. As
far as I know, nothing else on the server was relying on the presence of
suEXEC and it certainly wasn't my intent to install that Apache feature to
begin with. So, suEXEC has now been disabled. 

Now can we talk about the proper ownership of all the mailman files both IN
/usr/local/mailman directory structure and in the UserAccount/mailman
directory as well?

Or should I just put take a large dose of cyanide and go take a long nap
instead?

Thanks!

-----Original Message-----
From: Mark Sapiro [mailto:mark at msapiro.net] 
Sent: Tuesday, October 28, 2008 3:21 PM
To: webwitchcraft at webwitchcraft.com; mailman-users at python.org
Subject: RE: [Mailman-Users] What happened to my archive? Why isn't the
archive process running?

TGPlatt, WebMaster wrote:
>
>Could this be a group/owner-ship issue?

Yes. It took me a while, but I finally connected you with our exchanges
from last June/July :)

>It hadn't occurred to me to look for mailman's error log. Plus I had no
idea
>where it was. I've seen it mentioned but I'd seen nothing in the docs that
>said where to find it. But with a bit of looking, I found it in
>/usr/local/mailman/logs/error
>
> 
>
>The error log that was saved when we made our final September 28 backup on
>the old server was last updated 9/25. The more I look the more this looks
>like an ownership issue to me. It may be I screwed up somewhere back in
>July. In our current mailman directory structure a smattering of files
>throughout the mailman directory tree seem to be owned by root / mailman
>now; whereas in the old backup everything seems to have been owned by
>mailman / mailman or www-data (Debian's default Apache user) / mailman. On
>9/28, the error log was owned by mailman / mailman. Indeed everything
except
>mischief, subscribed and vet were owned by mailman / mailman back then.
>Those three files were different and were owned by "www-data" / mailman.
>Today in our running copy of mailman, all logs are owned by mailman/mailman
>except error which is owned by root / mailman and mischief, vet and
>subscribed which are owned by www-data / mailman.

There were undoubtedly problems on the old server because of SUExec
issues which I told you at the time was incompatible with Mailman's
security model since Mailman's CGI wrappers can't be SETGID under
SUExec and that's the whole point of the wrappers in the first place.

So ownership and permissions within Mailman have to be such that the
SUExec user can read and write.

But the qrunner processes also have to be able to read and write and
they will run as the mailman user:group.

Normally the owner doesn't matter because everything runs as group
mailman, but that may not be the case here.

>The reason I think this is an ownership issue is because when I look at the
>July - September error log I see lots of errors like this:
>
> 
>
>Jul 09 07:29:22 2008 (10592) Archive file access failure:
>
>        /usr/local/mailman/archives/private/ourlist.mbox/ourlist.mbox
[Errno
>13] Permission denied:
>'/usr/local/mailman/archives/private/ourlist.mbox/ourlist.mbox'
>
>Jul 09 07:29:22 2008 (10592) Uncaught runner exception: [Errno 13]
>Permission denied: '/usr/local/mailman/archives/private/ourlist
>
>.mbox/ourlist.mbox'
>
> 
>
>.and when I check the files in that directory, they're owned by root as a
>member of the group mailman.

That should be OK because ArchRunner should be running as group mailman
and the files shoud be group writable.

<snip>
>
>Sep 25 21:09:31 2008 (16599) SHUNTING:
>1222394969.723012+6122e5f84bb97875a24698a159961936bed8802b
>
> 
>
>Sadly, when I look at the qfiles/shunt directory in the 9/28 backup, the
>oldest file there seems to be from 09-20-2008. So it looks to me like it
was
>only keeping those shunt files 5 or 6 days before discarding them.

If you are running Mailman 2.1.11, there is a cron that runs daily and
by default it discards anything in qfiles/bad and qfiles/shunt older
than 7 days. From Defaults.py

# The length of time after which a qfiles/bad or qfiles/shunt file is
# considered to be stale.  Set to zero to disable culling of qfiles/bad
and
# qfiles/shunt entries.
BAD_SHUNT_STALE_AFTER = days(7)

# The pathname of a directory (searchable and writable by the Mailman
cron
# user) to which the culled qfiles/bad and qfiles/shunt entries will be
# moved.  Set to None to simply delete the culled entries.
BAD_SHUNT_ARCHIVE_DIRECTORY = None

>Here are the first and last error in the current error file:
>
> 
>
>Oct 18 10:43:54 2008 mailmanctl(25124): Site list is missing: mailman
>
>Oct 18 10:43:54 2008 (25124) Site list is missing: mailman
>
>Oct 18 18:38:42 2008 (14433) admin.py access for non-existent list: ourlist
>
>Oct 18 18:39:34 2008 (14458) admin.py access for non-existent list: ourlist
>
>Oct 18 20:34:08 2008 (19034) admin.py access for non-existent list: ourlist
>
>Oct 20 08:58:31 2008 (9822) admin.py access for non-existent list: ourlist 
>
>Oct 21 22:33:46 2008 (1431) Uncaught runner exception: [Errno 13]
Permission
>den
>
>ied: '/usr/local/mailman/archives/private/ourlist/index.html'
>
>Oct 21 22:33:46 2008 (1431) Traceback (most recent call last):
>
>  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop
>
>    self._onefile(msg, msgdata)
>
> 
>
>...
>
> 
>
>Oct 28 10:09:30 2008 (2589) Uncaught runner exception: [Errno 13]
Permission
>den
>
>ied: '/usr/local/mailman/archives/private/ourlist/index.html'
>
>Oct 28 10:09:30 2008 (2589) Traceback (most recent call last):
>
>  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop
>
>    self._onefile(msg, msgdata)
>
>  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 191, in _onefile
>
>    keepqueued = self._dispose(mlist, msg, msgdata)
>
>  File "/usr/local/mailman/Mailman/Queue/ArchRunner.py", line 73, in
>_dispose
>
>    mlist.ArchiveMail(msg)
>
>  File "/usr/local/mailman/Mailman/Archiver/Archiver.py", line 217, in
>ArchiveMa
>
>il
>
>    h.close()
>
>  File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 324, in
>close
>
>    self.write_TOC()
>
>  File "/usr/local/mailman/Mailman/Archiver/HyperArch.py", line 1097, in
>write_T
>
>OC
>
>    toc = open(os.path.join(self.basedir, 'index.html'), 'w')
>
>IOError: [Errno 13] Permission denied:
>'/usr/local/mailman/archives/private/ourlist/index.html'
>
> 
>
>Oct 28 10:09:30 2008 (2589) SHUNTING:
>1225213769.6014299+1a6c02d7003773ec8b731f9
>
>23e4d1e0eb795d0b5

So you still have permissions issues.

You could start with bin/check_perms which should get them right except
if the web server is still SUExec, the web interface may not work.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan