[Mailman-Users] Strange errors
Dan Szkola
szkola at tanis.cso.niu.edu
Fri Oct 21 22:40:12 CEST 2005
Mark Sapiro wrote:
> Dan Szkola wrote:
>
>
>>OK, I found something extremely odd. For some reason,
>>the Utils.pyc in /usr/local/mailman/Mailman/Logging
>>direcotry is being removed and then the compile of it
>>seems to fail and I'm left with a 0 byte file. It is
>>owned by the daemon user and later it gets compiled
>>and is owned by the mailman user.
> Presumably daemon is the user that sendmail uses to invoke the wrapper.
> When you tried invoking the wrapper manually and did not get the
> error, were you running it as the daemon user? If not, you might try
> that.
Did that, same thing:
$ id
uid=1(daemon) gid=1(other)
$ /usr/local/mailman/mail/mailman admin esstest </var/tmp/testmessage
HZ
TERM vt100
SHELL /usr/bin/sh
TZ US/Central
PYTHONPATH /usr/local/mailman
LOGNAME daemon
MAIL /var/mail/daemon
HOME /
before = /usr/local/mailman/scripts
before = /usr/local/mailman
before = /usr/local/lib/python24.zip
before = /usr/local/lib/python2.4/
before = /usr/local/lib/python2.4/plat-sunos5
before = /usr/local/lib/python2.4/lib-tk
before = /usr/local/lib/python2.4/lib-dynload
after = /usr/local/mailman/pythonlib
after = /usr/local/mailman
after = /usr/local/mailman/scripts
after = /usr/local/mailman
after = /usr/local/lib/python24.zip
after = /usr/local/lib/python2.4/
after = /usr/local/lib/python2.4/plat-sunos5
after = /usr/local/lib/python2.4/lib-tk
after = /usr/local/lib/python2.4/lib-dynload
after = /usr/local/lib/python2.4/site-packages
> The recompiling is strange in itself. Normally, if the .pyc is more
> recent than the .py, accessible and not corrupt, it is just used, so
> once you have a good one, why is python trying to recompile it? And if
> Python is recompiling this module when invoked in the 'odd' way, is it
> also doing others, and why does this cause a problem (if it is a
> cause)?
Very odd, I agree. A truss of the persistent queue runner that handled
one of the test mails shows this (12762 is the pid that mailman gets
when sendmail exec's it):
12762: open64("/usr/local/mailman/Mailman/Logging/Utils.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/Mailman/Logging/Utilsmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/Mailman/Logging/Utils.py",
O_RDONLY) = 66
12762: fstat64(66, 0xFFBF8928) = 0
12762: open64("/usr/local/mailman/Mailman/Logging/Utils.pyc",
O_RDONLY) = 256
12762: close(256) = 0
12762: fstat64(66, 0xFFBF83C8) = 0
12762: fstat64(66, 0xFFBF8270) = 0
12762: ioctl(66, TCGETA, 0xFFBF8354) Err#25 ENOTTY
12762: read(66, " # C o p y r i g h t ".., 8192) = 1912
12762: read(66, 0x001F902C, 8192) = 0
12762: unlink("/usr/local/mailman/Mailman/Logging/Utils.pyc") = 0
12762: open64("/usr/local/mailman/Mailman/Logging/Utils.pyc",
O_WRONLY|O_CREAT|O_TRUNC|O_EXCL, 0666) = 256
12762: fcntl(256, F_GETFD, 0xFEFE7F18) = 0
12762: stat64("/usr/local/mailman/Mailman/Logging/traceback",
0xFFBF7AD0) Err#2 ENOENT
12762: open64("/usr/local/mailman/Mailman/Logging/traceback.so",
O_RDONLY) Err#2 ENOENT
12762:
open64("/usr/local/mailman/Mailman/Logging/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/Mailman/Logging/traceback.py",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/Mailman/Logging/traceback.pyc",
O_RDONLY) Err#2 ENOENT
12762: stat64("/usr/local/mailman/pythonlib/traceback", 0xFFBF7AD0)
Err#2 ENOENT
12762: open64("/usr/local/mailman/pythonlib/traceback.so", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/mailman/pythonlib/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/pythonlib/traceback.py", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/mailman/pythonlib/traceback.pyc", O_RDONLY)
Err#2 ENOENT
12762: stat64("/usr/local/mailman/traceback", 0xFFBF7AD0) Err#2 ENOENT
12762: open64("/usr/local/mailman/traceback.so", O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/tracebackmodule.so", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/mailman/traceback.py", O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/traceback.pyc", O_RDONLY) Err#2 ENOENT
12762: stat64("/usr/local/mailman/scripts/traceback", 0xFFBF7AD0)
Err#2 ENOENT
12762: open64("/usr/local/mailman/scripts/traceback.so", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/mailman/scripts/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/scripts/traceback.py", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/mailman/scripts/traceback.pyc", O_RDONLY)
Err#2 ENOENT
12762: stat64("/usr/local/mailman/traceback", 0xFFBF7AD0) Err#2 ENOENT
12762: open64("/usr/local/mailman/traceback.so", O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/tracebackmodule.so", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/mailman/traceback.py", O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/mailman/traceback.pyc", O_RDONLY) Err#2 ENOENT
12762: stat64("/usr/local/lib/python24.zip/traceback", 0xFFBF7AD0)
Err#2 ENOENT
12762: open64("/usr/local/lib/python24.zip/traceback.so", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/lib/python24.zip/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python24.zip/traceback.py", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/lib/python24.zip/traceback.pyc", O_RDONLY)
Err#2 ENOENT
12762: stat64("/usr/local/lib/python2.4/traceback", 0xFFBF7AD0) Err#2
ENOENT
12762: open64("/usr/local/lib/python2.4/traceback.so", O_RDONLY)
Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/traceback.py", O_RDONLY) = 257
12762: close(257) = 0
12762: open64("/usr/local/lib/python2.4/traceback.pyc", O_RDONLY) = 257
12762: close(257) = 0
12762: stat64("/usr/local/lib/python2.4/plat-sunos5/traceback",
0xFFBF7AD0) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/plat-sunos5/traceback.so",
O_RDONLY) Err#2 ENOENT
12762:
open64("/usr/local/lib/python2.4/plat-sunos5/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/plat-sunos5/traceback.py",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/plat-sunos5/traceback.pyc",
O_RDONLY) Err#2 ENOENT
12762: stat64("/usr/local/lib/python2.4/lib-tk/traceback",
0xFFBF7AD0) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/lib-tk/traceback.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/lib-tk/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/lib-tk/traceback.py",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/lib-tk/traceback.pyc",
O_RDONLY) Err#2 ENOENT
12762: stat64("/usr/local/lib/python2.4/lib-dynload/traceback",
0xFFBF7AD0) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/lib-dynload/traceback.so",
O_RDONLY) Err#2 ENOENT
12762:
open64("/usr/local/lib/python2.4/lib-dynload/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/lib-dynload/traceback.py",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/lib-dynload/traceback.pyc",
O_RDONLY) Err#2 ENOENT
12762: stat64("/usr/local/lib/python2.4/site-packages/traceback",
0xFFBF7AD0) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/site-packages/traceback.so",
O_RDONLY) Err#2 ENOENT
12762:
open64("/usr/local/lib/python2.4/site-packages/tracebackmodule.so",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/site-packages/traceback.py",
O_RDONLY) Err#2 ENOENT
12762: open64("/usr/local/lib/python2.4/site-packages/traceback.pyc",
O_RDONLY) Err#2 ENOENT
You can see it actually unlink the compiled version and then look
for, find, and seemingly reject the traceback.py and traceback.pyc
that it finds. I thought it may be a too many open files problem
or a file descriptor limit problem because the FD returned on the
open64("/usr/local/mailman/Mailman/Logging/Utils.pyc", O_RDONLY)
was 256. But it doesn't return an error. I did see that the ulimit
does say 256 open files is the max, but that doesn't seem to be
the problem.
> Do any other Mailman .pyc files have way more recent dates than the
> corresponding .py, or are any others owned by 'daemon'?
>
> Maybe next time try something like
>
> find /usr/local/mailman -type f -a \( -mtime 0 -o -user daemon \)
Currently, I get the following pyc files:
398722 1 -rw-r--r-- 1 daemon mailman 613 May 31 08:48
/usr/local/mailman/scripts/paths.pyc
397000 3 -rw-r--r-- 1 www mailman 2613 Oct 21 08:04
/usr/local/mailman/pythonlib/email/Encoders.pyc
398921 13 -rw-r--r-- 1 www mailman 13256 Oct 21 08:04
/usr/local/mailman/pythonlib/email/_parseaddr.pyc
398829 2 -rw-r--r-- 1 mailman mailman 1308 Oct 21 10:45
/usr/local/mailman/Mailman/Logging/Utils.pyc
> I really don't understand what's happening since sendmail should always
> be invoking the wrapper with the same user:group and that works at
> first.
>
> In order to even more closely mimic sendmail, you could try
>
> cat file | /usr/local/mailman/mail/mailman admin listname
>
> instead of the above. Beyond that, the only difference I can see is the
> environment, but we say previously, after the wrapper got done with
> the environment, all that was there was
>
> PYTHONPATH /usr/local/mailman
> AGENT sendmail
>
> and PYTHONPATH is always put there by the wrapper and presumably AGENT
> is always put there by sendmail.
>
> So, why does it only seem to fail when sendmail invokes it and only
> after some successful invocations, and why does restarting sendmail
> fix it while restarting Mailman (which will recompile
> /usr/local/mailman/Mailman/Logging/Utils.py) doesn't fix it?
Again, I thought the restart fixing it pointed to the file descriptor
issue, but it doesn't look like that's it.
In any event, I'm getting too much flak over the lists being down, so
I have done the following. I enabled smrsh and removed the "expensive"
option from it and no longer run with persistent queue runners. Our
server probably isn't busy enough to warrant them anyway. If I continue
to see the problem with the new setup, I will continue to try other
things to fix it.
>
> Maybe we need to take this to comp.lang.python.
Maybe so. If we are really interested, I can try to do the exact same
setup on another Solaris 10 server and we can work with a non-production
server that way.
--
Dan Szkola
Sr Unix Systems Programmer
Northern Illinois University
More information about the Mailman-Users
mailing list