[Mailman-Users] Mailman Borks on Large Attachments
Lachlan Michael
lachlan at lkla.org
Mon Feb 4 08:59:38 CET 2008
Dear Mailman List,
I have a problem when using mailman. The problem manifests itself when a
user sends a largish attachment (say 300kB) such as a digital camera
picture to a list. The lists I run are small private groups and if
possible I would like to allow users to send these kind of attachments.
The server is question is vanilla FreeBSD 6.3 machine, using mailman
2.1.9_5 from ports with sendmail. The machine has 1GB of physical memory.
Python 2.5 is used (default settings). The maximum message size for this
list was set at 150kB, in an effort to stop this problem (which it didn't
seem to do), but I would like to set this limit higher (maybe to 1MB).
> uname -a
FreeBSD 6.3-PRERELEASE FreeBSD 6.3-PRERELEASE #0: Wed Jan 2 18:46:42 JST
2008
> pkg_info | grep mailman
mailman-2.1.9_5 A mailing list manager (MLM) with a user-friendly web
front
> pkg_info | grep python
python25-2.5.1_1 An interpreted object-oriented programming language
Mailman fails with a "memory error" (see logs below).
This error then caused ALL mail to ALL other lists on the same machine to
be stop being processed, although they were queued and could be sent
later. Eventually some days later I realized that mail was not getting
through and investigated.
A simple restart of mailman did not help, and I found a suspicious file in
qfiles. The suspicious file had a .bak extension I moved it away. Then a
restart of mailman allowed all queued messages to be sent correctly (with
the exception of the suspicious mail file). Using less I saw that the mail
included a large picture file. There are 11 members in the list that this
mail was sent to. The offending mail file size is shown below (ls -lh).
-rw-rw---- 1 mailnull mailman 599K Jan 30 23:36
1201703766.518079+a5beef9fbd8e4eea60ab77049b3bed4e877a98fc.bak
My questions is, is this a bug, a mis-configuration on my part, or
expected behaviour of mailman under these conditions for this machine
(i.e. am I hitting python or FreeBSD limits)?
My expectation would be that
1) the user gets an error that the file sent was too big
2) all other lists and other mail are not affected
I guess many people would be jumping up and down if this occurred
regularly, so I imagine that this a local configuration problem, but I
would appreciate if anybody had any advice for debugging or changing my
environment to remove this error.
Regards,
Lachlan
Error Logs (/usr/local/mailman/logs/error)
==========================================
Original Error (repeated several times)
Jan 30 23:36:06 2008 qrunner(1150): Traceback (most recent call last):
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/bin/qrunner", line 278, in <module>
Jan 30 23:36:06 2008 qrunner(1150): main()
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/bin/qrunner", line 238, in main
Jan 30 23:36:06 2008 qrunner(1150): qrunner.run()
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/Mailman/Queue/Runner.py", line 71, in run
Jan 30 23:36:06 2008 qrunner(1150): filecnt = self._oneloop()
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/Mailman/Queue/Runner.py", line 100, in _oneloop
Jan 30 23:36:06 2008 qrunner(1150): msg, msgdata =
self._switchboard.dequeue(filebase)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/Mailman/Queue/Switchboard.py", line 164, in dequeue
Jan 30 23:36:06 2008 qrunner(1150): msg =
email.message_from_string(msg, Message.Message)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/__init__.py", line 51, in
message_from_string
Jan 30 23:36:06 2008 qrunner(1150): return Parser(_class,
strict=strict).parsestr(s)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 75, in parsestr
Jan 30 23:36:06 2008 qrunner(1150): return self.parse(StringIO(text),
headersonly=headersonly)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:06 2008 qrunner(1150): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 240, in _parsebody
Jan 30 23:36:06 2008 qrunner(1150): msgobj = self.parsestr(part)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 75, in parsestr
Jan 30 23:36:06 2008 qrunner(1150): return self.parse(StringIO(text),
headersonly=headersonly)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:06 2008 qrunner(1150): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 240, in _parsebody
Jan 30 23:36:06 2008 qrunner(1150): msgobj = self.parsestr(part)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 75, in parsestr
Jan 30 23:36:06 2008 qrunner(1150): return self.parse(StringIO(text),
headersonly=headersonly)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:06 2008 qrunner(1150): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 265, in _parsebody
Jan 30 23:36:06 2008 qrunner(1150): msg = self.parse(fp)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:06 2008 qrunner(1150): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 240, in _parsebody
Jan 30 23:36:06 2008 qrunner(1150): msgobj = self.parsestr(part)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 75, in parsestr
Jan 30 23:36:06 2008 qrunner(1150): return self.parse(StringIO(text),
headersonly=headersonly)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:06 2008 qrunner(1150): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:06 2008 qrunner(1150): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 218, in _parsebody
Jan 30 23:36:06 2008 qrunner(1150): payload[start:terminator])
Jan 30 23:36:06 2008 qrunner(1150): MemoryError
Jan 30 23:36:07 2008 qrunner(41406): Traceback (most recent call last):
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/bin/qrunner", line 278, in <module>
Jan 30 23:36:07 2008 qrunner(41406): main()
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/bin/qrunner", line 238, in main
Jan 30 23:36:07 2008 qrunner(41406): qrunner.run()
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/Mailman/Queue/Runner.py", line 71, in run
Jan 30 23:36:07 2008 qrunner(41406): filecnt = self._oneloop()
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/Mailman/Queue/Runner.py", line 100, in _oneloop
Jan 30 23:36:07 2008 qrunner(41406): msg, msgdata =
self._switchboard.dequeue(filebase)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/Mailman/Queue/Switchboard.py", line 164, in dequeue
Jan 30 23:36:07 2008 qrunner(41406): msg =
email.message_from_string(msg, Message.Message)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/__init__.py", line 51, in
message_from_string
Jan 30 23:36:07 2008 qrunner(41406): return Parser(_class,
strict=strict).parsestr(s)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 75, in parsestr
Jan 30 23:36:07 2008 qrunner(41406): return
self.parse(StringIO(text), headersonly=headersonly)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:07 2008 qrunner(41406): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 240, in _parsebody
Jan 30 23:36:07 2008 qrunner(41406): msgobj = self.parsestr(part)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 75, in parsestr
Jan 30 23:36:07 2008 qrunner(41406): return
self.parse(StringIO(text), headersonly=headersonly)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:07 2008 qrunner(41406): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 240, in _parsebody
Jan 30 23:36:07 2008 qrunner(41406): msgobj = self.parsestr(part)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 75, in parsestr
Jan 30 23:36:07 2008 qrunner(41406): return
self.parse(StringIO(text), headersonly=headersonly)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:07 2008 qrunner(41406): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 265, in _parsebody
Jan 30 23:36:07 2008 qrunner(41406): msg = self.parse(fp)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:07 2008 qrunner(41406): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 240, in _parsebody
Jan 30 23:36:07 2008 qrunner(41406): msgobj = self.parsestr(part)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 75, in parsestr
Jan 30 23:36:07 2008 qrunner(41406): return
self.parse(StringIO(text), headersonly=headersonly)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 64, in parse
Jan 30 23:36:07 2008 qrunner(41406): self._parsebody(root, fp,
firstbodyline)
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/mailman/pythonlib/email/Parser.py", line 218, in _parsebody
Jan 30 23:36:07 2008 qrunner(41406): payload[start:terminator])
Jan 30 23:36:07 2008 qrunner(41406): File
"/usr/local/lib/python2.5/re.py", line 157, in split
Jan 30 23:36:07 2008 qrunner(41406): return _compile(pattern,
0).split(string, maxsplit)
Jan 30 23:36:07 2008 qrunner(41406): MemoryError
After a simple restart of Mailman several days later when investigating
why mail had stopped, the following error appeared in the logs several
times.
Feb 03 08:26:08 2008 (33316) Uncaught runner exception: out of memory
Feb 03 08:26:08 2008 (33316) Traceback (most recent call last):
File "/usr/local/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop
self._onefile(msg, msgdata)
File "/usr/local/mailman/Mailman/Queue/Runner.py", line 170, in _onefile
keepqueued = self._dispose(mlist, msg, msgdata)
File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in
_dispose
more = self._dopipeline(mlist, msg, msgdata, pipeline)
File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in
_dopipeline
sys.modules[modname].process(mlist, msg, msgdata)
File "/usr/local/mailman/Mailman/Handlers/SpamDetect.py", line 111, in
process
g.flatten(p)
File "/usr/local/mailman/pythonlib/email/Generator.py", line 101, in
flatten
self._write(msg)
File "/usr/local/mailman/pythonlib/email/Generator.py", line 129, in _write
self._dispatch(msg)
File "/usr/local/mailman/pythonlib/email/Generator.py", line 155, in
_dispatch
meth(msg)
File "/usr/local/mailman/pythonlib/email/Generator.py", line 226, in
_handle_multipart
g.flatten(part, unixfrom=False)
File "/usr/local/mailman/pythonlib/email/Generator.py", line 101, in
flatten
self._write(msg)
File "/usr/local/mailman/pythonlib/email/Generator.py", line 129, in _write
self._dispatch(msg)
File "/usr/local/mailman/pythonlib/email/Generator.py", line 155, in
_dispatch
meth(msg)
File "/usr/local/mailman/pythonlib/email/Generator.py", line 258, in
_handle_multipart
print >> self._fp, '\n--' + boundary + '--',
MemoryError: out of memory
Finally, the qrunner log suggests that it gave up after several re-tries.
Error Log (/usr/local/mailman/logs/qrunner)
============================================
Jan 30 23:36:06 2008 (1146) Master qrunner detected subprocess exit
(pid: 1150, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:07 2008 (41406) IncomingRunner qrunner started.
Jan 30 23:36:07 2008 (1146) Master qrunner detected subprocess exit
(pid: 41406, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:07 2008 (41407) IncomingRunner qrunner started.
Jan 30 23:36:07 2008 (1146) Master qrunner detected subprocess exit
(pid: 41407, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:07 2008 (41408) IncomingRunner qrunner started.
Jan 30 23:36:07 2008 (1146) Master qrunner detected subprocess exit
(pid: 41408, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:07 2008 (41409) IncomingRunner qrunner started.
Jan 30 23:36:07 2008 (1146) Master qrunner detected subprocess exit
(pid: 41409, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:08 2008 (41410) IncomingRunner qrunner started.
Jan 30 23:36:08 2008 (1146) Master qrunner detected subprocess exit
(pid: 41410, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:08 2008 (41411) IncomingRunner qrunner started.
Jan 30 23:36:08 2008 (1146) Master qrunner detected subprocess exit
(pid: 41411, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:08 2008 (41412) IncomingRunner qrunner started.
Jan 30 23:36:08 2008 (1146) Master qrunner detected subprocess exit
(pid: 41412, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:08 2008 (41413) IncomingRunner qrunner started.
Jan 30 23:36:08 2008 (1146) Master qrunner detected subprocess exit
(pid: 41413, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:08 2008 (41414) IncomingRunner qrunner started.
Jan 30 23:36:09 2008 (1146) Master qrunner detected subprocess exit
(pid: 41414, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:09 2008 (41415) IncomingRunner qrunner started.
Jan 30 23:36:09 2008 (1146) Master qrunner detected subprocess exit
(pid: 41415, sig: None, sts: 1, class: IncomingRunner, slice: 1/1)
[restarting]
Jan 30 23:36:09 2008 (1146) Qrunner IncomingRunner reached maximum restart
limit of 10, not restarting.
More information about the Mailman-Users
mailing list