HowTo send emails reliably in a Python multi-threading application?

[Gernot Hillier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi!

I'm the developer of a Linux ISDN application which uses embedded Python for 
controlling the communication. It starts several threads (i.e. one for each 
incoming call and for outgoing faxes) which run Python scripts in embedded 
Python interpreters which in turn do the real communication stuff. Any 
incoming data or confirmations of done jobs are sent via Email.

Most real hard problems I had in the last year were either related to sending 
emails or calling external tools via Pipes. I'll now try to roughly describe 
the structure of CapiSuite and the problems I experienced. I'll try to do it 
as shortly but understandably as possible. As all of it experiences 
"sometimes" I can't provide small, clear test cases, sorry...   :-((

All of the stuff coming now boils down to two questions:

1) How can one send mails reliable in a multi-threaded Python application?  
Should the email package be used? Should smtplib be used or is it better to 
call "sendmail" or "mail" directly? Should popen2 or os.system() be 
preferred?

2) How to call an external tool (or many of them) and connect them via pipes 
with each other and with the Python script in a multithreaded application? I 
read the discussion http://www.python.org/doc/current/lib/
popen2-flow-control.html several times now and think I followed the 
suggestions given there, but it seems still to cause trouble for me.

For those of you having some seconds, here's the rough description of my 
problems...

I'm using normal linuxthreads on Linux 2.4.

Currently I have two mail send functions in use:

a) sendMIMEMail which first of all does some format conversion using external 
commandline tools using popen2 and then constructs a mail using 
email.MIMEText, email.MIMEBase or email.MIMEAudio. This mail is sent via the 
"sendmail" commandline tool like this:

sendmail = popen2.Popen3("sendmail -t -f %s" % escape(mail_from))
if (sendmail.poll()!=-1):
	...
sendmail.tochild.write(msg.as_string())
sendmail.tochild.close()
sendmail.fromchild.close()
ret=sendmail.wait()

b) sendSimpleMail which creates a simple mail using email.MIMEText and then 
does the same as sendMIMEMail for sending it.

The problems I had so far:

- - when sendSimpleMail and sendMIMEMail are run nearly in parallel, then the 
system sometimes gets stuck with one thread which is showed in "ps ax" but 
does not exist according to gdb. It seems to be a try of one Python script to 
run "sendmail" - but it gets stuck either before the new process was really 
replaced by the started sendmail or after sendmail finished but before the 
thread is destroyed. This can be worked around according to one user by not 
using the mail package but sending the mail via
os.system("echo -e "+escape(text)+" | mail -s "+escape(mail_subject)+ " -r 
"+escape(mail_from)+" "+escape(mail_to)) in sendSimpleMail

- - sometimes users (mainly Red Hat users) got the error message "cannot 
unmarshal code objects in restricted execution mode" when the mail text was 
encoded to ASCII. This one was reliably worked around by calling "import 
encodings.ascii" at the begin of send*Mail.
(see http://lists.berlios.de/pipermail/capisuite-users/2003-July/000169.html 
and http://lists.berlios.de/pipermail/capisuite-users/2003-July/000175.html 
for details)

- - when a long document was converted by calling several commandline tools 
(format1->format2->format3 via 2 tools), then popen2-pipes between process 1 
and 2 were overflowed. This was worked around by using shell pipes between 
process1 + 2 (popen2.Popen3("command1 | command2"))

- - very, very seldom (1 out of 5000 received faxes), one user of me saw the 
following - at least with Python 2.2.2; I have no feedback about newer 
versions yet:
File "/usr/lib/python2.2/site-packages/cs_helpers.py", line 233, in 
sendMIMEMail
  sendmail.tochild.write(msg.as_string())
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Message.py", line 
107, in as_string
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line 
100, in flatten
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line 
135, in _write
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line 
167, in _write_headers
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line 
191, in _split_header
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line 
41, in _is8bitstring
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/encodings/__init__.py", 
line 43, in search_function
AttributeError: 'NoneType' object has no attribute 'get'



So to summarize it: my current approach to do the format conversions and the 
mail sending seems to be quite unstable, but I have no idea how to do it 
better. But I'm quite sure there must be a reliable way to do these kinds of 
things in Python. :-)) Perhaps there's also a general problem with my Python 
embedding stuff - but as everything else works very nice, I doubt this is the 
case...

I'll again have to excuse for the rough problem descriptions I can give yet 
(and the complete miss of small test cases but it's really hard to debug...), 
but I hope some of you could comment on my issues nevertheless. Any hint is 
greatly appreciated!!!

TIA!!!

- -- 
Bye,

Gernot
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQFAN01Ok997/GGeSeIRAiALAJ9lGCv2RnbZpgklXx/mPYjpd5M+DACdHJsE
/RSiWpe7mU4u9Lp9qsKp2CQ=
=tnPN
-----END PGP SIGNATURE-----