[Tutor] sockets, files, threads

[Marilyn Davis]

On Wed, 12 Jan 2005, Danny Yoo wrote:

Thank you so much for thinking about this Danny.

> > When stuff was read from the exim socket, it was stored in a tempfile,
> > so that I could release the exim process, then I lseek to the front of
> > the tempfile and have it handy.  I see from all my debugging and logging
> > that the file descriptor for this tempfile is 9.
> 
> Hi Marilyn,
> 
> Question: do you really need to use a tempfile?  If your daemon is
> persistent in memory, can it just write to a StringIO object?

Mail messages can be really big.  I was thinking that I'd be better
off if I put them on the disk.  Am I wrong?

Are we suspecting tempfile?  Might it have a shared resource?

> StringIO.StringIO elements act like files too, and may be easier to
> maintain than a tempfile.TemporaryFile.  If you can show us how you're
> constructing and using the TemporaryFile, we can try to trace any
> problematic usage.

OK.  I'll try to cut-and-paste together something tomorrow.  I have to
quit now.

> > The program then opens a pipe to exim to send mail.  I see that the
> > popen2.popen3 call returns 9 for the stdin file descriptor for the pipe.
> >
> > The tempfile (also file descriptor 9) is read for piping into exim and
> > it errors with "Bad file descriptor".
> 
> Oh!  This situation sounds like the 'tempfile' is being closed at some
> point, releasing the file descriptor back to the system.  If that is what
> is happening, then that's why the pipe has the same descriptor id: the
> call to pipe() just reuses a free file descriptor.

Yes.  So in my logfile, I log every creation, opening and closing of
everything and every write and read and seek.  And I just don't see
it.

> 
> I'd look for places where the tempfile might be close()d.  I'd also look
> for places where your reference to tempfile is reassigned, since that

Hmmmm.  I'll look for reassignments.  This is likely.  You mean the
tempfile would think its time to close if it got reassigned?

> would also signal a resource collection.
> 

> 
> How do you deal with threads?  Is the temporary file a global resource
> that the threads all touch?  If so, have you done any synchronization to

No.  Only the thread that creates it reads it.

> make sure that at most one thread can touch the temporary file at a time?
> What are the shared resources for the threads?

I can't think of any -- unless there are hidden things in os.popen? or
tempfile.mkstemp or open.

They do share a logfile.  But each entry opens it, appends it, and
closes it.

My threads seem to me to be very independent.

> 
> 
> The situation you mentioned,
> 
> > > Worse yet, the first 5 messages of my test go through the entire
> > > process without a problem, and then # 6 hits this -- but only if # 1
> > > message is really big.
> 
> is exactly the sort of thing I'd expect if two threads were contending for
> the same resource, so let's see if the bug has to do with this.

"Resource".  That's a file, a socket, a pipe, what else?

Well, I'll sleep and do some code review with your questions in mind.

> 
> Best of wishes to you!

Thank you!

I was looking at my use of file objects and file descriptors and I
wrote this sample program and was very surprised by the result --
which makes me think there's something here that I don't understand.
Where did my 'ooo' go?

#! /usr/bin/env python
import os

fobj = open('/tmp/xxx','w')
fobj.write('ooo\n')
fp = fobj.fileno()
os.write(fp,'x\n')
os.close(fp)
##########
#  OUTPUT:
[root at maildance tmp]# ./file.py
[root at maildance tmp]# cat xxx
x
[root at maildance tmp]# 

Thank you so much for your questions and thoughts.  We'll see what the
morning brings.

Marilyn