Threads and socket.setdefaulttimeout

[Steve Holden]

Russell Warren wrote:
> It appears that the timeout setting is contained within a process
> (thanks for the confirmation), but I've realized that the function
> doesn't play friendly with threads.  If I have multiple threads using
> sockets and one (or more) is using timeouts, one thread affects the
> other and you get unpredictable behavior sometimes.  I included a short
> script at the end of this that demonstrates the threading problem.
> 
When you say "one thread affects another", I see that your example uses 
the same function for both threads. IMHO it's much better to override 
the thread's run() method than to provide a callable at thread creating 
time. That way you can be sure each thread's execution is firmly in the 
context of the particular thread instance's namespace.

having said all this, I don't think that's your issue.

> I'm trying to get around this by forcing all locations that want to set
> a timeout to use a 'safe' call immediately prior to socket creation
> that locks out setting the timeout again until the lock is released.
> Something like this:
> 
> try:
>   SafeSetSocketTimeout(Timeout_s)
>   #lock currently acquired to prevent other threads sneaking in here
>   CreateSocket()
> finally:
>   ReleaseSocketTimeoutSettingLock()
> UseSocket()
> 
This seems extremely contorted, and I'm pretty sure we can find a better 
way.

> However - this is getting increasingly painful because I don't have
> easy access to all of the socket creations where I'd like to do this.
> The biggest pain right now is that I'm using xmlrpclib which has some
> seriously/frustratingly heavy use of __ prefixes that makes getting
> inside to do this at socket creation near impossible (at least I think
> so).  Right now the best I can do is surround the xmlrpclib calls with
> this (effectively putting the lock release after the UseSocket), but
> then other threads get hung up for the duration of the call or timeout,
> rather than just the simple socket creation.
> 
The threads' network calls should be yielding process control during 
their timeout period to allow other runnable threads to proceed. That's 
how it's always worked for me, anyway, and I've written an email sender 
with 200 parallel threads.

> It would be nice if the timeout were implemented as an argument in the
> socket constructor rather than having this global method.  Is there a
> reason for this?  I tried sifting through the cvs source and got lost -
> couldn't even find the call definition for socket(family, type, proto)
> and gave up...
> 
You are aware, I presume, that you can set a timeout on each socket 
individually using its settimeout() method?

> Does anybody have any idea of another way to do what I need (indpendent
> socket timeouts per thread), or have suggestions on how to break into
> xmlrpclib (actually down into httplib) to do the methdo I was trying?
> 
See above. However, this *does* require you to have access to the 
sockets, which is tricky if they are buried deep in some opaque object's 
methods.

> Related question: Is there some global way that I'm unaware of to make
> it so that some few lines of code are atomic/uninterruptable and no
> other thread can sneak in between?
> 
There are locks! I suspect what you need is a threading.Rlock object, 
that a thread has to hold to be able to modify the (global) default 
timeout. This isn't a full solution to your problem, though, as you have 
correctly deduced.

The problem arises when a single method call creates a socket and then 
tries to do something with it (like connect to a remote server), which 
*is* problematic. I have to say that I haven't ever used different 
timeout values for the sockets in different parallel threads, so this is 
a new problem to me.

> All suggestions appreciated!  Hopefully I'm just missing something
> obvious.
> 
> Russ
> 
> #--- This script confirms that settimeout's affect is across threads
> import threading, xmlrpclib, socket
> 
> def st():
>   socket.setdefaulttimeout(0.1)
> 
> try:
>   proxy = xmlrpclib.ServerProxy("http://localhost:10000")
>   print proxy.NonExistentCallThatShouldTimeout()
> except Exception, E:  print "Exception caught: %s" % (E,)
> 
> cbThread = threading.Thread(target = st)
> cbThread.start()
> 
> try:
>   print proxy.NonExistentCallThatShouldTimeout()
> except Exception, E:  print "Exception caught: %s" % (E,)
> 
> #Output is:
> #Exception caught: (10061, 'Connection refused')
> #Exception caught: timed out
> 
Here (unless I'm missing something obvious) it seems that your worker 
thread terminates immediately after setting the default timeout, and 
both of the proxy calls are made from the main thread, so I'm not 
particularly surprised at the results, given the global nature of the 
default socket timeout.

Maybe someone else can think of something that will help.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC                     www.holdenweb.com
PyCon TX 2006                  www.python.org/pycon/