CPython thread starvation

[Danyel Lawson]

Sprinkle time.sleep(0) liberally throughout your code where you think
natural processing breaks should be.  Even in while loops. It's lame
but is the only way to make Python multithreading task switch fairly.
Your compute intensive tasks need a time.sleep(0) in their loops. This
prevents starvation and makes overall processing and responsiveness
seem properly multithreaded. This is a hand optimization so you have
to play with the location and amount of time.sleep(0)s. You'll know
when you've found a problematic spot when the queues stop
growing/overflowing.

Put the dns lookup on a separate thread pool with it's own growing
queue with lots of time.sleep(0)s sprinkled in. The dns lookups don't
have to be real time and you can easily cache them with a timestamp
attached. This is the thread pool where more is better and threads
should be aggressively terminated for having a long running process
time. This also requires lots of hand tuning for dynamically managing
the number of threads needed to process the queue in a reasonable time
if you find it hard to aggressively kill threads. I think there is a
way to launch threads that only give them a maximum lifetime. The
problem you will hit while tuning may require allocating more file
handles for all the hung sockets.

The DNS lookup is one of those things that may make sense to run as a
separate daemon process that listens on a socket. You make one
connection that feeds in the ip addresses. The daemon process feeds
back ip address/host name combinations out of order. Your main
process/connection thread builds a serialized access dict with
timestamps. The main processes threads make their requests
asynchronously and sleep while waiting for the response to appear in
the dict. They terminate after a certain time if they don't see their
response. Requires hand/algorithmic tweaking for this to work
correctly across different machines.

On Fri, Apr 27, 2012 at 2:54 PM, John Nagle <nagle at animats.com> wrote:
>    I have a multi-threaded CPython program, which has up to four
> threads.  One thread is simply a wait loop monitoring the other
> three and waiting for them to finish, so it can give them more
> work to do.  When the work threads, which read web pages and
> then parse them, are compute-bound, I've had the monitoring thread
> starved of CPU time for as long as 120 seconds.
> It's sleeping for 0.5 seconds, then checking on the other threads
> and for new work do to, so the work thread isn't using much
> compute time.
>
>   I know that the CPython thread dispatcher sucks, but I didn't
> realize it sucked that bad.  Is there a preference for running
> threads at the head of the list (like UNIX, circa 1979) or
> something like that?
>
>   (And yes, I know about "multiprocessing".  These threads are already
> in one of several service processes.  I don't want to launch even more
> copies of the Python interpreter.  The threads are usually I/O bound,
> but when they hit unusually long web pages, they go compute-bound
> during parsing.)
>
>   Setting "sys.setcheckinterval" from the default to 10000 seems
> to have little effect.  This is on Windows 7.
>
>                                John Nagle
> --
> http://mail.python.org/mailman/listinfo/python-list