[Chicago] Best practices for this concurrent problem?

[Daniel Griffin]

Hi,

I have been playing with a program which essentially reads some elements
from a database then starts a "thread" for each element, this can be a few
elements or thousands. Inside that thread it does some processing then
starts n sockets to do tasks. The life of the thread is generally a few
seconds but can stretch up to hours or even days.

I think my options are to use threading, multi-processing or some sort of
async thing like twisted. I am really unsure which one to choose.  Here are
the results of my testing so far:

threads - GIL death when I spin up a ton of threads, I can limit how many
threads I make but I know I am not getting anywhere near full utilization of
the box.

processes - I dont think its a good idea to make processes that are short
lived, it seems too expensive. This is even worse on windows.

async - I would have to re-factor to make this work and havent tried yet.

summary - What is the best way to deal with (sometimes) large numbers of
"threads" that do a small amount of processing and a large amount of socket
io?

Thanks,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/chicago/attachments/20090913/07349239/attachment.htm>