newbee: Simple Backend Python Script Question

[Steve Holden]

joe shoemaker wrote:
> I need to create python script that is threaded. So the main program 
> will run in infinite loop and just retrieving messages and putting them 
> in a queue. (Main thread)
> 
> I need child threads from a pool to process the queue. When there is no 
> stuff in the queue, they go to the pool and become available but they 
> don't terminate. This has to be done continuously.
> 
> Main program need to keep putting stuff in the queue, when there are no 
> messages, then it sleeps for short time and check back to see any messages.
> 
> To do this, I guess you don't write joinAll(), so that the main threads 
> just don't wait for the child but goes to work.
> 
> am I right?
> 
Pretty much. The way I usually do this is to start a number of threads 
reading work items of some sort from a Queue.Queue. The thread will 
block automatically if there's nothing on the queue, resuming when 
something appears (and if I want orderly termination I use None as a 
dummy work unit, and the threads terminate when they receive a None, but 
any sentinel value would do).

> Also, child threads (a function that is threaded) will make connecitons 
> to the database. I am planning to use threadpool, so that threads reuse 
> the connections. So do you close the database connection at the end of 
> the function? If not then the connection will be opened forever?
> 
You don't want a thread pool, you want a connection pool, but there's 
little advantage to having one that is only shared between your threads. 
You really need a system-wide connection pool. Alternatively, you might 
find that your database module is thread-safe to the extent that 
different threads can use cursors created on the same connection without 
interference.

If you want each thread to be able to process transactions that are 
invisible to the other threads before they are committed you should 
ensure that you have a sufficient isolation level, which might imply the 
need for a connection-per-thread architecture, in which case you might 
expect some benefit from connection pooling.

regards
  Steve

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden

Sorry, the dog ate my .sigline