Threading questions

[Antal Rutz]

Magnus Lycka wrote:
> Antal Rutz wrote:
> 
>>Hi, All!
>>
>>I'm new to threading. I have some design questions:
>>Task: I collect data and store them in an RDBMS (mysql or pgsql)
>>
>>The question is how to do that with threading?
>>The data-collecting piece of the code runs in a thread.
>>
>>1. Open the db, and each thread writes the result immediately.
>>   (Sub-question: which is better: cursor object passed to the thread
>>   or stored in a global var.)
>>2. Threads return data, which is written to DB after that.
>>3. Threads write to global variable, after 'join()' data is written.
>>   (Can be used global (write-only) variables with threads at all?)
>>4. ?...
> 
> 
> I'm not sure why you need threading at all here, but I would avoid
> both globals and passing cursors between threads. I don't think all
> DB-API implementations can handle threading properly.

I 'snmpwalk' routers and some are very slow, so it's better to spread
the routers among data-collecting threads.

> I suspect it might be useful to do data collection in one thread,
> DB communication in another, and to use a Queue.Queue object to
> pass data from data collection to DB insertion thread.

Yes, I just figure out how the many-collecting-one-dbwriting-thread
design could be made. It doesn't seem to be too difficult.

Thanks for the hints!

-- 


--arutz