Psycopg and threads problem

[Ken Godee]

> 
> The script I'm working on does bulk inserts from multiple related XML 
> files (parsed using the sax parser) and take turns inserting small 
> batches of xml records from those files. The collection thread combines 
> these into 1 xml record, which is why it's so important that the inserts 
> are done in time.
> 
> I can't commit until all the data has been inserted and combined. The 
> commit shouldn't happen until the end of the main thread is reached.
> 
> If the server goes down, or there's another reason it can't continue 
> parsing, the whole transaction should rollback. Committing in between 
> would be 'problematic' (where in the XML files were we interupted? Hard 
> to tell).
> 
> Also, I don't think I can join the threads (as someone else suggested), 
> as they are still working in an Application instance (part of the SAX 
> parser). The threads are waiting until they're allowed to continue; by a 
> linked list of Events (so that I can remove events for threads that 
> finished - which shouldn't happen, but it may). Unless I misunderstand 
> thread joining, of course.
> 

I would probally pass a queue instance to the worker threads
and in the main thread create a timer method/function that periodically 
checks the queue.

When the worker thread is done doing its thing, do a commit and 
queue.put('done')

When the main thread checks the queue.get(0) = 'done'
have it do it's thing.

This is the beauty of queues.