[Tutor] glibc error while Python script runs - Solved

[Kent Johnson]

Hi Bernard,

I'm glad you got it working but kind of surprised at what you had to do. 
You shouldn't have to have a single thread to access the database. In 
your original desing were you sharing a connection between threads? That 
could cause trouble. But if each connection has its own thread and you 
are using transactions and isolation levels appropriately, they 
shouldn't stomp on each other.

There are some good recipes for queuing up tasks and getting callbacks 
that might be simpler than your dictionary solution. For example see
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/435883
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/203871

You could also use a second queue to return results to the main thread.

Kent

Bernard Lebel wrote:
> Hello,
> 
> For the record, in case anyone would like to know what happened with
> this issue...
> 
> 
> It seems that the problem was related to the way I managed MySQL
> connections. There was the main thread that would query every 5
> seconds the database to see if a someone wanted to abort the job
> running on the render node.
> 
> However, the actual rendering was managed by a child thread, who would
> perform many MySQL transactions, depending on the output produced by
> the rendering software.
> 
> So there were two separate threads doing MySQL queries. I noticed by
> reading my debug logs that some of these queries would become
> interlaced, that is, one query would try to update while another would
> try to select-fetch. In turned out that this would severe the
> connection with the database, and soon after Python would crash
> altogether.
> 
> So the solution was to start some sort of queue server in a separate
> thread. This queue would consist of a list, and each time the program
> would want to perform a MySQL operation, it would add it to the queue.
> Then, the caller would enter a while loop that would wait for the
> server to carry the operation.
> 
> The server would run a while loop, that check the queue once every
> iteration to see if there is something to do. When it finds something,
> it would carry the entire transaction, then put the result in a
> dictionary, using an ID number assigned when the transaction was
> submitted to the queue.
> 
> The "submitter" would then pick up the result, and return it to the
> original caller.
> The result is that no two queries would take place at the same time.
> 
> Once I did that, my program stopped crashing, and actually has been
> running very smooth since then. In fact my progam is now very very
> close to completion.
> 
> 
> 
> Kent: I took your suggestion and changed the program design and
> splitted the code into separate modules. It was tricky to "re-wire"
> everything together (it took me at least 4 days), but past that, I'm
> glad I did it.
> 
> 
> Thanks everyone for the help, it has been a great learning experience!
> 
> 
> Bernard