Funny behaviour of MySQLdb

[Alex Martelli]

Heiko Wundram wrote:

> Hi List!
> 
> I just tried the single-threaded way, and that works... Hmm... Nothing
> apparently different, except that I took out the base class
> SocketServer.ThreadingMixIn from the definition of the server class...
> 
> Strange...

Not really.  Rule of thumb: always ensure a single thread interacts
with any given external entity, be it a file, a database, the user,
or even a global shared variable subject to updates.

Have all other threads make requests to that guardian thread by
posting work requests on a Queue; the guardian thread loops getting
a request from the Queue, servicing it, returning the results back
on another Queue indicated as part of the work request.  Note that
this is quite simple to program -- Queue's handle their own
synchronization needs, so no further locks or whatever are needed.

In some cases you can get by with threading a bit freer than this,
but -- don't count on it.  It's anything BUT surprising if you get
anomalies any time two or more threads interact directly with the
same entity without a dedicated guardian/mediator thread.  Threads
are a minefield and it's well worth threading lightly there:-).

This is my own opinion, of course, but I expressed it quite
strongly in the Threading chapter of the forthcoming Python in
a Nutshell, and two threading experts (Aahz and Tim Peters)
kindly agreed to review the chapter and give me their opinions
about it -- neither objected to the general approach and Aahz
in fact especially commended the general suggestion (the same
rule of thumb as given in the first paragraph of this post).

I suspect I'm in for some flames about this, but I think it's
worth the price in order to spread the word.  If you are a
threading expert, know exactly what you're doing, and are
totally familiar with all components involved and their
behavior under multihreaded use, go right ahead, you need no
advice.  For the remaining 99.44% of use, dedicated guardian
threads and Queue's to mediate is THE way to maintain sanity
when we do have to multithread.  One last reminder: event
driven processing is often preferable to multithreading...

Be sure to check out www.twistedmatrix.com -- I used to be
an asyncore/asynchat fan (well, I still AM!), but the current
release of twisted.internet and twisted.protocols at least
(haven't used the upper layers) is rapidly winning me over.
twisted's architecture is just TOO much cleaner, and more
elegant and powerful, than asyncore's simple (and still quite
effective) one.  They both scale better than threading in
most cases (you can ALSO mix some threading in, of course --
haven't tried that, but twisted supports it directly -- but
I don't think it would be needed ALL that much).


There, a bit more fuel for the flames...

Alex