[Tutor] glibc error while Python script runs - Solved

[Bernard Lebel]

Hi Danny,

So have written a little test script.
The fact is that I want to be able to manage the same queue from
separate threads. Below is an example of what my real program is
doing:



from threading import Thread
from Queue import Queue
import time

class SQLServer:
	
	def __init__( self ):
		self.queueinput = Queue()
		self.queueoutput = Queue()
	

	def query( self, sInput ):
		
		# Puts an item to the input queue
		self.queueinput.put( sInput )
		
		# Wait for input queue to output result
		while True:
			
			# Get output from output queue
			sOutput = self.queueoutput.get()
			
			# Return result to caller
			return sOutput

	
	
	def shutdownOnIdle( self ):
		self.queueinput.put( "QUIT!" )


	
	# SERVER
	def serveSqlQueue( self ):
		
		while True:
			
			# Get a job from the input queue
			sQuery = self.queueinput.get()
			
			# Check if server has to quit
			if sQuery == "QUIT!": return
			
			print 'Server does %s' % sQuery
			
			self.queueoutput.put( sQuery )
	
	
	
	# START THE SERVER
	def startServer( self ):
		Thread( target = self.serveSqlQueue ).start()





def separateCaller():
	i = 1000
	while True:
		print '--Separate caller asks %i' % i
		oSeparateOutput = server.query( i )
		print '--Separate caller got %s' % str(oSeparateOutput)
		i += 1
		if i == 1004:
			break




if __name__ == '__main__':
	
	# Instantiate the server class
	server = SQLServer()
	
	
	
	# Start the server
	server.startServer()
	
	# Start the separate thread
	Thread( target = separateCaller ).start()
	
	
	
	print 'Main thread asks %s' % 'a'
	oMainOutput = server.query( 'a' )
	print 'Main thread got %s' % str(oMainOutput)
	
	print 'Main thread asks %s' % 'b'
	oMainOutput = server.query( 'b' )
	print 'Main thread got %s' % str(oMainOutput)
	
	print 'Main thread asks %s' % 'c'
	oMainOutput = server.query( 'c' )
	print 'Main thread got %s' % str(oMainOutput)
	
	server.shutdownOnIdle()




When I run this, I get this output:

Main thread asks a
--Separate caller asks 1000
Server does 1000
--Separate caller got 1000
--Separate caller asks 1001
Server does a
--Separate caller got a
--Separate caller asks 1002
Server does 1001
--Separate caller got 1001
--Separate caller asks 1003
Server does 1002
--Separate caller got 1002
Server does 1003
Main thread got 1003
Main thread asks b
Server does b
Main thread got b
Main thread asks c
Server does c
Main thread got c


As you can see, the main thread and the separateCaller function have
mixed results. How can I "synchronize" the queues?



Thanks
Bernard










On 1/19/06, Bernard Lebel <3dbernard at gmail.com> wrote:
> Thanks a lot Danny,
>
> That certainly does make sense. I'll look into implementing the Queue
> approach in my program tomorrow. I remember you recommending me this
> module as well not long ago, although in a different discussion (where
> I suspected problem with file access from multiple thread, but I guess
> it's pretty much a similar problem, isn't it? :-)
>
>
> Thanks again
> Bernard
>
>
>
> On 1/19/06, Danny Yoo <dyoo at hkn.eecs.berkeley.edu> wrote:
> >
> >
> > On Thu, 19 Jan 2006, Kent Johnson wrote:
> >
> >
> > > In your original desing were you sharing a connection between threads?
> > > That could cause trouble. But if each connection has its own thread and
> > > you are using transactions and isolation levels appropriately, they
> > > shouldn't stomp on each other.
> >
> > Hi Kent and Bernard,
> >
> > It's possible that MySQLdb was linked against the non-thread-safe
> > mysqlclient (rather than mysqlclient_r) library; that would probably cause
> > havoc.
> >
> >
> > > > So the solution was to start some sort of queue server in a separate
> > > > thread. This queue would consist of a list, and each time the program
> > > > would want to perform a MySQL operation, it would add it to the queue.
> >
> > We may want to use the Queue module here instead of a list; the way the
> > program is described now sounds like the server is busy-spinning when it
> > looks for new things to do.  The thing that makes busy-spinning slightly
> > not-nice is that it consumes CPU regardless if the system's doing anything
> > or not.
> >
> > But a more serious problem than busy-waiting is one of thread-safety: the
> > object that's used to communicate between two threads --- a shared list
> > --- might be unreliable if the list methods aren't thread-safe.  And I'm
> > not certain that all the list methods are thread-safe.
> >
> > That is, the problem with mutual exclusion may have just been pushed up
> > the program's structure, from the MySQL queries up to the shared queue
> > list access.  *grin*
> >
> >
> > The synchronized Queue described in:
> >
> >     http://www.python.org/doc/lib/module-Queue.html
> >
> > is designed to be a reliable communication medium between threads, and I
> > strongly recommend you look at it, especially because you've seen
> > first-hand the weird things that can happen with threads.  *grin*
> >
> >
> > Here's a small example that shows what a different Queue can make:
> >
> > ############################################
> > from threading import Thread
> >
> > class Server:
> >     def __init__(self):
> >         self.queue = []
> >
> >     def acceptJob(self, query):
> >         self.queue.append((query,))
> >
> >     def shutdownOnIdle(self):
> >         self.queue.append("QUIT!")
> >
> >     def jobLoop(self):
> >         while True:
> >             print "looping"
> >             if len(self.queue) > 0:
> >                nextJob = self.queue.pop(0)
> >                if nextJob == "QUIT!":
> >                    return
> >                print "I should do", nextJob
> >
> >     def startServer(self):
> >         Thread(target=self.jobLoop).start()
> >
> > if __name__ == '__main__':
> >     server = Server()
> >     server.startServer()
> >     server.acceptJob("a")
> >     server.acceptJob("b")
> >     server.acceptJob("c")
> >     server.shutdownOnIdle()
> > ############################################
> >
> >
> > Running this will show just how much work a busy-waiting thread does.
> >
> >
> > But if we change a few methods to use Queues instead of lists:
> >
> > ######
> > from threading import Thread
> > from Queue import Queue
> >
> > class Server:
> >     def __init__(self):
> >         self.queue = Queue()
> >
> >     def acceptJob(self, query):
> >         self.queue.put((query,))
> >
> >     def shutdownOnIdle(self):
> >         self.queue.put("QUIT!")
> >
> >     def jobLoop(self):
> >         while True:
> >             print "looping"
> >             nextJob = self.queue.get()
> >             if nextJob == "QUIT!":
> >                 return
> >             print "I should do", nextJob
> >
> >     def startServer(self):
> >         Thread(target=self.jobLoop).start()
> >
> > if __name__ == '__main__':
> >     server = Server()
> >     server.startServer()
> >     server.acceptJob("a")
> >     server.acceptJob("b")
> >     server.acceptJob("c")
> >     server.shutdownOnIdle()
> > ######
> >
> > and compare the output of this to the original implementation, it should
> > be clearer why Queues are cool.  *grin*
> >
> >
> > Does this make sense?  I hope this helps!
> >
> >
>