Message queues [Re: Best processor (i386) for Python performance?]
Ville Vainio
ville at spammers.com
Thu Aug 26 15:50:02 EDT 2004
>>>>> "David" == David Bolen <db3l at fitlinxx.com> writes:
David> I do think it can be tricky to determine just what case an
David> application falls into (and many oscillate between I/O and
David> CPU bound modes), and indeed a purely CPU bound Python
David> application (if in Python code and not a well-behaving
David> extension module) isn't going to be helped at all.
The sensible thing to do then is to use multiple processes, not just
multiple threads. Many Python apps use Queue.Queue anyway, and such an
approach is often easy to convert over to use processes instead of
threads.
In fact, it might be fun to have a trivial message queue
implementation in the standard library:
# server code
frow mq import *
q,results = MQueue(),MQueue()
# file has just a handle, like
# mq:123.12.12.54:67
q.publish(open("~/jobs","w"))
results.publish(open("~/result","w"))
spawn_server_if_needed()
while 1:
job = q.get()
res = my_handle_job( job )
results.put(res)
# client code
...
req, results = MQueue(open("~/job")), MQueue(open("~/results"))
req.put( ("easyjob", 34, 2.44) )
req.put( ("easyjob", 213, 2.44) )
...
Obviously these trivial mqueues could still be wrapped with additional
rendezvous functionality:
job = mq.Job(("hello",2))
rv = mq.Rendezvous(q,resultqueue)
rv.put(job)
res = job.result() # blocks until result is ready
Though this might be more in the territory of external
libs/frameworks... but hey, we've already got xml-rpc and web server
functionality ;-).
Inter-language systems should obviously something like Corba for this.
--
Ville Vainio http://tinyurl.com/2prnb
More information about the Python-list
mailing list