Message queues [Re: Best processor (i386) for Python performance?]

[Ville Vainio]

>>>>> "David" == David Bolen <db3l at fitlinxx.com> writes:

    David> I do think it can be tricky to determine just what case an
    David> application falls into (and many oscillate between I/O and
    David> CPU bound modes), and indeed a purely CPU bound Python
    David> application (if in Python code and not a well-behaving
    David> extension module) isn't going to be helped at all.

The sensible thing to do then is to use multiple processes, not just
multiple threads. Many Python apps use Queue.Queue anyway, and such an
approach is often easy to convert over to use processes instead of
threads.

In fact, it might be fun to have a trivial message queue
implementation in the standard library:

# server code

frow mq import *

q,results = MQueue(),MQueue()

# file has just a handle, like
# mq:123.12.12.54:67

q.publish(open("~/jobs","w"))   
results.publish(open("~/result","w"))
spawn_server_if_needed() 

while 1:
  job = q.get()
  res = my_handle_job( job )
  results.put(res)


# client code

...

req, results = MQueue(open("~/job")), MQueue(open("~/results")) 

req.put( ("easyjob", 34, 2.44) )
req.put( ("easyjob", 213, 2.44) )

...


Obviously these trivial mqueues could still be wrapped with additional
rendezvous functionality:

job = mq.Job(("hello",2))

rv = mq.Rendezvous(q,resultqueue)

rv.put(job)

res = job.result()  # blocks until result is ready

Though this might be more in the territory of external
libs/frameworks... but hey, we've already got xml-rpc and web server
functionality ;-).

Inter-language systems should obviously something like Corba for this.

-- 
Ville Vainio   http://tinyurl.com/2prnb