Async XMLRPC and job processing

[Adonis Vargas]

Sean Davis wrote:
> I would like to set up a server that takes XMLRPC requests and
> processes them asynchronously.  The XMLRPC server part is trivial in
> python.  The job processing part is the part that I am having trouble
> with.  I have been looking at how to use threadpool, but I can't see
> how to get that working.  I would like to have the XMLRPC part of
> things do something like:
> 
> def method1(a,b,c):
>     jobid=workRequest(long_method1,[a,b,c])
>     return(jobid)
> 
> def method2(a,b,c):
>     jobid=workRequest(long_method2,[a,b,c])
>     return(jobid)
> 
> def long_method1(a,b,c)
>     do lots of heavy computation, etc.
>     store results in files in a given directory, etc
>     return result
> 
> .... for any number of methods
> 
> Again, pretty straightforward.  However, I run into problems with the
> threadpool and xmlrpc server both waiting.  In particular, if I do
> something like:
> 
> server = SimpleXMLRPCServer.SimpleXMLRPCServer(.....)
> server.serve_forever()
> 
> Where can tell the threadpool that I have set up to wait
> indefinitely?  Both are blocking.
> 
> Thanks,
> Sean
> 

This site shows how to make it multi-threaded:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/425043

But now you have to contend with the integrity of your data, provided 
your going be storing data through these processes. You may want to look 
into the Queue module to create a sort of message queue so your data can 
be properly synchronized.

Hope this helps.

Adonis