multiprocessing module in async db query

[Philip Semanchuk]

On Mar 9, 2011, at 10:22 AM, Sheng wrote:

> Hi Philip,
> 
> multiprocessing.Queue is used to transfer data between processes, how
> it could be helpful for solving my problem? Thanks!

I misunderstood -- I thought transferring data between processes *was* your problem. If both of your functions are in the same process, I don't understand how multiprocessing figures into it at all.

If you want a function to start returning results before that function completes, and you want those results to be processed by other code *in the same process*, then you'll have to use threads. A Queue object for threads exists in the standard library too. You might find that useful.

HTH
Philip


> 
> On Mar 8, 6:34 pm, Philip Semanchuk <phi... at semanchuk.com> wrote:
>> On Mar 8, 2011, at 3:25 PM, Sheng wrote:
>> 
>>> This looks like a tornado problem, but trust me, it is almost all
>>> about the mechanism of multiprocessing module.
>> 
>> [snip]
>> 
>>> So the workflow is like this,
>> 
>>> get() --> fork a subprocess to process the query request in
>>> async_func() -> when async_func() returns, callback_func uses the
>>> return result of async_func as the input argument, and send the query
>>> result to the client.
>> 
>>> So the problem is the the query result as the result of sql_command
>>> might be too big to store them all in the memory, which in our case is
>>> stored in the variable "data". Can I send return from the async method
>>> early, say immediately after the query returns with the first result
>>> set, then stream the results to the browser. In other words, can
>>> async_func somehow notify callback_func to prepare receiving the data
>>> before async_func actually returns?
>> 
>> Hi Sheng,
>> Have you looked at multiprocessing.Queue objects?
>> 
>> HTH
>> Philip
> 
> -- 
> http://mail.python.org/mailman/listinfo/python-list