From pythonmarco at gmail.com  Thu Jan 13 21:03:39 2011
From: pythonmarco at gmail.com (Marco Hornung)
Date: Thu, 13 Jan 2011 15:03:39 -0500
Subject: [PyOP] how to use priority queue with multiprocessing
Message-ID: <87E38041-C753-40C1-BD37-0334717C543D@gmail.com>

Hey, 

------------------------------------------------------------------------------------------
question
------------------------------------------------------------------------------------------
How can I use a priority queue to schedule jobs within the "multiprocessing pool" module?

------------------------------------------------------------------------------------------
my scenario
------------------------------------------------------------------------------------------
I want to run several jobs on a server. The jobs are being sent by users. However, all jobs have a different priority, and high-priority jobs should be processed before any low-priority job gets touched.
Currently I just append all incoming jobs to the multiprocessing worker pool as follows:
### initialize worker pool
pool			= PriorityPool(processes=worker_count)
process_handles = []

### distribute function execution over several processes
for job_parameter in job_parameter_list:
	handle = pool.apply_async(process_function, [job_parameter,])
	process_handles.append(handle)

This will only put the jobs in some kind of a list - and execute the jobs in the order they come in. Is it possible to use a priority queue for the process-pool?

Kind Regards,
Marco

From jim at well.com  Thu Jan 27 20:19:18 2011
From: jim at well.com (jim)
Date: Thu, 27 Jan 2011 11:19:18 -0800
Subject: [PyOP] BayPIGgies meeting Tonight, Thursday, January 27,
 2011:  Introduction to CouchDB
Message-ID: <1296155958.1882.53.camel@jim-laptop>


BayPIGgies meeting Tonight, Thursday, January 27, 2011:  Introduction to
CouchDB 


This meeting's talk is 
Introduction to CouchDB 
by Luke Gostlings 

This talk introduces one 'NoSQL' solution, CouchDB, and how to get it
to 
play well with Python. Topics covered: 
* Introduction to CouchDB 
* A python ORM for CouchDB 
* Parsing CouchDB documents within python 
* Writing view functions in python 
* Map/reduce on CouchDB from python 
* Lessons learned from managing and distributing a live deployment 
  at scale under high load
 
Speaker Bio: Luke Gostlings
Luke is a lead engineer at about.me (recently acquired by AOL). 
His prior positions were in: network security research, online
payments, 
and small company stock offering markets. He has done contract work in 
the social and on-demand media spaces. He likes to dabble in NoSQL 
technologies, computer security, and financial markets. 
He has previously presented at CCCamp, San Francisco Startup Weekend, 
and RSAConference.
 
http://about.me/luke

.........................................

Meetings usually start with a Newbie Nugget, a short discussion of 
an essential Python feature, especially for those new to Python. 
Tonight's Newbie Nugget: 
Using zip() with Django, presented by Vicky Tuite



LOCATION 
Symantec Corporation
Symantec Vcafe
350 Ellis Street
Mountain View, CA 94043
http://maps.google.com/maps/ms?oe=utf-8&client=firefox-a&ie=UTF8&fb=1&split=1&gl=us&ei=w6i_Sfr6MZmQsQOzlv0v&hl=en&t=h&msa=0&msid=116202735295394761637.00046550c09ff3d96bff1&ll=37.397693,-122.053707&spn=0.002902,0.004828&z=18

BayPIGgies meeting information is available at 
http://www.baypiggies.net/ 


------------------------ Agenda ------------------------


..... 7:30 PM ...........................
General hubbub, inventory end-of-meeting announcements, 
any first-minute announcements. 


..... 7:35 PM to 7:45 PM ................
Tonight's Newbie Nugget: Using zip() with Django, 
presented by Vicky Tuite
 


..... 7:45 PM to 8:25 PM (or so) ................
The talk: Introduction to CouchDB 


..... 8:25 PM to 8:55 PM (or so) ................
Questions and Answers 


..... 8:55 PM to 9:30 PM (or so) ................
Mapping and Random Access

Mapping is a rapid-fire audience announcement of issues, 
hiring, events, and other topics. 

Random Access follows people immediately to allow follow 
up on the announcements and other interests.