[AstroPy] A question about multiprocessing

[Chris Beaumont]

I suspect you may want to use the Pool() and map functions instead:

def process_image(image):
   do stuff to a single image

p = multi.Pool()
images = glob.glob(DirImg + '*r.sdf')
results = p.map(process_image, images)

this just farms out each function call to processors as they become
available.

cheers,
Chris Beaumont


On Fri, Nov 4, 2011 at 11:51 AM, Chris Sontag <sontag at stsci.edu> wrote:

>  Hi Jean-Baptiste,
>
> I'm not sure this code is doing what you want.  Each of NPROC subprocesses
> runs CatDistort(), which does some processing on EACH image in your queue.
> So each image actually gets processed NPROC times (by every subprocess), is
> that your intention?  If not, send one image as the value for the args of
> CatDistort().
>
> Chris
>
>
> On 11/4/11 10:45 AM, Jean-Baptiste Marquette wrote:
>
> Dear Python gurus,
>
>  I'm not sure to be on the right mailing list, but I have the following
> problem:
>
>  I have this code ran from Eclipse/PyDev:
>
>  import multiprocessing as multi
>
>  def CatDistort(queue):
>     for Image in iter(queue.get, 'STOP'):
>  etc...
>
>  queue = multi.Queue()
>
>  for Image in glob.glob(DirImg + '*r.sdf'):
>     queue.put(Image)
>     print os.path.basename(Image), 'queued'
> queue.put('STOP')
>
>  Jobs = []
> for i in range(NPROC):
>     proc= multi.Process(target=CatDistort, args=[queue])
>     Jobs.append(proc)
>     proc.start()
>
>
>  for job in Jobs:
>     job.join(360)
>
>  print 'Process completed'
> exit(0)
>
>  The target and the queue work well, but the process never terminates. In
> other words, the final print is never displayed.
>
>  Any hint ?
> Thanks for your help,
>
>  Cheers
> Jean-Baptiste
>
>
>
> _______________________________________________
> AstroPy mailing listAstroPy at scipy.orghttp://mail.scipy.org/mailman/listinfo/astropy
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>


-- 
************************************
Chris Beaumont
Graduate Student
Institute for Astronomy
University of Hawaii at Manoa
2680 Woodlawn Drive
Honolulu, HI 96822
www.ifa.hawaii.edu/~beaumont
************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20111104/1bf115d4/attachment.html>