[AstroPy] A question about multiprocessing
Chris Beaumont
beaumont at hawaii.edu
Fri Nov 4 11:59:42 EDT 2011
I suspect you may want to use the Pool() and map functions instead:
def process_image(image):
do stuff to a single image
p = multi.Pool()
images = glob.glob(DirImg + '*r.sdf')
results = p.map(process_image, images)
this just farms out each function call to processors as they become
available.
cheers,
Chris Beaumont
On Fri, Nov 4, 2011 at 11:51 AM, Chris Sontag <sontag at stsci.edu> wrote:
> Hi Jean-Baptiste,
>
> I'm not sure this code is doing what you want. Each of NPROC subprocesses
> runs CatDistort(), which does some processing on EACH image in your queue.
> So each image actually gets processed NPROC times (by every subprocess), is
> that your intention? If not, send one image as the value for the args of
> CatDistort().
>
> Chris
>
>
> On 11/4/11 10:45 AM, Jean-Baptiste Marquette wrote:
>
> Dear Python gurus,
>
> I'm not sure to be on the right mailing list, but I have the following
> problem:
>
> I have this code ran from Eclipse/PyDev:
>
> import multiprocessing as multi
>
> def CatDistort(queue):
> for Image in iter(queue.get, 'STOP'):
> etc...
>
> queue = multi.Queue()
>
> for Image in glob.glob(DirImg + '*r.sdf'):
> queue.put(Image)
> print os.path.basename(Image), 'queued'
> queue.put('STOP')
>
> Jobs = []
> for i in range(NPROC):
> proc= multi.Process(target=CatDistort, args=[queue])
> Jobs.append(proc)
> proc.start()
>
>
> for job in Jobs:
> job.join(360)
>
> print 'Process completed'
> exit(0)
>
> The target and the queue work well, but the process never terminates. In
> other words, the final print is never displayed.
>
> Any hint ?
> Thanks for your help,
>
> Cheers
> Jean-Baptiste
>
>
>
> _______________________________________________
> AstroPy mailing listAstroPy at scipy.orghttp://mail.scipy.org/mailman/listinfo/astropy
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
--
************************************
Chris Beaumont
Graduate Student
Institute for Astronomy
University of Hawaii at Manoa
2680 Woodlawn Drive
Honolulu, HI 96822
www.ifa.hawaii.edu/~beaumont
************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20111104/1bf115d4/attachment.html>
More information about the AstroPy
mailing list