[Multiprocessing-sig] stop and restart processes

[Jesse Noller]

There is no similar control in 2.6, and we can not backport that
functionality to 2.6 which is in security fix mode only. If I remember
correctly, Celery (http://celeryproject.org/) implements something
like what you want internally.

Jesse

On Mon, Aug 29, 2011 at 4:34 PM, Shawn Gong <SGONG at mdacorporation.com> wrote:
> To clarify: I am using python 2.6.6, not 2.7
> So 'maxtasksperchild' is not available. Is there something similar?
>
> thanks,
> Shawn
>
>
> -----Original Message-----
> From: multiprocessing-sig-bounces+sgong=mdacorporation.com at python.org [mailto:multiprocessing-sig-bounces+sgong=mdacorporation.com at python.org] On Behalf Of Shawn Gong
> Sent: Monday, August 29, 2011 3:40 PM
> To: 'multiprocessing-sig at python.org'
> Subject: [Multiprocessing-sig] stop and restart processes
>
> Hi list,
>
> My Linux server has 24 CPUs. When I multiprocess a large job with more than 800 runs (para_list has >800 entries), I run out memory. There could be a memory leak somewhere.
>
> Is there a way to stop after running 10 processes on one CPU and restart, so that memory leak can be avoided?
>
> thanks,
> Shawn
>
> Codes:
>    args = [(arg1, arg2, ...) for arg1, arg2 in para_list]
>
>    # run My_calculation asynchronously
>    pool = multiprocessing.Pool(processes = cpu_used)
>    results = [pool.apply_async(My_calculation, a) for a in args]
>
>    # get the results: waits for each to finish and logs any errors
>    for idx, r in enumerate(results):
>        try:
>            r.get()
>        except:
>            log.debug(traceback.format_exc())
>
>
> _______________________________________________
> Multiprocessing-sig mailing list
> Multiprocessing-sig at python.org
> http://mail.python.org/mailman/listinfo/multiprocessing-sig
> _______________________________________________
> Multiprocessing-sig mailing list
> Multiprocessing-sig at python.org
> http://mail.python.org/mailman/listinfo/multiprocessing-sig
>