[IPython-dev] Parallel computing segfault behavior

[MinRK]

I get an EngineError when an engine dies running a task:

http://nbviewer.ipython.org/gist/minrk/8679553

I think this is the desired behavior.


On Tue, Jan 28, 2014 at 2:18 PM, Patrick Fuller <patrickfuller at gmail.com>wrote:

> Hi,
>
> Has there been any discussion around how ipython parallel handles
> segfaulting?
>
> To make this question more specific, the following code will cause some
> workers to crash. All results will become unreadable (or at least
> un-iterable), and future runs require a restart of the cluster. Is this
> behavior intended, or is it just something that hasn’t been discussed?
>
> from IPython.parallel import Clientfrom random import random
> def segfaulty_function(random_number, chance=0.25):
>     if random_number < chance:
>         import ctypes
>         i = ctypes.c_char('a')
>         j = ctypes.pointer(i)
>         c = 0
>         while True:
>             j[c] = 'a'
>             c += 1
>         return j
>     else:
>         return random_number
>
> view = Client(profile="something-parallel-here").load_balanced_view()
> results = view.map(segfaulty_function, [random() for _ in range(100)])
> for i, result in enumerate(results):
>     print i, result
>
> Backstory: Recently I’ve been working with a large monte carlo library
> that segfaults for, like, no reason at all. It’s due to some weird
> underlying random number issue and happens once every 5-10 thousand runs. I
> currently have each worker spin out a child process to isolate the
> occasional segfault, but this seems excessive. (I'm also trying to fix the
> source of the segfaults, but debugging is a slow process.)
>
> Thanks,
> Pat
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140128/fa19da0a/attachment.html>