[Numpy-discussion] failure to register ufunc loops for user defined types

[Mark Wiebe]

On Mon, Dec 5, 2011 at 9:37 AM, mark florisson <markflorisson88 at gmail.com>wrote:

> On 5 December 2011 17:25, Mark Wiebe <mwwiebe at gmail.com> wrote:
> > On Sun, Dec 4, 2011 at 11:37 PM, Geoffrey Irving <irving at naml.us> wrote:
> >>
> >> <snip>
> >>
> >>
> >> Back to the bugs: here's a branch with all the changes I needed to get
> >> rational arithmetic to work:
> >>
> >>    https://github.com/girving/numpy
> >>
> >> I discovered two more after the last email.  One is another simple 0
> >> vs. 1 bug, and another is somewhat optional:
> >>
> >> commit 730b05a892371d6f18d9317e5ae6dc306c0211b0
> >> Author: Geoffrey Irving <irving at naml.us>
> >> Date:   Sun Dec 4 20:03:46 2011 -0800
> >>
> >>    After loops, check for PyErr_Occurred() even if needs_api is 0
> >>
> >>    For certain types of user defined classes, casting and ufunc loops
> >>    normally run without the Python API, but occasionally need to throw
> >>    an error.  Currently we assume that !needs_api means no error occur.
> >>    However, the fastest way to implement such loops is to run without
> >>    the GIL normally and use PyGILState_Ensure/Release if an error
> occurs.
> >>
> >>    In order to support this usage pattern, change all post-loop checks
> >> from
> >>
> >>        needs_api && PyErr_Occurred()
> >>
> >>    to simply
> >>
> >>        PyErr_Occurred()
> >
> >
> > To support this properly, I think we would need to convert needs_api
> into an
> > enum with this hybrid mode as another case. While it isn't done
> currently, I
> > was imagining using a thread pool to multithread the trivially
> data-parallel
> > operations when needs_api is false, and I suspect the
> > PyGILState_Ensure/Release would trigger undefined behavior in a thread
> > created entirely outside of the Python system.
>
> PyGILState_Ensure/Release can be safely used by non-python threads
> with the only requirement that the GIL has been initialized previously
> in the main thread (PyEval_InitThreads).
>

Is there a way this could efficiently be used to propagate any errors back
to the main thread, for example using TBB as the thread pool? The innermost
task code which calls the inner loop can't call PyErr_Occurred() without
first calling PyGILState_Ensure itself, which would kill utilization.

Maybe this is an ABI problem in NumPy that needs to be fixed, to mandate
that inner loops always return an error code and disallow them from setting
the Python exception state without returning failure.

-Mark


>
> > For comparison, I created a
> > special mechanism for simplified multi-threaded exceptions in the nditer
> in
> > the 'errmsg' parameter:
> >
> >
> http://docs.scipy.org/doc/numpy/reference/c-api.iterator.html#NpyIter_GetIterNext
> >
> > Worth considering is also the fact that the PyGILState API is
> incompatible
> > with multiple embedded interpreters. Maybe that's not something anyone
> does
> > with NumPy, though.
> >
> > -Mark
> >
> >>
> >>
> >> Geoffrey
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20111205/9fd5fca3/attachment.html>