[C++-sig] question on boost.python exception mechanisms

[Holger Joukl]

Hi,

> From: "Niall Douglas" <s_sourceforge at nedprod.com>
> On 17 Apr 2013 at 17:13, Holger Joukl wrote:
>
> > // the global per-thread exception storage
> > boost::unordered_map<pthread_t, boost::exception_ptr> exception_map;
>
> You can't assume pthread_t is of integral type. You can a thread_t
> (from C11) I believe. You may not care on your supported platforms
> though.

I see. If it isn't an integral type I suppose I'd need to supply a custom
hash implementation to be able to use unordered_map.

> >     throw std::runtime_error("throwing up");
>
> If you're going to use Boost's exception_ptr implementation, you
> really ought to throw using Boost's exception throw macro. Otherwise
> stuff become unreliable.

Right. This should then rather be

    throw boost::enable_current_exception(std::runtime_error("throwing
up"));

Looks like unfortunately Boost.Python does not use this itself so a
packaged_task/future will throw unknown_exception for the deferred
invocation of a boost::python::error_already_set-throwing call.
I could work around this in my case by wrapping the original call in
try-catch and re-raise like

    try {
         this->get_override("onMsg")(bp::ptr(listener), boost::ref(msg));
    } catch (const bp::error_already_set& e) {
        // enable support for cloning the current exception, to avoid
having
        // a packaged_task/future throw an unknown_exception
        boost::enable_current_exception(bp::error_already_set());
    }

> > [...]
>
> If you're just going to do this, I'd suggest you save yourself some
> hassle and look into packaged_task which is a std::function combined
> with a std::future. It takes care of the exception trapping and
> management for you. Do bear in mind there is absolutely no reason you
> can't use a future within a single thread to traverse some state over
> third party binary blob code, indeed I do this in my own code at
> times e.g. if Qt, which doesn't like exceptions, stands between my
> exception throwing code at the end of a Qt signal and my code which
> called Qt.
>
> > [...]
>
> Why not using thread local storage?

Using thread local storage of a queue of deferred futures might then look
s.th. like this:

$ cat dispatch_main_packaged_task_tls_queue.cpp
// File: dispatch_main_packaged_task_tls_queue.cpp

#include <iostream>
#include <exception>
#include <stdexcept>
#include <boost/thread.hpp>
#include <boost/move/move.hpp>     // header-only
#include <boost/bind.hpp>          // header-only
#include <boost/lockfree/spsc_queue.hpp>          // header-only
#include <boost/lexical_cast.hpp>
#include "dispatch.h"


// global thread local store of void-returning task results
// (per-thread queue of futures)
class CURRENT_THREAD_STATE {
public:
    typedef boost::unique_future<void> unique_future_t;
    typedef boost::shared_ptr<unique_future_t> unique_future_ptr_t;
    typedef boost::lockfree::spsc_queue<
        unique_future_ptr_t, boost::lockfree::capacity<10> >
future_queue_t;
    typedef boost::thread_specific_ptr< future_queue_t > tls_t;

private:
    // the internal per-thread futures queue storage
    static tls_t _current_thread_tls;

public:
    // package a task and defer it
    // (to a thread-local queue of of void-returning futures)
    template <typename Fn, typename A0>
    static void deferred(Fn func, A0 const & arg1) {
        // need a nullary function object as an arg to packaged_task so
        // use boost::bind
        boost::packaged_task<void> pt(boost::bind(func, arg1));
        unique_future_t fut = pt.get_future();
        future_queue_t * queue_ptr = _current_thread_tls.get();

        if (queue_ptr == NULL) {
             queue_ptr = new future_queue_t;
             _current_thread_tls.reset(queue_ptr);
        }

        unique_future_ptr_t fut_ptr(new unique_future_t(boost::move(fut)));
        queue_ptr->push(fut_ptr);
        pt();
    }

    // retrieve the deferred task results
    static void get_deferred(void) {
        future_queue_t * queue_ptr = _current_thread_tls.get();
        if (queue_ptr != NULL) {
            unique_future_ptr_t fut_ptr;
            if (queue_ptr->pop(fut_ptr)) {
                if (fut_ptr) {
                    fut_ptr->get();
                }
            }
        }

    }
};

// don't forget the definition of the static member variable
CURRENT_THREAD_STATE::tls_t CURRENT_THREAD_STATE::_current_thread_tls;


void callback(cb_arg_t arg) {
    std::cout << "--> CPP callback arg=" << arg << std::endl;
    std::cout << "<-- CPP callback arg=" << arg << std::endl;
}


// this callback raises an exception
void callback_with_exception(cb_arg_t arg) {
    std::cout << "--> exception-throwing CPP callback arg=" << arg <<
std::endl;
    std::string errormsg("throwing up ");
    errormsg += arg;
    std::cout << "errormsg=" << errormsg << std::endl;
    throw std::runtime_error(errormsg);
    std::cout << "<-- exception-throwing CPP callback arg=" << arg <<
std::endl;
}


// this calls the exception raising callback but has packaged_task manage
// the result and possible exceptions
void guarded_callback_with_exception(cb_arg_t arg) {
    std::cout << "--> guarded exception-throwing CPP callback arg=" << arg
<< std::endl;
    CURRENT_THREAD_STATE::deferred(&callback_with_exception, arg);
    std::cout << "<-- guarded exception-throwing CPP callback arg=" << arg
<< std::endl;
}


int main(void) {
    std::cout << "--> CPP main" << std::endl;
    cb_arg_t s = "CPP main callback argument";

    std::cout << std::endl << "invoking callback" << std::endl;
    invoke(&callback, s);

    std::cout << std::endl << "invoking guarded callback" << std::endl;
    invoke(&guarded_callback_with_exception, s);
    try {
        std::cout << "rethrowing exception from global tls" << std::endl;
        CURRENT_THREAD_STATE::get_deferred();
    } catch (const std::exception& exc) {
        std::cout << "caught rethrown callback exception: " << exc.what()
<< std::endl;
    }

    for (int i=0; i<3; ++i) {
        std::string cb_string_arg = std::string("CPP main cb arg ") +
boost::lexical_cast<std::string>(i);
        std::cout << std::endl << "invoking guarded callback (run " << i <<
")" << std::endl;
        invoke(&guarded_callback_with_exception, cb_string_arg.c_str());
    }
    for (int i=0; i<3; ++i) {
        try {
            std::cout << "rethrowing exception from global tls (run " << i
<< ")" << std::endl;
            CURRENT_THREAD_STATE::get_deferred();
        } catch (const std::exception& exc) {
            std::cout << "caught rethrown callback exception: " << exc.what
() << std::endl;
        }
    }

    // this is expected to segfault iff the underlying C lib isn't compiled
with
    // exception support
    std::cout << std::endl << "invoking exception-throwing callback" <<
std::endl;
    std::cout << "(will segfault iff c lib is not exception-enabled)" <<
std::endl;
    try {
        invoke(&callback_with_exception, s);
    } catch (const std::exception& exc) {
        std::cout << "caught callback exception: " << exc.what() <<
std::endl;
    }

    std::cout << std::endl;
    std::cout << "<-- CPP main" << std::endl;
    return 0;
}

0 $ BOOST_VERSION=1.53.0; BOOST_DIR=boost_${BOOST_VERSION//./_}; echo
$BOOST_DIR; /apps/local/gcc/4.6.1/bin/g++ -pthreads -I. -L. -R.
-R /apps/prod/gcc/4.6.1/lib/
-R /var/tmp/BUILD/gcc/$BOOST_DIR/stage/gcc-4.6.1/py2.7/boost/$BOOST_VERSION/lib

-L /var/tmp/BUILD/gcc/$BOOST_DIR/stage/gcc-4.6.1/py2.7/boost/$BOOST_VERSION/lib
 -I /var/tmp/BUILD/gcc/$BOOST_DIR/ -ldispatch -lboost_thread -lboost_system
-lboost_atomic dispatch_main_packaged_task_tls_queue.cpp && ./a.out
boost_1_53_0
--> CPP main

invoking callback
--> invoke(298752, CPP main callback argument)
--> CPP callback arg=CPP main callback argument
<-- CPP callback arg=CPP main callback argument
<-- invoke(298752, CPP main callback argument)

invoking guarded callback
--> invoke(299312, CPP main callback argument)
--> guarded exception-throwing CPP callback arg=CPP main callback argument
--> exception-throwing CPP callback arg=CPP main callback argument
errormsg=throwing up CPP main callback argument
<-- guarded exception-throwing CPP callback arg=CPP main callback argument
<-- invoke(299312, CPP main callback argument)
rethrowing exception from global tls
caught rethrown callback exception: throwing up CPP main callback argument

invoking guarded callback (run 0)
--> invoke(299312, CPP main cb arg 0)
--> guarded exception-throwing CPP callback arg=CPP main cb arg 0
--> exception-throwing CPP callback arg=CPP main cb arg 0
errormsg=throwing up CPP main cb arg 0
<-- guarded exception-throwing CPP callback arg=CPP main cb arg 0
<-- invoke(299312, CPP main cb arg 0)

invoking guarded callback (run 1)
--> invoke(299312, CPP main cb arg 1)
--> guarded exception-throwing CPP callback arg=CPP main cb arg 1
--> exception-throwing CPP callback arg=CPP main cb arg 1
errormsg=throwing up CPP main cb arg 1
<-- guarded exception-throwing CPP callback arg=CPP main cb arg 1
<-- invoke(299312, CPP main cb arg 1)

invoking guarded callback (run 2)
--> invoke(299312, CPP main cb arg 2)
--> guarded exception-throwing CPP callback arg=CPP main cb arg 2
--> exception-throwing CPP callback arg=CPP main cb arg 2
errormsg=throwing up CPP main cb arg 2
<-- guarded exception-throwing CPP callback arg=CPP main cb arg 2
<-- invoke(299312, CPP main cb arg 2)
rethrowing exception from global tls (run 0)
caught rethrown callback exception: throwing up CPP main cb arg 0
rethrowing exception from global tls (run 1)
caught rethrown callback exception: throwing up CPP main cb arg 1
rethrowing exception from global tls (run 2)
caught rethrown callback exception: throwing up CPP main cb arg 2

invoking exception-throwing callback
(will segfault iff c lib is not exception-enabled)
--> invoke(298904, CPP main callback argument)
--> exception-throwing CPP callback arg=CPP main callback argument
errormsg=throwing up CPP main callback argument
terminate called after throwing an instance of 'std::runtime_error'
  what():  throwing up CPP main callback argument
Abort (core dumped)

(core dump expected for the unguarded case as I did not compile the
C lib with exception support)


Notes:
- not generalized, supports only 1-argument void-returning callables
- Boost doesn't list Boost.Atomic as a non-header-only lib but you need to
compile & link with it (for lockfree)
- need -lboost_system when using -lboost_thread
- could also use a per-thread future instead of a queue of futures if
there's no possibility of or no interest in a subsequent result overwriting
a previous result

> Keep a count of nesting levels, so if a callback calls a callback
> which calls a callback etc.

Ok so no out-of-the-box functionality.

> Another possibly useful idea is to have the C libs dispatcher
> dispatch packaged_task's into a per-thread lock free queue which you
> then dispatch on control return when the C library isn't in the way
> anymore. Call it deferred callbacks :)

Interesting. So I'd basically transfer the actual callback invocation to
exception-capable-land.

I feel a bit out of my depths here now wrt to all that boost magic
but I've definitely learned a bunch :)

Still quite unsure if I shouldn't just keep it really simple
and go with the original idea of catching boost::python::error_already_set
and re-raising if (PyErr_Occurred()) (and not add a dependency to
Boost.Thread, Boost.System and possibly Boost.Atomic)...

Thanks again for this wealth of useful hints and suggestions,
Holger

Landesbank Baden-Wuerttemberg
Anstalt des oeffentlichen Rechts
Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz
HRA 12704
Amtsgericht Stuttgart