[C++-sig] Serialization/pickle
Neal Becker
ndbecker2 at gmail.com
Tue Dec 6 22:44:31 CET 2005
David Abrahams wrote:
> The following message is a courtesy copy of an article
> that has been posted to gmane.comp.lib.boost.user as well.
>
> "Drumheller, Michael" <michael.drumheller at boeing.com> writes:
>
>>>> From: David Abrahams <dave <at> boost-consulting.com>
>>>> Subject: Re: Boost.python serialization
>>>> Newsgroups: gmane.comp.lib.boost.user
>>>> Date: 2005-11-29 23:28:17 GMT (6 days, 17 hours and 9 minutes ago)
>>>>
>>>> "Drumheller, Michael" <michael.drumheller <at> boeing.com> writes:
>>>>
>>>> > Thank you for the help. I have seen that link (the one you
>>>> > supplied) before, but I posted my question because that link in
>>>> > particular does not mention the words "Boost.Serialization" or
>>>> > "Archive" at all.
>>>>
>>>> That's because the Boost Serialization library is unrelated.
>>
>> I understand that it is technically unrelated. I am asking for
>> information about about patterns of actual usage.
>
> I'm afraid I don't know that.
>
>>>> > I suppose what I was really getting at, and was probably not very
>>>> > clear about (sorry :| ) was whether the specific approach (by
>>>> > N. Becker) of using a stringstream wrapped with a
>>>> > boost::archive::binary_oarchive is a standard idiom.
>>>>
>>>> "Standard?"
>>
>> OK: "common," "popular," "preferred," "regarded-as-best-practice,"
>> "effective."
>
> Ditto.
>
>>>> > (Basically, I would have thought that "python pickle boost::archive"
>>>> > would be a million-hit Google query, but it's only about a dozen. I
>>>> > find that weird. Do people just not serialize their C++ extensions
>>>> > very often?)
>>>>
>>>> Yes, they do it very often. There's usually no need to touch
>>>> Boost.Serialization in order to do so, though.
>>
>> My extensions refer to one another, i.e., they form a significant object
>> hierarchy in and of themselves. E.g., on the C++ side I might have an
>> instance x of class X, which contains a vector of shared_ptrs p1,...,pn
>> to instances y1,...,yn of another C++ class Y. At pickle-time there may
>> be Python object z with a member u bound to to x and members w1,...,wm
>> bound to a subset of the y1,...,yn. Is it even feasible to expect to be
>> able to simple pickle.dump z and have it all work?
>
> Yes. Of course you have to do some work in your wrapping code to say
> how X gets pickled.
>
>> Please keep in mind: I am just getting started wading into serializing
>> a pretty complicated set of strongly interdependent Python and C++
>> objects and I am just trying to get my bearings. If this is the wrong
>> forum to be asking these questions, please tell me.
>
> Well, the C++-sig _might_ be more appropriate:
> http://www.boost.org/more/mailing_lists.htm#cplussig (cross-posted there)
>
> We (Boost Consulting) are actually planning to do something like this
> with one of our clients, but we haven't gotten started with it yet. I
> think everything should "just work" as long as you take care not to
> try to serialize the same object both from the C++ side (using
> Boost.Serialization) and from the Python side (using pickle). Because
> each system implements its own object tracking, you could end up
> representing the same object twice.
>
I have 2 examples that might interest you. I haven't looked at this code
for a while, so I don't exactly remember it.
The first is a wrapper for mersenne_twister. First, I patched
mersenne_twisted.hpp, adding:
friend class boost::serialization::access;
template<class Archive>
inline void save (Archive &ar, const unsigned int) const {
for (int j = 0; j < state_size; ++j) {
const UIntType x = compute (j);
ar << boost::serialization::make_nvp("item", x);
}
}
template<class Archive>
inline void load (Archive &ar, const unsigned int) {
for (int j = 0; j < state_size; ++j) {
ar >> boost::serialization::make_nvp ("item", x[j]);
}
i = state_size;
}
template<class Archive>
void serialize(Archive & ar, const unsigned int file_version) {
boost::serialization::split_member(ar, *this, file_version);
}
Then, I use this in my wrapper:
typedef boost::mt19937 rng_t;
struct mt_pickle_suite : python::pickle_suite {
static python::object getstate (const rng_t& rng) {
std::ostringstream os;
boost::archive::binary_oarchive oa(os);
oa << rng;
return python::str (os.str());
}
static void
setstate(rng_t& rng, python::object entries) {
python::str s = python::extract<python::str> (entries)();
std::string st = python::extract<std::string> (s)();
std::istringstream is (st);
boost::archive::binary_iarchive ia (is);
ia >> rng;
}
};
The second example is a wrapper for boost ublas vector:
namespace boost {
namespace serialization {
template<class T>
struct implementation_level<std::complex<T> >
{
typedef mpl::integral_c_tag tag;
// typedef mpl::int_<primitive_type> type;
typedef mpl::int_<object_serializable> type;
BOOST_STATIC_CONSTANT(
int,
value = implementation_level::type::value
);
};
template<class T>
struct tracking_level<std::complex<T> >
{
typedef mpl::integral_c_tag tag;
typedef mpl::int_<track_never> type;
BOOST_STATIC_CONSTANT(
int,
value = tracking_level::type::value
);
};
}
}
namespace boost { namespace serialization {
template<class Archive, class T>
inline void serialize (Archive &ar, std::complex<T>& z, const
unsigned int file_version) {
ar & boost::serialization::make_nvp ("real", real(z));
ar & boost::serialization::make_nvp ("imag", imag(z));
// ar & real(z);
// ar & imag(z);
}
}
}
namespace boost {
namespace serialization {
template<class T>
struct implementation_level<ublas::vector<T> >
{
typedef mpl::integral_c_tag tag;
// typedef mpl::int_<primitive_type> type;
typedef mpl::int_<object_serializable> type;
BOOST_STATIC_CONSTANT(
int,
value = implementation_level::type::value
);
};
}
}
template<class Archive, class U>
inline void save (Archive &ar, const ublas::vector<U> &v, const unsigned
int) {
unsigned int count = v.size();
ar << BOOST_SERIALIZATION_NVP (count);
typename ublas::vector<U>::const_iterator it = v.begin();
while (count-- > 0) {
ar << boost::serialization::make_nvp ("item", *it++);
}
}
template<class Archive, class U>
inline void load (Archive &ar, ublas::vector<U> &v, const unsigned int) {
unsigned int count;
ar >> BOOST_SERIALIZATION_NVP (count);
v.resize (count);
typename ublas::vector<U>::iterator it = v.begin();
while (count-- > 0) {
ar >> boost::serialization::make_nvp ("item", *it++);
}
}
namespace boost { namespace serialization {
template<class Archive, class U>
inline void serialize (Archive &ar, ublas::vector<U>& v, const unsigned int
file_version) {
boost::serialization::split_free (ar, v, file_version);
}
}
}
namespace python = boost::python;
template <typename MatrixType>
struct vector_pickle_suite : pickle_suite
{
static
tuple
getinitargs(const MatrixType &m)
{
return make_tuple(m.size());
}
static python::object getstate (const MatrixType& v) {
std::ostringstream os;
boost::archive::binary_oarchive oa(os);
oa << v;
return python::str (os.str());
}
static void
setstate(MatrixType& v, python::object entries) {
python::str s = python::extract<python::str> (entries)();
std::string st = python::extract<std::string> (s)();
std::istringstream is (st);
boost::archive::binary_iarchive ia (is);
ia >> v;
}
};
More information about the Cplusplus-sig
mailing list