[C++-sig] profiling C++ extensions

[David Abrahams]

----- Original Message -----
From: "greg Landrum" <greglandrum at mindspring.com>
To: <c++-sig at python.org>
Sent: Saturday, March 02, 2002 7:22 PM
Subject: Re: [C++-sig] profiling C++ extensions


>
> Thanks for the response Dave.  I've posted to the python list and will
see
> if I get anything useful.  Based upon some googling I did this
morning, I'm
> not hopeful.
>
> However, I did a bit of screwing around and I have narrowed my
depickling
> problems down a bit.  I'm now pretty sure that the inefficiency is
taking
> place in the Boost layer.  Here's why I think this.
>
> I've got a C++ library which I've wrapped with both Boost and CXX (I
did
> the CXX wrapper long ago and then switched to Boost because I like it
a
> *lot* better than CXX).  I'm storing instances of this class
(molecules) in
> ZODB, which means I have to be able to pickle it.  That's no
problem...
> I've got the pickling stuff working just fine.  However, depickling
> molecules is really slow.  I wanted to do the profiling to track down
the
> source of this slowness.
>
> What I did instead is set up some boundary cases.  First I tried
depickling
> 1e5 2-tuples, to get a feeling for pickle overhead, this takes about
1.4
> seconds on my machine.  Then I tried instantiating 1e5 instances of
> a  python class (just an empty class).  This takes about 0.3 seconds.
Then
> I tried instantiating 1e5 instances of my molecule class using its
default
> constructor (which doesn't really do any work).  If I use the CXX
version
> of the library, this takes 0.9 seconds.  Doing the same thing with the
> Boost wrapped version of the library takes 6.7 seconds.  This is a
killer
> for me.  I need to be able to, on a regular basis, loop through 1e5 or
1e6
> molecules and this object construction overhead is overwhelming the
actual
> processing I'm doing.  (the bright side is that I don't need to worry
about
> optimizing my own code any further hahahahah <whimper>)
>
> Dave, I'm happy to do what I can to figure out how to solve this
problem
> (I'd really rather keep using Boost), but I'd like to get some feeling
from
> you as to whether or not it's even solvable.

I don't have a reason to be pessimistic, if that helps, but Ralf
Kunstleve is the Boost.Python pickle guy, and he knows a bit more than
I. Also I think he's planning to address pickling for Boost.Python V2 in
the near future.

I wonder what your CXX pickling interface looks like as compared to your
Boost interface? How many python functions get called with the Boost
version?

I guess actual profile info would be more helpful. Have you tried
Intel's profiling tools?
http://www.intel.com/software/products/vtune/index.htm?iid=ipp_home+soft
_vtune&

I think there's a free beta.