[Numpy-discussion] Profiling numpy ? (parts written in C)

[David Cournapeau]

John Hunter wrote:
>>>>>> "David" == David Cournapeau <david at ar.media.kyoto-u.ac.jp> writes:
>     David> At the end, in the original context (speeding the drawing
>     David> of spectrogram), this is the problem. Even if multiple
>     David> backend/toolkits have obviously an impact in performances,
>     David> I really don't see why a numpy function to convert an array
>     David> to a RGB representation should be 10-20 times slower than
>     David> matlab on the same machine.
>
> This isn't exactly right.  When matplotlib converts a 2D grayscale
> array to rgba, a lot goes on under the hood.  It's all numpy, but it's
> far from single function and it involves many passes through the
> data.  In principle, this could be done with one or two passes through
> the data.  In practice, our normalization and colormapping abstractions
> are so abstract that it is difficult (though not impossible) to
> special case and optimize.
>
Well, we managed to have more than a 100 % speed increase for to_rgba 
function already, with the help from Eric Firing. Before, both functors 
Normalize and Colormap were expensive; I think we got something like a 
2-3x speed increase in normalize (now, the main problem is the clip 
function, which lead to the discussion about numpy clip), and now, the 
Colormap functor takes 75 % of the time of to_rgba function with an 
array of 8000x256 samples:

        1    0.000    0.000    0.832    0.832 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:275(expose_event)
        1    0.010    0.010    0.831    0.831 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtkagg.py:71(_render_figure)
        1    0.000    0.000    0.821    0.821 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_agg.py:385(draw)
        1    0.000    0.000    0.819    0.819 
/home/david/local/lib/python2.4/site-packages/matplotlib/figure.py:511(draw)
        1    0.000    0.000    0.817    0.817 
/home/david/local/lib/python2.4/site-packages/matplotlib/axes.py:1043(draw)
        1    0.000    0.000    0.648    0.648 
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:1927(imshow)
        3    0.000    0.000    0.573    0.191 
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:883(gca)
        1    0.000    0.000    0.572    0.572 
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:950(ishold)
        1    0.007    0.007    0.510    0.510 
/home/david/local/lib/python2.4/site-packages/matplotlib/image.py:173(draw)
        1    0.109    0.109    0.503    0.503 
/home/david/local/lib/python2.4/site-packages/matplotlib/image.py:109(make_image)
        4    0.000    0.000    0.491    0.123 
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:903(gcf)
        1    0.000    0.000    0.491    0.491 
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:818(figure)
        1    0.000    0.000    0.491    0.491 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtkagg.py:36(new_figure_manager)
        1    0.024    0.024    0.482    0.482 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:401(__init__)
        1    0.000    0.000    0.458    0.458 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtkagg.py:25(_get_toolbar)
        1    0.001    0.001    0.458    0.458 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:496(__init__)
        1    0.000    0.000    0.458    0.458 
/home/david/local/lib/python2.4/site-packages/matplotlib/backend_bases.py:1112(__init__)
        1    0.010    0.010    0.458    0.458 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:557(_init_toolbar)
        1    0.029    0.029    0.448    0.448 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:595(_init_toolbar2_4)
        1    0.419    0.419    0.419    0.419 
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:967(__init__)
        1    0.002    0.002    0.393    0.393 
/home/david/local/lib/python2.4/site-packages/matplotlib/cm.py:50(to_rgba)
        1    0.111    0.111    0.307    0.307 
/home/david/local/lib/python2.4/site-packages/matplotlib/colors.py:570(__call__)

Of this 300 ms spent in Colormap functor, 200 ms are taken by the take 
function: this is the function which I think can be speed up considerably.
> The top-level routine is
>
>     def to_rgba(self, x, alpha=1.0):
>         '''Return a normalized rgba array corresponding to x.
>         If x is already an rgb or rgba array, return it unchanged.
>         '''
>         if hasattr(x, 'shape') and len(x.shape)>2: return x
>         x = ma.asarray(x)
>         x = self.norm(x)
>         x = self.cmap(x, alpha)
>         return x
>
> which implies at a minimum two passes through the data, one for norm
> and one for cmap.
>
> In 99% of the use cases, cmap is a LinearSegmentedColormap though
> users can define their own as long as it is callable.  My guess is
> that the expensive part is Colormap.__call__, the base class for
> LinearSegmentedColormap.  We could probably write some extension code
> that does the following routine in one pass through the data.  But it
> would be hairy.  In a quick look and rough count, I see about 10
> passes through the data in the function below.
So with above points, I agree that self.norm and self.cmap are the slow 
parts, but I think this can be much improved by improving the 
corresponding numpy functions. I think there is space to improve things 
without touching matplotlib,

cheers,

David