Python is slow?

[Robert Singer]

On Tue, 23 Sep 2008 06:23:12 -0700 (PDT), sturlamolden
<sturlamolden at yahoo.no> wrote:

>I have recently been playing with a kd-tree for solving the "post
>office problem" in a 12-dimensional space. This is pure cpu bound
>number crunching, a task for which I suspected Python to be
>inefficient.

Well, python is not a number crunching language. However much we would
like it to be (we would ? :-). No scripting language is.
Developing time is shorter, I agree, but when you have, for example a
problem which takes 8,31 minutes to go through in optimized fortran
code (like the one I had the other day), then that hardly matters.

>
>My prototype in Python 2.5 using NumPy required 0.41 seconds to
>construct the tree from 50,000 samples. Unfortunately, searching it
>felt a bit slow, finding the 11 nearest-neighbours of 1,000 points
>took 29.6 seconds (and there were still 49,000 to go). Naturally, I
>blamed this on Python. It would be 100 times faster if I used C++,
>right?


Not necessarily.
Before resorting to rewriting the problem try psyco. It speeds up
things sometimes.
Also, (I'm not that familiar with python yet, so I don't know how to
do it in python), try finding the bottlenecks of your calculation. Are
the loops where most of the processing time is wasted, or disk
accessing, or ... ?

>
>After having a working Python prototype, I resorted to rewrite the
>program in C++. The Python prototype took an hour to make, debug and
>verify. The same thing in C++ took me almost a day to complete, even
>with a working prototype as model. To my surprise, the resulting beast
>of C++ required 64.3 seconds to construct the same kd-tree. Searching
>the tree was not faster either, 1,000 points required 38.8 seconds. I
>wasted a day, only to find my Python prototype being the faster.



>
>We may conclude that I'm bad at programming C++, but I suspect that is
>not the case here. Albeit micro-benchmarks may indicate that Python is
>100-200 times slower than C++, they may not be applicable to the real
>world. Python can be very efficient. And when combined with libraries
>like NumPy, beating it's performance with hand-crafted C++ is
>difficult. At least, my 10 years experience programming scientific
>software in various languages was not sufficient to beat my own Python
>prototype with C++.
>
>That is not to say I have never seen C++ run a lot faster than Python.
>But it tends to be very short pieces of CPU bound code, no more than a
>function or two. But as the problem grows in complexity, C++
>accumulates too much of its own bloat.
>

Well, personally, I try to combine fortran (being a fortran programmer
by trade) with python (in the last few years), as I find fortran to
be, by two grades, more comfortable for solving scientific problems
then c (or python for that matter, although it has its merits).
Starting from ith his capabilities for "normal" array handling, to
optimisation and easy readability, to whatnot.


Best regards
Bob