[pypy-dev] your thoughts on low level optimizations

[wlavrijsen at lbl.gov]

Hi Gertjan,

> With PyPy at the horizon, I am not so sure anymore. For one I'm not sure if
> PyPy will ever be able to use the C module, or use it efficiently

it's only the crossing into and out of a C extension module through cpyext
that is less than optimal. If that crossing does not happen often, the penalty
of the cross-language call overhead does not matter. Another option that you
have is to write an "extension" module in Python or RPython that just calls
into the C code through ctypes or libffi. That will be efficient.

Just keep the C module clean and simple: clear memory ownership rules, no use
of CPython internals, callbacks only on the Python side, etc. That way, any
extra code needed for glue is simple to write and maintain.

> I am also not certain if I should maybe forget about C altogether and rely
> on the JIT to compile the Python for loops that I have always tried to avoid.
> Would you go as far as to say that there will be no more reason for low level
> programming whatsoever? Or would you advise to write the component in RPython
> and use the translator to compile it?

Whatever works best, they all have their pro's and cons. With C, you can share
between PyPy and CPython and true low-level coding is by far the easiest in it.
RPython allows you to provide hints to the JIT which may make a big difference
in performance, but it isn't modular: RPython components become part of pypy-c
after the translation. Programming in Python is the cleanest, shares easily,
but performance behavior may be surprising, especially in tight loops.

I realize that that isn't an answer, but I don't think there is a single one
answer that is valid in all cases. Your original approach is most likely to
still be best: stick to Python until you find a performance problem, only then
look for and evaluate your other options.

> Regarding today's blog post, I have the feeling that this is meant more for
> wrapping existing C++ libraries than for starting new ones, is that correct?

Well, of course you can still write a new C++ library and wrap that too. And
in the process make it Python-bindings-friendly. :) But since your original
code was C, ctypes is likely easier to use since you can code in the needed
reflection info (you can also run Reflex over the C code and use that to
generate the Python-side annotations in a build process).

> Or if not, and it is, in fact, an option to consider, will this be able to
> work in CPython, too?

The code is based on an equivalent that exists for CPython. However, that is
not a standalone package, but part of ROOT (http://root.cern.ch). It could be
standalone, but I never saw the need: if you do not already have reflection
information libraries ("dictionaries") as we did for I/O, then SWIG or an
equivalent tool works just as well and is already available.

Best regards,
            Wim

P.S. I noticed on your web site that you're working on a project for Oce. I
      used to work there in times gone by. :)
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net