[pypy-dev] Paper on the HOPE JIT for Python

[Yury V. Zaytsev]

On Sun, 2015-06-21 at 11:52 +0200, Laura Creighton wrote:
> 
> 
> http://www.sciencedirect.com/science/article/pii/S2213133714000687 

I'm curious whether I'm the only one to find this paper disappointing?

To summarize, the authors developed a translation system into C++ for a
very restricted subset of Python, as well as some machinery to
automagically compile and make translated functions available to the
Python land. How justified is it to call this a method-level JIT, I find
rather questionable; following this line of thinking, even Cython
suddenly also becomes a JIT (see pyximport).

Anyways, the claimed advantage of this exercise is that they are able to
deduce the types, and so idiomatic Python code doesn't need to be
supplemented with type annotations. However, if one looks at the
benchmark codes, HOPE-friendly functions do not look much like idiomatic
Python code, so this argument is quite far fetched.

Amusingly, the benchmarks show that Cython beats HOPE in most cases, and
when it doesn't, it's either a very questionable case (such as the
"simplify" benchmark, don't get me started here...), or strange and
probably suboptimal Cython implementation ("star" benchmark, where older
NumPy mode instead of memory views is used, np.sqrt() is called in a
supposedly tight loop, unclear whether the call to np.sum() is actually
faster than unrolling, etc.).

In as far as PyPy is concerned, it so appears that the time it takes to
run most of their benchmarks about 100 times (as they claim they did) is
generally << 1 second, and most loops were like 1000 iterations anyways,
so most likely JIT didn't kick in any case (if I remember correctly, the
heuristic was some odd number of iterations that is slightly higher than
1000). I couldn't find what exactly they ran on PyPy and how they did it
from a cursory look, so it's unclear.

In conclusion, I'm afraid there is not much at all to learn from this
publication...

... except that, maybe, this kind of papers should be reviewed by
experts on the subject, and not by domain scientists (in this case,
astrophysicists), who are generally unlikely to reject such works, as
long as it looks reasonably "scientific" and the grammar is good enough.

-- 
Sincerely yours,
Yury V. Zaytsev