[pypy-dev] ctypes rawffi and ffi

[Elefterios Stamatogiannakis]

I'm exploring pypy's code so as to speed up callbacks from C code, so as 
to speed up sqlite module's UDF.

I have some questions:

  - What are the differences between ctypes, rawffi and ffi. Where 
should each one of them be used?

- I see that ctypes is build on top of rawffi and ffi. If one wishes to 
work around ctypes (so as to not have ctype's overhead) which of the 
rawffi or ffi should he use? Which of the two is faster at runtime?

  - How can i create a null pointer with _ffi?

And some remarks:

By only modifying pypy's sqlite module code, i managed to speed up 
sqlite's callbacks by 30% (for example there is a "for i in 
range(nargs)" line in  _sqlite3. _convert_params, which is a hot path).

Also the following line in _ctypes/function.py ._wrap_callable

args = [argtype._CData_retval(argtype.from_address(arg)._buffer)
                         for argtype, arg in zip(argtypes, args)]

Occupies a large percentage of the overall callback time (around 60-70%).

Assuming that pypy JITs all of the above callback code. Is it a problem 
having all these memory allocations for each callback (my test does 10M 
callbacks)? Is there a way to avoid as much as possible all these memory 
allocations.

Right now CPython runs my test (10M callbacks) in 1.2 sec and pypy needs 
from 9 to 14 secs. I suspect that the large spread of pypy's run times 
are due to GC.

Thank you in advance for your answers.

lefteris.