[python-committers] The state of our copies of libffi (was: Redoing the C API?)

[Zachary Ware]

On Thu, Mar 3, 2016 at 12:38 PM, Brett Cannon <brett at python.org> wrote:
> [...] the maintenance issue we have with ctypes (or at least that's
> my hang-up with it). I think we still have not figured out what code we have
> patched and so no one has been willing to update to a newer version of
> libffi. I think Zachary looked into it and got some distance but never far
> enough to feel comfortable with trying to update things.

Since I've been invoked, I'll try to explain our libffi situation as
far as I understand it.  This is just a history lesson, I don't really
have an opinion on what should be done with it.  I will opine that we
have some seriously old unmaintained code here, and the whole libffi
situation in cpython is far more complex than is ideal.

We actually have four separate copies of libffi:

Modules/_ctypes/libffi: This is a mostly-vanilla copy of libffi-3.1
(released 19May2014), lightly patched according to
Modules/_ctypes/libffi.diff.  This one is used for any non-OSX posix
build that doesn't use `--with-system-ffi`.  doko has done a pretty
good job keeping this one relatively up to date.

Modules/_ctypes/libffi_osx: This is a "lightly patched" copy from
somewhere before libffi-2.0, probably.  It has barely been touched
since 2009.  I've been given to understand that it has modifications
necessary to allow building fat binaries on OSX (Ned or Ronald would
know better than I), but I don't know if such modifications may have
made it upstream since pre-2.0.  This one is used for all OSX builds
that don't use `--with-system-ffi`.

Modules/_ctypes/libffi_msvc: This is a heavily patched copy from
27January2004 and is used for all Windows builds.  The patches include
additional functionality, namely
(IIRC) returning the number of arguments expected when calling a
foreign function with the wrong number of arguments to allow ctypes to
raise a ValueError instead of crashing.  I'm fairly certain those
modifications have not been even submitted upstream, and the
intervening twelve years (and my utter lack of experience working with
asm) scare me away from trying to forward-port the patches from
libffi_msvc.  I did attempt to use a vanilla libffi on Windows
sometime in 2014/2015, but ran into the issue that it doesn't have the
features ctypes wants, which made a few fairly basic tests fail.

Modules/_ctypes/libffi_arm_wince: I don't know why we even have this.
Nobody has touched it since ctypes was merged into cpython in 2006.

-- 
Zach