[Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

[M.-A. Lemburg]

Guido van Rossum wrote:
> On Sun, Aug 28, 2011 at 11:23 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> sorry for hooking in here with my usual Cython bias and promotion. When the
>> question comes up what a good FFI for Python should look like, it's an
>> obvious reaction from my part to throw Cython into the game.
>>
>> Terry Reedy, 28.08.2011 06:58:
>>>
>>> Dan, I once had the more or less the same opinion/question as you with
>>> regard to ctypes, but I now see at least 3 problems.
>>>
>>> 1) It seems hard to write it correctly. There are currently 47 open ctypes
>>> issues, with 9 being feature requests, leaving 38 behavior-related issues.
>>> Tom Heller has not been able to work on it since the beginning of 2010 and
>>> has formally withdrawn as maintainer. No one else that I know of has taken
>>> his place.
>>
>> Cython has an active set of developers and a rather large and growing user
>> base.
>>
>> It certainly has lots of open issues in its bug tracker, but most of them
>> are there because we *know* where the development needs to go, not so much
>> because we don't know how to get there. After all, the semantics of Python
>> and C/C++, between which Cython sits, are pretty much established.
>>
>> Cython compiles to C code for CPython, (hopefully soon [1]) to Python+ctypes
>> for PyPy and (mostly [2]) C++/CLI code for IronPython, which boils down to
>> the same build time and runtime kind of dependencies that the supported
>> Python runtimes have anyway. It does not add dependencies on any external
>> libraries by itself, such as the libffi in CPython's ctypes implementation.
>>
>> For the CPython backend, the generated code is very portable and is
>> self-contained when compiled against the CPython runtime (plus, obviously,
>> libraries that the user code explicitly uses). It generates efficient code
>> for all existing CPython versions starting with Python 2.4, with several
>> optimisations also for recent CPython versions (including the upcoming 3.3).
>>
>>
>>> 2) It is not trivial to use it correctly.
>>
>> Cython is basically Python, so Python developers with some C or C++
>> knowledge tend to get along with it quickly.
>>
>> I can't say yet how easy it is (or will be) to write code that is portable
>> across independent Python implementations, but given that that field is
>> still young, there's certainly a lot that can be done to aid this.
> 
> Cythin does sound attractive for cross-Python-implementation use. This
> is exciting.
> 
>>> I think it needs a SWIG-like
>>> companion script that can write at least first-pass ctypes code from the .h
>>> header files. Or maybe it could/should use header info at runtime (with the
>>> .h bundled with a module).
>>
>> From my experience, this is a "nice to have" more than a requirement. It has
>> been requested for Cython a couple of times, especially by new users, and
>> there are a couple of scripts out there that do this to some extent. But the
>> usual problem is that Cython users (and, similarly, ctypes users) do not
>> want a 1:1 mapping of a library API to a Python API (there's SWIG for that),
>> and you can't easily get more than a trivial mapping out of a script. But,
>> yes, a one-shot generator for the necessary declarations would at least help
>> in cases where the API to be wrapped is somewhat large.
> 
> Hm, the main use that was proposed here for ctypes is to wrap existing
> libraries (not to create nicer APIs, that can be done in pure Python
> on top of this). In general, an existing library cannot be called
> without access to its .h files -- there are probably struct and
> constant definitions, platform-specific #ifdefs and #defines, and
> other things in there that affect the linker-level calling conventions
> for the functions in the library. (Just like Python's own .h files --
> e.g. the extensive renaming of the Unicode APIs depending on
> narrow/wide build) How does Cython deal with these? I wonder if for
> this particular purpose SWIG isn't the better match. (If SWIG weren't
> universally hated, even by its original author. :-)

SIP is an alternative to SWIG:

     http://www.riverbankcomputing.com/software/sip/intro
     http://pypi.python.org/pypi/SIP

and there are a few others as well:

     http://wiki.python.org/moin/IntegratingPythonWithOtherLanguages

>>> 3) It seems to be slower than compiled C extension wrappers. That, at
>>> least, was the discovery of someone who re-wrote pygame using ctypes. (The
>>> hope was that using ctypes would aid porting to 3.x, but the time penalty
>>> was apparently too much for time-critical code.)
>>
>> Cython code can be as fast as C code, and in some cases, especially when
>> developer time is limited, even faster than hand written C extensions. It
>> allows for a straight forward optimisation path from regular Python code
>> down to the speed of C, and trivial interaction with C code itself, if the
>> need arises.
>>
>> Stefan
>>
>>
>> [1] The PyPy port of Cython is currently being written as a GSoC project.
>>
>> [2] The IronPython port of Cython was written to facility a NumPy port to
>> the .NET environment. It's currently not a complete port of all Cython
>> features.
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2011-10-04: PyCon DE 2011, Leipzig, Germany                36 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/