ctype performance benchmark

[Nick Craig-Wood]

Wai Yip <aurora00 at gmail.com> wrote:
>  I started with ctypes because it is the battery included with the
>  Python standard library. My code is very fluid and I'm looking for
>  easy opportunity to optimize it. One example is to find the longest
>  common prefix among two strings. Right now I am comparing it character
>  by character with pure Python. It seems like an ideal low hanging
>  fruit. With ctype I can rewriting a few lines of Python into a few
>  lines of C. All the tools are available and no third party library is
>  needed.

Yes ctypes is very easy for this and I've used it like this in the
past too.  It makes interfacing C code very easy.

You might consider just writing a plain bit of C code.  If you make a
setup.py too (not hard) then it is very easy to distribute and use on
other platforms.

The Python C API is very easy to use - documentation is extensive.
You need to be very careful with references and the error checking is
boring and tedious!

>  It turned out the performance fail to match pure Python. This function
>  is called million of times in an inner loop. The overhead overwhelm
>  any performance gain with C. Eventually I have found success grouping
>  the data into larger chunk for each API call. This is what I
>  originally planned to do anyway. I am only sharing my experience here
>  that doing fine grained ctype function call has its limitation.

Interesting but not too unexpected - that translation of types is
non-trivial.

>  I have looked into Pyrex before and I was quite confused by the fusion
>  of C and Python langauage. Perhaps it is time for me to give it a
>  second look. I just heard of Cython now and it also look interesting.
>  I think it is helpful for both project to articulate more clearly what
>  they are, how they achieve the performance gain and to give more
>  guidance on good use cases. On the other hand, perhaps it is just me
>  are confused because I don't know enough of the Python internal.

Pyrex evolved/was forked into Cython.

Cython allows you to write a subset of python code that is compiled
into C.  You can also optionally provide some type annotations to get
it to produce better C code at the cost of making your code look less
like python.

I find Cython alternatively brilliant and frustrating!  Sometimes it
works really well, and sometimes I spend hours working out why my
program doesn't compile, or to find out the exact syntax I need to
interface with such and such a library.

I did a C library interfacing project with both ctypes and cython
recently as a comparison.  It came out to be about the same number of
lines of code for both.  I decided to run with the ctypes version as
it was part python.

Here is a short Cython example which interfaces with opendir /
closedir / readdir which python doesn't do natively.  Note the way you
interface with C structures (the real ones are used in the C code -
the annotations just tell cython how to use them).  Note also that
Directory is a class defined in C which I don't think you can't do
with ctypes without a wrapper class.

This compiles into 1320 lines of C, which in turn compile into 11380
bytes of shared object (when stripped).

import cython

cdef extern from "dirent.h":
    struct dirent:
        char d_name[0]
    ctypedef struct DIR
    DIR *opendir(char *name)
    int closedir(DIR *dirp)
    dirent *readdir(DIR *dirp)

cdef extern from "errno.h":
    int errno

cdef extern from "string.h":
    char *strerror(int errnum)

cdef class Directory:
    """Represents an open directory"""

    cdef DIR *handle

    def __init__(self, path):
        self.handle = opendir(path)
        if self.handle is NULL:
            raise OSError(errno, "Failed to open directory: %s" % strerror(errno))

    def readdir(self):
        """Read the next name in the directory"""
        cdef dirent *p
        p = readdir(self.handle)
        if p is NULL:
            return None
        return p.d_name

    def close(self):
        """Close the directory"""
        if self.handle is not NULL:
            closedir(self.handle)
            self.handle = NULL

    def __dealloc__(self):
        self.close()


-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick