Extension module question

[Stefan Behnel]

Evan Driscoll, 15.01.2012 08:37:
> As I hinted at in an earlier email, I'm working on a module which will
> allow calling readdir() (and FindFirstFile on Windows, hopefully pretty
> uniformly) from Python. The responses I got convinced me that it was a
> good idea to write a C-to-Python bridge as an extension module.

An even better idea is to write an extension module in Cython. Much faster
and simpler to learn and do.


> What I'm not sure about is how to store pointers to *C* stuff between
> calls. In particular, opendir() returns a DIR* which you then need to
> pass to calls to readdir() in the future (and closedir()).
> 
> So I've got this:
> 
>     static PyObject*
>     py_opendir(PyObject* self, PyObject* args)
>     {
>         const char* dirname = 0;
>         if (!PyArg_ParseTuple(args, "s", &dirname)) {
>             return NULL;
>         }
>         // Eventually want to Py_BEGIN_ALLOW_THREADS here

Cython allows you to do that by simply putting it into a "with nogil" block.


>         DIR* directory = opendir(dirname);
> 
>         PyObject out = PyBuildValue( ???, directory );
>         return out;
>     }
> 
> but I don't know what to build. (I might want to wrap it in a custom
> handle class or something, but I still need to know how to build the
> value I eventually store in an attribute of that class.)

I suggest you write an extension type and store the pointer in it directly.

Untested Cython code example:

    filesystem_encoding = sys.getfilesystemencoding()

    cdef class Directory:
        cdef DIR* c_directory
        def __cinit__(self, directory):
            if isinstance(directory, unicode):
                 directory = directory.encode(filesystem_encoding)
            cdef char* c_dirname = directory # raises TypeError on failure
            with nogil:
                self.c_directory = opendir(c_dirname)

        def __iter__(self):
            cdef char* name
            cdef size_t name_length
            for name in however_you_list_the_content_of(self.c_directory):
                name_length = length_which_you_may_know_of(name)
                yield name[:name_length].decode(filesystem_encoding)

and so on. Note how Cython does all sorts of things automatically for you
here, e.g. type conversions and the corresponding error handling as well as
all those nasty details of the C-level extension type implementation. Also
note that I'm using __cinit__() instead of __init__() for safety. See here:

http://docs.cython.org/src/userguide/special_methods.html#initialisation-methods-cinit-and-init

To implement the same interface for Unices and Windows, I suggest you write
two separate extension modules and hide them in a Python package that does
the appropriate platform specific imports at runtime.

Stefan