[Cython] Core dump when Python is built without debug symbols

[Nikita Nemkin]

On Thu, 30 May 2013 17:23:39 +0600, Marin Atanasov Nikolov  
<dnaeon at gmail.com> wrote:

> Hello,
>
> First, apologies if this is not exactly the right place to post this,  
> but I
> have  tried already cython-users@ and my post is still pending somewhere
> for approval, so I've decided to give it a shot here.
>
> I'm working on Cython wrappers for a C library and everything was going
> smooth on my dev machine, until I've decided to install the wrappers on
> another machine.
>
> The only difference between these machines is that my dev machine has
> Python built with debug symbols and the other machine does not have debug
> symbols in Python.
>
> The issue I'm having and was trying to solve for the past few days is  
> that
> if I run my Cython wrappers on a machine where Python does not have debug
> symbols it core dumps ugly. If I run the same code on another machine  
> with
> Python + debug symbols then everything is running perfect.
>
> I've done some debugging and tracing as well, and noticed that the core
> dump happens at PyTuple_Size() when I have Python *without* debug  
> symbols.
> Running Python + debug symbols shows that the interpreter takes a
>  different path and does not core dump (does not even call PyTuple_Size()
> for some reason)
>
> Anyway, I've uploaded the backtrace and some gdb tracing in here:
>
> * http://users.unix-heaven.org/~dnaeon/cython-wrappers/wrappers-trace.txt
>
> The code that I have you can find in here:
>
> * http://users.unix-heaven.org/~dnaeon/cython-wrappers/src/
>
> What really puzzles me is why it core dumps when Python does not have  
> debug
> symbols and why it takes a different path (and not core dumping) when
> Python is built with debug symbols.
>
> I'd say that if it core dumps because of a NULL pointer or something
> similar, then having Python with or without debug symbols should not  
> make a
> difference, but it does for some reason.
>
> I've been working on this for the past few days and I'm out of ideas now.
>
> Any help/hints/feedback is much appreciated.

In pkg-db.pxi, this line:

         return Pkg(<object>pkg)

and in pkg-pkg.pxi this line:

     def __cinit__(self, pkg):
         self._pkg = <c_pkg.pkg *>pkg

are the problem.

You are trying to pass a C pointer to __cinit__. What actually
happens is that, because of the <object> cast, Cython treats pkg as
as Python object _itself_ and tries to do some object operations on it.

Two ways to solve this (really common) problem:

1) Pass your pointer as a real python object containing the integer value
    of the pointer:

     def __cinit__(self, pkg):
         self._pkg = <c_pkg.pkg *><Py_ssize_t>pkg  # or Py_intptr_t

     return Pkg(<object><Py_ssize_t>pkg)

     Note the <Py_ssize_t> cast. When used on a C pointer, it makes a C
     integer that Cython convert to python int object. And when <Py_ssize_t>
     is used on a python int object, Cyton extracts its integer value
     as a C-level value, so it is castable to a C-level pointer.

     Using Py_ssize_t (or Py_intptr_t or your platform intptr_t) guarantees
     that the pointer will fit.

2) Don't use __cinit__. Declare your own cdef init method and call that.
    Since it's cdef method, it can take any C-level arguments, pointerw,
    whatever without the need to convert to python objects and back.

    @cython.final  # small optimization
    cdef _init(self, c_pkg.pkg * pkg):
         self._pkg = pkg

    # cdef __cinit__ not needed. Maybe use it to set default
    # attribute values.

    cdef Pkg pkg = Pkg.__new__(Pkg)  # fast object creation. Pkg() also  
works.
    pkg._init(pkg)

    (When you use the __new__ trick, __cnit__ is called without
    arguments and __init__ is not called at all. The benefit to
    this ugly syntax is that __new__ is faster than creating the object
    with traditional Pyhton syntax.)

Also, if all your initialization is pointer assigment, you can
just do it directly:

    cdef Pkg pkg = Pkg.__new__(Pkg)  # fast object creation
    pkg._pkg = pkg


There is a problem related to cython-dev. I think the code you
wrote should have never compiled in the first place. <PyObject*>
cast must be required if you want to reinterpret a C pointer
as object or vice versa.


Best regards,
Nikita Nemkin