[Cython] Metaclass bug corrupts the heap

[Robert Bradshaw]

Thanks for the careful analysis. See
https://github.com/cython/cython/commit/e8b54dec4cdc94764abda4a107bc37dcd58e6f13

On Tue, Sep 10, 2013 at 3:26 PM, Dénes Vadász <python2 at dvadasz.hu> wrote:
> Hello,
>
> according to valgrind the following Cython fragment causes a heap corruption
> (with Python 2.7.5 and Cython 0.19.1 on Ubuntu):
>
> cdef class MetaClass(type):
>     cdef int i
>
> class MyClass(object):
>     __metaclass__ = MetaClass
>
>
>
> Please find below the result of a many hour investigation (originally
> triggered by random crashes and heap corruptions of a bigger Cython program
> that uses metaclasses).
>
> MetaClass is compiled to a C structure that extends PyTypeObject (a cc. 200
> byte structure on a 32-bit architecture):
>
> struct __pyx_obj_4meta_MetaClass {
>   PyTypeObject __pyx_base;
>   int i;
> };
>
>
> Instances of PyTypeObject are supposed to be allocated statically when
> initializing extension modules, because the structure does not support
> (among others) garbage collection. However, when MyClass is created, an
> instance of struct __pyx_obj_4meta_MetaClass (cc. 200 + 4 bytes) is
> dynamically allocated by the Python memory management machinery. The
> machinery then tries to initialize the allocated memory. The problem is that
> it expects that the first member of struct __pyx_obj_4meta_MetaClass is of
> type PyHeapTypeObject (a cc. 500 byte structure) and after a type cast it
> writes to the tail members of the assumed PyHeapTypeObject, i.e. way beyond
> the allocated cc. 200 + 4 bytes, corrupting the object heap. This corruption
> is nicely reported by valgrind.
>
> The proposed fix is to make __pyx_obj_4meta_MetaClass extend
> PyHeapTypeObject:
>
> struct __pyx_obj_4meta_MetaClass {
>   PyHeapTypeObject __pyx_base;
>   int i;
> };
>
>
> This can be achieved by adding the below 2 lines to Compiler/Builtin.py:
>
> 382a383,384
>>         elif name == 'type':
>>             objstruct_cname = 'PyHeapTypeObject'
>
>
> So that the init_builtin_types function becomes:
>
> def init_builtin_types():
>     global builtin_types
>     for name, cname, methods in builtin_types_table:
>         utility = builtin_utility_code.get(name)
>         if name == 'frozenset':
>             objstruct_cname = 'PySetObject'
>         elif name == 'bool':
>             objstruct_cname = None
>        elif name == 'type':
>            objstruct_cname = 'PyHeapTypeObject'
>         else:
>             objstruct_cname = 'Py%sObject' % name.capitalize()
>         the_type = builtin_scope.declare_builtin_type(name, cname, utility,
> objstruct_cname)
>         builtin_types[name] = the_type
>         for method in methods:
>             method.declare_in_type(the_type)
>
>
> After patching my Cython installation with the above, valgrind stopped to
> complain and there were no more crashes.
>
> Please consider adding this fix in the next release of Cython.
>
> Regards
>
> Dénes Vadász
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>