[pypy-dev] cpyext: Detecting pypy and other issues

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/14/2011 08:20 AM, Armin Rigo wrote:
> Ah, sorry.  I think I got mistaken in my previous mail.  The point is
> that you are debugging *only* a C extension, and not interested in
> debugging any of the rest of PyPy.

Correct.  More accurately I have a C extension that works perfectly with
CPython including very extensive testing and being valgrind clean.  However
with pypy things are a complete mess.

> Then all we said about GCs makes no sense

The point of mentioning GCs was to get deterministic behaviour and hopefully
to get valgrind working.

> You disable it
> with "make no_obmalloc", and then they are just normal malloc()ed and
> free()d objects from the C point of view.

Would you be able to add a target to the Makefile whose purpose is to make a
good pypy for C extension debugging.  ie no obmalloc, plentiful debug
symbols, ideally doing a GC run immediately before and after every call to
cpyext wrapped functions.

> This is independent of the
> GC, which anyway only handles the objects that are in native RPython
> format.

The GC matters in terms of being deterministic.  Here are 3 examples of
problems I have encountered when running my test suite (pure python code in
a separate file being a subclass of unittest with lots of test* methods):

- - My C level object has a member that is initialised with PyList_New(0).  A
while later a crash happens because it is garbage (a ref count in the tens
of millions when it will never go above 1 and is Py_CLEAR'ed in the destructor).

- - One early test calls a method implemented in C with an integer parameter.
 The pyarg_parse wants a string and hence should raise a type error but
doesn't.  In gdb I want to confirm what cpyext is passing in hence a request
for a 'pyo' macro or similar for gdb.

- - If I use gdb in the above and immediately hit 'continue' (ie not doing
anything else) then pypy fails the test suite on the grounds that particular
unittest method it was just executing doesn't exist

> or it would if we had _PyObject_Dump() in cpyext.

_PyObject_Dump() also spews the string to stdout IIRC.  Should I create a
ticket asking for this?

> Note that the objects that C extension modules see are "placeholder"
> PyObjects, that indirectly contain a reference to the "real" RPython
> object.

gdb only shows two members for a PyObject - a refcount and a type but no
pointer to the RPython object.  Having just those two fields isn't useful
for debugging hence the need for a 'pyo' or similar gdb macro so you know
what is hiding behind the PyObject.

> I'm pretty sure that the
> troubles you get are simply due to the placeholders being freed too
> early.

The item is allocated in an object initialiser and is Py_CLEAR'ed in the
destructor.  The destructor being run multiple times could cause a problem.
 It is a list of weakrefs so weakref handling having issues could also
grief.  I'm very sure the code is correct, but certainly believe that it
could be outwitted if the garbage collector ran at inopportune times.  Using
valgrind would tell me when the PyObject part is having issues but it would
also be nice if the rest of pypy played nice with valgrind as I mentioned
using the macros in the previous email as that would then make it clear who
exactly was responsible for any allocations, overwrites and frees of memory
that could end up at the same spot.

> That's the case for which
> adding a debugging mode in PyPy would be useful.

It is far preferable for the tool to analyse itself and complain at the
first opportunity rather than play happy go lucky.  I certainly believe that
many existing extensions will hit issues be they due to bugs in the
extensions, bugs in the CPython documentation and bugs in pypy.  It is
better to have a way of proactively finding them versus corrupting memory or
crashing as that just makes pypy look bad.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk3PPgQACgkQmOOfHg372QQZRQCgm5Pe2iE6FOBdSk7SF4dad7A5
EnIAoN8CsJK399q4PPjMDt/2msKTHBDm
=6ved
-----END PGP SIGNATURE-----