[pypy-dev] cpyext: Detecting pypy and other issues

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> Are you using callgrind or using valgrind for memory checks? The
>> former should work, the latter is rather pointless I think, because
>> RPython manages memory on it's own.

The point is to use valgrind.

>> If you have a bug in cpyext somehow (or pypy does, which is also a
>> possibility) you won't get deterministic behavior due to way memory
>> works in C.

Errr, the way memory works in C is exactly why valgrind does work.

>> No, never use refcounting GC for anything. What's the problem with say minimark?

That has nothing to do with the issue at hand.

To recap the whole thread, I tried porting my CPython extension to pypy.  I
wanted to ensure I did everything right but there turned out to be no
documentation (eg which mailing list is appropriate, common gotchas, how to
compile/configure pypy in such a way as to make gdb and valgrind play nice).
 It was suggested that I write as much of that doc as possible at which
point the blanks would be filled in for me.  So I wrote it and no blanks
have been filled in ...

https://docs.google.com/document/pub?id=1Ve9B6Gz5laV63zYFOlb_Lhe8ZBIw2V1756lquott6JI

You'll note that for cpython I show how to compile it in such a way as to
disable all memory optimisations and recycling so that it works perfectly
with valgrind.

With pypy my symptoms are that pypy crashes after trying to work on one of
my objects that the debugger shows is complete junk.  The distributed pypy
has no debug symbols.  Just compiling pypy following instructions results in
function names but no line numbers.  Running gdb and just hitting 'continue'
on a break point gives very different behaviour than running without gdb.

In order for me or anyone else having issues with cpyext we need some hints
as to how to proceed.  Things that would help greatly:

- - How to do a debug build that retains all symbols and line number information

- - Some gdb macros to make debugging easier.  For example CPython comes with
a .gdbinit with nice macros like 'pyo' so you can see what a PyObject *
represents

- - How to disable all memory optimisations and make it work with valgrind

- - How to get deterministic behaviour - ie exactly the same thing happens
each time whether you run gdb or not

Trying the refcount gc was to address the last two since it should result in
objects being freed the moment they are not used and not being dependent on
when GC is run or what address spaces it is run over.

I don't care what the solutions are, and attempted to find some by myself
which didn't work.  Trying to debug issues like this when none of the normal
tools work is exceptionally difficult especially given the non-deterministic
behaviour of pypy with cpyext.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk3MxdEACgkQmOOfHg372QTWcgCgpLcZS1KniZlRCwgHJr/qe8Oo
7Z4AniicHqC892bsFigTL8YMvMTMNlN6
=lHMW
-----END PGP SIGNATURE-----