[pypy-dev] Fuzzing with Fusil and JIT crash

[Victor Stinner]

Hi,

Because of the last posts on your blog (a JIT better than just a PoC), I 
choosed to give a new try to PyPy. I'm trying PyPy with Fusil the fuzzer 
because it's my project, so I can easily fix bugs in Fusil, and also because 
it's a great tool to find bugs in PyPy :-)

I already found some bugs/crashs (see PyPy's bugtracker), but it's not bad 
(there are few bugs if you compare it to other projects). Since PyPy is open 
source, I can write patches to fix the issues ;-)

Slowly, I know PyPy enough to fix RPython bugs (I mean to fix modules written 
in RPython). But now my problem are the JIT crashes. I'm unable to locate the 
crash. There is not enough informations:

   BROKEN PROFILER DATA!
   ~~~ Crash in JIT!
   ~~~ <AssertionError object at 0x8449878>
   BROKEN PROFILER DATA!
   Fatal RPython error: CrashInJIT

Ok, nice, a crash. But where? What is the assertion? Sometimes, it fails with 
a TypeError (same output except the exception type).

gdb doesn't help:

   (gdb) where
   #0  0xb7fe1424 in __kernel_vsyscall ()
   #1  0xb7c1d3d0 in raise () from /lib/i686/cmov/libc.so.6
   #2  0xb7c20a85 in abort () from /lib/i686/cmov/libc.so.6
   #3  0x0804ce1e in main ()

"main()", that's all. No file name or line number :-/

On IRC, we asked me to recompiled PyPy using the hidden "make debug_exc" 
command. But it doesn't change anything. I'm not sure that the make did really 
enable the hidden debug features. How can I check it? Does it output something 
special? Always or only on a crash. On a crash, the output is the same using 
pypy-c-jit ("make") or pypy-c-jit-debug ("make debug_exc").

I will retry to recompile PyPy from scratch using directly the right commmand 
("make debug_exc").

Note: Don't try to compile PyPy C backend using -O0 or -O1. Only "-O3" and "-
O2 -fomit-frame-pointer" are supported.

Note2: "make clean" doesn't remove testing_1.s. Someone told me that it should 
be fixed now.

Oh, another problem: on make failure, all .s files are removed. It's not 
possible to debug the error. I added ".PRECIOUS: %.s" in the Makefile. Can it 
be done by default?


---


If you would like to try Fusil, it's available in Debian, Ubuntu, Mandriva, 
OpenEmbedded, Arch Linux, MacPort, Gentoo, ... For better performances and 
fewer false positives, clone the Mercurial repository. Fusil website:

    http://bitbucket.org/haypo/fusil/wiki/Home


To reproduce the JIT crash, it takes 20 to 60 seconds using the following 
command:

   export PYTHONPATH=~/prog/HG/fusil/:~/prog/HG/python-ptrace/
   ./pypy-c-jit-debug --jit threshold=10 \
      ~/prog/HG/fusil/fuzzers/fusil-python \
      --success=100 --blacklist=logging,compileall,this,socket

(socket is blacklisted because of the issue #465, a crash in _ssl)

You don't need to install Fusil if you change the PYTHONPATH.

-- 
Victor Stinner
http://www.haypocalc.com/