[pypy-dev] Running PyPy on top of CPython

[Matti Picus]

Hi Omer. I think much of what Armin wrote is already in the 
documentation like 
https://doc.pypy.org/en/latest/contributing.html#architecture and 
https://doc.pypy.org/en/latest/contributing.html#test-driven-development. 
Could you see where things can be improved?

Matti


On 11/1/22 4:17 pm, Omer Katz wrote:
> Hi Armin,
>
> Would you mind adding the above to the documentation?
> I think it'll be useful for newcomers.
>
> Best Regards,
> Omer Katz
>
> ‫בתאריך יום ב׳, 10 בינו׳ 2022 ב-18:03 מאת ‪Armin Rigo‬‏ 
> <‪armin.rigo at gmail.com <mailto:armin.rigo at gmail.com>‬‏>:‬
>
>     Hi,
>
>     On Mon, 10 Jan 2022 at 15:56, M A <teammember0x01 at gmail.com
>     <mailto:teammember0x01 at gmail.com>> wrote:
>     > I think you are confused. I just read in PyPy's documentation
>     that PyPy could be run from CPython. This sounds like something
>     that could help save me time by seeing if my changes work. I'm am
>     not sure why you think I am ignoring the tests. Yes I have tried
>     them. I am seeing if there is a more efficient way of trying out
>     my code without having to wait a long time of PyPy to recompile.
>
>     Sorry for not explaining myself clearly.  I've been trying all along
>     to tell you that you don't need to recompile PyPy, ever. As long as
>     not all the tests I listed are passing, then there is basically no
>     point.  (except that it feels good to see PyPy half-working instead of
>     not working at all, of course, but don't try to debug that)
>
>     The documentation says "PyPy can be run on top of CPython": that's
>     almost what all the tests are doing.  They run not the whole PyPy, but
>     instead some small example RPython code (sometimes written as RPython
>     in the test, sometimes at a lower level).  The point is that they test
>     the JIT backend by running it as normal Python code, on top of
>     CPython.  When working on the JIT backend, you don't want to run the
>     whole PyPy on top of CPython running with the JIT; while possible in
>     theory, it is far too slow to be practical.  Instead, run the tests,
>     which test the same thing but with a small bit of RPython code instead
>     of the many-thousand-of-lines-of-code of PyPy.
>
>     Let me try to explain myself in a different way.
>     What we're trying to do here is fix a JIT compiler emitting machine
>     code, at the level of RPython instead of Python.  That means that any
>     error is likely to cause not just a problem for some specific Python
>     function which you can find by running Python code; instead, bugs
>     cause random hard-to-debug crashes that show up at half-random
>     points, with no obvious one-to-one correspondence to what Python code
>     is running.  This could be caused for example by the wrong registers
>     being used or accidentally overwritten in some cases, leading to a
>     nonsense value propagating further, until the point of crash. Most
>     tests listed above check all basic cases and problems we encountered
>     so far.  The last test is producing and running random exemples, in an
>     attempt to find the random rare-case issues.  It has been the case
>     that this last test found a remaining bug after running for hours, but
>     I think that after half a day we can conclude that there is no bug
>     left to be found.  In the past, there were very rare bugs that went
>     uncaught by random testing too; these were a real pain to debug---I
>     once spent more than two weeks straight on a pair of them, running
>     "gdb" inside a translated PyPy, hacking at the generated C code to
>     dump more information to try to figure it out.  But these bugs were
>     more than just in the JIT backend (e.g. some interaction between the
>     JIT and the GC), and they are fixed now.  I'm telling this story to
>     explain why I do not recommend going down that path for the simpler
>     bugs that are already caught by the tests!
>
>     So, once more, I recommend working on this by running the tests, and
>     fixing them, until all tests pass.  Once all tests pass (and not
>     before) you can try to translate the whole PyPy, and at this point it
>     will likely work on the first try.
>
>     Sorry for not managing to get this message across to you in the past.
>     I hope I have done so now.
>
>
>     A bientôt,
>
>     Armin.
>     _______________________________________________
>     pypy-dev mailing list
>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>     https://mail.python.org/mailman/listinfo/pypy-dev
>     <https://mail.python.org/mailman/listinfo/pypy-dev>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev