[Python-Dev] Re: Another test_compiler mystery

[Tim Peters]

More clues.

As expected, commenting out the tests in test_parser.py that contain
deeply nested tuples makes the problems go away.

A surprise:  adding code to every function in parsermodule.c to do
Py_FatalError() if the Windows stack-check fails doesn't make any
difference.  So it's not C recursion in parsermodule.c that causes
this (although I bet it *could*, it simply doesn't in this test case).

Not a surprise:   Grzegorz Makarewicz's patch to call the stack-check
code in pymalloc catches the problem "in time".  But we could put the
same patch anywhere in Python that gets called frequently, and catch
it in time.  That's too expensive to bear.

The Windows stack-check code just isn't called often enough, and the
recursion limit is "too large" now to live without a reliable stack
check.  Py_EnterRecursiveCall() in fact rarely calls the Windows
stack-check code.  At the time the Windows stack-check code first
*would* have complained (had it been called then), the tstate
recursion depth was 694, and _Py_CheckRecursionLimit was 973.

The funky way this code works, that means we can do ~150 more levels
of Python call before the Windows stack-check code is called again,
and that's too late.  We only checked that there's still room for 2K
pointers on the stack, and 150 Python-level calls means about 450
C-level calls in this test:  the stack at this point is hundreds (&
hundreds) of repetitions of

PyEval_EvalFrame ->
fast_function ->
call_function ->
PyEval_Frame -> etc etc etc

and PyEval_EvalFrame C frames are fat enough on their own for 150 of
them to blow what remains of the stack.

The easiest workaround for Windows is still to boost the stack size. 
I'm going to do that if nobody screams.

Looks like nobody has an explanation yet for why 2.3.4 consistently
yielded MemoryError but 2.4a2 mixes those with spurious KeyError and
SyntaxError exceptions.  That could be coincidence in this specific
test driver, though -- lookdict has always wiped out exceptions.

where's-stackless-when-you-need-it-ly y'rs  - tim