setrecursionlimit

[Oscar Benjamin]

On 18 May 2016 at 17:11, Steven D'Aprano <steve at pearwood.info> wrote:
> The documentation for setrecursion limit warns against setting the limit too
> high:
>
>     [quote]
>     The highest possible limit is platform-dependent. A user may need to
>     set the limit higher when they have a program that requires deep
>     recursion and a platform that supports a higher limit. This should
>     be done with care, because a too-high limit can lead to a crash.
>     [end quote]
>
> https://docs.python.org/3/library/sys.html#sys.setrecursionlimit
>
> Indeed, if you set the recursion limit too high, you can smash the memory
> heap and get a segfault. How exactly does that work?
>
> Why doesn't setrecursionlimit() raise an exception when you try to set it
> too high? For example:
>
> sys.setrecursionlimit(2000000000)
>
> succeeds on my system, even though that's a ludicrously high number. (It is
> more than half the number of bytes of memory my computer has.)
>
>
> So why can't Python tell if I'm setting the limit too high?
>
> (I'm assuming that if it could, it would.)

It's not altogether impossible to do this but it requires ducking
underneath the C level to the machine code level. The C standard
doesn't define the exact number of bytes that should be used by a
stack frame. The exact number depends on the calling convention used
for your OS/hardware/compiler combination and also on the
optimisation/debug settings used by the compiler. A good optimiser
might be able to completely eliminate your function so that it doesn't
touch the stack for example.

Also it depends ultimately on the code path that leads to
PyEval_EvalFrameEx calling itself recursively. The recursion here is
indirect and there are multiple paths for it, depending on whether the
function is called with no arguments or whether it is a generator etc.
Then after that you need to consider extension modules. For example a
numpy array can store Python objects and perform operations on them.
If we have an ndarray of Fractions then there's no way at compile time
to know how much stack space ndarray.__add__ (implemented with a load
of complex C code) will need before calling e.g. Fraction.__add__.

Given the extension module case it's clearly impossible to compute the
hardware-stack memory requirements of a Python-level frame at compile
time. Christian has already explained why this wouldn't really work at
runtime either. There isn't even a portable way at the C level to
query the current value of the stack pointer. And even if you could
you'd have to make hardware-specific assumptions to be able to use
that information.

My understanding (although I have no direct experience here) is that
Stackless Python is an alternative implementation that gets around all
of these problems by avoiding the recursion altogether.

--
Oscar