[New-bugs-announce] [issue23654] infinite loop in faulthandler._stack_overflow

[Matt Frank]

New submission from Matt Frank:

When the faulthandler module is compiled at -O3 (the default for non-debug builds) with a compiler that does tailcall optimization the Modules/faulthandler.c:stack_overflow() function may become an infinite loop that does not expand the stack.  This puts the interpreter into an infinite loop with 100% CPU utilization that won't respond to SIGINT.  (But sending SIGTERM will get it to exit.)

The Intel compiler (15.0.1 on Linux) seems to be able to prove this optimization "safe".  (And thus cause the function to turn into an infinite loop.)  While gcc 4.8.2 and clang 3.4.2 do not currently optimize this call (because their pointer analysis isn't quite strong enough to deal with the pointer to the depth argument), there's no guarantee that they won't be able to optimize it in their next versions.

This patch adds a test between the recursive call and the return statement which makes it not a tail call.

----------
components: Extension Modules
files: faulthandler-infloop.patch
keywords: patch
messages: 237993
nosy: WanderingLogic
priority: normal
severity: normal
status: open
title: infinite loop in faulthandler._stack_overflow
type: behavior
versions: Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file38463/faulthandler-infloop.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23654>
_______________________________________