[issue39367] readline module core dumps Python 3.8.1 when calling exit()

[STINNER Victor]

STINNER Victor <vstinner at python.org> added the comment:

I'm able to reproduce the issue on Fedora 31, but only if the 3rd party readline module is loaded, instead of the stdlib "readline" module.

The problem is that https://pypi.org/project/readline/ misuses the Python memory allocator API. The GIL must be held to call PyMem_Malloc(). This project should be modified to use PyMem_RawMalloc().

https://docs.python.org/dev/c-api/memory.html#raw-memory-interface
vs
https://docs.python.org/dev/c-api/memory.html#memory-interface

It's a bug in a 3rd party module, I close the issue.


== packaged Python 3.8 ==

vstinner at apu$ python3.8 -m pip install --user readline

Python still uses the stdlib module by default (sys.path order):

vstinner at apu$ python3.8 
Python 3.8.1 (default, Dec 19 2019, 00:00:00) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> readline
<module 'readline' from '/usr/lib64/python3.8/lib-dynload/readline.cpython-38-x86_64-linux-gnu.so'>

I'm using uninstall just to know the path, but then cancel:

vstinner at apu$ python3.8 -m pip uninstall readline
Uninstalling readline-6.2.4.1:
  Would remove:
    /home/vstinner/.local/lib/python3.8/site-packages/readline-6.2.4.1-py3.8.egg-info
    /home/vstinner/.local/lib/python3.8/site-packages/readline.cpython-38-x86_64-linux-gnu.so
Proceed (y/n)? n

I reproduce the issue if I force to prefer the 3rd party readline:

vstinner at apu$ PYTHONPATH=/home/vstinner/.local/lib/python3.8/site-packages/ python3.8 
Python 3.8.1 (default, Dec 19 2019, 00:00:00) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
munmap_chunk(): invalid pointer
Abandon (core dumped)

== Python 3.9 compiled manually in debug mode ==

(...)
vstinner at apu$ PYTHONPATH=/home/vstinner/python/master/env/lib/python3.9/site-packages/ gdb -args env/bin/python 
GNU gdb (GDB) Fedora 8.3.50.20190824-26.fc31
(gdb) run

>>> import readline
Fatal Python error: Python memory allocator called without holding the GIL
Python runtime state: initialized

Current thread 0x00007ffff7c1f740 (most recent call first):
<no Python frame>

Program received signal SIGABRT, Aborted.
0x00007ffff7c60625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install libxcrypt-4.4.10-2.fc31.x86_64 ncurses-libs-6.1-12.20190803.fc31.x86_64
(gdb) where
#0  0x00007ffff7c60625 in raise () from /lib64/libc.so.6
#1  0x00007ffff7c498d9 in abort () from /lib64/libc.so.6
#2  0x0000000000560b43 in fatal_error (prefix=0x0, msg=0x6b4c40 "Python memory allocator called without holding the GIL", status=-1) at Python/pylifecycle.c:2249
#3  0x0000000000560b6f in Py_FatalError (msg=0x6b4c40 "Python memory allocator called without holding the GIL") at Python/pylifecycle.c:2259
#4  0x000000000047d988 in _PyMem_DebugCheckGIL () at Objects/obmalloc.c:2320
#5  0x000000000047d9a0 in _PyMem_DebugMalloc (ctx=0x7cd950 <_PyMem_Debug+48>, nbytes=17) at Objects/obmalloc.c:2328
#6  0x000000000047b958 in PyMem_Malloc (size=17) at Objects/obmalloc.c:605
#7  0x00007fffeaa48dc0 in call_readline (sys_stdin=0x7ffff7de67e0 <_IO_2_1_stdin_>, sys_stdout=0x7ffff7de7500 <_IO_2_1_stdout_>, prompt=0x7fffeaa9c480 ">>> ")
    at Modules/3.x/readline.c:1122
#8  0x000000000069e8a4 in PyOS_Readline (sys_stdin=0x7ffff7de67e0 <_IO_2_1_stdin_>, sys_stdout=0x7ffff7de7500 <_IO_2_1_stdout_>, prompt=0x7fffeaa9c480 ">>> ")
    at Parser/myreadline.c:358
#9  0x0000000000615155 in tok_nextc (tok=0x988f00) at Parser/tokenizer.c:848
#10 0x0000000000615f7d in tok_get (tok=0x988f00, p_start=0x7fffffffd118, p_end=0x7fffffffd110) at Parser/tokenizer.c:1128
#11 0x0000000000617876 in PyTokenizer_Get (tok=0x988f00, p_start=0x7fffffffd118, p_end=0x7fffffffd110) at Parser/tokenizer.c:1774
#12 0x0000000000612862 in parsetok (tok=0x988f00, g=0x7f1ce0 <_PyParser_Grammar>, start=256, err_ret=0x7fffffffd280, flags=0x7fffffffd27c) at Parser/parsetok.c:251
#13 0x0000000000612672 in PyParser_ParseFileObject (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename='<stdin>', enc=0x7fffeab24340 "utf-8", g=0x7f1ce0 <_PyParser_Grammar>, start=256, 
    ps1=0x7fffeaa9c480 ">>> ", ps2=0x7fffeaa9c520 "... ", err_ret=0x7fffffffd280, flags=0x7fffffffd27c) at Parser/parsetok.c:186
#14 0x0000000000567716 in PyParser_ASTFromFileObject (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename='<stdin>', enc=0x7fffeab24340 "utf-8", start=256, ps1=0x7fffeaa9c480 ">>> ", 
    ps2=0x7fffeaa9c520 "... ", flags=0x7fffffffd470, errcode=0x7fffffffd314, arena=0x7fffeab2d7c0) at Python/pythonrun.c:1387
#15 0x0000000000564fc1 in PyRun_InteractiveOneObjectEx (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename='<stdin>', flags=0x7fffffffd470) at Python/pythonrun.c:239
#16 0x0000000000564cbd in PyRun_InteractiveLoopFlags (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename_str=0x69f43a "<stdin>", flags=0x7fffffffd470) at Python/pythonrun.c:121
#17 0x0000000000564b80 in PyRun_AnyFileExFlags (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename=0x69f43a "<stdin>", closeit=0, flags=0x7fffffffd470) at Python/pythonrun.c:80
#18 0x000000000041e910 in pymain_run_stdin (config=0x819240, cf=0x7fffffffd470) at Modules/main.c:467
#19 0x000000000041ec10 in pymain_run_python (exitcode=0x7fffffffd4bc) at Modules/main.c:556
#20 0x000000000041ece6 in Py_RunMain () at Modules/main.c:632
#21 0x000000000041ed60 in pymain_main (args=0x7fffffffd520) at Modules/main.c:662
#22 0x000000000041edda in Py_BytesMain (argc=1, argv=0x7fffffffd648) at Modules/main.c:686
#23 0x000000000041d796 in main (argc=1, argv=0x7fffffffd648) at ./Programs/python.c:16

----------
nosy: +vstinner
resolution:  -> third party
stage:  -> resolved
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39367>
_______________________________________