[issue31532] Py_GetPath, Py_SetPath memory corruption due to mixed PyMem_New micex with PyMem_Raw_Free

[Hartmut Goebel]

New submission from Hartmut Goebel:

When using

Py_GetPath();
Py_SetPath(pypath_w);

near the beginning of a program, Py_SetPath crashes with memory corruption if run on a systems with glibc's MALLOC_CHECK enabled.

The reason is: Py_GetPath and Py_SetPath use different memory interfaces.

* Py_GetPath() calls calculate_path(), which uses PyMem_New() to allocate the memory (see https://github.com/python/cpython/blob/v3.6.0/Modules/getpath.c#L738, assigned to `buf` first)

* But Py_SetPath() uses PyMem_RawFree() to free the memory (see https://github.com/python/cpython/blob/v3.6.0/Modules/getpath.c#L830).


This error DOES NOT occur on Windows, since for win32 there is a separate implementation, which uses PyMem_RawMalloc (see https://github.com/python/cpython/blob/v3.6.0/PC/getpathp.c#L378).

This error also DOES NOT occur in Python <= 3.5, since PYMEM_FUNCS have been the same as PYRAW_FUNCS (see https://github.com/python/cpython/blob/v3.5.0/Objects/obmalloc.c#L148). This was changed in v3.6.0: PYMEM_FUNCS now is PYOBJ_FUNCS, see https://github.com/python/cpython/blob/v3.6.0/Objects/obmalloc.c#L159.

This error only occurs when running on a system with glibc's MALLOC_CHECK enabled, which seems to be the default st least on CentOS, Fedora and ArchLinux.

Example code and relevant source see below.


Example stack trace
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*** glibc detected *** ./dist/jcollect3: double free or corruption (out): 0x00007fde271ab030 ***
======= Backtrace: =========
/lib64/libc.so.6[0x37f0675e66]
/lib64/libc.so.6[0x37f06789b3]
/tmp/_MEIhKg3kx/libpython3.6m.so.1.0(Py_SetPath+0x15)[0x7fde2d20e825]
./dist/jcollect3[0x4043e2]
./dist/jcollect3[0x402db2]
./dist/jcollect3[0x402fb0]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x37f061ed5d]
./dist/jcollect3[0x401a9e]
======= Memory map: ========


How to reproduce
~~~~~~~~~~~~~~~~~~~~~~~~~

An example for this problem is the bootloader of PyInstaller (as of 2017-08-01).

* Make sure you are running on a Linux distribution having glibc's MALLOC_CHECK enabled.

* Set up a virtual environment (just too keep your system clean)

pip install https://github.com/pyinstaller/pyinstaller/archive/develop@%7B2017-08-01%7D.zip
echo 'print("Hello")' > test.py
pyinstaller test.py
dist/test/test  # run the frozen script

The relevant source of PyInstaller's bootloader is at <https://github.com/pyinstaller/pyinstaller/blob/develop@%7B2017-08-01%7D/bootloader/src/pyi_pythonlib.c#L466>

----------
components: Interpreter Core
messages: 302624
nosy: htgoebel
priority: normal
severity: normal
status: open
title: Py_GetPath, Py_SetPath memory corruption due to mixed PyMem_New micex with PyMem_Raw_Free
type: crash
versions: Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31532>
_______________________________________