[New-bugs-announce] [issue22207] Test for integer overflow on Py_ssize_t: explicitly cast to size_t

[STINNER Victor]

New submission from STINNER Victor:

Python contains a lot of tests like this one:

    if (length > PY_SSIZE_T_MAX / 4)
         return PyErr_NoMemory();

where length type is Py_ssize_t.

This test uses signed integers. There is usually a "assert(length > 0);" before.

The issue #22110 enabled more compiler warnings and there are now warnings when the test uses an unsigned number. Example:

   if (size > PY_SSIZE_T_MAX - PyBytesObject_SIZE) ...

where PyBytesObject_SIZE is defined using offsetof() which returns a size_t.

I propose to always cast Py_ssize_t length to size_t to avoid undefined behaviour (I never know if the compiler chooses signed or unsigned at the end) to ensure that the test also fail for negative number. For example, the following test must fail for negative size:

   if ((size_t)size > (size_t)PY_SSIZE_T_MAX - PyBytesObject_SIZE) ...

Attached patch changes bytesobject.c, tupleobject.c and unicodeobject.c (and asdl.c). If the global approach is accepted, more files should be patched.

----------
files: test_overflow_ssize_t.patch
keywords: patch
messages: 225369
nosy: haypo, neologix, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Test for integer overflow on Py_ssize_t: explicitly cast to size_t
versions: Python 3.5
Added file: http://bugs.python.org/file36381/test_overflow_ssize_t.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22207>
_______________________________________