[issue46913] UBSAN: test_ctypes, test_faulthandler and test_hashlib are failing

[STINNER Victor]

STINNER Victor <vstinner at python.org> added the comment:

Reproducer of test_ctypes undefined behavior:
---
from ctypes import *

class BITS(Structure):
    _fields_ = [("A", c_int, 1),
                ("B", c_int, 2),
                ("C", c_int, 3),
                ("D", c_int, 4),
                ("E", c_int, 5),
                ("F", c_int, 6),
                ("G", c_int, 7),
                ("H", c_int, 8),
                ("I", c_int, 9),

                ("M", c_short, 1),
                ("N", c_short, 2),
                ("O", c_short, 3),
                ("P", c_short, 4),
                ("Q", c_short, 5),
                ("R", c_short, 6),
                ("S", c_short, 7)]

b = BITS()
a = getattr(b, "M")
---

>     GET_BITFIELD(val, size); // <==== HERE

I expanded the macro:
---
    if (NUM_BITS(size)) {
        size_t s = sizeof(val)*8;
        Py_ssize_t low = LOW_BIT(size);
        Py_ssize_t nbits = NUM_BITS(size);
        // val=0, size=65553 = (1 << 16) + 17
        // s=16, low=17, nbits=1
        // s - low - nbits = (size_t)-2
        val <<= (s - low - nbits);
        val >>= (s - nbits);
    }
---

The problem is that (s - low - nbits) is negative (-2), but becomes a very large number since it's unsigned (s is unsigned: "sizeof(v)*8" in the macro).

C code:
---
#include <stdio.h>

int main()
{
    short s = 0x7fff;
    size_t z = (size_t)-2;
    s <<= z;
    printf("s = %hi\n", s);
    return 0;
}
---

GCC and clang disagree :-D
---
$ gcc x.c -o x -O3 -Wall -Wextra -Wconversion && ./x
s = 0

$ clang x.c -o x -O3 -Wall -Wextra -Wconversion && ./x
s = -5784
---

Moreover, runtime the binary built by clang produces a different result at each run...
---
$ ./x
s = -4824
$ ./x
s = 4120
$ ./x
s = -22344
$ ./x
s = -26744
$ ./x
s = -18184
---

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue46913>
_______________________________________