[New-bugs-announce] [issue37603] parsetok(): Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed, when running get-pip.py

[STINNER Victor]

New submission from STINNER Victor <vstinner at redhat.com>:

bpo-16806 introduced a regression with the following change:

commit 995d9b92979768125ced4da3a56f755bcdf80f6e (refs/bisect/bad)
Author: Anthony Sottile <asottile at umich.edu>
Date:   Sat Jan 12 20:05:13 2019 -0800

    bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)

get-pip.py does now crash:

$ ./python get-pip.py --help
python: Parser/parsetok.c:266: parsetok: Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed.
Aborted (core dumped)

Python 3.8 is also affected.

get-pip.py comes from: https://bootstrap.pypa.io/get-pip.py

Reproducing depends if REALLOC() returns a new pointer, see below.

I attached get-pip2.py: simplified version which crash when using "./python get-pip2.py --help". If it doesn't crash, duplicate lines to make the file larger.

--

The root issue seems to be that tok->multi_line_start isn't updated on REALLOC() in tok_nextc():

                newbuf = (char *)PyMem_REALLOC(newbuf,
                                               newsize);
                if (newbuf == NULL) {
                    tok->done = E_NOMEM;
                    tok->cur = tok->inp;
                    return EOF;
                }
                tok->buf = newbuf;
                tok->cur = tok->buf + cur;
                tok->line_start = tok->cur;

I guess that multi_line_start should also be updated there?

                tok->multi_line_start = tok->cur;

----------
components: Interpreter Core
keywords: 3.8regression
messages: 348008
nosy: lukasz.langa, vstinner
priority: release blocker
severity: normal
status: open
title: parsetok(): Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed, when running get-pip.py
versions: Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37603>
_______________________________________