[issue26107] PEP 511: code.co_lnotab: use signed line number delta to support moving instructions in an optimizer

[STINNER Victor]

STINNER Victor added the comment:

We have many unit test in the Python test suite which rely on exact line numbers. Examples:

* test_sys_settrace
* test_pdb
* test_dis

I know them because they were all broken when my fatoptimizer project had bugs related to line numbers :-)

With my patch, the full Python test suite pass whereas my patch doesn't modify any test.


> I afraid this patch can cause problems with code tracing where it is assumed that lines are increased monotonically and *instr_lb <= frame->f_lasti < *instr_ub. We should carefully analyze the effect of the patch on the tracing.

First, my patch has no impact on frame->f_lasti.

The trace module and test_sys_settrace use frame.f_lineno which PyFrame_GetLineNumber(). This function returns f->f_lineno if the frame has a trace function, or PyCode_Addr2Line(). PyCode_Addr2Line() and PyFrame_GetLineNumber() still work with my patch.

When you trace a program, "negative line delta" and "negative instruction offset" are not new in Python: it's a basic requirement to support loop, when you compare two instructions seen by the tracer.

To be clear, my patch does *not* introduce negative line number delta in co_lnotab. It only *adds support* for negative line number delta. If a tool decodes co_lnotab using 8-bit unsigned number for line number delta, the tool still works even with the patch. It only starts to return wrong line numbers if you start debugging a program which has negative line numbers.

If you use fatoptimizer, you get such negative delta. But if you use an optimizer, you should be prepared to some subtle differences. The good practice is to disable all optimizers when you debug code. It's also really hard (or impossible) to debug C code optimized with -O3. I always use gcc -O0 to debug CPython.


> Before committing you must ask Guido for approval. AFAIK his had objections against code transformations that make debugging harder.

Are you aware of tools decoding directly co_lnotab?

Oh, I forgot the old Misc/gdbinit script which *does* decode directly co_lnotab. Does anyone still use it? If yes, it should also be updated.

I failed to reproduce the bug with Misc/gdbinit, beacuse bug only occurs if you debug a program which uses negative line number, and CPython doesn't produce negative line number in co_lnotab by default... So it would be "nice" to also support negative line number in Misc/gdbinit, but maybe it's ok to let this old script dying? :-D

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26107>
_______________________________________