[issue3975] PyTraceBack_Print() doesn't respect # coding: xxx header

[STINNER Victor]

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Thanks for your remarks amaury. I improved my patch:
 - PyMem_FREE(found_encoding) is called just after PyFile_FromFd()
 - Create static subfunction _Py_FindSourceFile(): find a file in 
sys.path
 - Consider that sys.path contains only unicode (and not bytes) string
 - Clear exception on "truncated = PyUnicode_FromUnicode(p, len);" 
error, but continue the execution
 - I added PyUnicode_check(lineobj)
 - Replace open(filename) by open(namebuf) (while searching the full 
path of the file) <= fix a regression introduced by my patch

Should I stop on the first error instead of using PyErr_Clear()? I 
would like to display the traceback even if an function failed.

Sum up of the patch version 4:
 - use open(O_RDONLY | O_BINARY) + PyFile_FromFd() instead of fopen()
 - open the file in unicode mode using the Python encoding found in 
the "#coding:..." header
 - consider sys.path as a list of unicode strings (and not of bytes 
strings)

I used _PyUnicode_AsStringAndSize() to convert unicode to string to be 
consistent with tb_printinternal(). As you noticed, it uses UTF-8 
which should is on Windows :-/ I propose to open a new issue when this 
one will be closed :-)

Added file: http://bugs.python.org/file11736/traceback_unicode-4.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3975>
_______________________________________