[Python-Dev] Where are universal newlines handled in the parser/compiler?

[Brett Cannon]

If you import a module that uses '\r\n' line endings, Python does the
right thing. But if you read in the bytes for the same file and then
pass it to compile() you get an unhelpful syntax error pointing at a
blank line.

Normally I would say one should just open the source file as 'r'
instead of 'rb', but with source code that does not work well as their
can be a source encoding set. Lib/test/test_pep263.py is the perfect
example of this; Windows newlines with a koi8-r encoding.

What I would like to do is get compile() to work properly with a bytes
stream just as if Python itself was handling the compilation through
import and from a file directly. But before I try to dig into the
parser to figure out where the translation of newlines occurs (or
where the translation option is set), I thought I would ask to see if
anyone just happened to know (I have already spent a few hours
figuring out why Latin-1 encodings were not working with compile() so
I don't want to go diving into the maze of function calls in the
parser again).

-Brett