[Python-Dev] eval and triple quoted strings

[Benjamin Peterson]

2013/6/17 Guido van Rossum <guido at python.org>:
> On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2013/6/17 Guido van Rossum <guido at python.org>:
>>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>> What exactly does the parser handles better than the io module? Is it
>>> just the coding cookies? I suppose that works as long as the file is
>>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
>>> would fail pretty badly if it was UTF-16 (and yes, that's an
>>> abominable encoding for other reasons :-).
>>
>> The coding cookie is the main one. In fact, if you can't parse that,
>> you don't really know what encoding to open the file with at all.
>> There's also small things like BOM handling (you have to use the
>> utf-16-sig encoding with TextIO to get it removed) and defaulting to
>> UTF-8 (which the io module doesn't do) which is better left to the
>> parser.
>
> Maybe there are some lessons here that the TextIO module could learn?

UTF-8 by default would be great, but that ship has sailed.

Reading Python coding cookies is outside the purview of TextIOWrapper.
However, it would be good to have a function in the stdlib to read a
python source file to Unicode; I've definitely implemented that
several times.



--
Regards,
Benjamin