[New-bugs-announce] [issue7313] 2to3 fails when source file starts with BOM

[Gabriel Genellina]

New submission from Gabriel Genellina <gagsl-py2 at yahoo.com.ar>:

If the source file is encoded in UTF-8 with a BOM, 
2to3 cannot parse it:

C:\APPS\python\trunk\PCbuild>.\python ..\Tools\scripts
\2to3 c:\temp\test_2to3.py
root: Generating grammar tables from C:\APPS\python
\trunk\lib\lib2to3\PatternGrammar.txt
root: Writing grammar tables to C:\APPS\python\trunk
\lib\lib2to3\PatternGrammar2.7.0.alpha.0.pickle
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Can't parse c:\temp\test_2to3.py: 
ParseError: bad token: type=55, value=u'\ufeff', 
context=(u'', (1, 0)
)
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't parse c:\temp\test_2to3.py: 
ParseError: bad token: type=55, value=u'\ufeff', 
context=(u'', (1, 0)
)


Tested with 3.1.1 and trunk.

----------
components: 2to3 (2.x to 3.0 conversion tool)
files: test_2to3_bom.py
messages: 95176
nosy: gagenellina
severity: normal
status: open
title: 2to3 fails when source file starts with BOM
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2
Added file: http://bugs.python.org/file15319/test_2to3_bom.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7313>
_______________________________________