[Python-checkins] r79069 - python/branches/py3k/Doc/library/tokenize.rst
benjamin.peterson
python-checkins at python.org
Thu Mar 18 23:43:41 CET 2010
Author: benjamin.peterson
Date: Thu Mar 18 23:43:41 2010
New Revision: 79069
Log:
show a common usage of detect_encoding
Modified:
python/branches/py3k/Doc/library/tokenize.rst
Modified: python/branches/py3k/Doc/library/tokenize.rst
==============================================================================
--- python/branches/py3k/Doc/library/tokenize.rst (original)
+++ python/branches/py3k/Doc/library/tokenize.rst Thu Mar 18 23:43:41 2010
@@ -98,7 +98,17 @@
but disagree, a SyntaxError will be raised. Note that if the BOM is found,
``'utf-8-sig'`` will be returned as an encoding.
- If no encoding is specified, then the default of ``'utf-8'`` will be returned.
+ If no encoding is specified, then the default of ``'utf-8'`` will be
+ returned.
+
+ :func:`detect_encoding` is useful for robustly reading Python source files.
+ A common pattern for this follows::
+
+ def read_python_source(file_name):
+ with open(file_name, "rb") as fp:
+ encoding = tokenize.detect_encoding(fp.readline)[0]
+ with open(file_name, "r", encoding=encoding) as fp:
+ return fp.read()
Example of a script re-writer that transforms float literals into Decimal
More information about the Python-checkins
mailing list