[New-bugs-announce] [issue23840] tokenize.open() leaks an open binary file on TextIOWrapper error

[STINNER Victor]

New submission from STINNER Victor:

If TextIOWrapper constructor fails in tokenize.open(), the open binary file is not closed and a ResourceWarning is emited.

Try attached patch:

$ python3 -Wd ~/tokenize_open_bug.py
tokenize.open failed: unknown encoding for 'test.py': xxx
/home/haypo/tokenize_open_bug.py:13: ResourceWarning: unclosed file <_io.BufferedReader name='test.py'>
  print("tokenize.open failed: %s" % err)

The fix is quite simple: add "try: ... except: buffer.close(); raise". If someone wants to fix this issue, an unit test must be added, test based on my script and ensures that the buffer is closed (ex: mock tokenize._builtin_open and checks that close() was called).

----------
files: tokenize_open_bug.py
keywords: easy
messages: 239786
nosy: haypo
priority: normal
severity: normal
status: open
title: tokenize.open() leaks an open binary file on TextIOWrapper error
versions: Python 3.4, Python 3.5
Added file: http://bugs.python.org/file38781/tokenize_open_bug.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23840>
_______________________________________