[issue8226] sys.setfilesystemencoding("xxx"); open("a") => stack overflow

[STINNER Victor]

New submission from STINNER Victor <victor.stinner at haypocalc.com>:

sys.setfilesystemencoding() doesn't check if the argument is a valid encoding name.

If the filesystem encoding is invalid, open("a") goes into an unlimited loop. The default recursion limit is 1000, but the example crash at 930: too bad :-) Each loop allocate ~9000 bytes: Linux creates a 8 MB stack by default, and so ~9000x930 uses all the stack.

Using a lower recursion limit, we can see the loop:
----------
$ ./python -c 'import sys; sys.setrecursionlimit(30); sys.setfilesystemencoding("xxx"); open("x")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 83, in search_function
    norm_encoding = normalize_encoding(encoding)
  File "/home/SHARE/SVN/py3k/Lib/encodings/__init__.py", line 55, in normalize_encoding
    if isinstance(encoding, bytes):
RuntimeError: maximum recursion depth exceeded while calling a Python object
----------

----------
components: Interpreter Core
messages: 101656
nosy: haypo
severity: normal
status: open
title: sys.setfilesystemencoding("xxx"); open("a") => stack overflow
type: crash
versions: Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8226>
_______________________________________