[New-bugs-announce] [issue31979] Simplify converting non-ASCII strings to int, float and complex

[Serhiy Storchaka]

New submission from Serhiy Storchaka <storchaka+cpython at gmail.com>:

This issue is inspired by the tweet https://twitter.com/dabeaz/status/925787482515533830

    >>> a = 'n'
    >>> b = 'ñ'
    >>> sys.getsizeof(a)
   50
    >>> sys.getsizeof(b)
   74
    >>> float(b)
   Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
   ValueError: could not convert string to float: 'ñ'
    >>> sys.getsizeof(b)
   77

See also a discussion on Python-list (https://mail.python.org/pipermail/python-list/2017-November/728149.html) and Stack Overflow (https://stackoverflow.com/questions/47062184/why-does-the-size-of-this-python-string-change-on-a-failed-int-conversion).


When convert a non-ASCII string which don't contain non-ASCII decimal digits and spaces, this will fail, but will change the size of the input string due to creating an internal UTF-8 representation.

This can look surprising for beginners, but there is nothing wrong here. There are many cases in which an internal UTF-8 representation is created implicitly.

But looking on the code I have found that it is too complicated. Parsers to int, float and complex call _PyUnicode_TransformDecimalAndSpaceToASCII() which transforms non-ASCII decimal digits and spaces to ASCII. This functions uses the general function fixup() which takes a transformation function, creates a new string object, apply the transformation. It checks if the transformation produces a string with larger maximal character code than the original string and creates a new string object and repeat the transformation in that case. Finally, if the resulting string is equal to the original string, destroy the resulting string and returns the original string. In the past fixup() was used for implementing methods like upper(). But now _PyUnicode_TransformDecimalAndSpaceToASCII() is only the user of fixup(), and it doesn't need all complicated logic of fixup(). For example, this transformation never produces wider strings.

The proposed PR simplifies the code by getting rid of fixup() and fix_decimal_and_space_to_ascii(). The semantic of _PyUnicode_TransformDecimalAndSpaceToASCII() has been changed. It now always produces ASCII string (this also simplifies caller places). If non-ASCII characters which is not a decimal digit and is not a space is encountered, the rest of the string is replaced with '?' which will cause an error in parsers.

The only visible behavior change (except not changing the size of the original string) is changing the exception raised by float() and complex() when the string contains lone surrogates from UnicodeEncodeError to ValueError (the same as for other malformed strings). int() already contained a special case for this and raised a ValueError.

Unpatched:

>>> int('\ud800')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '\ud800'
>>> float('\ud800')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 0: surrogates not allowed
>>> complex('\ud800')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 0: surrogates not allowed

Patched:

>>> int('\ud800')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '\ud800'
>>> float('\ud800')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: could not convert string to float: '\ud800'
>>> complex('\ud800')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: complex() arg is a malformed string

This PR saves around 80 lines of code.

----------
assignee: serhiy.storchaka
components: Unicode
messages: 305824
nosy: ezio.melotti, haypo, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Simplify converting non-ASCII strings to int, float and complex
type: enhancement
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31979>
_______________________________________