[New-bugs-announce] [issue16254] PyUnicode_AsWideCharString() increases string size

[David Beazley]

New submission from David Beazley:

The PyUnicode_AsWideCharString() function is described as creating a new buffer of type wchar_t allocated by PyMem_Alloc() (which must be freed by the user).   However, if you use this function, it causes the size of the original string object to permanently increase.  For example, suppose you had some extension code like this:

static PyObject *py_receive_wchar(PyObject *self, PyObject *args) {
  PyObject *obj;
  wchar_t *s;
  Py_ssize_t len;

  if (!PyArg_ParseTuple(args, "U", &obj)) {
    return NULL;
  }
  if ((s = PyUnicode_AsWideCharString(obj, &len)) == NULL) {
    return NULL;
  }
  /* Do nothing */
  PyMem_Free(s);
  Py_RETURN_NONE;
}

Now, try an experiment (assume that the above extension function is available as 'receive_wchar'). 

>>> s = "Hell"*1000
>>> len(s)
4000
>>> import sys
>>> sys.getsizeof(s)
4049
>>> receive_wchar(s)
>>> sys.getsizeof(s)
20053
>>>

It seems that PyUnicode_AsWideCharString() may be filling in the wstr field of the associated PyASCIIObject structure from PEP393 (I haven't verified).  Once filled, it never seems to be discarded.

Background:  I am trying to figure out how to convert from Unicode to (wchar_t, int *) that doesn't cause a permanent increase in the memory footprint of the original Unicode object.  Also, I'm trying to stay away from deprecated Unicode APIs.

----------
components: Extension Modules, Interpreter Core, Unicode
messages: 173089
nosy: dabeaz, ezio.melotti
priority: normal
severity: normal
status: open
title: PyUnicode_AsWideCharString() increases string size
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16254>
_______________________________________