[Python-checkins] r72313 - in python/branches/py3k: Doc/library/codecs.rst Doc/library/os.rst Include/unicodeobject.h Lib/test/test_codecs.py Lib/test/test_os.py Misc/NEWS Modules/_io/fileio.c Modules/posixmodule.c Modules/python.c Objects/unicodeobject.c Python/codecs.c Python/pythonrun.c configure configure.in pyconfig.h.in
martin.v.loewis
python-checkins at python.org
Tue May 5 06:43:18 CEST 2009
Author: martin.v.loewis
Date: Tue May 5 06:43:17 2009
New Revision: 72313
Log:
Issue #5915: Implement PEP 383, Non-decodable Bytes in
System Character Interfaces.
Modified:
python/branches/py3k/Doc/library/codecs.rst
python/branches/py3k/Doc/library/os.rst
python/branches/py3k/Include/unicodeobject.h
python/branches/py3k/Lib/test/test_codecs.py
python/branches/py3k/Lib/test/test_os.py
python/branches/py3k/Misc/NEWS
python/branches/py3k/Modules/_io/fileio.c
python/branches/py3k/Modules/posixmodule.c
python/branches/py3k/Modules/python.c
python/branches/py3k/Objects/unicodeobject.c
python/branches/py3k/Python/codecs.c
python/branches/py3k/Python/pythonrun.c
python/branches/py3k/configure
python/branches/py3k/configure.in
python/branches/py3k/pyconfig.h.in
Modified: python/branches/py3k/Doc/library/codecs.rst
==============================================================================
--- python/branches/py3k/Doc/library/codecs.rst (original)
+++ python/branches/py3k/Doc/library/codecs.rst Tue May 5 06:43:17 2009
@@ -322,6 +322,8 @@
| ``'backslashreplace'`` | Replace with backslashed escape sequences |
| | (only for encoding). |
+-------------------------+-----------------------------------------------+
+| ``'utf8b'`` | Replace byte with surrogate U+DCxx. |
++-------------------------+-----------------------------------------------+
In addition, the following error handlers are specific to a single codec:
@@ -333,7 +335,7 @@
+------------------+---------+--------------------------------------------+
.. versionadded:: 3.1
- The ``'surrogates'`` error handler.
+ The ``'utf8b'`` and ``'surrogates'`` error handlers.
The set of allowed values can be extended via :meth:`register_error`.
Modified: python/branches/py3k/Doc/library/os.rst
==============================================================================
--- python/branches/py3k/Doc/library/os.rst (original)
+++ python/branches/py3k/Doc/library/os.rst Tue May 5 06:43:17 2009
@@ -51,6 +51,30 @@
``'ce'``, ``'java'``.
+.. _os-filenames:
+
+File Names, Command Line Arguments, and Environment Variables
+-------------------------------------------------------------
+
+In Python, file names, command line arguments, and environment
+variables are represented using the string type. On some systems,
+decoding these strings to and from bytes is necessary before passing
+them to the operating system. Python uses the file system encoding to
+perform this conversion (see :func:`sys.getfilesystemencoding`).
+
+.. versionchanged:: 3.1
+ On some systems, conversion using the file system encoding may
+ fail. In this case, Python uses the ``utf8b`` encoding error
+ handler, which means that undecodable bytes are replaced by a
+ Unicode character U+DCxx on decoding, and these are again
+ translated to the original byte on encoding.
+
+
+The file system encoding must guarantee to successfully decode all
+bytes below 128. If the file system encoding fails to provide this
+guarantee, API functions may raise UnicodeErrors.
+
+
.. _os-procinfo:
Process Parameters
@@ -688,12 +712,8 @@
.. function:: getcwd()
- Return a string representing the current working directory. On Unix
- platforms, this function may raise :exc:`UnicodeDecodeError` if the name of
- the current directory is not decodable in the file system encoding. Use
- :func:`getcwdb` if you need the call to never fail. Availability: Unix,
- Windows.
-
+ Return a string representing the current working directory.
+ Availability: Unix, Windows.
.. function:: getcwdb()
@@ -800,10 +820,8 @@
entries ``'.'`` and ``'..'`` even if they are present in the directory.
Availability: Unix, Windows.
- This function can be called with a bytes or string argument. In the bytes
- case, all filenames will be listed as returned by the underlying API. In the
- string case, filenames will be decoded using the file system encoding, and
- skipped if a decoding error occurs.
+ This function can be called with a bytes or string argument, and returns
+ filenames of the same datatype.
.. function:: lstat(path)
Modified: python/branches/py3k/Include/unicodeobject.h
==============================================================================
--- python/branches/py3k/Include/unicodeobject.h (original)
+++ python/branches/py3k/Include/unicodeobject.h Tue May 5 06:43:17 2009
@@ -198,6 +198,7 @@
# define PyUnicode_FromStringAndSize PyUnicodeUCS2_FromStringAndSize
# define PyUnicode_FromUnicode PyUnicodeUCS2_FromUnicode
# define PyUnicode_FromWideChar PyUnicodeUCS2_FromWideChar
+# define PyUnicode_FSConverter PyUnicodeUCS2_FSConverter
# define PyUnicode_GetDefaultEncoding PyUnicodeUCS2_GetDefaultEncoding
# define PyUnicode_GetMax PyUnicodeUCS2_GetMax
# define PyUnicode_GetSize PyUnicodeUCS2_GetSize
@@ -296,6 +297,7 @@
# define PyUnicode_FromStringAndSize PyUnicodeUCS4_FromStringAndSize
# define PyUnicode_FromUnicode PyUnicodeUCS4_FromUnicode
# define PyUnicode_FromWideChar PyUnicodeUCS4_FromWideChar
+# define PyUnicode_FSConverter PyUnicodeUCS4_FSConverter
# define PyUnicode_GetDefaultEncoding PyUnicodeUCS4_GetDefaultEncoding
# define PyUnicode_GetMax PyUnicodeUCS4_GetMax
# define PyUnicode_GetSize PyUnicodeUCS4_GetSize
@@ -693,25 +695,6 @@
PyObject *unicode,
const char *errors);
-/* Decode a null-terminated string using Py_FileSystemDefaultEncoding.
-
- If the encoding is supported by one of the built-in codecs (i.e., UTF-8,
- UTF-16, UTF-32, Latin-1 or MBCS), otherwise fallback to UTF-8 and replace
- invalid characters with '?'.
-
- The function is intended to be used for paths and file names only
- during bootstrapping process where the codecs are not set up.
-*/
-
-PyAPI_FUNC(PyObject*) PyUnicode_DecodeFSDefault(
- const char *s /* encoded string */
- );
-
-PyAPI_FUNC(PyObject*) PyUnicode_DecodeFSDefaultAndSize(
- const char *s, /* encoded string */
- Py_ssize_t size /* size */
- );
-
/* Returns a pointer to the default encoding (normally, UTF-8) of the
Unicode object unicode and the size of the encoded representation
in bytes stored in *size.
@@ -1252,6 +1235,33 @@
const char *errors /* error handling */
);
+/* --- File system encoding ---------------------------------------------- */
+
+/* ParseTuple converter which converts a Unicode object into the file
+ system encoding, using the PEP 383 error handler; bytes objects are
+ output as-is. */
+
+PyAPI_FUNC(int) PyUnicode_FSConverter(PyObject*, void*);
+
+/* Decode a null-terminated string using Py_FileSystemDefaultEncoding.
+
+ If the encoding is supported by one of the built-in codecs (i.e., UTF-8,
+ UTF-16, UTF-32, Latin-1 or MBCS), otherwise fallback to UTF-8 and replace
+ invalid characters with '?'.
+
+ The function is intended to be used for paths and file names only
+ during bootstrapping process where the codecs are not set up.
+*/
+
+PyAPI_FUNC(PyObject*) PyUnicode_DecodeFSDefault(
+ const char *s /* encoded string */
+ );
+
+PyAPI_FUNC(PyObject*) PyUnicode_DecodeFSDefaultAndSize(
+ const char *s, /* encoded string */
+ Py_ssize_t size /* size */
+ );
+
/* --- Methods & Slots ----------------------------------------------------
These are capable of handling Unicode objects and strings on input
Modified: python/branches/py3k/Lib/test/test_codecs.py
==============================================================================
--- python/branches/py3k/Lib/test/test_codecs.py (original)
+++ python/branches/py3k/Lib/test/test_codecs.py Tue May 5 06:43:17 2009
@@ -1516,6 +1516,34 @@
self.assertEquals(codecs.raw_unicode_escape_decode(r"\u1234"), ("\u1234", 6))
self.assertEquals(codecs.raw_unicode_escape_decode(br"\u1234"), ("\u1234", 6))
+class Utf8bTest(unittest.TestCase):
+
+ def test_utf8(self):
+ # Bad byte
+ self.assertEqual(b"foo\x80bar".decode("utf-8", "utf8b"),
+ "foo\udc80bar")
+ self.assertEqual("foo\udc80bar".encode("utf-8", "utf8b"),
+ b"foo\x80bar")
+ # bad-utf-8 encoded surrogate
+ self.assertEqual(b"\xed\xb0\x80".decode("utf-8", "utf8b"),
+ "\udced\udcb0\udc80")
+ self.assertEqual("\udced\udcb0\udc80".encode("utf-8", "utf8b"),
+ b"\xed\xb0\x80")
+
+ def test_ascii(self):
+ # bad byte
+ self.assertEqual(b"foo\x80bar".decode("ascii", "utf8b"),
+ "foo\udc80bar")
+ self.assertEqual("foo\udc80bar".encode("ascii", "utf8b"),
+ b"foo\x80bar")
+
+ def test_charmap(self):
+ # bad byte: \xa5 is unmapped in iso-8859-3
+ self.assertEqual(b"foo\xa5bar".decode("iso-8859-3", "utf8b"),
+ "foo\udca5bar")
+ self.assertEqual("foo\udca5bar".encode("iso-8859-3", "utf8b"),
+ b"foo\xa5bar")
+
def test_main():
support.run_unittest(
@@ -1543,6 +1571,7 @@
CharmapTest,
WithStmtTest,
TypesTest,
+ Utf8bTest,
)
Modified: python/branches/py3k/Lib/test/test_os.py
==============================================================================
--- python/branches/py3k/Lib/test/test_os.py (original)
+++ python/branches/py3k/Lib/test/test_os.py Tue May 5 06:43:17 2009
@@ -7,6 +7,7 @@
import unittest
import warnings
import sys
+import shutil
from test import support
# Tests creating TESTFN
@@ -698,9 +699,44 @@
self.assertRaises(os.error, os.setregid, 0, 0)
self.assertRaises(OverflowError, os.setregid, 1<<32, 0)
self.assertRaises(OverflowError, os.setregid, 0, 1<<32)
+
+ class Pep383Tests(unittest.TestCase):
+ filenames = [b'foo\xf6bar', 'foo\xf6bar'.encode("utf-8")]
+
+ def setUp(self):
+ self.fsencoding = sys.getfilesystemencoding()
+ sys.setfilesystemencoding("utf-8")
+ self.dir = support.TESTFN
+ self.bdir = self.dir.encode("utf-8", "utf8b")
+ os.mkdir(self.dir)
+ self.unicodefn = []
+ for fn in self.filenames:
+ f = open(os.path.join(self.bdir, fn), "w")
+ f.close()
+ self.unicodefn.append(fn.decode("utf-8", "utf8b"))
+
+ def tearDown(self):
+ shutil.rmtree(self.dir)
+ sys.setfilesystemencoding(self.fsencoding)
+
+ def test_listdir(self):
+ expected = set(self.unicodefn)
+ found = set(os.listdir(support.TESTFN))
+ self.assertEquals(found, expected)
+
+ def test_open(self):
+ for fn in self.unicodefn:
+ f = open(os.path.join(self.dir, fn))
+ f.close()
+
+ def test_stat(self):
+ for fn in self.unicodefn:
+ os.stat(os.path.join(self.dir, fn))
else:
class PosixUidGidTests(unittest.TestCase):
pass
+ class Pep383Tests(unittest.TestCase):
+ pass
def test_main():
support.run_unittest(
@@ -714,7 +750,8 @@
ExecTests,
Win32ErrorTests,
TestInvalidFD,
- PosixUidGidTests
+ PosixUidGidTests,
+ Pep383Tests
)
if __name__ == "__main__":
Modified: python/branches/py3k/Misc/NEWS
==============================================================================
--- python/branches/py3k/Misc/NEWS (original)
+++ python/branches/py3k/Misc/NEWS Tue May 5 06:43:17 2009
@@ -12,6 +12,8 @@
Core and Builtins
-----------------
+- Implement PEP 383, Non-decodable Bytes in System Character Interfaces.
+
- Issue #5890: in subclasses of 'property' the __doc__ attribute was
shadowed by classtype's, even if it was None. property now
inserts the __doc__ into the subclass instance __dict__.
Modified: python/branches/py3k/Modules/_io/fileio.c
==============================================================================
--- python/branches/py3k/Modules/_io/fileio.c (original)
+++ python/branches/py3k/Modules/_io/fileio.c Tue May 5 06:43:17 2009
@@ -245,7 +245,7 @@
return -1;
stringobj = PyUnicode_AsEncodedString(
- u, Py_FileSystemDefaultEncoding, NULL);
+ u, Py_FileSystemDefaultEncoding, "utf8b");
Py_DECREF(u);
if (stringobj == NULL)
return -1;
Modified: python/branches/py3k/Modules/posixmodule.c
==============================================================================
--- python/branches/py3k/Modules/posixmodule.c (original)
+++ python/branches/py3k/Modules/posixmodule.c Tue May 5 06:43:17 2009
@@ -493,12 +493,14 @@
char *p = strchr(*e, '=');
if (p == NULL)
continue;
- k = PyUnicode_FromStringAndSize(*e, (int)(p-*e));
+ k = PyUnicode_Decode(*e, (int)(p-*e),
+ Py_FileSystemDefaultEncoding, "utf8b");
if (k == NULL) {
PyErr_Clear();
continue;
}
- v = PyUnicode_FromString(p+1);
+ v = PyUnicode_Decode(p+1, strlen(p+1),
+ Py_FileSystemDefaultEncoding, "utf8b");
if (v == NULL) {
PyErr_Clear();
Py_DECREF(k);
@@ -534,6 +536,37 @@
return d;
}
+/* Convert a bytes object to a char*. Optionally lock the buffer if it is a
+ bytes array. */
+
+static char*
+bytes2str(PyObject* o, int lock)
+{
+ if(PyBytes_Check(o))
+ return PyBytes_AsString(o);
+ else if(PyByteArray_Check(o)) {
+ if (lock && PyObject_GetBuffer(o, NULL, 0) < 0)
+ /* On a bytearray, this should not fail. */
+ PyErr_BadInternalCall();
+ return PyByteArray_AsString(o);
+ } else {
+ /* The FS converter should have verified that this
+ is either bytes or bytearray. */
+ Py_FatalError("bad object passed to bytes2str");
+ /* not reached. */
+ return "";
+ }
+}
+
+/* Release the lock, decref the object. */
+static void
+release_bytes(PyObject* o)
+{
+ if (PyByteArray_Check(o))
+ o->ob_type->tp_as_buffer->bf_releasebuffer(NULL, 0);
+ Py_DECREF(o);
+}
+
/* Set a POSIX-specific error from errno, and return NULL */
@@ -558,10 +591,11 @@
static PyObject *
-posix_error_with_allocated_filename(char* name)
+posix_error_with_allocated_filename(PyObject* name)
{
- PyObject *rc = PyErr_SetFromErrnoWithFilename(PyExc_OSError, name);
- PyMem_Free(name);
+ PyObject *rc = PyErr_SetFromErrnoWithFilename(PyExc_OSError,
+ bytes2str(name, 0));
+ release_bytes(name);
return rc;
}
@@ -728,17 +762,19 @@
static PyObject *
posix_1str(PyObject *args, char *format, int (*func)(const char*))
{
- char *path1 = NULL;
+ PyObject *opath1 = NULL;
+ char *path1;
int res;
if (!PyArg_ParseTuple(args, format,
- Py_FileSystemDefaultEncoding, &path1))
+ PyUnicode_FSConverter, &opath1))
return NULL;
+ path1 = bytes2str(opath1, 1);
Py_BEGIN_ALLOW_THREADS
res = (*func)(path1);
Py_END_ALLOW_THREADS
if (res < 0)
- return posix_error_with_allocated_filename(path1);
- PyMem_Free(path1);
+ return posix_error_with_allocated_filename(opath1);
+ release_bytes(opath1);
Py_INCREF(Py_None);
return Py_None;
}
@@ -748,17 +784,20 @@
char *format,
int (*func)(const char *, const char *))
{
- char *path1 = NULL, *path2 = NULL;
+ PyObject *opath1, *opath2;
+ char *path1, *path2;
int res;
if (!PyArg_ParseTuple(args, format,
- Py_FileSystemDefaultEncoding, &path1,
- Py_FileSystemDefaultEncoding, &path2))
+ PyUnicode_FSConverter, &opath1,
+ PyUnicode_FSConverter, &opath2))
return NULL;
+ path1 = bytes2str(opath1, 1);
+ path2 = bytes2str(opath2, 1);
Py_BEGIN_ALLOW_THREADS
res = (*func)(path1, path2);
Py_END_ALLOW_THREADS
- PyMem_Free(path1);
- PyMem_Free(path2);
+ release_bytes(opath1);
+ release_bytes(opath2);
if (res != 0)
/* XXX how to report both path1 and path2??? */
return posix_error();
@@ -1560,8 +1599,8 @@
int (*wstatfunc)(const Py_UNICODE *, STRUCT_STAT *))
{
STRUCT_STAT st;
- char *path = NULL; /* pass this to stat; do not free() it */
- char *pathfree = NULL; /* this memory must be free'd */
+ PyObject *opath;
+ char *path;
int res;
PyObject *result;
@@ -1590,25 +1629,24 @@
#endif
if (!PyArg_ParseTuple(args, format,
- Py_FileSystemDefaultEncoding, &path))
+ PyUnicode_FSConverter, &opath))
return NULL;
- pathfree = path;
-
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
res = (*statfunc)(path, &st);
Py_END_ALLOW_THREADS
if (res != 0) {
#ifdef MS_WINDOWS
- result = win32_error("stat", pathfree);
+ result = win32_error("stat", path);
#else
- result = posix_error_with_filename(pathfree);
+ result = posix_error_with_filename(path);
#endif
}
else
result = _pystat_fromstructstat(&st);
- PyMem_Free(pathfree);
+ release_bytes(opath);
return result;
}
@@ -1625,6 +1663,7 @@
static PyObject *
posix_access(PyObject *self, PyObject *args)
{
+ PyObject *opath;
char *path;
int mode;
@@ -1644,13 +1683,14 @@
are also valid. */
PyErr_Clear();
}
- if (!PyArg_ParseTuple(args, "eti:access",
- Py_FileSystemDefaultEncoding, &path, &mode))
+ if (!PyArg_ParseTuple(args, "O&i:access",
+ PyUnicode_FSConverter, &opath, &mode))
return 0;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
attr = GetFileAttributesA(path);
Py_END_ALLOW_THREADS
- PyMem_Free(path);
+ release_bytes(opath);
finish:
if (attr == 0xFFFFFFFF)
/* File does not exist, or cannot read attributes */
@@ -1663,13 +1703,14 @@
|| (attr & FILE_ATTRIBUTE_DIRECTORY));
#else
int res;
- if (!PyArg_ParseTuple(args, "eti:access",
- Py_FileSystemDefaultEncoding, &path, &mode))
+ if (!PyArg_ParseTuple(args, "O&i:access",
+ PyUnicode_FSConverter, &opath, &mode))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
res = access(path, mode);
Py_END_ALLOW_THREADS
- PyMem_Free(path);
+ release_bytes(opath);
return PyBool_FromLong(res == 0);
#endif
}
@@ -1750,11 +1791,11 @@
#ifdef MS_WINDOWS
return win32_1str(args, "chdir", "y:chdir", win32_chdir, "U:chdir", win32_wchdir);
#elif defined(PYOS_OS2) && defined(PYCC_GCC)
- return posix_1str(args, "et:chdir", _chdir2);
+ return posix_1str(args, "O&:chdir", _chdir2);
#elif defined(__VMS)
- return posix_1str(args, "et:chdir", (int (*)(const char *))chdir);
+ return posix_1str(args, "O&:chdir", (int (*)(const char *))chdir);
#else
- return posix_1str(args, "et:chdir", chdir);
+ return posix_1str(args, "O&:chdir", chdir);
#endif
}
@@ -1779,6 +1820,7 @@
static PyObject *
posix_chmod(PyObject *self, PyObject *args)
{
+ PyObject *opath = NULL;
char *path = NULL;
int i;
int res;
@@ -1809,9 +1851,10 @@
are also valid. */
PyErr_Clear();
}
- if (!PyArg_ParseTuple(args, "eti:chmod", Py_FileSystemDefaultEncoding,
- &path, &i))
+ if (!PyArg_ParseTuple(args, "O&i:chmod", PyUnicode_FSConverter,
+ &opath, &i))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
attr = GetFileAttributesA(path);
if (attr != 0xFFFFFFFF) {
@@ -1826,22 +1869,23 @@
Py_END_ALLOW_THREADS
if (!res) {
win32_error("chmod", path);
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
- PyMem_Free(path);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
#else /* Py_WIN_WIDE_FILENAMES */
- if (!PyArg_ParseTuple(args, "eti:chmod", Py_FileSystemDefaultEncoding,
- &path, &i))
+ if (!PyArg_ParseTuple(args, "O&i:chmod", PyUnicode_FSConverter,
+ &opath, &i))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
res = chmod(path, i);
Py_END_ALLOW_THREADS
if (res < 0)
- return posix_error_with_allocated_filename(path);
- PyMem_Free(path);
+ return posix_error_with_allocated_filename(opath);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
#endif
@@ -1877,18 +1921,20 @@
static PyObject *
posix_lchmod(PyObject *self, PyObject *args)
{
- char *path = NULL;
+ PyObject *opath;
+ char *path;
int i;
int res;
- if (!PyArg_ParseTuple(args, "eti:lchmod", Py_FileSystemDefaultEncoding,
- &path, &i))
+ if (!PyArg_ParseTuple(args, "O&i:lchmod", PyUnicode_FSConverter,
+ &opath, &i))
return NULL;
+ path = bytes2str(opath, 1)
Py_BEGIN_ALLOW_THREADS
res = lchmod(path, i);
Py_END_ALLOW_THREADS
if (res < 0)
- return posix_error_with_allocated_filename(path);
- PyMem_Free(path);
+ return posix_error_with_allocated_filename(opath);
+ release_bytes(opath);
Py_RETURN_NONE;
}
#endif /* HAVE_LCHMOD */
@@ -1902,18 +1948,20 @@
static PyObject *
posix_chflags(PyObject *self, PyObject *args)
{
+ PyObject *opath;
char *path;
unsigned long flags;
int res;
- if (!PyArg_ParseTuple(args, "etk:chflags",
- Py_FileSystemDefaultEncoding, &path, &flags))
+ if (!PyArg_ParseTuple(args, "O&k:chflags",
+ PyUnicode_FSConverter, &opath, &flags))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
res = chflags(path, flags);
Py_END_ALLOW_THREADS
if (res < 0)
- return posix_error_with_allocated_filename(path);
- PyMem_Free(path);
+ return posix_error_with_allocated_filename(opath);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
}
@@ -1928,18 +1976,20 @@
static PyObject *
posix_lchflags(PyObject *self, PyObject *args)
{
+ PyObject *opath;
char *path;
unsigned long flags;
int res;
- if (!PyArg_ParseTuple(args, "etk:lchflags",
- Py_FileSystemDefaultEncoding, &path, &flags))
+ if (!PyArg_ParseTuple(args, "O&k:lchflags",
+ PyUnicode_FSConverter, &path, &flags))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
res = lchflags(path, flags);
Py_END_ALLOW_THREADS
if (res < 0)
- return posix_error_with_allocated_filename(path);
- PyMem_Free(path);
+ return posix_error_with_allocated_filename(opath);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
}
@@ -1953,7 +2003,7 @@
static PyObject *
posix_chroot(PyObject *self, PyObject *args)
{
- return posix_1str(args, "et:chroot", chroot);
+ return posix_1str(args, "O&:chroot", chroot);
}
#endif
@@ -1996,19 +2046,21 @@
static PyObject *
posix_chown(PyObject *self, PyObject *args)
{
- char *path = NULL;
+ PyObject *opath;
+ char *path;
long uid, gid;
int res;
- if (!PyArg_ParseTuple(args, "etll:chown",
- Py_FileSystemDefaultEncoding, &path,
+ if (!PyArg_ParseTuple(args, "O&ll:chown",
+ PyUnicode_FSConverter, &opath,
&uid, &gid))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
res = chown(path, (uid_t) uid, (gid_t) gid);
Py_END_ALLOW_THREADS
if (res < 0)
- return posix_error_with_allocated_filename(path);
- PyMem_Free(path);
+ return posix_error_with_allocated_filename(opath);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
}
@@ -2045,19 +2097,21 @@
static PyObject *
posix_lchown(PyObject *self, PyObject *args)
{
- char *path = NULL;
+ PyObject *opath;
+ char *path;
int uid, gid;
int res;
- if (!PyArg_ParseTuple(args, "etii:lchown",
- Py_FileSystemDefaultEncoding, &path,
+ if (!PyArg_ParseTuple(args, "O&ii:lchown",
+ PyUnicode_FSConverter, &opath,
&uid, &gid))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
res = lchown(path, (uid_t) uid, (gid_t) gid);
Py_END_ALLOW_THREADS
if (res < 0)
- return posix_error_with_allocated_filename(path);
- PyMem_Free(path);
+ return posix_error_with_allocated_filename(opath);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
}
@@ -2113,7 +2167,7 @@
return posix_error();
if (use_bytes)
return PyBytes_FromStringAndSize(buf, strlen(buf));
- return PyUnicode_Decode(buf, strlen(buf), Py_FileSystemDefaultEncoding,"strict");
+ return PyUnicode_Decode(buf, strlen(buf), Py_FileSystemDefaultEncoding,"utf8b");
}
PyDoc_STRVAR(posix_getcwd__doc__,
@@ -2146,7 +2200,7 @@
static PyObject *
posix_link(PyObject *self, PyObject *args)
{
- return posix_2str(args, "etet:link", link);
+ return posix_2str(args, "O&O&:link", link);
}
#endif /* HAVE_LINK */
@@ -2171,6 +2225,7 @@
HANDLE hFindFile;
BOOL result;
WIN32_FIND_DATA FileData;
+ PyObject *opath;
char namebuf[MAX_PATH+5]; /* Overallocate for \\*.*\0 */
char *bufptr = namebuf;
Py_ssize_t len = sizeof(namebuf)-5; /* only claim to have space for MAX_PATH */
@@ -2260,9 +2315,16 @@
}
#endif
- if (!PyArg_ParseTuple(args, "et#:listdir",
- Py_FileSystemDefaultEncoding, &bufptr, &len))
+ if (!PyArg_ParseTuple(args, "O&:listdir",
+ PyUnicode_FSConverter, &opath))
return NULL;
+ if (PyObject_Size(opath)+1 > MAX_PATH) {
+ PyErr_SetString(PyExc_ValueError, "path too long");
+ Py_DECREF(opath);
+ return NULL;
+ }
+ strcpy(namebuf, bytes2str(opath, 0));
+ len = PyObject_Size(opath);
if (len > 0) {
char ch = namebuf[len-1];
if (ch != SEP && ch != ALTSEP && ch != ':')
@@ -2324,6 +2386,7 @@
#ifndef MAX_PATH
#define MAX_PATH CCHMAXPATH
#endif
+ PyObject *oname;
char *name, *pt;
Py_ssize_t len;
PyObject *d, *v;
@@ -2333,11 +2396,13 @@
FILEFINDBUF3 ep;
APIRET rc;
- if (!PyArg_ParseTuple(args, "et#:listdir",
- Py_FileSystemDefaultEncoding, &name, &len))
+ if (!PyArg_ParseTuple(args, "O&:listdir",
+ PyUnicode_FSConverter, &oname))
return NULL;
+ name = bytes2str(oname);
+ len = PyObject_Size(oname);
if (len >= MAX_PATH) {
- PyMem_Free(name);
+ release_bytes(oname);
PyErr_SetString(PyExc_ValueError, "path too long");
return NULL;
}
@@ -2350,7 +2415,7 @@
strcpy(namebuf + len, "*.*");
if ((d = PyList_New(0)) == NULL) {
- PyMem_Free(name);
+ release_bytes(oname);
return NULL;
}
@@ -2363,7 +2428,7 @@
if (rc != NO_ERROR) {
errno = ENOENT;
- return posix_error_with_allocated_filename(name);
+ return posix_error_with_allocated_filename(oname);
}
if (srchcnt > 0) { /* If Directory is NOT Totally Empty, */
@@ -2393,11 +2458,11 @@
} while (DosFindNext(hdir, &ep, sizeof(ep), &srchcnt) == NO_ERROR && srchcnt > 0);
}
- PyMem_Free(name);
+ release_bytes(oname);
return d;
#else
-
- char *name = NULL;
+ PyObject *oname;
+ char *name;
PyObject *d, *v;
DIR *dirp;
struct dirent *ep;
@@ -2408,14 +2473,15 @@
arg_is_unicode = 0;
PyErr_Clear();
}
- if (!PyArg_ParseTuple(args, "et:listdir", Py_FileSystemDefaultEncoding, &name))
+ if (!PyArg_ParseTuple(args, "O&:listdir", PyUnicode_FSConverter, &oname))
return NULL;
+ name = bytes2str(oname, 1);
if ((dirp = opendir(name)) == NULL) {
- return posix_error_with_allocated_filename(name);
+ return posix_error_with_allocated_filename(oname);
}
if ((d = PyList_New(0)) == NULL) {
closedir(dirp);
- PyMem_Free(name);
+ release_bytes(oname);
return NULL;
}
for (;;) {
@@ -2429,7 +2495,7 @@
} else {
closedir(dirp);
Py_DECREF(d);
- return posix_error_with_allocated_filename(name);
+ return posix_error_with_allocated_filename(oname);
}
}
if (ep->d_name[0] == '.' &&
@@ -2447,18 +2513,16 @@
w = PyUnicode_FromEncodedObject(v,
Py_FileSystemDefaultEncoding,
- "strict");
- if (w != NULL) {
- Py_DECREF(v);
+ "utf8b");
+ Py_DECREF(v);
+ if (w != NULL)
v = w;
- }
else {
- /* Ignore undecodable filenames, as discussed
- * in issue 3187. To include these,
- * use getcwdb(). */
- PyErr_Clear();
- Py_DECREF(v);
- continue;
+ /* Encoding failed to decode ASCII bytes.
+ Raise exception. */
+ Py_DECREF(d);
+ d = NULL;
+ break;
}
}
if (PyList_Append(d, v) != 0) {
@@ -2470,7 +2534,7 @@
Py_DECREF(v);
}
closedir(dirp);
- PyMem_Free(name);
+ release_bytes(oname);
return d;
@@ -2482,10 +2546,8 @@
static PyObject *
posix__getfullpathname(PyObject *self, PyObject *args)
{
- /* assume encoded strings won't more than double no of chars */
- char inbuf[MAX_PATH*2];
- char *inbufp = inbuf;
- Py_ssize_t insize = sizeof(inbuf);
+ PyObject *opath;
+ char *path;
char outbuf[MAX_PATH*2];
char *temp;
#ifdef Py_WIN_WIDE_FILENAMES
@@ -2519,13 +2581,17 @@
PyErr_Clear();
}
#endif
- if (!PyArg_ParseTuple (args, "et#:_getfullpathname",
- Py_FileSystemDefaultEncoding, &inbufp,
- &insize))
- return NULL;
- if (!GetFullPathName(inbuf, sizeof(outbuf)/sizeof(outbuf[0]),
- outbuf, &temp))
- return win32_error("GetFullPathName", inbuf);
+ if (!PyArg_ParseTuple (args, "O&:_getfullpathname",
+ PyUnicode_FSConverter, &opath))
+ return NULL;
+ path = bytes2str(opath, 1);
+ if (!GetFullPathName(path, sizeof(outbuf)/sizeof(outbuf[0]),
+ outbuf, &temp)) {
+ win32_error("GetFullPathName", path);
+ release_bytes(opath);
+ return NULL;
+ }
+ release_bytes(opath);
if (PyUnicode_Check(PyTuple_GetItem(args, 0))) {
return PyUnicode_Decode(outbuf, strlen(outbuf),
Py_FileSystemDefaultEncoding, NULL);
@@ -2542,7 +2608,8 @@
posix_mkdir(PyObject *self, PyObject *args)
{
int res;
- char *path = NULL;
+ PyObject *opath;
+ char *path;
int mode = 0777;
#ifdef Py_WIN_WIDE_FILENAMES
@@ -2563,9 +2630,10 @@
are also valid. */
PyErr_Clear();
}
- if (!PyArg_ParseTuple(args, "et|i:mkdir",
- Py_FileSystemDefaultEncoding, &path, &mode))
+ if (!PyArg_ParseTuple(args, "O&|i:mkdir",
+ PyUnicode_FSConverter, &opath, &mode))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
/* PyUnicode_AS_UNICODE OK without thread lock as
it is a simple dereference. */
@@ -2573,17 +2641,18 @@
Py_END_ALLOW_THREADS
if (!res) {
win32_error("mkdir", path);
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
- PyMem_Free(path);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
#else
- if (!PyArg_ParseTuple(args, "et|i:mkdir",
- Py_FileSystemDefaultEncoding, &path, &mode))
+ if (!PyArg_ParseTuple(args, "O&|i:mkdir",
+ PyUnicode_FSConverter, &opath, &mode))
return NULL;
+ path = bytes2str(opath, 1);
Py_BEGIN_ALLOW_THREADS
#if ( defined(__WATCOMC__) || defined(PYCC_VACPP) ) && !defined(__QNX__)
res = mkdir(path);
@@ -2592,8 +2661,8 @@
#endif
Py_END_ALLOW_THREADS
if (res < 0)
- return posix_error_with_allocated_filename(path);
- PyMem_Free(path);
+ return posix_error_with_allocated_filename(opath);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
#endif
@@ -2685,7 +2754,7 @@
Py_INCREF(Py_None);
return Py_None;
#else
- return posix_2str(args, "etet:rename", rename);
+ return posix_2str(args, "O&O&:rename", rename);
#endif
}
@@ -2700,7 +2769,7 @@
#ifdef MS_WINDOWS
return win32_1str(args, "rmdir", "y:rmdir", RemoveDirectoryA, "U:rmdir", RemoveDirectoryW);
#else
- return posix_1str(args, "et:rmdir", rmdir);
+ return posix_1str(args, "O&:rmdir", rmdir);
#endif
}
@@ -2713,9 +2782,9 @@
posix_stat(PyObject *self, PyObject *args)
{
#ifdef MS_WINDOWS
- return posix_do_stat(self, args, "et:stat", STAT, "U:stat", win32_wstat);
+ return posix_do_stat(self, args, "O&:stat", STAT, "U:stat", win32_wstat);
#else
- return posix_do_stat(self, args, "et:stat", STAT, NULL, NULL);
+ return posix_do_stat(self, args, "O&:stat", STAT, NULL, NULL);
#endif
}
@@ -2781,7 +2850,7 @@
#ifdef MS_WINDOWS
return win32_1str(args, "remove", "y:remove", DeleteFileA, "U:remove", DeleteFileW);
#else
- return posix_1str(args, "et:remove", unlink);
+ return posix_1str(args, "O&:remove", unlink);
#endif
}
@@ -2853,7 +2922,8 @@
PyObject *arg;
PyUnicodeObject *obwpath;
wchar_t *wpath = NULL;
- char *apath = NULL;
+ PyObject *oapath;
+ char *apath;
HANDLE hFile;
long atimesec, mtimesec, ausec, musec;
FILETIME atime, mtime;
@@ -2875,9 +2945,10 @@
PyErr_Clear();
}
if (!wpath) {
- if (!PyArg_ParseTuple(args, "etO:utime",
- Py_FileSystemDefaultEncoding, &apath, &arg))
+ if (!PyArg_ParseTuple(args, "O&O:utime",
+ PyUnicode_FSConverter, &oapath, &arg))
return NULL;
+ apath = bytes2str(oapath, 1);
Py_BEGIN_ALLOW_THREADS
hFile = CreateFileA(apath, FILE_WRITE_ATTRIBUTES, 0,
NULL, OPEN_EXISTING,
@@ -2885,10 +2956,10 @@
Py_END_ALLOW_THREADS
if (hFile == INVALID_HANDLE_VALUE) {
win32_error("utime", apath);
- PyMem_Free(apath);
+ release_bytes(oapath);
return NULL;
}
- PyMem_Free(apath);
+ release_bytes(oapath);
}
if (arg == Py_None) {
@@ -2929,7 +3000,8 @@
return result;
#else /* Py_WIN_WIDE_FILENAMES */
- char *path = NULL;
+ PyObject *opath;
+ char *path;
long atime, mtime, ausec, musec;
int res;
PyObject* arg;
@@ -2952,9 +3024,10 @@
#endif /* HAVE_UTIMES */
- if (!PyArg_ParseTuple(args, "etO:utime",
- Py_FileSystemDefaultEncoding, &path, &arg))
+ if (!PyArg_ParseTuple(args, "O&O:utime",
+ PyUnicode_FSConverter, &opath, &arg))
return NULL;
+ path = bytes2str(opath, 1);
if (arg == Py_None) {
/* optional time values not given */
Py_BEGIN_ALLOW_THREADS
@@ -2964,18 +3037,18 @@
else if (!PyTuple_Check(arg) || PyTuple_Size(arg) != 2) {
PyErr_SetString(PyExc_TypeError,
"utime() arg 2 must be a tuple (atime, mtime)");
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
else {
if (extract_time(PyTuple_GET_ITEM(arg, 0),
&atime, &ausec) == -1) {
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
if (extract_time(PyTuple_GET_ITEM(arg, 1),
&mtime, &musec) == -1) {
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
ATIME = atime;
@@ -2993,9 +3066,9 @@
#endif /* HAVE_UTIMES */
}
if (res < 0) {
- return posix_error_with_allocated_filename(path);
+ return posix_error_with_allocated_filename(opath);
}
- PyMem_Free(path);
+ release_bytes(opath);
Py_INCREF(Py_None);
return Py_None;
#undef UTIME_ARG
@@ -3030,6 +3103,22 @@
PyMem_Free(array[i]);
PyMem_DEL(array);
}
+
+int fsconvert_strdup(PyObject *o, char**out)
+{
+ PyObject *bytes;
+ Py_ssize_t size;
+ if (!PyUnicode_FSConverter(o, &bytes))
+ return 0;
+ size = PyObject_Size(bytes);
+ *out = PyMem_Malloc(size+1);
+ if (!*out)
+ return 0;
+ /* Don't lock bytes, as we hold the GIL */
+ memcpy(*out, bytes2str(bytes, 0), size+1);
+ Py_DECREF(bytes);
+ return 1;
+}
#endif
@@ -3044,6 +3133,7 @@
static PyObject *
posix_execv(PyObject *self, PyObject *args)
{
+ PyObject *opath;
char *path;
PyObject *argv;
char **argvlist;
@@ -3053,10 +3143,11 @@
/* execv has two arguments: (path, argv), where
argv is a list or tuple of strings. */
- if (!PyArg_ParseTuple(args, "etO:execv",
- Py_FileSystemDefaultEncoding,
- &path, &argv))
+ if (!PyArg_ParseTuple(args, "O&O:execv",
+ PyUnicode_FSConverter,
+ &opath, &argv))
return NULL;
+ path = bytes2str(opath, 1);
if (PyList_Check(argv)) {
argc = PyList_Size(argv);
getitem = PyList_GetItem;
@@ -3067,28 +3158,27 @@
}
else {
PyErr_SetString(PyExc_TypeError, "execv() arg 2 must be a tuple or list");
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
if (argc < 1) {
PyErr_SetString(PyExc_ValueError, "execv() arg 2 must not be empty");
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
argvlist = PyMem_NEW(char *, argc+1);
if (argvlist == NULL) {
- PyMem_Free(path);
+ release_bytes(opath);
return PyErr_NoMemory();
}
for (i = 0; i < argc; i++) {
- if (!PyArg_Parse((*getitem)(argv, i), "et",
- Py_FileSystemDefaultEncoding,
- &argvlist[i])) {
+ if (!fsconvert_strdup((*getitem)(argv, i),
+ &argvlist[i])) {
free_string_array(argvlist, i);
PyErr_SetString(PyExc_TypeError,
"execv() arg 2 must contain only strings");
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
@@ -3100,7 +3190,7 @@
/* If we get here it's definitely an error */
free_string_array(argvlist, argc);
- PyMem_Free(path);
+ release_bytes(opath);
return posix_error();
}
@@ -3116,6 +3206,7 @@
static PyObject *
posix_execve(PyObject *self, PyObject *args)
{
+ PyObject *opath;
char *path;
PyObject *argv, *env;
char **argvlist;
@@ -3129,10 +3220,11 @@
argv is a list or tuple of strings and env is a dictionary
like posix.environ. */
- if (!PyArg_ParseTuple(args, "etOO:execve",
- Py_FileSystemDefaultEncoding,
- &path, &argv, &env))
+ if (!PyArg_ParseTuple(args, "O&OO:execve",
+ PyUnicode_FSConverter,
+ &opath, &argv, &env))
return NULL;
+ path = bytes2str(opath, 1);
if (PyList_Check(argv)) {
argc = PyList_Size(argv);
getitem = PyList_GetItem;
@@ -3158,10 +3250,8 @@
goto fail_0;
}
for (i = 0; i < argc; i++) {
- if (!PyArg_Parse((*getitem)(argv, i),
- "et;execve() arg 2 must contain only strings",
- Py_FileSystemDefaultEncoding,
- &argvlist[i]))
+ if (!fsconvert_strdup((*getitem)(argv, i),
+ &argvlist[i]))
{
lastarg = i;
goto fail_1;
@@ -3243,7 +3333,7 @@
Py_XDECREF(vals);
Py_XDECREF(keys);
fail_0:
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
#endif /* HAVE_EXECV */
@@ -3261,6 +3351,7 @@
static PyObject *
posix_spawnv(PyObject *self, PyObject *args)
{
+ PyObject *opath;
char *path;
PyObject *argv;
char **argvlist;
@@ -3272,10 +3363,11 @@
/* spawnv has three arguments: (mode, path, argv), where
argv is a list or tuple of strings. */
- if (!PyArg_ParseTuple(args, "ietO:spawnv", &mode,
- Py_FileSystemDefaultEncoding,
- &path, &argv))
+ if (!PyArg_ParseTuple(args, "iO&O:spawnv", &mode,
+ PyUnicode_FSConverter,
+ &opath, &argv))
return NULL;
+ path = bytes2str(opath, 1);
if (PyList_Check(argv)) {
argc = PyList_Size(argv);
getitem = PyList_GetItem;
@@ -3287,24 +3379,23 @@
else {
PyErr_SetString(PyExc_TypeError,
"spawnv() arg 2 must be a tuple or list");
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
argvlist = PyMem_NEW(char *, argc+1);
if (argvlist == NULL) {
- PyMem_Free(path);
+ release_bytes(opath);
return PyErr_NoMemory();
}
for (i = 0; i < argc; i++) {
- if (!PyArg_Parse((*getitem)(argv, i), "et",
- Py_FileSystemDefaultEncoding,
- &argvlist[i])) {
+ if (!fsconvert_strdup((*getitem)(argv, i),
+ &argvlist[i])) {
free_string_array(argvlist, i);
PyErr_SetString(
PyExc_TypeError,
"spawnv() arg 2 must contain only strings");
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
}
@@ -3324,7 +3415,7 @@
#endif
free_string_array(argvlist, argc);
- PyMem_Free(path);
+ release_bytes(opath);
if (spawnval == -1)
return posix_error();
@@ -3349,6 +3440,7 @@
static PyObject *
posix_spawnve(PyObject *self, PyObject *args)
{
+ PyObject *opath;
char *path;
PyObject *argv, *env;
char **argvlist;
@@ -3364,10 +3456,11 @@
argv is a list or tuple of strings and env is a dictionary
like posix.environ. */
- if (!PyArg_ParseTuple(args, "ietOO:spawnve", &mode,
- Py_FileSystemDefaultEncoding,
- &path, &argv, &env))
+ if (!PyArg_ParseTuple(args, "iO&OO:spawnve", &mode,
+ PyUnicode_FSConverter,
+ &opath, &argv, &env))
return NULL;
+ path = bytes2str(opath, 1);
if (PyList_Check(argv)) {
argc = PyList_Size(argv);
getitem = PyList_GetItem;
@@ -3393,10 +3486,8 @@
goto fail_0;
}
for (i = 0; i < argc; i++) {
- if (!PyArg_Parse((*getitem)(argv, i),
- "et;spawnve() arg 2 must contain only strings",
- Py_FileSystemDefaultEncoding,
- &argvlist[i]))
+ if (!fsconvert_strdup((*getitem)(argv, i),
+ &argvlist[i]))
{
lastarg = i;
goto fail_1;
@@ -3486,7 +3577,7 @@
Py_XDECREF(vals);
Py_XDECREF(keys);
fail_0:
- PyMem_Free(path);
+ release_bytes(opath);
return res;
}
@@ -3504,6 +3595,7 @@
static PyObject *
posix_spawnvp(PyObject *self, PyObject *args)
{
+ PyObject *opath;
char *path;
PyObject *argv;
char **argvlist;
@@ -3514,10 +3606,11 @@
/* spawnvp has three arguments: (mode, path, argv), where
argv is a list or tuple of strings. */
- if (!PyArg_ParseTuple(args, "ietO:spawnvp", &mode,
- Py_FileSystemDefaultEncoding,
- &path, &argv))
+ if (!PyArg_ParseTuple(args, "iO&O:spawnvp", &mode,
+ PyUnicode_FSConverter,
+ &opath, &argv))
return NULL;
+ path = bytes2str(opath);
if (PyList_Check(argv)) {
argc = PyList_Size(argv);
getitem = PyList_GetItem;
@@ -3529,24 +3622,23 @@
else {
PyErr_SetString(PyExc_TypeError,
"spawnvp() arg 2 must be a tuple or list");
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
argvlist = PyMem_NEW(char *, argc+1);
if (argvlist == NULL) {
- PyMem_Free(path);
+ release_bytes(opath);
return PyErr_NoMemory();
}
for (i = 0; i < argc; i++) {
- if (!PyArg_Parse((*getitem)(argv, i), "et",
- Py_FileSystemDefaultEncoding,
- &argvlist[i])) {
+ if (!fsconvert_strdup((*getitem)(argv, i),
+ &argvlist[i])) {
free_string_array(argvlist, i);
PyErr_SetString(
PyExc_TypeError,
"spawnvp() arg 2 must contain only strings");
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
}
@@ -3561,7 +3653,7 @@
Py_END_ALLOW_THREADS
free_string_array(argvlist, argc);
- PyMem_Free(path);
+ release_bytes(opath);
if (spawnval == -1)
return posix_error();
@@ -3583,6 +3675,7 @@
static PyObject *
posix_spawnvpe(PyObject *self, PyObject *args)
{
+ PyObject *opath
char *path;
PyObject *argv, *env;
char **argvlist;
@@ -3598,9 +3691,10 @@
like posix.environ. */
if (!PyArg_ParseTuple(args, "ietOO:spawnvpe", &mode,
- Py_FileSystemDefaultEncoding,
- &path, &argv, &env))
+ PyUnicode_FSConverter,
+ &opath, &argv, &env))
return NULL;
+ path = bytes2str(opath);
if (PyList_Check(argv)) {
argc = PyList_Size(argv);
getitem = PyList_GetItem;
@@ -3626,10 +3720,8 @@
goto fail_0;
}
for (i = 0; i < argc; i++) {
- if (!PyArg_Parse((*getitem)(argv, i),
- "et;spawnvpe() arg 2 must contain only strings",
- Py_FileSystemDefaultEncoding,
- &argvlist[i]))
+ if (!fsconvert_strdup((*getitem)(argv, i),
+ &argvlist[i]))
{
lastarg = i;
goto fail_1;
@@ -3710,7 +3802,7 @@
Py_XDECREF(vals);
Py_XDECREF(keys);
fail_0:
- PyMem_Free(path);
+ release_bytes(opath);
return res;
}
#endif /* PYOS_OS2 */
@@ -4549,12 +4641,12 @@
posix_lstat(PyObject *self, PyObject *args)
{
#ifdef HAVE_LSTAT
- return posix_do_stat(self, args, "et:lstat", lstat, NULL, NULL);
+ return posix_do_stat(self, args, "O&:lstat", lstat, NULL, NULL);
#else /* !HAVE_LSTAT */
#ifdef MS_WINDOWS
- return posix_do_stat(self, args, "et:lstat", STAT, "U:lstat", win32_wstat);
+ return posix_do_stat(self, args, "O&:lstat", STAT, "U:lstat", win32_wstat);
#else
- return posix_do_stat(self, args, "et:lstat", STAT, NULL, NULL);
+ return posix_do_stat(self, args, "O&:lstat", STAT, NULL, NULL);
#endif
#endif /* !HAVE_LSTAT */
}
@@ -4570,16 +4662,18 @@
{
PyObject* v;
char buf[MAXPATHLEN];
+ PyObject *opath;
char *path;
int n;
int arg_is_unicode = 0;
- if (!PyArg_ParseTuple(args, "et:readlink",
- Py_FileSystemDefaultEncoding, &path))
+ if (!PyArg_ParseTuple(args, "O&:readlink",
+ PyUnicode_FSConverter, &opath))
return NULL;
+ path = bytes2str(opath, 1);
v = PySequence_GetItem(args, 0);
if (v == NULL) {
- PyMem_Free(path);
+ release_bytes(opath);
return NULL;
}
@@ -4592,16 +4686,16 @@
n = readlink(path, buf, (int) sizeof buf);
Py_END_ALLOW_THREADS
if (n < 0)
- return posix_error_with_allocated_filename(path);
+ return posix_error_with_allocated_filename(opath);
- PyMem_Free(path);
+ release_bytes(opath);
v = PyBytes_FromStringAndSize(buf, n);
if (arg_is_unicode) {
PyObject *w;
w = PyUnicode_FromEncodedObject(v,
Py_FileSystemDefaultEncoding,
- "strict");
+ "utf8b");
if (w != NULL) {
Py_DECREF(v);
v = w;
@@ -4623,7 +4717,7 @@
static PyObject *
posix_symlink(PyObject *self, PyObject *args)
{
- return posix_2str(args, "etet:symlink", symlink);
+ return posix_2str(args, "O&O&:symlink", symlink);
}
#endif /* HAVE_SYMLINK */
@@ -4811,7 +4905,8 @@
static PyObject *
posix_open(PyObject *self, PyObject *args)
{
- char *file = NULL;
+ PyObject *ofile;
+ char *file;
int flag;
int mode = 0777;
int fd;
@@ -4835,17 +4930,17 @@
}
#endif
- if (!PyArg_ParseTuple(args, "eti|i",
- Py_FileSystemDefaultEncoding, &file,
+ if (!PyArg_ParseTuple(args, "O&i|i",
+ PyUnicode_FSConverter, &ofile,
&flag, &mode))
return NULL;
-
+ file = bytes2str(ofile, 1);
Py_BEGIN_ALLOW_THREADS
fd = open(file, flag, mode);
Py_END_ALLOW_THREADS
if (fd < 0)
- return posix_error_with_allocated_filename(file);
- PyMem_Free(file);
+ return posix_error_with_allocated_filename(ofile);
+ release_bytes(ofile);
return PyLong_FromLong((long)fd);
}
@@ -5289,20 +5384,27 @@
wchar_t *s1, *s2;
wchar_t *newenv;
#else
+ PyObject *os1, *os2;
char *s1, *s2;
char *newenv;
#endif
PyObject *newstr;
size_t len;
- if (!PyArg_ParseTuple(args,
#ifdef MS_WINDOWS
+ if (!PyArg_ParseTuple(args,
"uu:putenv",
-#else
- "ss:putenv",
-#endif
&s1, &s2))
return NULL;
+#else
+ if (!PyArg_ParseTuple(args,
+ "O&O&:putenv",
+ PyUnicode_FSConverter, &os1,
+ PyUnicode_FSConverter, &os2))
+ return NULL;
+ s1 = bytes2str(os1, 1);
+ s2 = bytes2str(os2, 1);
+#endif
#if defined(PYOS_OS2)
if (stricmp(s1, "BEGINLIBPATH") == 0) {
@@ -5345,6 +5447,8 @@
PyOS_snprintf(newenv, len, "%s=%s", s1, s2);
if (putenv(newenv)) {
Py_DECREF(newstr);
+ release_bytes(os1);
+ release_bytes(os2);
posix_error();
return NULL;
}
@@ -5365,6 +5469,10 @@
#if defined(PYOS_OS2)
}
#endif
+#ifndef MS_WINDOWS
+ release_bytes(os1);
+ release_bytes(os2);
+#endif
Py_INCREF(Py_None);
return Py_None;
}
@@ -6688,6 +6796,7 @@
static PyObject *
win32_startfile(PyObject *self, PyObject *args)
{
+ PyObject *ofilepath;
char *filepath;
char *operation = NULL;
HINSTANCE rc;
@@ -6729,20 +6838,21 @@
#endif
normal:
- if (!PyArg_ParseTuple(args, "et|s:startfile",
- Py_FileSystemDefaultEncoding, &filepath,
+ if (!PyArg_ParseTuple(args, "O&|s:startfile",
+ PyUnicode_FSConverter, &ofilepath,
&operation))
return NULL;
+ filepath = bytes2str(ofilepath, 1);
Py_BEGIN_ALLOW_THREADS
rc = ShellExecute((HWND)0, operation, filepath,
NULL, NULL, SW_SHOWNORMAL);
Py_END_ALLOW_THREADS
if (rc <= (HINSTANCE)32) {
PyObject *errval = win32_error("startfile", filepath);
- PyMem_Free(filepath);
+ release_bytes(ofilepath);
return errval;
}
- PyMem_Free(filepath);
+ release_bytes(ofilepath);
Py_INCREF(Py_None);
return Py_None;
}
Modified: python/branches/py3k/Modules/python.c
==============================================================================
--- python/branches/py3k/Modules/python.c (original)
+++ python/branches/py3k/Modules/python.c Tue May 5 06:43:17 2009
@@ -14,6 +14,93 @@
return Py_Main(argc, argv);
}
#else
+static wchar_t*
+char2wchar(char* arg)
+{
+ wchar_t *res;
+#ifdef HAVE_BROKEN_MBSTOWCS
+ /* Some platforms have a broken implementation of
+ * mbstowcs which does not count the characters that
+ * would result from conversion. Use an upper bound.
+ */
+ size_t argsize = strlen(arg);
+#else
+ size_t argsize = mbstowcs(NULL, arg, 0);
+#endif
+ size_t count;
+ unsigned char *in;
+ wchar_t *out;
+#ifdef HAVE_MBRTOWC
+ mbstate_t mbs;
+#endif
+ if (argsize != (size_t)-1) {
+ res = (wchar_t *)PyMem_Malloc((argsize+1)*sizeof(wchar_t));
+ if (!res)
+ goto oom;
+ count = mbstowcs(res, arg, argsize+1);
+ if (count != (size_t)-1)
+ return res;
+ PyMem_Free(res);
+ }
+ /* Conversion failed. Fall back to escaping with utf8b. */
+#ifdef HAVE_MBRTOWC
+ /* Try conversion with mbrtwoc (C99), and escape non-decodable bytes. */
+
+ /* Overallocate; as multi-byte characters are in the argument, the
+ actual output could use less memory. */
+ argsize = strlen(arg) + 1;
+ res = PyMem_Malloc(argsize*sizeof(wchar_t));
+ if (!res) goto oom;
+ in = (unsigned char*)arg;
+ out = res;
+ memset(&mbs, 0, sizeof mbs);
+ while (argsize) {
+ size_t converted = mbrtowc(out, (char*)in, argsize, &mbs);
+ if (converted == 0)
+ /* Reached end of string; null char stored. */
+ break;
+ if (converted == (size_t)-2) {
+ /* Incomplete character. This should never happen,
+ since we provide everything that we have -
+ unless there is a bug in the C library, or I
+ misunderstood how mbrtowc works. */
+ fprintf(stderr, "unexpected mbrtowc result -2\n");
+ return NULL;
+ }
+ if (converted == (size_t)-1) {
+ /* Conversion error. Escape as UTF-8b, and start over
+ in the initial shift state. */
+ *out++ = 0xdc00 + *in++;
+ argsize--;
+ memset(&mbs, 0, sizeof mbs);
+ continue;
+ }
+ /* successfully converted some bytes */
+ in += converted;
+ argsize -= converted;
+ out++;
+ }
+#else
+ /* Cannot use C locale for escaping; manually escape as if charset
+ is ASCII (i.e. escape all bytes > 128. This will still roundtrip
+ correctly in the locale's charset, which must be an ASCII superset. */
+ res = PyMem_Malloc((strlen(arg)+1)*sizeof(wchar_t));
+ if (!res) goto oom;
+ in = (unsigned char*)arg;
+ out = res;
+ while(*in)
+ if(*in < 128)
+ *out++ = *in++;
+ else
+ *out++ = 0xdc00 + *in++;
+ *out = 0;
+#endif
+ return res;
+oom:
+ fprintf(stderr, "out of memory\n");
+ return NULL;
+}
+
int
main(int argc, char **argv)
{
@@ -40,31 +127,9 @@
oldloc = strdup(setlocale(LC_ALL, NULL));
setlocale(LC_ALL, "");
for (i = 0; i < argc; i++) {
-#ifdef HAVE_BROKEN_MBSTOWCS
- /* Some platforms have a broken implementation of
- * mbstowcs which does not count the characters that
- * would result from conversion. Use an upper bound.
- */
- size_t argsize = strlen(argv[i]);
-#else
- size_t argsize = mbstowcs(NULL, argv[i], 0);
-#endif
- size_t count;
- if (argsize == (size_t)-1) {
- fprintf(stderr, "Could not convert argument %d to string\n", i);
+ argv_copy2[i] = argv_copy[i] = char2wchar(argv[i]);
+ if (!argv_copy[i])
return 1;
- }
- argv_copy[i] = (wchar_t *)PyMem_Malloc((argsize+1)*sizeof(wchar_t));
- argv_copy2[i] = argv_copy[i];
- if (!argv_copy[i]) {
- fprintf(stderr, "out of memory\n");
- return 1;
- }
- count = mbstowcs(argv_copy[i], argv[i], argsize+1);
- if (count == (size_t)-1) {
- fprintf(stderr, "Could not convert argument %d to string\n", i);
- return 1;
- }
}
setlocale(LC_ALL, oldloc);
free(oldloc);
Modified: python/branches/py3k/Objects/unicodeobject.c
==============================================================================
--- python/branches/py3k/Objects/unicodeobject.c (original)
+++ python/branches/py3k/Objects/unicodeobject.c Tue May 5 06:43:17 2009
@@ -1530,6 +1530,53 @@
}
}
+/* Convert the argument to a bytes object, according to the file
+ system encoding */
+
+int
+PyUnicode_FSConverter(PyObject* arg, void* addr)
+{
+ PyObject *output = NULL;
+ Py_ssize_t size;
+ void *data;
+ if (PyBytes_Check(arg) || PyByteArray_Check(arg)) {
+ output = arg;
+ Py_INCREF(output);
+ }
+ else {
+ arg = PyUnicode_FromObject(arg);
+ if (!arg)
+ return 0;
+ output = PyUnicode_AsEncodedObject(arg,
+ Py_FileSystemDefaultEncoding,
+ "utf8b");
+ Py_DECREF(arg);
+ if (!output)
+ return 0;
+ if (!PyBytes_Check(output)) {
+ Py_DECREF(output);
+ PyErr_SetString(PyExc_TypeError, "encoder failed to return bytes");
+ return 0;
+ }
+ }
+ if (PyBytes_Check(output)) {
+ size = PyBytes_GET_SIZE(output);
+ data = PyBytes_AS_STRING(output);
+ }
+ else {
+ size = PyByteArray_GET_SIZE(output);
+ data = PyByteArray_AS_STRING(output);
+ }
+ if (size != strlen(data)) {
+ PyErr_SetString(PyExc_TypeError, "embedded NUL character");
+ Py_DECREF(output);
+ return 0;
+ }
+ *(PyObject**)addr = output;
+ return 1;
+}
+
+
char*
_PyUnicode_AsStringAndSize(PyObject *unicode, Py_ssize_t *psize)
{
@@ -4154,11 +4201,22 @@
collstart-startp, collend-startp, &newpos);
if (repunicode == NULL)
goto onError;
- if (!PyUnicode_Check(repunicode)) {
- /* Implementation limitation: byte results not supported yet. */
- PyErr_SetString(PyExc_TypeError, "error handler should return unicode");
+ if (PyBytes_Check(repunicode)) {
+ /* Directly copy bytes result to output. */
+ repsize = PyBytes_Size(repunicode);
+ if (repsize > 1) {
+ /* Make room for all additional bytes. */
+ if (_PyBytes_Resize(&res, ressize+repsize-1)) {
+ Py_DECREF(repunicode);
+ goto onError;
+ }
+ ressize += repsize-1;
+ }
+ memcpy(str, PyBytes_AsString(repunicode), repsize);
+ str += repsize;
+ p = startp + newpos;
Py_DECREF(repunicode);
- goto onError;
+ break;
}
/* need more space? (at least enough for what we
have+the replacement+the rest of the string, so
@@ -5123,11 +5181,24 @@
collstartpos, collendpos, &newpos);
if (repunicode == NULL)
return -1;
- if (!PyUnicode_Check(repunicode)) {
- /* Implementation limitation: byte results not supported yet. */
- PyErr_SetString(PyExc_TypeError, "error handler should return unicode");
+ if (PyBytes_Check(repunicode)) {
+ /* Directly copy bytes result to output. */
+ Py_ssize_t outsize = PyBytes_Size(*res);
+ Py_ssize_t requiredsize;
+ repsize = PyBytes_Size(repunicode);
+ requiredsize = *respos + repsize;
+ if (requiredsize > outsize)
+ /* Make room for all additional bytes. */
+ if (charmapencode_resize(res, respos, requiredsize)) {
+ Py_DECREF(repunicode);
+ return -1;
+ }
+ memcpy(PyBytes_AsString(*res) + *respos,
+ PyBytes_AsString(repunicode), repsize);
+ *respos += repsize;
+ *inpos = newpos;
Py_DECREF(repunicode);
- return -1;
+ break;
}
/* generate replacement */
repsize = PyUnicode_GET_SIZE(repunicode);
@@ -5691,7 +5762,7 @@
if (repunicode == NULL)
goto onError;
if (!PyUnicode_Check(repunicode)) {
- /* Implementation limitation: byte results not supported yet. */
+ /* Byte results not supported, since they have no decimal property. */
PyErr_SetString(PyExc_TypeError, "error handler should return unicode");
Py_DECREF(repunicode);
goto onError;
Modified: python/branches/py3k/Python/codecs.c
==============================================================================
--- python/branches/py3k/Python/codecs.c (original)
+++ python/branches/py3k/Python/codecs.c Tue May 5 06:43:17 2009
@@ -829,6 +829,82 @@
}
}
+static PyObject *
+PyCodec_UTF8bErrors(PyObject *exc)
+{
+ PyObject *restuple;
+ PyObject *object;
+ Py_ssize_t start;
+ Py_ssize_t end;
+ PyObject *res;
+ if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
+ Py_UNICODE *p;
+ Py_UNICODE *startp;
+ char *outp;
+ if (PyUnicodeEncodeError_GetStart(exc, &start))
+ return NULL;
+ if (PyUnicodeEncodeError_GetEnd(exc, &end))
+ return NULL;
+ if (!(object = PyUnicodeEncodeError_GetObject(exc)))
+ return NULL;
+ startp = PyUnicode_AS_UNICODE(object);
+ res = PyBytes_FromStringAndSize(NULL, end-start);
+ if (!res) {
+ Py_DECREF(object);
+ return NULL;
+ }
+ outp = PyBytes_AsString(res);
+ for (p = startp+start; p < startp+end; p++) {
+ Py_UNICODE ch = *p;
+ if (ch < 0xdc80 || ch > 0xdcff) {
+ /* Not a UTF-8b surrogate, fail with original exception */
+ PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
+ Py_DECREF(res);
+ Py_DECREF(object);
+ return NULL;
+ }
+ *outp++ = ch - 0xdc00;
+ }
+ restuple = Py_BuildValue("(On)", res, end);
+ Py_DECREF(res);
+ Py_DECREF(object);
+ return restuple;
+ }
+ else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
+ unsigned char *p;
+ Py_UNICODE ch[4]; /* decode up to 4 bad bytes. */
+ int consumed = 0;
+ if (PyUnicodeDecodeError_GetStart(exc, &start))
+ return NULL;
+ if (PyUnicodeDecodeError_GetEnd(exc, &end))
+ return NULL;
+ if (!(object = PyUnicodeDecodeError_GetObject(exc)))
+ return NULL;
+ if (!(p = (unsigned char*)PyBytes_AsString(object))) {
+ Py_DECREF(object);
+ return NULL;
+ }
+ while (consumed < 4 && consumed < end-start) {
+ /* Refuse to escape ASCII bytes. */
+ if (p[start+consumed] < 128)
+ break;
+ ch[consumed] = 0xdc00 + p[start+consumed];
+ consumed++;
+ }
+ Py_DECREF(object);
+ if (!consumed) {
+ /* codec complained about ASCII byte. */
+ PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
+ return NULL;
+ }
+ return Py_BuildValue("(u#n)", ch, consumed, start+consumed);
+ }
+ else {
+ wrong_exception_type(exc);
+ return NULL;
+ }
+}
+
static PyObject *strict_errors(PyObject *self, PyObject *exc)
{
@@ -864,6 +940,11 @@
return PyCodec_SurrogateErrors(exc);
}
+static PyObject *utf8b_errors(PyObject *self, PyObject *exc)
+{
+ return PyCodec_UTF8bErrors(exc);
+}
+
static int _PyCodecRegistry_Init(void)
{
static struct {
@@ -918,6 +999,14 @@
surrogates_errors,
METH_O
}
+ },
+ {
+ "utf8b",
+ {
+ "utf8b",
+ utf8b_errors,
+ METH_O
+ }
}
};
Modified: python/branches/py3k/Python/pythonrun.c
==============================================================================
--- python/branches/py3k/Python/pythonrun.c (original)
+++ python/branches/py3k/Python/pythonrun.c Tue May 5 06:43:17 2009
@@ -262,6 +262,22 @@
_PyImportHooks_Init();
+#if defined(HAVE_LANGINFO_H) && defined(CODESET)
+ /* On Unix, set the file system encoding according to the
+ user's preference, if the CODESET names a well-known
+ Python codec, and Py_FileSystemDefaultEncoding isn't
+ initialized by other means. Also set the encoding of
+ stdin and stdout if these are terminals. */
+
+ codeset = get_codeset();
+ if (codeset) {
+ if (!Py_FileSystemDefaultEncoding)
+ Py_FileSystemDefaultEncoding = codeset;
+ else
+ free(codeset);
+ }
+#endif
+
if (install_sigs)
initsigs(); /* Signal handling stuff, including initintr() */
@@ -285,22 +301,6 @@
#ifdef WITH_THREAD
_PyGILState_Init(interp, tstate);
#endif /* WITH_THREAD */
-
-#if defined(HAVE_LANGINFO_H) && defined(CODESET)
- /* On Unix, set the file system encoding according to the
- user's preference, if the CODESET names a well-known
- Python codec, and Py_FileSystemDefaultEncoding isn't
- initialized by other means. Also set the encoding of
- stdin and stdout if these are terminals. */
-
- codeset = get_codeset();
- if (codeset) {
- if (!Py_FileSystemDefaultEncoding)
- Py_FileSystemDefaultEncoding = codeset;
- else
- free(codeset);
- }
-#endif
}
void
Modified: python/branches/py3k/configure
==============================================================================
--- python/branches/py3k/configure (original)
+++ python/branches/py3k/configure Tue May 5 06:43:17 2009
@@ -1,5 +1,5 @@
#! /bin/sh
-# From configure.in Revision: 71731 .
+# From configure.in Revision: 72144 .
# Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.61 for python 3.1.
#
@@ -16299,11 +16299,12 @@
+
for ac_func in alarm setitimer getitimer bind_textdomain_codeset chown \
clock confstr ctermid execv fchmod fchown fork fpathconf ftime ftruncate \
gai_strerror getgroups getlogin getloadavg getpeername getpgid getpid \
getpriority getpwent getspnam getspent getsid getwd \
- kill killpg lchmod lchown lstat mkfifo mknod mktime \
+ kill killpg lchmod lchown lstat mbrtowc mkfifo mknod mktime \
mremap nice pathconf pause plock poll pthread_init \
putenv readlink realpath \
select sem_open sem_timedwait sem_getvalue sem_unlink setegid seteuid \
Modified: python/branches/py3k/configure.in
==============================================================================
--- python/branches/py3k/configure.in (original)
+++ python/branches/py3k/configure.in Tue May 5 06:43:17 2009
@@ -2403,7 +2403,7 @@
clock confstr ctermid execv fchmod fchown fork fpathconf ftime ftruncate \
gai_strerror getgroups getlogin getloadavg getpeername getpgid getpid \
getpriority getpwent getspnam getspent getsid getwd \
- kill killpg lchmod lchown lstat mkfifo mknod mktime \
+ kill killpg lchmod lchown lstat mbrtowc mkfifo mknod mktime \
mremap nice pathconf pause plock poll pthread_init \
putenv readlink realpath \
select sem_open sem_timedwait sem_getvalue sem_unlink setegid seteuid \
Modified: python/branches/py3k/pyconfig.h.in
==============================================================================
--- python/branches/py3k/pyconfig.h.in (original)
+++ python/branches/py3k/pyconfig.h.in Tue May 5 06:43:17 2009
@@ -419,6 +419,9 @@
/* Define this if you have the makedev macro. */
#undef HAVE_MAKEDEV
+/* Define to 1 if you have the `mbrtowc' function. */
+#undef HAVE_MBRTOWC
+
/* Define to 1 if you have the `memmove' function. */
#undef HAVE_MEMMOVE
More information about the Python-checkins
mailing list